I'm not so Smart
Yea, I figured it was "smarter" than I, so I went into the original folders where the original photos were located and then deleted the rejected photos.
However, it would be a nice option to be able to do this from the collection instead.