You need to look closely at the Color Label Set feature mentioned above. Also, if you accidentally make photos white, notice that the name text is still shown in the Library Filter, so in my screenshot you can see that I must have used AB at some point. Using the Library Filter should help you tidy things up.
That should help resolve this, and please feel even freer to ignore what follows.....
When you apply a colour label, LR isn't actually recording yellow or blue or whatever. It's seeing that you have chosen yellow, then looks to the Color Label Set's current setup and says "ah, yellow means I must record 'To email' in the colour label text field". You see this in Library Filter and also in the Metadata panel.
It does the reverse later when you go back to the images. It finds 'To email', then looks up its meaning in the Color Label Set. If you've not changed the setup, it says "Ah, yellow". But if you've done something to change the Color Label Set, it no longer understands that 'To email' should be mapped to a colour and just displays white.
Personally, I have always thought this was overcomplicated, even if it does allow compatibility with Bridge or other software. So
I have always preferred to name green "Green" and remember that it means "multi-frame" (HDR, panorama) or red "Red" and remember it means "!!!!AAAARRRRGGHHH!!!!".