I have the same kind of problems (Windows 10 , LR 11.5).
I have been using masking for years with LR, PS, and picture window pro. With LR, I noticed for example : select the subject, invert, then you still see parts of the subject remain selected, and I mean not on the borders of the subject.