I think probably the most efficient way to make a mask of neutral colors, is to do it in Lab using "blend if". Set the range in the a and b channels accordingly. The mask itself may have to be created based on the resulting transparency (I've done this before, but can't recall the details right now).
The advantage of doing it in Lab over RGB is to bypass the luminance component, which otherwise will complicate things in RGB.