Color blind is kind of a misnomer, but squirrels and deer do not see colors in the way we do. Their vision is mostly shades of grey. So you can certainly wear your orange hat or vest while squirrel hunting... or a red or blue shirt (if it's not too light colored). Remember, though, that bright colors that reflect a lot of light will "flash" in the woods, and any game will see that. That includes a shiny new orange vest or hat. Dull colors are usually OK.
I always used to run a new orange hat through the washer a time or two before wearing it in the field (without detergent). I'd also rub it on burnt tree trunks to dull it a little more. Human eyes can still see it, but it helps knock down that glare for other animals.