I can recommend Shea Moisture Tea Tree After Shave & Bump Preventer. You may be able to find it in your local grocery store. It's an excellent astringent.
I have used this in the past and it does seem to work on ingrown hairs and other bumps a good bit. I was able to pick mine up at Target.