There are folks in NYC who no matter where they went to school would never wear a button down shirt or a pair of khakis.
A lot of people in Manhattan wear button downs with suits, although it's not my preference. I do wear them for casual or business casual purposes. I'll confess I'm not a huge fan of khakis.