Edit: Try switching to face lathering for a while as well and see how that goes. If you don't want to, try working the lather longer on your face.
+1 Face lathering was the way forward for me. (Nickshaves is the main youtube face latherer I've found)