Hi,
I've got an iPhone 4S, and also had this issue...couldn't hear a caller unless I put them on speaker phone. My problem was pretty sporadic, and kinda baffling. I live in the Pacific Northwest, it rains a lot here. This holds relevance because I finally figured out that the difficulty only happens when my phone is a bit wet from the rain. Makes sense, because the earphone jack is located at the top of the handset, so it's likely to end up with a drop or two in the socket.
The trouble always vanishes once I'm back inside, and the phone has dried off.
Hope this helps.