When a plug is inserted in the 3.5mm input jack, it breaks the connection from the amplifier to the speakers, in order to avoid interference or damage.

This is happening inside the jack: The inserted plug bends the contacts slightly, pushing them away from the contacts connected to the amplifier. I am holding the plug over the schematic diagram at the exact point to illustrate this. (Ann.: This schematic probably is not exact, but it is what I would expect it to look like.)

When the plug is removed from the jack, the contacts are supposed to go back into the original position, where they touch the contacts coming from the receiver. Obviously, for some reason, this does not always work, and they do not make the circuit, leaving the respective earpiece without an amplifier signal.