Insert the plastic spudger in between the speaker and the headset directly in line with one of the exposed screws of the headset casing, since that is where the four clamps that hold the speaker to the headset are located. These clamps are not visible until after speaker removal.

Make sure to insert the spudger in between the foam buffer-ring and the headset casing instead of in between the speaker and the foam buffer-ring.

Reassembly may be tricky, there is one clamp smaller than others, so you have to align it correctly.