(This is with the newer model — not the one depicted in the image.) I could not pull the battery cartridge out with my fingers. I wound up using a pair of vise-grip pliers to grasp the cartridge “handle” — with the pliers gripping the short ends of the handle. Just be careful to adjust the pliers so they don’t crack the handle — use just enough force.
By the way, I called Munchkin first, and they suggested using a flat-head screwdriver around the edges — didn’t work. However, the support rep said if that didn’t work, they would provide a free one-time courtesy replacement.