The bottom of the battery housing has holes, which allow you to push the battery out. The flat side of a spudger fits perfectly through the widest opening, marked here in red.

Insert the flat end of a spudger (or another plastic prying tool) into the bottom of the battery housing and push the battery out the other side.