Don't try to force the connector off or you might hurt yourself on the metal edges of the enclosure. If the connector doesn't come off easily, continue prying up with your spudger until the connector is loose enough to easily disconnect.

Grasp the battery connector by its edges and pull straight up to disconnect it.