Hi @rulily
Here's a [https://www.youtube.com/watch?v=hzo7OQ2CtZg|video] that shows how to access the battery so that it can be replaced. There doesn't seem to be a "Part 2" video that shows the actual replacement/reassembly but it seems straightforward enough once you have the replacement batteries.
Here's a [https://www.aliexpress.com/item/32916717957.html|supplier] of the batteries. There may be others I didn't look too hard but some of the others I did find are battery manufacturers who have minimum order requirements of more than just 4. Just search for ''604134 LiPO battery,'' as you may find some that I missed.