My controllers won't turn on unless I use a USB cable

2 of my 3 Xbox one controllers (I got them used) have the same issue:

They won't turn on with the Xbox button, no matter which battery I use (normal battery or play and charge battery).

They turn on and work perfectly fine with a USB cable.

I can then unplug them and they keep working wirelessly. I can then switch them off by holding the Xbox button.

So it's not an issue with the Xbox button nor with the battery contact.

I updated the firmware and did a reset of both controllers, no change.

It's weird since they both have this same issue, and I can't find anybody online that has it.

Do you guys have any idea of what it could be coming from?