As I said in an earlier answer, the connector is called JST GH, so the problem is solved for me.
To elaborate on the original problem:
The available AirPort cards for the Mac Pro don't support 802.11ac, and the Bluetooth modules don't support BT LE, which means they are also incompatible with macOS' handoff features.
The solution is to replace the original AirPort card with a combined Airport/BT card from an iMac via an additional adapter. The original BT connector on the Mac Pro will stay empty, but on the Mac Pro 4,1 / 5,1 you need to deliver an USB signal to the adapter. If you don't want to route a cable from an external USB port back to the inside or have an internal USB port on a PCIe card, you can use the original BT connector on the mainboard to provide the USB signal via a selfmade cable.
Since I didn't want to destroy the original cable, I was looking for the name of the connector for the new cable.