Where did you place the antenna? The Mac Pro is a metal box and will likely shield an antenna which is located inside of it, unless it's in a similar location to the original antenna.
In terms of connections, you should carefully check that the connector for the antenna is actually the correct one for the bluetooth module. There are a few different small-form-factor connectors that these types of cards use, and while they generally look alike, they aren't all compatible with each other.