Common to do this on the Mac Minis. Grab a micro soldering iron, and solder it back on at the anchor points.
If the connector is destroyed, replace it with a new one http://uk.rs-online.com/web/p/pcb-headers/7000833/?searchTerm=78171-0004&relevancy-data=636F3D3126696E3D4931384E4B6E6F776E41734D504E266C753D656E266D6D3D6D61746368616C6C7061727469616C26706D3D5E285C642B5B2D5C2E5C732F5D292B285C642B293F247C5E5C642B2426706F3D313326736E3D592673743D4B4559574F52445F53494E474C455F4F525F4D554C54495F4E554D455249432677633D424F5448267573743D37383137312D30303034267374613D37383137312D3030303426
If some of the pads are pulled off, you will need to solder wires to the correct location, and will need a schematic and board view.
Good luck!