To remove the bumper, on either sides are little pieces of platic just like the ones we just undid, just push it away from the controller. I like to use the screwdriver for this ones since its a little smaller and gets behind the plastic easier.

I also added a pic of the bumper so you can see how it looks if you need to buy a new one (mine broke within a year).