With the PSP's faceplate removed, position the PSP so that the analogue stick is easily attainable.

Gently pull up on the analogue stick. It should just pop off from the rest of the device.

In order to clean it correctly, take a cotton swab w/ rubbing alcohol and clean the base of the analogue stick.

The connection point on the PSP should be done as well (where the analogue stick meets the PSP).