Solvents used for cleaning brake hardware are harmful to skin. Always wear gloves when dealing with these chemicals.

Thoroughly spray the inside of the brake drum with a brake cleaning solvent.

Once the solvent has had a chance to work its magic (10 minutes or so), use a rough-grit sandpaper (100 grit or so) to clean the inner diameter of the drum. Rinse and repeat until you are satisfied with the cleanliness of the drum.