Wear a face mask, and then using a dremel, carefully remove some of the nylon all the way around until the bearing slides all the way down. This took about an hour. The stem was mounted on a vertical stick so it could be rotated while using the dremel.

Use 150 sandpaper to smooth out the edges of the dremel wheel.

Verify that the new bearing fits square to the mark of the old bearing. Note that the mark will be worn down by the dremel , so keep re-marking it with the old bearing.