Replace your Shuffle's belt clip.
Before proceeding, be sure your Shuffle is switched off.
Use a combination of iPod opening tools and metal spudgers to pry the rear panel away from the Shuffle by the seam nearest the ON/OFF switch.
Remove the rear panel from the Shuffle.
Remove the four Phillips screws securing the belt clip to the rear panel.
Remove the belt clip from the rear panel.