I think the Silent Switch was very poor constructed but yours may not be broken.
Here is the fix way:
Open your iPhone and remove #1~3 connector to remove Display Assy.
Check that small Silver Prong on the Display Assy (which is located above Silent Switch unit) wasn't bend too deep or so. If so, try to bend them back enough to touch the Silent Switch when display fitted in.
And also you need to do something like tuning on this. Look carefully that Silent Switch unit springs back when you going into Silent Mode. I think that cause it to touch Silver Prong and vibration made by instant electrical circuit for a little while.
So tune it up by a slight bending or something.
Be careful not to break Silent Switch unit. As you know it is included in Audio Jack Connector Flex Cable. The problem is that it may expensive and take so much time to be fitted in.
And there should be several difference on replacement parts. I think that won't cause a big problem but fix original one is the best way.
But note that if you have something wrong with units on Flex Cable, for example Sleep Button or Volume Button, you have to replace it.
Good Luck!