That may depend on the specific device; significant force *was* needed to eject it in the case of the phone I'm working on. Putting it back in, however, was very easy and didn't require any significant effort.
FWIW, I found it easier to replace during re-assembly by turning the phone front-side up, as on the Sprint-locked one I was working on, the SIM card faces that way. Otherwise, it tends to fall out (down) while trying to re-insert it if the phone is face down.
While "This may require a significant amount of force." may vary by individual device, I found it to be true. I used a straightened paper clip held with a (really good) pair of needle nose pliers and pushed straight in until it fully popped out.
Also, depending on the condition of the old (broken?) display, watch for any debris/glass fragments that may obstruct re-assembly. I found that the replacement I was installing wouldn't sit flat, until I checked closely and found a *tiny* sand grain sized bit of glass off to one side of the home button. Once that was removed, the new display fit just fine. It probably can't be emphasized enough not to force anything; if it doesn't fit right, check it - and keep checking until you find the obstruction.