The comment Kal left about lifting the retaining flap is key during reassembly!
I, too, totally missed that there is a plastic “bar” covering the end of the pins. Without that lifted up, it’s %#*@ near impossible to push the connector in since the flap/bar is putting pressure downwards.
With the retaining flap down, it comes out without too much effort. But putting it back in is a whole different story. Be sure to lift it up!