One of your ribbon cables is probably partially broken, and only works at a certain angle. It's also possible that the light sensor is broken...but that is probably unlikely. I would try with a display that you are certain is working (a new one). Depending on where you got it from, they may send you a replacement. Best of luck!
No, do not remove the black rubber. There is a special 3M clear double-sided adhesive made specifically for this device--you should use that to reattach the digitizer and screen assembly. (Look it up as "iPod touch 4 adhesive" on eBay or Amazon.) It often comes in the full size with the outer frame pre-cut but not removed; that's what you will need. Good luck! Hope this helps.
