That's more likely due to slight misalignment (which may be due to poor quality --> poor fitting parts.) It's just the LCD being pushed against the midframe. It could be poor quality parts. I've also found if the iPod's bezel has any dents in it this can cause the display to be out of place very slightly causing similar issues. Also make sure the new glue stickers you apply are in the correct position and not overlapping the frame.
It is displaying the symptoms of the video card going bad. Due to lead free solder and inadequate cooling in these imacs, the BGA array of solder balls connecting the video card (7300GT or 7600GT) to the logicboard fail due to whiskers and cracks. You'll need to replace it, best prices are on ebay at around $150-175. Try to get a 7600gt as these are much less prone to failure and if not get a more recent rom version of the 7300gt card. Installation will requite torx screw drivers, and reapplying thermal paste. In the mean time, dl smcfancontrol and raise the CPU and HDD fan speeds to help the card. Booting into safe mode will also help.
Mayer is right.. have done a few of these myself. The install isn't too hard if you know what you're doing. Putting a 17 mobo in won't work at all though, if only for practical reasons, you won't be able to fit it in due to different bracket and screw placements. What you need is a new graphics chip, either the 7300gt or the much better 7600gt. However, these are both hard to come by and can be very expensive when you do. If you're still around I can offer you a replacement for around $150 with a trade in of your old card. Let me know.