How far can you upgrade this iMac?

I would invest in maxing the RAM to 16 GB and swapping out the HDD for a SSD or, adding the SSD as a second drive and make it the boot drive.

The best CPU this system supported was a an Intel i7 2600S. Sadly the systems thermals, power supply & CPU microcode held in the firmware can’t support anything newer than Intel i7 Sandy Bridge CPU

As far as the GPU you would need to find a MXM AMD Radeon HD 6770M with 1GB of RAM or AMD Radeon HD 6970M with 2GB of RAM GPU bd which has the custom Apple firmware. Both are almost impossible to find as many failed or not sold as far as the better 6970 with 2GB of RAM.

So while that will make your system sing, the costs and the usability of the system is getting questionable as most games now require Metal2 which these GPU’s don’t offer. People have managed to use a few other boards but you will need to program the firmware which is not something most people can do.

Maybe it’s time to look at getting a newer M series system, I would wait till Apple drops the expected M4 models this fall (Oct Apple event expected)