Can I exceed maximum RAM?

Sorry, 8 GB is the physical systems limit, the CPU's address lines are limited to 8 GB.

If your system is slow it might not be a memory issue at all!

Have you cleaned out the old cache & log files? Here's one tool that can make this easy: Disk Doctor.

The next issue is checking the health of your drive, but before you do that you should make sure you're files are fully backed up. Using an external bootable disk boot up your system and run Disk Utility from it to test & repair your internal drive.

So far we've done the basics. Now the hard part, you need to leave at least 1/4 of the drive unused. Since you are working with larger files I think you'll likely need more space than that, see if you can free up 1/3 of the disk. You'll likely need to defrag the drive as well. You'll need something like this app: Drive Genius this will re-link the blocks so things will be quicker.

While that gets us as far as we can go with what your system came with you might want to look at going with a newer drive like a SSHD or if you can swing it a SSD. Seagate makes a good SSHD that will work here: Seagate Laptop SSHD I would go with the drive with 32 GB of Flash.

If you can afford a SSD that would be the best for a performance. Here you'll likely need a 1 or 2 TB drive (as you're working with large files). Sadly this will be expensive! Samsung makes a larger SSD that will work here: Samsung 850 EVO

If you have an external FireWire drive you could go with a smaller SSD by placing your work on the external.