Sadly, you are trying to push your system beyond what it can support! This series Max memory is only 2GB.
Here’s the specs for your system: [https://everymac.com/systems/apple/imac/specs/imac-core-2-duo-1.83-17-inch-specs.html|iMac 17-Inch 1.83 GHz Core 2 Duo (iMac5,2)] This model uses the onboard intergraded graphics engine (IG). This is the reason your system is limited! The iMac5,1 models have a dedicated graphics setup (Radeon X1600) thus it has its own graphics memory and don’t need to lock the RAM address space for the internal graphics engine like your system. As an example [https://everymac.com/systems/apple/imac/specs/imac-core-2-duo-2.0-17-inch-specs.html|iMac 17-Inch 2.0 GHz Core 2 Duo (iMac5,1)] in this model it can address 3 GB of RAM (using either one 2GB and one 1GB DIMMs or using two 2GB DIMM’s) Only 3 GB is accessible by the OS.