The best explanation of this I have yet found is here; with this article as a supplement.
This explanation makes it pretty clear that even if Windows 32-bit doesn't report the memory, it is getting used by some part of the hardware.
If you had less memory, less would be reported since the hardware still requires that reserved memory space.
Note that PCI-E maps video memory into the 4G addressable space and thus leaves less system memory available for applications. So a 768m graphics card reduces the memory space "seen" by the OS as available to applications in terms of both physical memory and process address space.
Still, any physical RAM above 2G means standard 32-bit processes stand a chance of fitting entirely into RAM together with the OS which would mean zero page faults to virtual memory pages on the hard disk. That is certainly good for performance so it is worth the effort to try to stick to the mobo/chipset combos that support addressing 4G of RAM.
Finally, notice the potential for PAE and DEP errors. This helps to explain some of the DEP errors the community sees from time to time.