Josh Williams wrote a nice summary of the big performance issues when considering x64.

It's very succinct but it hits the main hightlights, especially worth noticing is the effect cache size reduction due to pointer inflation competing against the general goodness of having more memory.

I twisted his arm and made him post it so I could link to it.  Thanks Josh :)