As I alluded to previously, this has gotten to be a more and more interesting question lately. One of the things I've been kicking around is a sort of flowchart so that people who don't study this stuff will come to the correct conclusion more often. It's also a question of what counter-measures are in place, and which version. For example, David Litchfield made it clear that with the Visual C++ 7.0 version of /GS, all it takes is to whack an exception handler and cause an exception to take control of an application. But then in the 7.1 compiler, there's SafeSEH, which does a decent job of thwarting this attack, and in the 8.0 compiler, the exception record is even more well guarded. So here's some notes about what things will and won't do –
ASLR – as I said last post, this is a speed bump. It makes it harder to write worms, but it won't help much (sometimes at all) against local attacks. It's also true that if the exploit depends on a relative offset instead of an absolute offset, this may not be much help at all. If you abuse exception handlers, you can undo any good this might have gotten you.
SafeSEH – keeps exception handlers from getting whacked, but if you load a DLL that doesn't have this, it's exception handlers could get whacked, and just about anything in the DLL could be fair game for being called. It's a good countermeasure, but know the limits. Note – if it is a 64-bit binary, then the exception handlers are hard-coded in the binary, are read-only, and this attack is foiled.
NX – probably one of the most potent of the countermeasures, when it can be turned on. Theoretically, you could VirtualProtect something into being executable, but as had been pointed out, this is practically difficult. Can be thwarted by return into libc, or even a return into your code that happens to do just what the attacker wants. This plus ASLR can make life really difficult for the exploit writers.
64-bit – this changes the game quite a bit. The default calling convention is fastcall – with a twist – in fastcall, the first parameter is passed in a register on 32-bit, but on 64-bit, the first _4_ parameters are passed in registers. A whole lot of stuff is different, and harder to attack. Unbreakable? I don't think so. Harder? I think it will be, especially until everyone gets the hang of it.
What's still out there? There's certainly data-driven attacks – if you can munge up the argument to LoadLIbrary, that's the same as running completely arbitrary code.
Interesting stuff – Low rights (protected mode IE) is a really cool feature. Limitations – it can still read your data, but can't trash your system or even your personal settings. Something else really cool I've written articles on in the past are restricted tokens. I've learned more about these in the last couple of years (and some more in the last month or so). You can use these to create a process running as essentially nobody, which is a neat thing to be able to do. Yeah, yeah, UNIX has had it for ages, and I still can't chroot, but I can still do some cool stuff with it, and I'm personally more concerned about what can be done with Windows security. I'll spend more time on this later.