Startup, Shutdown and related matters

RE: Startup, Shutdown & related matters

David McNamee — Wed, 20 Aug 2003 19:47:00 GMT

Great post, Chris! With all the security discussions going on right now, your "security addendum" deserves to be in its own separate post. Some folks might miss it with it tacked on the end of a CLR internals discussion.

RE: Startup, Shutdown & related matters

M. Keith Warren — Wed, 20 Aug 2003 21:12:00 GMT

Congrats Chris, you have won the award for the web longest blog entry! Great stuff!

RE: Startup, Shutdown & related matters

Anonymous — Wed, 20 Aug 2003 23:18:00 GMT

I agree. I'm gonna pull it out and post it on my blog. It deserves to be seen. Great stuff Chris!

RE: Startup, Shutdown & related matters

Phil — Thu, 21 Aug 2003 00:22:00 GMT

Awesome post, Chris! I look forward to your detailed and insightfull posts on misunderstood topics. I think that a separate security post would be great!

RE: Startup, Shutdown & related matters

Jamie Cansdale — Thu, 21 Aug 2003 00:52:00 GMT

What an amazing bit of synchronicity! A couple of days ago I was going to email you about a shutdown issue that has been driving me nuts. I even got to the point of putting together a simple test case to show you. The test case loads Adam Nathans CliSpy into a new app domain, waits for a few seconds, calls Application.Exit(…) and finally AppDomain.Unload(…). The problem I’m having is with AppDomain.Unload(..). It claims that it can’t be unloaded because a thread can’t be unwound out of it. Worst of all the attempted unload was causing the process to freeze at a later point during shutdown. After a bit of experimentation I discovered what you were saying about app domains not being unloaded on shutdown. This meant that if I know the process was being shutdown, I could just skip the unload attempt. It is still annoying if certain applications can render an app domain immortal! Is there anything else I can do to encourage this app domain to unload? I’m certainly willing to use P/Invoke if there’s something I could do that wouldn’t be too damaging to my code Karma.. ;o) I’ve put the test code and exe here.. http://www.managedaddins.net/downloads/ClrSpyShutdown.zip Thanks once again for a great article!

RE: Startup, Shutdown & related matters

Marco Russo — Thu, 21 Aug 2003 00:58:00 GMT

Great post, Chris! One question: when you talk about "Monitor.Block" primitive, what are you talking about?

RE: Startup, Shutdown & related matters

Chris Brumme — Thu, 21 Aug 2003 02:09:00 GMT

Monitor.Block -- sorry, at some point before shipping V1 we changed the name of this primitive from Block to Monitor.Wait. This fits in better with the names given to the equivalent operations on a Win32 waitable object.

RE: Startup, Shutdown & related matters

Richard Tallent — Thu, 21 Aug 2003 07:30:00 GMT

Bah! I just use the End statement in VB.NET. Works just fine for me. ;) Seriously, very good post / article / tome...

RE: Startup, Shutdown & related matters

Jim Argeropoulos — Fri, 22 Aug 2003 16:19:00 GMT

Chris can you point to a good "doing it right" information source on the below quote: "Incidentally, I find it disturbing that there’s often little discipline in how managed locks like Monitors are used. These locks are so convenient, particularly when exposed with language constructs like C# lock and VB.NET SyncLock (which handle backing out of the lock during exceptions), that many developers ignore good hygiene when using them. For example, if code uses multiple locks then these locks should typically be ranked so that they are always acquired in a predictable order. This is one common technique for avoiding deadlocks."

RE: Startup, Shutdown & related matters

Chris Brumme — Fri, 22 Aug 2003 16:58:00 GMT

I can't point to any docs on "doing it right." There must be some, but I've never read them. Here are some common ideas: 1) Rank all your locks, so they are always taken in a consistent order. This prevents lock inversions, which is a classic source of deadlock. Don't cheat by having some spin locks or by using an AutoResetEvent to achieve lock behavior. If they act like locks, they all need to be ranked. Another common technique for cheating is to allow multiple locks to be taken in any order "at the same level". That's like not having any leveling. You have to be rigorous here. 2) Don't call out to someone else's code while you hold a lock. Unlike #1, this lock must sometimes be broken. For example, if you have virtual methods on your unsealed base class, you might find it necessary to hold a lock while you call that virtual. If so, try to educate whoever specializes your base, so they understand and conform to your locking discipline. Unless the other party understands your rules, he is likely to acquire his own locks. Since his locks are not formally ranked with respect to your locks, this will likely break rule #1. 3) Consider wrapping Monitor or whatever lock you are using. Both Monitor and the OS CRITICAL_SECTION implicitly support recursive acquisition. This is convenient. But generally it is unhygienic. Instead, your wrapper for Monitor can bump a counter to indicate that the lock is held. For most locks, you should blow up if a recursive acquisition is attempted. You may want to conditionally compile this check, so it disappears in your retail product. 4) Don't build your own locks. It's unlikely that you will pick appropriate spin counts for the multi-processor machine you are running on. It is unlikely that you will yield correctly when executing on a hyper-threaded P4. It is unlikely that you will properly rely on fiber-affinity rather than thread-affinity for your lock ownership, when running in a fiber-scheduled environment like SQL Server. It is unlikely that you will correctly notify the CLR of your locked regions, so that upcoming reliability work will respect the critical nature of your locks. Frankly, it's just very hard to build a good lock. (Monitor is significantly better now than it was in V1). 5) Consider a stress mode to break loose any race conditions in your code. For example, we sometimes run in a mode where we yield the processor immediately after releasing a lock. That's so, if we are publishing data inside the lock and then we finish preparing it outside the lock, we are more likely to find these bugs. Of course, it's handy to test on a machine with lots of processors, where you have many threads -- perhaps artificially -- racing through the same operations. You could add this Yield idea using the wrapper that I suggested in #3 above. Over time, the CLR and tools like Visual Studio should be doing more to help developers with this sort of thing. For example, we could track what data is touched under what locks. As the program runs, we would prune the set of locks that potentially cover a piece of shared data. If that set becomes empty, the data is probably unprotected and we've found a bug. I've seen a lot of very interesting variants of this sort of thread-safety test mechanism. Some are based on static analysis and others are based on dynamic analysis. Microsoft internally has invested in both approaches. I have to admit that I don't know how well they work in practice. For example, managed applications recycle their shared state at a high rate, thanks to the GC. So we might never detect unlocked access before the data is reclaimed, even if the program has a bug. Nevertheless, they are an intriguing direction and we might see something in our toolset a decade from now.

RE: Startup, Shutdown & related matters

KiwiBlue — Fri, 22 Aug 2003 17:50:00 GMT

Congrats, Chris. This is by far the best stuff from Microsofties since Victor Stone's musings :)

RE: Startup, Shutdown & related matters

Dmitriy Zaslavskiy — Fri, 22 Aug 2003 21:01:00 GMT

Chris your rss is not being updated. Any reason ?

RE: Startup, Shutdown & related matters

Braet — Fri, 22 Aug 2003 23:24:00 GMT

Oh my god. What sort of abomination of code requires all these evil hacks and has so many problems? Why, win32 of course. I think I just lost SAN. *runs away to POSIX*

RE: Startup, Shutdown & related matters

Craig — Sat, 23 Aug 2003 00:06:00 GMT

Holy crap - this reminds me of my past - and the mess that is Windows. An absolute hard rule of an OS is that an application - especially a user application, not an OS-related utility - should NOT be responsible for maintaining OS integrity. This is violated so many ways in Windows it's truly sad. The violation of this rule combined with the low skillz of many Windows programmers is the true reason for the problems in the Windows world. Any why the heck is it that an in-use DLL or EXE *still* cannot be updated in Windows? The reboot-after-patch (but not patched until reboot) is cause for many of the security issues with MSBlaster. My wife just moves the "Press OK to reboot" screen off the desktop since she's too busy to incur the painful context switch of a reboot. Yes, for those of you that don't know, every Unix-variant since the 70's or so can have a file replaced while in use - and beautifully so. A service can be updated while running. New connections will get the new copy. Running referenced maintain the old. Get with the times. And putting metadata in the filesystem is NOT the answer... Craig

RE: Startup, Shutdown & related matters

Ross — Tue, 26 Aug 2003 15:51:00 GMT

Sheesh write a book already !! :) Seriously though, an interesting article. Thanks for spending the time enlightening all us poor souls stuck in MS-land ... I'm off to re-work some code ;)

RE: Startup, Shutdown & related matters

Chris Brumme — Wed, 27 Aug 2003 05:28:00 GMT

Craig, The operating system's integrity is not violated by the loader lock issues. The only risk is that an application will hork itself, just as it could with any other user-mode lock. This particular lock causes so many problems because of all the bad behaviors that have grown up around it, like the CRT's leak detection code. I agree that hot patching of OS libraries is an important feature for an OS to provide. This is particularly important these days, since the rate of patching is increasing. I don't think I want to pay for hot patching of user libraries in user processes. Partly this is because I don't see a lot of patches to user code. Partly this is because I'm more willing to recycle a user process if I need to apply a patch to user code. And mostly this is because the interactions between user libraries are much more fine-grained. There aren't a lot of DLL exports from kernel32.dll or ntdll.dll. And all those exports are nice flat APIs. But the exports out of user libraries are more "intricate". Because of object-oriented programming, you see dependencies on VTable offsets or instance field offsets or global state layout being communicated across DLLs. I don't want to give up those rich dependencies (now that I have managed code to solve the historical C++ brittle class problem). So I'm willing to give up hot patching for these cases. Obviously my goal in this blog is not to start a religious OS war. I don't know enough about different OS'es to even have something interesting to say. I really want to focus on how the CLR works. Sometimes it's hard to talk about CLR internals without also discussing how the underlying OS works. As with any mature code base, compatibiity requirements prevent Win32's application model from being entirely clean. But I believe it has a compelling value proposition for customers and for developers. The phenomenal adoption of Win32 supports that belief.

RE: Startup, Shutdown & related matters

Craig — Fri, 29 Aug 2003 12:43:00 GMT

OK Chris, I have to take your word that a process can only "hork" itself with loader lock issues. I was more drawing on my own experiences with Windows middleware when talking about app/os integrity. In regards to hot patching, I believe this comes to the very core of the disparity in reliability between Windows and Unix. Patching a running process is - of course - a very nasty and dangerous thing to do. Let's just call it practically impossible. Not that I should be telling this to anyone at MS, ;^) but the real edge to Unix is a combination of filesystem semantics and the process model. I already mentioned the semantics - at least for local file systems. An open file can be overwritten - be it a .so (dll), executable, or a 12GB database file. The trick is that the "files" are just references and a file name is just a named reference. When you "overwrite" a file, you're really creating a new file and changing the named reference - the file path - which is how most things find the file. After dereferencing the name, the process just has a handle on the actual file. The file name(s) (one file can have multiple names via hard links) can change all over the place - and even be deleted. The file space is not reclaimed until the file's refcount is 0. Now, the reason this works well is the model/convention used by Unix programs and services (no distinction in Unix, really). Processes are King. And this is good (shock, horror). But realize how much lighter they are and that they've had the s*** optimized out of them under Unix, partly because that's all there was for years. Only systems like VMS, Windows, and app programing have really required thread technology. Why? Processes are too fricking heavy in these systems. And people don't want to wait for apps to save files, print, and such. But when you think about it, realize how much harder it is to write a Windows service, which loads at system start time, and ends at reboot? You leak even 1 byte per transaction and you're going to be in trouble. One thread causes nasty, and 1000 of connections are dropped - not to mention the security implications. And this only gets worse when a VM and Gc are introduced to the mix ala CLR. Only the most rigorous time-tested and high-loaded services run multi-threaded under Unix. Apache is the prime example. But even here, multi-threaded execution is optional. And yes, it's still very fast. The nice thing about the fork-per-connection model for Unix, is that each transaction runs in a bubble. It has it's connection. It has its file handles (shared libs included), and it has it's job. If the service or constituent libraries, config files, etc. are updated, and the daemon (the fork-er) restarted, new connections will get new code/config. Old ones will keep running - but only long enough to service their connection - since they just have their job to perform: take care of the connection. Without both of these things, I'm not sure how hot updates can safely happen. Sorry for the off-topic rant. Hope I kept it technical enough to be considered non-religious. I still don't understand how people are able to keep up with all this stuff in Windows. I think they need to relax the hiring process up there and get some more dumies/typical programmers in there. |^]

RE: Startup, Shutdown & related matters

Blake — Mon, 08 Sep 2003 19:18:00 GMT

Craig You miss the mark on two very key points. The first one is in your characterization of how almost all Unix services fork a process per connection. This is just completely wrong. Only the older/naive/not-scalable Unix services fork per connection. The newer, higher performance and more scalable ones all use threads or async I/O. You then miss the mark completely with this line "And this only gets worse when a VM and Gc are introduced to the mix ala CLR". If you insist on being a Unix-bigot, reconsider that same assertion 'ala' your favorite Java app server instead. Managed environments and application servers go a _long_ way towards addressing the complexity of writing scalable services.

RE: Startup, Shutdown & related matters

Scott — Thu, 18 Sep 2003 19:12:00 GMT

Wonder what happened to that "five 9's" uptime that MS was parading around a few years back? Oh well, have to find another marketing fallacy for this year.

re: Some reasons not to do anything scary in your DllMain

The Old New Thing — Wed, 28 Jan 2004 09:46:00 GMT

Apartments and Pumping in the CLR

cbrumme's WebLog — Mon, 02 Feb 2004 19:52:00 GMT

Apartments and Pumping in the CLR - Translate to Japanese 無茶してみる

福井厚のBlog — Mon, 16 Feb 2004 04:29:00 GMT

?????Blog ?????????
Apartments and Pumping in the CLR - Translate to Japanese ??????
???????????????????????????

?????????????????????????????????????(T_T)
???????????????????????

Finalization

cbrumme's WebLog — Sat, 21 Feb 2004 06:27:00 GMT

Apartments and Pumping in the CLR - Translate to Japanese 無茶してみるその４

福井厚のBlog — Sun, 22 Feb 2004 19:57:00 GMT

?????Blog ?????????
?????? ??3?????? Apartments and Pumping in the CLR ???
?????????????????????(^_^;) ????????????????????

Apartments and Pumping in the CLR - Translate to Japanese 無茶してみる

福井厚のBlog — Sun, 22 Feb 2004 19:58:00 GMT

?????Blog ?????????
Apartments and Pumping in the CLR - Translate to Japanese ??????
???????????????????????????

?????????????????????????????????????(T_T)
???????????????????????

re: IServiceProvider and Plugin Architecture

Something Clever — Sun, 09 May 2004 06:29:00 GMT

Maybe you can help me

Pedro Abreu — Wed, 30 Jun 2004 18:28:00 GMT

Hi

I'm quite new at windows forms programming and i don´t really understand most parts of your article.

I'm having a problem wich is:

I have a form that works like a browser and it works fine... at least until i leave the application and it throw me an error wich look like this:

0xc0020001

(NOTE: If i don´t open the browser form it doesn't throw any error)

I copy the same form into onother project and the error was no longer!

Does any one has any sugestions of what might be leading my application to crash on exit?!?

Thanks and sorry my english ;)

re: Startup, Shutdown & related matters

Chris Brumme — Wed, 30 Jun 2004 18:50:00 GMT

I just answered a similar question at http://blogs.msdn.com/cbrumme/archive/2003/04/15/51318.aspx. The offer I made there extends to you also.

One way to dodge a bullet

Dan — Mon, 12 Jul 2004 21:07:00 GMT

It's been so long since the original 'blog entry that I don't know if anyone will see this, but I have a handy workaround that I used to dodge a particular bullet when loading a mixed-mode (IJW) DLL: "/DELAYLOAD:YourMixedMode.dll". Pass that flag to the linker, and link with delayimp.obj, and as long as none of the other DLLs in your process call into the mixed-mode DLL during startup, the CLR won't get loaded until long after all the other DLLs are properly initialized. This protects you from mscorwks.dll calling into, say, shell32.dll before shell32 has been initialized...

What's on my blog-todo list.

Mike Stall's .NET Debugging Blog — Sun, 28 Aug 2005 07:25:24 GMT

At this point, I've got what seems to be an endlessly long list of things I'd like to eventually blog...

More Thread.Abort Goodness

Greg Young — Sat, 11 Feb 2006 12:41:10 GMT

More Thread.Abort Goodness

Greg Young — Sat, 11 Feb 2006 12:47:11 GMT

Weiser Lock - Lock Links

Weiser Lock - Lock Links — Mon, 03 Apr 2006 04:51:16 GMT

PingBack from http://lock.fythd.com/archive/weiser-lock/

Lock » Definition of lock - Merriam-Webster Online Dictionary

Lock » Definition of lock - Merriam-Webster Online Dictionary — Tue, 04 Jul 2006 11:33:11 GMT

PingBack from http://lock.espbx.com/archive/definition-of-lock-merriam-webster-online-dictionary/

__LOCKUP_www.lockup666.com___ - Lock Links

__LOCKUP_www.lockup666.com___ - Lock Links — Mon, 31 Jul 2006 06:26:54 GMT

PingBack from http://lock.fythd.com/archive/__lockup_wwwlockup666com___/

cbrumme’s WebLog : Startup, Shutdown & related matters - Lock Links

cbrumme’s WebLog : Startup, Shutdown & related matters - Lock Links — Thu, 12 Oct 2006 14:04:00 GMT

PingBack from http://lock.fythd.com/archive/cbrummes-weblog-startup-shutdown-related-matters/

Finalization « Jiddy’s Weblog

Finalization « Jiddy’s Weblog — Thu, 29 May 2008 23:55:10 GMT

PingBack from http://jiddy55.wordpress.com/2008/05/29/finalization/

cbrumme s WebLog Startup Shutdown and related matters | Paid Surveys

cbrumme s WebLog Startup Shutdown and related matters | Paid Surveys — Sat, 30 May 2009 01:11:45 GMT

PingBack from http://paidsurveyshub.info/story.php?title=cbrumme-s-weblog-startup-shutdown-and-related-matters

cbrumme s WebLog Startup Shutdown and related matters | Outdoor Ceiling Fans

cbrumme s WebLog Startup Shutdown and related matters | Outdoor Ceiling Fans — Sun, 31 May 2009 20:38:30 GMT

PingBack from http://outdoorceilingfansite.info/story.php?id=22726

cbrumme s WebLog Startup Shutdown and related matters | Menopause Relief

cbrumme s WebLog Startup Shutdown and related matters | Menopause Relief — Tue, 09 Jun 2009 03:02:33 GMT

PingBack from http://menopausereliefsite.info/story.php?id=17

cbrumme s WebLog Startup Shutdown and related matters | Cellulite Creams

cbrumme s WebLog Startup Shutdown and related matters | Cellulite Creams — Tue, 09 Jun 2009 05:37:13 GMT

PingBack from http://cellulitecreamsite.info/story.php?id=11019

cbrumme s WebLog Startup Shutdown and related matters | Green Tea Fat Burner

cbrumme s WebLog Startup Shutdown and related matters | Green Tea Fat Burner — Tue, 09 Jun 2009 22:11:12 GMT

PingBack from http://greenteafatburner.info/story.php?id=357

cbrumme s WebLog Startup Shutdown and related matters | debt solutions

cbrumme s WebLog Startup Shutdown and related matters | debt solutions — Tue, 16 Jun 2009 03:40:50 GMT

PingBack from http://debtsolutionsnow.info/story.php?id=6729

cbrumme s WebLog Startup Shutdown and related matters | alternative dating

cbrumme s WebLog Startup Shutdown and related matters | alternative dating — Wed, 17 Jun 2009 10:39:25 GMT

PingBack from http://topalternativedating.info/story.php?id=1427

cbrumme s WebLog Startup Shutdown and related matters | patio set

cbrumme s WebLog Startup Shutdown and related matters | patio set — Thu, 18 Jun 2009 05:14:56 GMT

PingBack from http://patiosetsite.info/story.php?id=595

Startup, Shutdown and related matters

RE: Startup, Shutdown & related matters

RE: Startup, Shutdown & related matters

RE: Startup, Shutdown & related matters

RE: Startup, Shutdown & related matters

RE: Startup, Shutdown & related matters

RE: Startup, Shutdown & related matters

RE: Startup, Shutdown & related matters

RE: Startup, Shutdown & related matters

RE: Startup, Shutdown & related matters

RE: Startup, Shutdown & related matters

RE: Startup, Shutdown & related matters

RE: Startup, Shutdown & related matters

RE: Startup, Shutdown & related matters

RE: Startup, Shutdown & related matters

RE: Startup, Shutdown & related matters

RE: Startup, Shutdown & related matters

RE: Startup, Shutdown & related matters

RE: Startup, Shutdown & related matters

RE: Startup, Shutdown & related matters

re: Some reasons not to do anything scary in your DllMain

Apartments and Pumping in the CLR

Apartments and Pumping in the CLR - Translate to Japanese 無茶してみる

Finalization

Apartments and Pumping in the CLR - Translate to Japanese 無茶してみる その４

Apartments and Pumping in the CLR - Translate to Japanese 無茶してみる

re: IServiceProvider and Plugin Architecture

Maybe you can help me

re: Startup, Shutdown & related matters

One way to dodge a bullet

What's on my blog-todo list.

More Thread.Abort Goodness

More Thread.Abort Goodness

Weiser Lock - Lock Links

Lock » Definition of lock - Merriam-Webster Online Dictionary

__LOCKUP_www.lockup666.com___ - Lock Links

cbrumme’s WebLog : Startup, Shutdown & related matters - Lock Links

Finalization « Jiddy’s Weblog

cbrumme s WebLog Startup Shutdown and related matters | Paid Surveys

cbrumme s WebLog Startup Shutdown and related matters | Outdoor Ceiling Fans

cbrumme s WebLog Startup Shutdown and related matters | Menopause Relief

cbrumme s WebLog Startup Shutdown and related matters | Cellulite Creams

cbrumme s WebLog Startup Shutdown and related matters | Green Tea Fat Burner

cbrumme s WebLog Startup Shutdown and related matters | debt solutions

cbrumme s WebLog Startup Shutdown and related matters | alternative dating

cbrumme s WebLog Startup Shutdown and related matters | patio set

Apartments and Pumping in the CLR - Translate to Japanese 無茶してみるその４