Fabulous Adventures In Coding : Metablogging

Sorry about the CAPTCHA

Eric Lippert — Thu, 10 Sep 2009 16:28:00 GMT

A quick metablogging note. Those of you who comment on this blog (6700+ comments and counting, thank you all) have probably noticed that it now has a CAPTCHA, that little "please prove you're a human" test before the comment is posted.

I understand why. The MSDN and TechNet blog sites are high-value targets for unwanted commercial advertisers, for attackers who wish to attempt to influence search engines to drive traffic to their sites, and for vandals. The people who run security for this site have their hands full; we've experienced some pretty serious denials of service based on ham-fisted spammer attacks. Adding a CAPTCHA to regulate comments massively slows down the rate of successful comment spam.

That said, I'm not thrilled about this. I find CAPTCHA-style solutions distasteful for several reasons:

The benign commenter -- precisely the kind of person we want to encourage -- is forced to do additional work. This is a small but nonzero disincentive to writing comments.
Sometimes mistakes will be made; providing new ways that computers can tell us on a daily basis that we are failures seems irksome.
The assumption of innocence is changed to an assumption of guilt; the benign commenter must prove their innocence. Every time I have to fill out a CAPTCHA I feel a small but real insult; I'm a trustworthy person, so trust me already. As Joel Spolsky once pointed out, it's like walking into a train station and the first thing you see is the NO SKATEBOARDING NO PANHANDLING NO THIS NO THAT NO THE OTHER THING sign. Its unwelcoming. It makes you feel attacked and guilty and reminds you that there is evil in the world.
There are accessibility concerns. Not everyone who uses computers has decent vision but that doesn't make them evil robots. They deserve as much as a chance as everyone else to contribute and have to overcome plenty of obstacles already; let's not throw more in their way.
And so on

So, sorry about that, commenters. I don't like it any more than you do, but there's not much I can do about it; I don't run the blog servers. The only thing I can control is how purple the text is.

Fabulous Adventures In Russian

Eric Lippert — Fri, 05 Jun 2009 17:18:00 GMT

I am pleased to announce that a bunch of our Russian MVPs who (1) are awesome people and (2) apparently have some free time on their hands, have started translating this blog into Russian, of all things. So if you want some Фантастичные приключения на русском, check it out!

http://blogs.msdn.com/ruericlippert/default.aspx

Thanks to all the Russian MVPs and particularly to developer evangelist Gaidar Magdanurov for organizing and overseeing this effort.

When Five Hundred Posts You Reach

Eric Lippert — Thu, 28 May 2009 16:55:00 GMT

… look this good you will not. But man, I tell you, the memory goes.

In the July 1985 issue of Fantasy and Science Fiction, Isaac Asimov wrote:

Yesterday I sat down to write my 321st essay for Fantasy and Science Fiction. […] It went swimmingly. I was pleased at the ease with which I worked out its construction. It practically wrote itself and I scarcely had to look anything up. I whistled while I worked. And then, when I reached the last page and launched into my climactic paragraphs, I thought to myself: Why does this suddenly sound familiar to me? […] Hoping earnestly that my memory had misfired, I looked it up. It turned out to be essay #182, first published in the December 1973 issue. There it was. That earlier essay was essentially what I had just written.

The other day I wrote a blog post. It went great. It was strongly opinionated and about an interesting topic. I did a web search to make sure that I had a particular fact straight and… I found my own previous article on the subject. Apparently the late Dr. Asimov and I have at least one thing in common. (I also have something in common with writer Neil Gaiman, but that’s another story.)

It seemed to me looking back that I started off with a huge backlog of old emails answering real user questions which I could use for blog articles, and indeed, when I got started there, I was posting a couple a day. And then over the years I both got busy with other things (writing and editing books, for example) and ran out of canned emails to turn into blog posts, so my rate of writing dropped off. And then lately it seems like I’ve gotten more of a groove back and am doing a couple a week.

I decided to see whether my memory was correct by computing the twenty-post moving average of number of days between successive posts: (Yes, I am a geek, but you already knew that.)

Yep, I think in this case my memory was correct.

Five hundred posts from me and six thousand comments from you all in less than six years seems like rather a lot. I want to take this opportunity to say thanks to everyone who has given me great feedback, interesting questions and technical and moral support over the years. You rock.

Enough chit-chat. On to more fabulous adventures!

Events and Races

Eric Lippert — Wed, 29 Apr 2009 15:00:00 GMT

Here’s a question similar to one I saw on stackoverflow the other day. Suppose you have an event:

public event Action Foo;

The standard pattern for firing this event is:

Action temp = Foo;
if (temp != null)
temp();

What the heck is up with that? Why not just call “Foo()” ?

First off, this pattern ensures that the thing that is invoked is not the null delegate reference, which would cause a null reference exception to be thrown. But if that’s what we want, then surely we could have skipped the temporary variable – just “if (Foo != null) Foo();” would do. Why the temp?

The temporary variable ensures that this is “thread safe”. If the event’s handlers are being modified on another thread it is possible for Foo to be non-null at the point of the check, then set to null on a different thread, and then the invocation throws.

Using the temporary variable effectively makes a copy of the current set of event handlers. Remember, multi-cast delegates are immutable; when you add or remove a handler, you replace the existing multi-cast delegate object with a different delegate object that has different behaviour. You do not modify an existing object, you modify the variable that stores the event handler. Therefore, stashing away the current reference stored in that variable into a temporary variable effectively makes a copy of the current state.

Is that clear? Make sure it is, because now things start to get really confusing.

A common criticism of this pattern is that it trades one race condition for another. Let’s consider that scenario again more carefully:

The event handler contains a single handler, H. Thread Alpha makes a copy of the delegate to H in temp and determines that it is not null. Thread Beta decides that H must no longer be called when the event is fired, so it sets the handler to null. Thread Beta then assumes that H will never be called, and destroys a bunch of state that H needs to execute successfully. Thread Alpha then regains control and executes H, which behaves crazily since its necessary state has been destroyed. Thread Beta’s attempt to ensure that H is not called has been defeated by the race condition.

A frequently-stated principle of good software design is that code which calls attention to bugs by throwing exceptions is better than code which hides bugs by muddling on through and doing the wrong thing. This code has a race condition that results in incorrect behaviour. Perhaps an application of that principle is to stop using a temporary and instead crash if it races. That is to say, this code has a failure mode; surely it is better to highlight that failure with a crisp exception than to turn it into crazy wrong behaviour.

That seems plausibly argued, I agree, but the conclusion is wrong.

Suppose we remove the temporary variable but keep the null check. Does that solve the problem? No! We still have the same race conditions. Suppose the event handler delegate contains a reference to H. Thread Alpha checks to see whether it is null; it is not. Thread Alpha pushes the object to invoke on the runtime stack. Between the push of the delegate value and the call to invoke it, thread Beta sets the event handler to null. And once again, H will be invoked after thread Beta has removed the handler.

Suppose we remove the temporary and the null check and just invoke the delegate directly. Does that help? No, we still have the same race condition. The contents of the event handling variable can still be changed between the push of the delegate object onto the stack and the invocation of the delegate.

Removing the code around the call site does not decrease the number of race conditions in the code, and it does increase the number of crashes. Worse, doing so makes the race condition harder to detect by shrinking the window in which the race can occur without eliminating it.

Removing the null check and/or the temporary is a bad idea; the already-bad situation is only made worse.

Essentially there are two problems here that are being conflated. The two problems are:

1) The event handler delegate can be null at any time.
2) A “stale” handler can be invoked even after it has been removed.

These two problems are actually orthogonal and have different solutions. The onus for solving the first problem is laid upon the code which does the invocation; it is required to ensure that it does not dereference null. The store-in-temporary-and-test pattern ensures that null is never dereferenced. (There are other ways to solve this problem; for example, initializing the handler to have an empty action that is never removed. But doing a null check is the standard pattern.)

The onus for solving the second problem is laid upon the code being invoked; event handlers are required to be robust in the face of being called even after the event has been unsubscribed. In the scenario I described, the bug is actually in H. It needs to be robust enough to check to see whether the state it needs is still there, and bail out cleanly if it is not. (Or, alternatively, some additional locking mechanism needs to be implemented which ensures that the code which fires the event cooperates with the code that changes the event handlers to ensure the desired behaviour.)

The point though is that the "null ref problem" and the "stale handler problem" are two separate problems that are easily confused because the symptoms of both arise at the exact same call site code. You've got to solve both problems if you want to do threadsafe events. How you do so is up to you; just don't confuse the solution of one problem for a solution of the other.

(Thanks to Microsofties Levi Broderick, Chris Burrows, Curt Hagenlocher and Wolf Logan; this article is based on a conversation about their analysis of this pattern.)

Boring Metablogging, Part Two

Eric Lippert — Tue, 07 Oct 2008 18:54:00 GMT

I try to not blog about blogging. Metablogging is boring. The last time I blogged about blogging was on the subject of "Can I take your blog post and republish it?" My answer then was "yes... but". That is, yes, but please tell me, please do not maliciously quote me out of context, please link back to the original, and please don't expect me to correct errors in your copy.

In the world we live in now, a world with various ways of poisoning search engines, that seems a bit naive. Of course, there's little I can do to stop anyone from copying everything I write onto their own servers and then tricking search engines into sending clicks their way, so saying "please don't" is unlikely to help.

My position on blog post republishing is thus basically the same, though I would add one more caveat to that list.

Everything I write at work is copyright Microsoft Corporation. I am not a lawyer of any stripe, certainly not a copyright lawyer, and though I certainly am a Voice Of Microsoft on the subject of the C# compiler, I am definitely not the Voice Of Microsoft on the subject of copyright law.

I write this blog so that the information in it can be disseminated as efficiently as possible to those people whom it helps in their jobs; I appreciate that the whole point of syndicated content is that it can be picked up, aggregated, and republished, and I certainly do not want to work against that or discourage the benign use of such services. But I also do not want to create additional work for myself or liabilities for others. So, please use good judgment and be willing to take responsibility when you disseminate someone else's copyrighted content.

I'd also like to contrast republishing of blog posts with publishing of private emails. This comes up because I recently dashed off an email to a customer in which I made some brief, off-the-cuff comments about some proposed C# language features that had been mentioned on a third-party newsgroup. The customer asked if he could post my emailed comments onto said newsgroup.

I said "please do not", and here's why.

First off, my comments were brief and off-the-cuff. I take a fair amount of time with blog postings to make sure that the prose is polished, the content is complete and accurate, the order in which topics are presented is sensible, that the whole piece flows well. I don't always succeed, but I try. In short, I apply a lower standard to brief emails than I do to public, archived, in-the-world-forever postings.

Second, by allowing the reader to post on my behalf, I would be encouraging him to indulge in hearsay. I have seen my fair share of internet newgroup flame wars, and believe me "Oh yeah, how do I know this is really from a guy at Microsoft?" is an argument you don't want to get into.

Third, all the copyright issues which I mentioned above come up again, even worse. The various third-party discussion sites all have their own policies about who owns the content posted on them and that is just a mess that I do not want to get into or encourage anyone else to get into.

Fourth, by posting my comments in a forum that I don't have posting access to, I have no ability to respond there to further questions, criticisms or follow-ups. I have no ability to correct mistakes in my posting or to add explanatory details should something prove confusing. I have no ability to engage in conversation. I likely have no way of even being informed when a response is made. This sounds suspiciously like its making more work for me in return for less value. It sounds like a good deed once more failing to go unpunished.

So: if you've gotten an email from me and you think that it might make a good newsgroup posting, here is what I encourage you to do:

Send me an email suggesting that I write the email up as a blog post.
After I do so, publish a link to that post in whatever newsgroup you like.

If that's the way it goes then all the objections disappear. I am publishing Microsoft-owned content written by me on a Microsoft-owned site, so there's no copyright issue or hearsay issue. I can make corrections, additions, updates, at will. I can be as prolix as I want to be on my own schedule. I can get reader comments and (time permitting!) respond to some of them. (I have gotten over five thousand comments on this blog; I cannot possibly respond to them all.)

OK, that's enough metablogging for the next four years or so. Back to some fabulous adventures!

Aargh! Part Eight, plus Boring Metablogging

Eric Lippert — Fri, 23 Jul 2004 21:51:00 GMT

Q: What's a pirate's second favourite mode of transportation?

A: A caaaaargh! Preferably a Jaguaaaaargh, but an early Oldsmobile Cutlass will do.

Q: Very amusing -- but what's a pirate's favourite mode of transportation?

A: A pirate ship, silly.

Gripe #10: Don't use _alloca

_alloca allocates memory off the stack instead of the heap. It's very convenient to use when you need a temporary buffer and don't want to worry about freeing it, but I try to avoid it whenever possible for two reasons:

(1) it wrecks the ability for the compiler to make optimizations based on how much stack a function uses, and

(2) when combined with the previous point, it leads to terrible, terrible crashes. If the user asks the machine for lots and lots of heap then the program churns away trying to allocate lots of memory before finally throwing an exception or returning null. The worst that can happen is not too bad. The worst that can happen if the stack is overallocated is that the stack guard page is hit for the second time and the process is terminated. (The exact behaviour of the win32 stack under low-stack conditions is interesting, and I will blog about it at some point; the script engines do some fancy footwork in this department to keep the stack from blowing.)

Here's the thing -- if you have an upper bound on the amount of stack you're going to use, you might as well just ask for it. If you don't have an upper bound -- if the upper bound is determined by something at runtime and it could be arbitrarily big -- then use the heap, because really big stack allocations are too risky.

For example, I'll often see code where _alloca is used to hold a temporary copy of a string when it is being converted from eight bit ASCII to UTF-16. Bad idea, particularly if the string can come from an untrusted internet script. The string could be a million bytes long, and blow the stack! Aaargh! It's drivin' me nuts!

The thing is, stack allocation is incredibly fast, and in many situations the common scenario is for the allocation to be small even if that's not guaranteed. Therefore I often use a "best of both worlds" strategy if I have data that is usually small but might be huge. First, I declare a stack buffer twice the size I expect to use. Then detect if more than that is required and allocate it off the heap if it is. (Of course, you have to remember to clean up the heap buffer if you do that.)

*********************************

You'll note how few new hit television series are called "Extreme Television Editing", or "New Issues In Satellite Uplink Futures Contract Law" -- though come to think of it, either of those would be more likely to catch my interest than the vast majority of the inaptly named "reality" television. Therefore I try to avoid talking about the blog in the blog -- I personally find blogs about blogging boring. I'll keep this short then.

A number of people have asked me questions like "can I quote your blog in my blog?", "can I post your article in my wiki?", "can I use this information in my MSDN Magazine article?" etc.

I'm putting this information on the web because I want it to reach the people who can use it, so by all means, disseminate the information widely. I ask only four things:

1) Please give me a heads-up before you do.
2) Please attribute the source, preferably with a link back to the original.
3) Please don't quote me out of context.
4) Please understand that by doing so, you remove my ability to correct mistakes in your copy.

UPDATE: I have posted a follow-up to these guidelines.

Once More Into The Breach

Eric Lippert — Thu, 11 Dec 2003 22:05:00 GMT

This is the new home of Eric Lippert's Fabulous Adventures In Coding. Eventually the content from http://blogs.gotdotnet.com/ericli will be moved over here

Heads up: Fabulous Adventures will be moving

Eric Lippert — Fri, 05 Dec 2003 21:46:00 GMT

A number of people have pointed out to me over the last few months that this blog server is a little flaky. There are occasionally "out of memory" errors and other failures reported, sometimes my posts fail to actually propagate to the server properly, and so on.

FYI, some time over the next month I'll be moving my blog over to http://weblogs.asp.net/ -- that site has the appropriate human and technical resources to handle the considerable (and unexpected!) response that have been overwhelming the blogs.gotdotnet beta site.

I apologize in advance for the inconvenience. I'll see what I can do about moving existing entries over, putting in the appropriate redirects, etc.