Michael Stuart's Weblog

Incommunicado for TWO YEARS...Yes, I have contributed to the gene pool

2007-10-16T00:24:00Z

My charming and uber-tolerant wife has given me a beautiful baby girl, Selkie. This happened 21 months ago, but it's only now I find a spare 30 seconds to blog about it.

LOL, though, because I was that guy who variously poked fun at my friends with children or scorned them for being behind on the newest movies/games. Well, I haven't bought a new XBox. I don't own Halo 3 yet. I haven't even finished Halo 2, let alone Half-Life 2.

About the only thing I have completed is my second bass horn...and that took two years. More on exponential horns and bass later.

The great consolation and hope I can offer prospective fathers is HOLY CRUD it's fun. I'm enjoying her more every day; she's at that n^4 cognitive stage where they pick up half a dozen new words a day. You can practically HEAR the synapses firing as new dendrites sprout. I'm surprised I don't have to mount one of those lovely copper-flower Zalman cooling fans on her head because that CPU must be dissipating 100W easily :)

Spent a year doing Java, enjoyed it, lovely language, but back to C# 2 and 3. C# is definitely the better language now; Java will be with us for lo many years, but I believe there's more room in the CLR's architecture for neater and neater tricks in the future.

Recent fun projects:

Asynchronous Command Execution Service (ACES)--reads ServiceAgents from any "queue" source, which depending on the provider can be MSMQ, MQSeries, a DB table, whatever. The ServiceAgent adheres to IServiceAgent and gets de-serialized from the queue provider, then called in on Execute(). The ACES manages a common thread pool, assigns agents to them, monitors their health, kills them if they go off to never-never land, etc. The beauty of it is, it can read from as many sources as you configure, and can be distributed across as many machines as you like. We're using it to do everything from asynchronous exception logging, code audit logging, to running 24-hour queries for report generation.

RulesEngine--not exactly a green-field application, but tailored well to our needs. Takes any expression with arithmetic, set, boolean, and statistical operators and evaluates them. Dynamically parses DataSet tables and columns, fetches the data, and operates on it.

Leaving the nest at Microsoft...Bigger Things Afoot

2005-06-19T00:47:00Z

After five fine years, I'm leaving Microsoft.

I couldn't have hoped for a better experience. I've learned more in five years than I did in four of medical school; deep technical sk1llz, consulting skills, how businesses operate, how to develop _products_ not just projects...an amazing amount, a firehose.

Microsoft is one of those rare places that inspire real passion in people. I have a Windows XP logo tatto'ed on my right shoulder...yes, it's real and it's permanent (to paraphrase Terri Hatcher in Seinfeld). I've yelled myself hoarse at Steve Ballmer's annual ra-fests; I was right there at "developers developers developers developers..." Most of the time, I got up in the morning excited about going to work.

It wasn't all rosy and wonderful; like any company there are imperfections. I'd like to post a constructive commentary on that in a later blog.

I'm going to a super-cool small software company here in Houston called "PROS Pricing Solutions". They are the forerunners in a field variously named but herein "revenue optimization". Basically, I have something that is valuable NOW, but in a while it will expire; what should I sell it for to maximize my profit? Applies to airline seats, hotel rooms, transmission capacity, you name it. Takes serious, heavy-duty math and code to grind an answer. My PhD buddy John Salch (was in Rice grad school while I was in med school) will be filling me in on the gory details.

I'll be tech-lead doing hard-core development--my favorite thing to do in the whole wide world. I haven't met everyone on my team yet; but judging from my interviewers (three PhD's and a very smart developer) they will be stellar.

It's going to be a pleasure to focus on ONE THING, and do it exceedingly well, producing a product.

The company is about fifteen minutes' drive from my house...hey maybe I'll get one of those uber-dork Segway thingies, you know, "Ginger" :)

I'll also get to play with...er, um...alternate tech such as ...er, um....the "J" word and the "U" word. I'm really keen though; it will be nice to become multi-lingual and multi-platform.

Stay tuned. I will have MUCH to blog on, I'm sure, as I get immersed in new tech and new problems.

ex-msft Michael

WARNING POLITICS Stupidity of War On Some Drugs

2005-06-02T17:53:00Z

** WARNING: PERSONAL LIBERTARIAN POLITICS AHEAD **

Here goes--my first rabidly political post. Oh, goody! I'm drooling with glee.

The recent hub-bub about lowering the drinking age in Wisconsin for military folks got me all hopped-up frothing and raving again.

Once again I'm struck by the complete absurdity of the War On Some Drugs (WOSD). Our Puritan heritage allows us the nifty double-think trick of legalizing the two most awfully harmful, blatantly addictive drugs known to man...because they're both quite damaging and, we feel, those who experience pleasure MUST be punished later. That is--ethanol causes hangovers, a just retribution for enjoying oneself; tobacco, well, we know how THOSE sinners are punished!

Before I go any further: I argue these points from a completely abstract viewpoint, since I'm a total non-user and non-drinker. I really do have the betterment of mankind at heart :)

So back to the War On Some Drugs. We're in the completely laughable position of fighting a battle we've fought before--Prohibition--that we've been fighting for 30 years; recognizing it, and yet persisting in the insanity. We've created a hugely profitable business with wonderful profit margins; what fool would want to destroy such a lucrative market? The drug suppliers sell natural or off-patent substances, manufactured or harvested generically, which by themselves are worth pennies a dose--at 1000*cost. Sure, they lose the occasional 10% to the Keystone Cops, who trumpet each bust on the evening news like they've shut down drug Y for all of North America this month. In fact it just means Mitsubishi-brand Ecstacy won't be available that evening, we'll have to settle for those crummy Smurf X tabs. Or go sniff glue and gulp Robitussin DM.

Law enforcement, meantime, benefits almost as richly as the suppliers. Sure, most of the officers are honest people just doing their work; they probably even believe in it though how I don't know. Some, I'm sure, gain handsomely themselves; no enterprise this profitable fought by underpaid civil servants escapes corruption. Even the honest ones--well, they're employed by the WOSD, aren't they? And those agencies--DEA, BATF--what will they do if we go sane?

Due process? Out the window; if it's drugs, your stuff (money, home, vehicles, freedom) are confiscated first; you essentially sue to get your own stuff back. I seem to remember the "logic" for this trampling of the Constitution was that those awful, rich cocaine suppliers can afford such great attorneys, why, we've gotta make sure all they can use is our crappy public defender here in Podunkville...Gestapo, anyone? Yep, civil rights are just too inconvenient when you're fighting the noble War On Some Drugs.

The two legal intoxicants kill somewhere around 450,000 people per year; roughly 370,000 from smoking, and another 80,000 or so from drinking. Meanwhile, while true statistics for drug deaths are hard to find, even the most imaginative interpretations of drug-related deaths..."Well ma'am, I'm sorry your son died. He died of drugs; at the time he was hit by a bus, there may have been marijuana in the city somewhere." But guaranteed, deaths due to all illegal drugs together are an order of magnitude less.

When we instituted Prohibition (why don't we remember how badly THAT sucked), ethanol deaths increased dramatically; due to impurity, on-the-sly consumption in dangerous situations, crimes over turf battles, the list goes on. It was a disaster and we quickly came to our senses. Now granted--ethanol wouldn't pass muster with the FDA today by a long shot, but at least what you buy isn't adulterated. The same logic applies to drugs. Today's market is filled with adulterated product made in shoddy labs or diluted with impurities. It's consumed on the sly creating dangerous situations. It's so profitable that it drives a huge criminal industry, and costs us a huge enforcement engine that's ineffectual except to maintain inflated price levels.

Ah--but if we legalize (aka "decriminalize"), the World Will End and people will simply Spiral Out Of Control. Nope. People are remarkably smart if they're given free choice and the information to exercise it. Publish the truth about every substance; an MSDS (Material Safety Data Sheet) and a proper drug insert--just like those pharmaceutical commercials where beautiful people smile and wink knowingly, then the announcer comes on rapid fire and says things like "Do not take Foovamine if you are a person. Or a dog. Or a mammal at all. May cause blindness, itching, auricular necrosis, and supraventricular tachycardia. Talk to your doctor immediately if your fingers turn black and fall off."

I bet that with decriminalization we'd initially see a huge upswing in use--maybe one to five years--followed by a huge decline, probably to current levels or less. Deaths would decrease dramatically; high quality, well-documented substances would curtail the usual junkie overdoses and adulterant toxicity cases. People would exercise their free choice--and learn to protect themselves. Certainly we'd avoid the thousands of unnecessary violence-related deaths; fighting over turf, fighting over money, drug busts gone bad, the whole damn mess.

Hey--we didn't used to have prohibition! It's a fairly recent thing. Let's get rid of it and spend the money on a manned mission to Mars or something.

And oh yeah--the decent people who voluntarily visit forsaken overseas hell-holes and risk their lives for esoteric things like our principles--can we PLEASE let them have a beer?

SQL Injection--Everyone Knows It, No-one FIXES IT!

2005-06-02T04:42:00Z

Does the following input into a web application text box strike appropriate fear into your heart? Nay, I should say, dost though poop thine pants gazing there-upon?

' or 1=1;
'; use master; xp_cmdshell 'net user SomeUser SomeUsersPassword /add'; --

I love the last one. I just added a user to your SQL box. Now I will paaaarr--tay.

I remember vividly the vomit-comet, bowel-dropping sensation I had when Jesper Johanssen used this as the opening salvo in a complete network penetration at last year's TechEd. I think the talk was titled "10 Easy Steps to Penetrating Your Network"--Jesper sorry if I butchered the title. It was an excellent talk.

That second line works on quite secure-appearing networks; ones with proper external firewalls, a nice tidy DMZ with your uber-secure web serveren, and an inner firewall where your pristine, champagne-sipping SQL boxen sit.

The moral of this frightening tale? Protect yourself against SQL Injection! If you don't know what it is, Google it.

I've seen so many complacent customers with SA passwords in cleartext in web.config. Start there--use CryptoUtility to at least protect your connection string. (http://workspaces.gotdotnet.com/cryptoutility) But before you go encrypting it--use least privilege! SA is really hitting the fly with a hammer.

Use stored procedures. I was asked recently, "...well if I pass a varchar that came from user input, and then use string concatenation in the stored procedure, then use sp_executesql to run it...that's safe, right?" NO. The beauty of stored proc params is they're not executed; if you do string concat-execute you circumvent that.

Yep, there are times we can't use sp's for everything and have to generate SQL on the fly. But remember that all user input is evil and untrustable! There are some nice reg-ex based utilities out there that do a good job of scrubbing user input for nasties...but they're not perfect.

Now I'm going to go back to looking at my HoneyPot box in my DMZ. I like to watch script-kiddies come in with cute little hacks like:

2005-05-30 14:40:48 67.167.243.58 - <my ip> 80 GET /scripts/root.exe /c+dir 404 -
2005-05-30 14:40:48 67.167.243.58 - <my ip> 80 GET /MSADC/root.exe /c+dir 404 -
2005-05-30 14:40:48 67.167.243.58 - <my ip> 80 GET /c/winnt/system32/cmd.exe /c+dir 404 -
2005-05-30 14:40:48 67.167.243.58 - <my ip> 80 GET /d/winnt/system32/cmd.exe /c+dir 404 -
2005-05-30 14:40:48 67.167.243.58 - <my ip> 80 GET /scripts/..%5c../winnt/system32/cmd.exe /c+dir 404 -

Aren't they just precious? Ah, the eternal optimism of the young...hey kids, doesn't it seem odd that the machine's name is "HoneyPot"? I blame the schools these days. They just don't teach these young'uns to READ, especially great classics like "The Cuckoo's Egg". What especially cracks me up are the ones who, day after day, from the same damn IP addresses, run full port scans against my ISA 2005 external firewall.

New Version of CryptoUtility

2005-05-31T19:31:00Z

Please have a look at:

http://workspaces.gotdotnet.com/busybeebuilder

http://workspaces.gotdotnet.com/cryptoutility

CryptoUtility: My friend J Sawyer and I published CryptoUtility as part of an article on high-security crypto in a real world application last year, in the November 2004 MSDN magazine....see:

http://msdn.microsoft.com/msdnmag/issues/04/11/CryptoUtility/default.aspx

Anyway, J didn't really, er, um...LIKE...my UI. Yes, I admit it... My name is Michael Stuart, and I do not design pretty UI's.

J did up a very nice property-control admin UI, and I've revamped the solution file to include it and got rid of a bunch of superfluous other stuff. The new Admin Wizard is very nice! Thanks J.

The rest of the CryptoUtility--that is, the core Crypto stuff--is unchanged and still rock-solid.

The MSI installs buildable source, binaries, and the documentation. I've updated the documentation to work with the new admin wizard UI.

The key new feature is that the Admin Wizard creates a setup file for CryptoUtility, complete with everything necessary to install it on a blank machine. It then password-encrypts the setup config file; you take that file, and copies of the CryptoUtility binaries, and put them on a USB key. Then, you run around the various servers in your farm, plug the USB key in, copy the binaries to whereever, and run the wizard in command-line mode.

Why is this better? Because only one person, the admin, ever has access to the symmetric crypto key for your server farm. He or she knows the password to the config file containing the ONLY cleartext copy of that key. It's a secure install process--much better than the old admin utility.

New Versions--CryptoUtility and BusyBeeBuilder

2005-05-31T19:18:00Z

Please have a gander at:

http://workspaces.gotdotnet.com/busybeebuilder

http://workspaces.gotdotnet.com/cryptoutility

I've updated my two (currently) favorite pet projects. More on CryptoUtility in the next post...here's BusyBee:

BusyBeeBuilder: My automated build solution. About 18 months ago I became frustrated with the complexity of some automated build solutions, and with the over-simple alternatives. There had to be a middle ground--and BusyBeeBuilder is it.

I wrote it as essentially a sequential workflow management framework. It uses the Plug-In pattern extensively, allowing you to write very simple "IBuildActionWorker" implementations to do build tasks. BusyBee comes with most of the useful ones already--.NET Solution file build, SourceSafe label and get, run NUnit's, run FxCop, move files, move/copy directories, archive, and send email to the team.

It runs as a service and it can build multiple applications simultaneously; it's running at several quite large places very successfully.

It implements a very good monitoring model, so BusyBee allocates a certain number of minutes to each task; if the task exceeds its allocation, BusyBee kills it cleanly...no more hung instances of DevEnv.exe holding up your builds.

There's a Remote Control, so developers can pull up the Remote, connect to a server, and kick of builds on demand. It echoes back progress on each step--BTW, uses eventing over a Remoting channel and implements the two-way proxy thing to mutually hide the server and client types from each other...kind of fun.

The config file is very literal and simple to set up; it specifies a Build Target and some global settings like where to put the log file, and then has a sequence of BuildActions. Those actions' Workers are specified by name, type, assembly, and they're passed a name-value collection of parameters.

Please join the workspace. The MSI you'll find there, version 2.2, installs source, binaries, documentation (still kind of sparse, but OK). It also installs two ZIP files that you can use to run most of the NUnits, which are included.

I'm glad to offer help on the GotDotNet workspace's forum/discussion. Please let me know if you use BusyBee!

Funny story...funny ha-ha on me :)

2005-05-13T17:19:00Z

A good friend of mine--whose company I'm consulting for right now--emailed me this joke. Too funny...and kind of hit home :)

Thanks for keeping me humble, Ryan!

>       A cowboy was herding his herd in a remote pasture when suddenly
>       a brand-new BMW advanced out of a dust cloud towards him.
>
>       The driver, a young man in a Brioni suit, Gucci shoes, Ray Ban
>       sunglasses and YELL tie, leans out the window and asks the
>       cowboy, "If I tell you exactly how many cows and calves you have
>       in your herd, will you give me a calf?"
>
>       The cowboy looks at the man, obviously a yuppie, then looks at
>       his peacefully grazing herd and calmly answers, "Sure. Why
>       not?"
>
>       The yuppie parks his car, whips out his Dell notebook computer,
>       connects it to his AT&T cell phone and surfs to a NASA page on the
>       Internet, where he calls up a GPS satellite navigation system to
>       get an exact fix on his location which he then feeds to another
>       NASA satellite that scans the area in an ultra-high-resolution
>       photo.
>
>       The young man then opens the digital photo in Adobe Photoshop
>       and exports it to an image processing facility in Hamburg,
>       Germany. Within seconds, he receives an email on his Palm Pilot
>       that the image has been processed and the data stored.
>
>       He then accesses a MS-SQL database through an ODBC connected
>       Excel spreadsheet with hundreds of complex formulas. He uploads
>       all of this data via an email on his Blackberry, and after a few
>       minutes, receives a response.
>
>       Finally, he prints out a full-color, 150-page report on his hi-
>       tech, miniaturized HP LaserJet printer and finally turns to the
>       cowboy and says, "You have exactly 1586 cows and calves."
>
>       "That's right. Well, I guess you can take one of my calves,"
>       says the cowboy. He watches the young man select one of the
>       animals and looks on amused as the young man stuffs it into the
>       trunk of his car.
>
>       Then the cowboy says to the young man, "Hey, if I can tell you
>       exactly what your business is, will you give me back my calf?"
>
>       The young man thinks about it for a second and then says,
>       "Okay, why not?"
>
>       "You're a consultant." says the cowboy. "Wow! That's correct,"
>       says the yuppie, "but how did you guess that?"
>
>       "No guessing required," answered the cowboy. "You showed up
>       here even though nobody called you; you want to get paid for an
>       answer I already knew; to a question I never asked; and you
>       don't know anything about my business."
>
>       "Now give me back my DOG."

Swallowing The Red Pill

2005-05-03T19:19:00Z

As I alluded in my last post, I've officially swallowed the Red Pill...at least this month. For the 3 developers who haven't seen the Matrix--the red pill pops you out of the Matrix into reality, the blue pill returns you to blissful ignorance and a life of computer-generated fantasy...the latter much like owning an XBox.

I want to emphasize some core philosophical points I re-learned on my last project:

use supported technologies/products
focus on the final goal, not the path to it
spend engineering effort where it helps the business
when you pay for expert advice, listen to it

Use supported technologies/products: Open source tools like NUnit, CruiseControl, NDoc, etc. are just super. They're indispensable to me. But you'll notice, they're not really embedded in the actual output; they're tools we use during development. The flip side is code that'll show up in release builds; there's a different animal. How much do you truly trust what you've included? How bulletproof is it? Chances are, you really don't know. I know how hard it is to truly test code; I mean TEST it! Beat it up. Run it for 72 hours at absolute maximum load. Gather perf-counter and exception data for that whole time. Verify accuracy of results during that run. Black-box testing--positive input, negative input, boundary input, garbage input, etc. Does it leak handles or memory? Does it degrade over time? Is it fragmenting the heap(s)? Is it chewing up threadpool threads? Is it hiding exceptions but spewing out garbage?

It's really hard to honestly test this stuff and trust completely in the outcome...I mean, trust to the extent that I'll gladly say "Why YES, Ms. CIO, you can confidently put your $25million/day website on this application. No really."

I've done this extent of testing, front-to-back, honestly about five times in the past five years. It's expensive, time-consuming, and many customers just won't buy off on it or won't dedicate the hardware and people to make it happen.

When you go plugging random Mom & Pop LLC code into your code, you should know if they've done this testing, or do it yourself.

On the other hand--look at quasi-products like PAG's EnterpriseLibrary. It is BULLETPROOF, and whenever it's discovered it's not, it's fixed. You have a "neck to wring"--Microsoft's. It has huge community support--extensions, bug fixes, discussions, and its developers' blogs. It's had the horrendous brutalizing testing done already--in fact you can SEE the unit tests right there, and read the tester's articles on MSDN.

Focus on the Final Goal: Too often on a big project we (developers+architects) get lost on the design, the abstractions, the "frameworks"...in other words, navel-gazing academic purism of the worst degree. The fact is, every design will evolve during development. Period. There is NO AMOUNT of pretty UML and exhaustive user interviews that will change this--which is why I'm so in love with XP's pragmatic weekly iterative approach. Every day, ask yourself: "Did we do something today that contributes directly to the end goal?" Or "Did we implement a feature? Did we fix a bug?" At the end of the week, when you do an interim gold release, will the users see measurable progress and no regressions?

I FREAK OUT when I see folks designing "frameworks", without an application inside it that serves as a weekly user-visible benchmark of actual progress. You're asking for trouble with that old-school waterfall mentality; you can't possibly anticipate every need for the framework unless you are simultaneously loading it with an application. Much worse, if you're developing a "framework" with multiple apps in mind. The specifications become so vague, the abstractions so broad, and the timelines so long that they basically never get done. What have you saved? You may have reuse, but the effort to get there was more than the time you would have spent adapting smaller, more modular "application blocks" which were more focused on a particular problem.

I guess that last part means: think first about pattern, template, and process reuse; then about modular "application block" reuse--like CryptoUtility, or all the awesome PAG blocks; THEN maybe think about reusing an entire framework...

Spend Engineering Effort Where It Helps The Business: Stop! Why are you writing a thread pool? Sure it's fun. I've written one or two of those; and other meaningless things like workflow engines. But unless you're a software house, it's unlikely you're doing anyone a favor with such a duplication.

By the same token, don't do too many "smart" things--I kick myself routinely looking back on a piece of over-engineered code that gets called maybe once a day. "do the absolute simplest thing that could possibly work".

But DO put your engineering best into TESTING, then more TESTING. The bulk of your work comes AFTER you are code- and feature-complete. Have you load tested? Stress tested? Soak tested? I'm assuming of course you already had thorough TDD-style unit tests which were run every time you automatically built, at least once a day. This stuff really takes time because it depends on other groups' cooperation. You have to interface with the infrastructure folks, with management; you're asking for an isolated test-network, with lots of machines and software and setup. You'll have devs and testers hanging out looking glazed poring over perf-mon logs and trace files and dumps. You'll have bleary-eyed coders who aren't used to using LoadRunner writing scripts--and wondering if they're representative of real-life data.

And before all that--you've added testing hooks, right? You can turn on and off detailed tracing? You have performance counters, dozens of them? You have MSI installers that go BANG and put it in right?

That's where the real engineering effort goes. It's not algorithms; those things you find in books for the most part. It's the hard tedious stuff, that stuff above, that counts.

When You Pay for Expert Advice, HEED IT: I'm astonished at how many companies will gladly shell out $10,000 a week for consultants, and then blandly ignore their advice; that's one of those "ya outta be fired" sins. Let's pretend there's an adequate screening process for said consultant, and they know what they're talking about--as proven by a track record of customers willing to give testimony, and a list of successful projects. They work for a reputable and preferably the vendor company. Probably would be nice if they're published, too. And be sure they're in their area of expertise--and if they're not, they're willing to admit it and bring in other expertise.

Those things said, you now have an expert. And you're paying for it. Will you listen? People on the team may question the advice, and that's good; the consultant should hear it and ably defend his/her thesis. But when push comes, chances are they're right. If they're not why are you paying ten grand a week??

The best part about a vendor consultant--the company's reputation is on the line, not just his/hers. And that company will move mountains to see a good outcome; use that to your advantage! Chances are they actually know how to get the best out of their own products....funny how that works :)

Man. Couple more posts like this and I'll just be moving right into management. Think I'll plug that jack-thingy back into my head now and jump a couple of skyscrapers....

Ferrari vs. Chevy: pragmatic engineering

2005-05-02T19:20:00Z

I love cars. It's the thing I'm most likely to spend stupid amounts of money on, and only the guiding hand of my frugal wife keeps me from mortgaging future college funds on a shiny 400HP...whatever. Too many 400HP cars around to count these days--topic for a future blog.

Let's talk software engineering by analogy with engine...er, engineering. This is a rough analogy but it works during two-hour talks where the audience doesn't have time to examine my hand-waving and see where the analogy breaks down :)

Ferrari engines are exquisitely engineered; ground-up development for the most part. They borrow from their F1 experience with exotic piston coatings, high-strength Exotic Alloy Du Jour for connecting rods, sodium-filled or titanium valves, aluminum blocks, you name it. They run insane compression ratios at stupendous RPM's--over 8500 in the latest ones. They reach and surpass what used to be a magical barrier--a specific output of more than 100HP per Liter (nice mixed US/SI measurements). That number's not so impressive against the 200+ that modern motorcycles churn out--but they have a huge advantage, since it's much easier to get high specific output from an engine of smaller sizes. Output decreases as the square of engine dimensions, while mass decreases as the cube. A 4-liter engine making 430 HP is quite an achievement, considering it has to be streetable and last at least 50,000 miles with no major hiccups. F1 is a different game; they're around 300HP/Liter now, but they might as well be turbine engines they're spinning so fast.

This is the equivalent of finely-crafted C++ with some assembly thrown in for the critical parts. I think of Mike Abrash's book on optimizing code; down at the level of thinking about registers, L1 cache, locality of reference type stuff. Getting that last 0.1%. Worrying about reordering. Thinking about memory barriers. Cursing kernel-user transitions. Pain, pain, and more pain. Building a beach, one grain at a time, with a toothpick.

By comparison, the venerable Chevy small-block; truly an engineering tour-de-force, but not of the same kind. This thing's been around what, 40, 50 years? It's evolved, but the basic layout is the same. For crying out loud, its camshaft is still nestled between the banks and operates its valves with freakin pushrods! It's archaic by comparison. And yet, it is triumphant in every way. Volume, cost, reliability, total output. Granted, it doesn't reach the Ferrari's high specific output in streetable trim. They're in the neighborhood of 70 HP/Liter; almost lazy by modern standards. But it gets better gas mileage--even though they're 5.7 liters against the Ferrari's 3.6 or 4-ish. They rev less; less frictional waste.

Why is the Chevy the hands-down winner, when technically the highly-tuned engine is superior? PRAGMATISM and STANDARDIZATION. The Chevy has had decades of effort poured into it, millions of man-hours from both the actual developers and the vast hordes of third-party developers. It's a standard. Its engineering weaknesses have been conquered through sheer brute force over years, so you hardly care anymore. If you can ignore the thought of pushrods banging up and down at 7000 RPM, your Corvette Z06 sure does a number on the Ferrari next to you. Doesn't sound as good. But it's about 1/4 the cost, and if said pushrods fail you won't shed tears too long.

So you say: "What does this have to do with CODE, and why are you not taking Lithium?"

Well it upsets my tummy, on the second question. On the first, this has everything to do with Web Services, YAGNI, KISS, and HST.

Web Services are bandwidth hogs compared to binary protocols. Their serialization models leave a bit to be desired. They are a little or quite a bit slower than binary at present. But who cares! Web Services are the Chevy Smallblock of the RPC world. Their advantages are so compelling that we will though sheer brute engineering force MAKE them work. Their "specific output", perhaps expressed as "objects/kilobyte", will probably never match a binary protocol. But who cares! Over time, every vendor, every software house, will become so invested in the magic of Web Services that we'll get around the limitations. Or live with them. I don't hear Corvette owners boo-hoo'ing over pushrods; I do see them able to send their kids to college on the $120K they saved!

YAGNI--"Ya Ain't Gonna Need It"--and KISS, HST ("Hooking S*** Together") are what most of us do for corporate and even ISV development. We're not interested in the last 0.1% We love elegance, but if it costs 4X as much it's out the door. What we need is pragmatic, robust, standardized engineering. I hate to say it, because it's Not The Developers Way, but cheapest usually IS best.

Man did I ever swallow the Red Pill today. I hope I'll soon return to my blue-pill nirvana of fine software engineering unconstrained by practicality, cost, schedule, or maintenance. 9000 RPM code; titanium con-rods and molybdenum piston skirts. Nope, the pushrods and iron win over time--management was right after all.

P.S. these analogies are TOTALLY my opinion, do not reflect at all on the excellent (in their own ways) engines/makers, and have nothing whatsoever to do with my employer. My views are my own, and they are insane. I do not speak for my employer.

More RTCClient API 1.3

2005-04-28T22:04:00Z

Hassan writes:

Wednesday, April 27, 2005 7:57 PM by Hassan Khawaja

# re: Real-Time, Server Push, Pub/Sub Network Apps in .NET--Buellar? Buellar?

I am dealing with a somewhat related issue, basically creating an AD employee search directory app in asp.net. I want to be able to show the users' presence next to their names.
We are using LCS2005, and I am trying to develop using RTC 1.3 in C#.

So far I am pretty much stuck with getting COMException errors with some weird HRESULT codes. Interestingly a winforms app using the same code works just fine. It's the RTPresence app in the SDK samples.

Have you been able to write something for the web using RTC 1.3? I'd appreciate if you could guide me in the right direction.

Thanks so much,
Hassan.

and I replied:

Thursday, April 28, 2005 3:04 PM by methylamine

# re: Real-Time, Server Push, Pub/Sub Network Apps in .NET--Buellar? Buellar?

Hey Hassan--

Yeah that's kinda what I'm talking about--the non-zero HRESULTS come flyin' back and it's just poopy.

You've gotta pay attention to which thread you created the original instance of IRTCClient2 on; never create or pass outside that thread.
Technically the team says they've fixed the marshalling but I can't make it work in C#, and I don't want to drop into C++ just for a wrapper.

I created the Interop assembly with the RTCCOreLib.dll; just use the RTCCORE.IDL file in the SDK IDL directory. DON'T use the one in System32; remember this is a redirected DLL so the actual 1.3 dll is in some weird SxS directory.

Also, be sure the .MANIFEST file is correct and available in your BIN dir.

Hope this helps!

I do hope it helps; I spent quite a few hours getting it all set up. The basic steps to using RTCClient API v1.3 from C# are:

1) generate an interop file from the RTCCORE.IDL file in "C:\Program Files\RTC Client API v1.3 SDK\SDK\IDL", call it a good name like "Interop.RtcCore.dll"

how to do step one:
use "midl rtccore.idl" to get a tlb
use "tlbimp rtccore.tlb /out:Interop.RtcCore.dll" to get the interop assembly; optionally, sign it at this time too

2) reference said interop assembly from a C# project

3) Now the delicious part; you must create this from an STA thread, ALWAYS, and you must never pass references to the objects you create outside that thread's context. This means no delegates on background threads, no putting references in a collection that can be hit by other threads, etc etc. It's quite painful. I created and managed said thread as follows:

   m_staThread = new Thread( new ThreadStart( this.InitializeInternal ) );
   m_staThread.ApartmentState = ApartmentState.STA;
   m_staThread.IsBackground = true;
   m_staThread.Name = String.Format( "RTCClient_STAThread_{0}", g_instanceCount );

   m_staThread.Start();

4) on that thread you need to initialize an instance of IRTCClient2 as follows:

    // Run the required message loop to pump messages from the RTC RtcWrapper
    //  COM object to this thread.
    m_context = new ApplicationContext();

    // trap messages with our filter, looking for WM_PUMPRTCMESSAGE--which
    // we'll redirect to invoke a delegate
    Application.AddMessageFilter( this );

    // create the RTCClient instance and initialize it
    RTCClientClassInitializeEx( INITIALIZEEX_PARAMS_BITMASK );
    SetEventFilter( EVENT_FILTER_BITMASK );

    //MSTUART: moved here from before new context above 04132005 to mitigate race when
    // outside, initializing thread gets signalled we're ready too soon.
    // OK all done, signal to whomever's waiting in Initialize()
    m_completedWaitHandle.Set();

// start message loop on this STA thread
Application.Run( m_context );

All that goo above is done _on the STA thread_! And the initializeEx is called as follows:

m_client = new RTCClientClass();

    m_client.InitializeEx( rtcifConstantsBitMask );

    // snag events
    m_client.IRTCEventNotification_Event_Event += new IRTCEventNotification_EventEventHandler( ClientEventHandler );
    // set up to listen
    m_client.ListenForIncomingSessions = RTC_LISTEN_MODE.RTCLM_BOTH;
    // media types
    //m_client.SetPreferredMediaTypes( RTCConstants.RTCMT_ALL, true );

5) Now if you want to call this from many threads, you'll have to create the plumbing for marshalling calls onto your brand-new nifty Windows message loop you created above. Since you're an IFilter, you get first chance at those messages; look for ones that pertain to you, and do only them.

It's a little painful, really. I'm looking forward to nice friendly wrappers or rewrites for this. Not having them, I had to write my own. When it came time to do the server, I had to think a bit about limiting the number of sessions per RTCClient instance--sending is O(n). So I have many instances of my wrapper (which encapsulates the STA and serializes access from outside threads), each of which is limited to 20 sessions.

It all works but it feels unnatural.

At the end of the day, our web service polling implementation offers so many advantages--scalability, ease of development, ease of maintenance, traverses Firewalls and NAT, etc. etc. etc. that it's probably the way we'll go.

We had already implemented a Heartbeat-Sequence type 'reliability layer' to give some guarantee of message delivery, and we'd designed for a plug-in model; substituting any technology is easy with this design.

Frankly I think most solutions that need to be WAN-friendly--that is, NAT, firewall, no incoming connections, etc--will be better off with HTTP(s) polling type implementations. Inside the LAN it's a totally different picture; but I hate "asymmetric" solutions, I'd much rather double up my engineering effort on one solid solution than have one for inside and one for outside to gain some fractional speed advantage inside.

Given how monstrously scalable web servers have become, and given the great gobs of bandwidth available--remember all that dark fiber we laid in the 90's?--it Just Works.

It's not quite "YAGNI"--Ya Aint' Gonna Need It--but it's close.

Real-Time, Server Push, Pub/Sub Network Apps in .NET--Buellar? Buellar?

2005-04-13T17:50:00Z

My best friend's boyfriend's cousin said she heard it passed out in the 31 flavors last night. Guess it's pretty bad.

Yep this is awful. It has to be the most under-served, and one of the most desirable, features in enterprise applications...YET, there are relatively few good options.

All we need here is a server which answers subscription requests to "topics", enrolling said client in that "topic" in a pub-sub way. Then the server sends out messages on those topics either to everyone (who filter locally) or specifically to its subscribers. It's also sending out heartbeats with sequence numbers so clients know when they've missed something, and can request the missed set.

It's just NOT THAT HARD, at least in the trivial case. But let's start adding constraints:

low client footprint, ideally NO extra install except .NET and perhaps an interop assembly that uses base-OS features
must work across WAN
WAN may be highly latent, deliver things out of order, etc.
messages will not be more frequent than one per second, time resolution (one second) is fixed
*MAY* have to communicate through NAT on the client end
*MAY* have to negotiate firewalls, though this is a very distant scenario
server must be absolutely reliable when serving up to 1000 simultaneous clients on a dozen or so "topics"

Now we've got some issues. In fact they're kinda nasty; so nasty that we're seriously contemplating a web service polling solution. It just WORKS.

SO: What have we been trying? The best candidate is SIP, using the RTCClient API, v1.3. But sheesh. It's a really really steep hill, mostly because a) it's a COM object, so there's some Interop fudge that happens and b) it's an STA thread model, which is just super-poopy.

I've written wrapper upon wrapper; first, there's the wrapper that creates the actual IRTCClient2 instance on an STA thread; that thread, BTW, creates and enters its own application context via Application.Run(). Now we've got a messaging loop; step 2, we have to implement IMessageFilter, register our own message type, and intercept the message loop watching for it. Oh yeah--if the message is intended for us, we then invoke the delegate that was passed to one of our own internal methods. We just do the simple thing and force the outside calling thread to block while we work, and return the results on a private member which the caller picks up on its way out the door. We could have gone all elegant and done a purely asynch with a callback for notification but I've spent too much time on it already.

All standard com/win32 stuff, but really a rude shock when you've had the luxury of doing pure .NET for years.

Then there's a huge management layer on top of all this; how many remote clients per RTC instance, how many RTC instances; then there's the reliability layer that does sequences, heartbeats, missed message retrieval and history storage.

NOW do you see why I'm ready to throw up my hands and say "Enough already! Let's just do a gosh-darn POLLING implementation!!~%!$%!$$##" (the last part is mumbled curses)

Question for dear readers: how have ya'll dealt with this need for real-time pub/sub server-push type communication in .NET?

Fun Variant on State Design Pattern

2005-04-10T20:41:00Z

OK so I'm working on quite a fun project; it's a proof of concept for a near-real-time, pub-sub, server to client push type communicaton subsystem. Phew.

Essentially it's a big fat wrapper around a few candidate technologies including SIP, a.k.a. our RTCClient API, v1.3. THAT (RTCClient) will be the subject of another blog I'm sure. Getting a multithreaded .NET server app to manage and use multiple STA-threaded COM objects was quite fun...

The heart of the wrapper is a quasi-reliable-messaging protocol. Because the underlying communication API's don't all offer reliability, we had to engineer our own. It doesn't have to be as bulletproof as MSMQ or other reliable systems; it just has to be Pretty Darn Good.

I went with the obvious solution; messages with sequence numbers, and a heartbeat system telling all clients on a regular schedule what the latest sequence number for a topic is. Clients can detect missed messages and request historical updates. There's some complexity as we have to account for out-of-order receiving, rolling over Int32.MaxValue sequence number cleanly, etc.

So where's the State Pattern?

Right, enough preamble. I had this really nasty method with a TON of if-then's, some with composite tests--lots of &&'s and ||'s scattered around. I **FEAR** such methods from painful experience going back to age 10 and writing a PacMan in assembly.

I busted out C#Refactory (http://www.xtreme-simplicity.net/, best 100 bucks you'll ever spend on a coding tool). Sure enough, the Cyclomatic Complexity of said method was 10--and 10 is a magic number. That is the cyclomatic complexity at which it is almost _certain_ you will create a bug when modifying the method. And, the fates help that poor developer who comes after me and has to mentally compile that pile of doo I wrote!!

Here's where the State pattern comes in. Separate state representation from behavior, and allow flexibility and extensibility of the system. In fact it just recognizes the key fact: those long-winded if-then methods we hate to debug are in fact finite state machines, and should be written that way...Take a peek at http://msdn.microsoft.com/msdnmag/issues/01/07/patterns/ for a great example of when/how to use State.

My case wasn't really amenable to the iterative type of State machine there though. Most State implementations assume the vending machine analogy, or the bank account status, etc. You send messages to the FSM and it migrates between nodes in response.

Mine had to work with a known set of inputs, present from the outset. I had to get the information--the "language" I'd feed the FSM--and ask the FSM to find the right output state for that input right away.

So I created a State abstract base class, with properties to describe the State--in this case, HaveMissedMessages, IsBadState, and TrueSequenceNumber. Those three describe fully what conditions we can be in. See the abbreviated source below.

But here's the departure from usual State implementations: I'm going to call it Statically Found State because I'm naming-impaired:

I don't pass my Context into the State. In fact State knows _nothing_ about its users; most State patterns have to know about their context--their user, the behavior thing above them. They also often mimic all the methods of the context; so their signature can be quite cluttered. They don't JUST have the properties that define the current State.

Instead I have a static method defined on the State base and all derivatives, "FindOutputState". Each class implements its own version of FindOutputState. Each class knows what inputs correspond to which of its immediate neighbor nodes. It then calls FindOutputState on the appropriate next neighbor in the graph.

That next neighbor, again, knows only about its immediate neighbors and what inputs would transition to one of them.

Eventually you have a final state with no exits--you've gone to ground and the FSM ends.

In this system the output states are InSynchState--we have not missed any messages--OutOfSynchState--we've missed some but we're recoverable--and BadState, which happens when you're so far off it's ridiculous.

The inputs are InternalSequence, which is what we believe is our sequence, ExternalSequence--what we received--and whether the message was a Message or a Heartbeat. The latter is important because the semantics are subtly different.

I abbreviated the code below a little 'cause this editor blows every CRLF into a fat double-spaced linefeed. The important parts--implementations of FindOutputState--are bold and indented.

Why I Love the State Pattern:

It separates state from behavior. It formally expresses each node of the FSM, rather than burying them in the scope of some if-then clause. I can add new States and inject it between existing nodes with little effort. The very existence of each State derivate class verbalizes that node formally, calling out an anticipated intermediate.

Note that the most complex FindOutputState has a CC of only 4--much easier to NUnit test. In fact using the State pattern is wonderful for TDD, because you can test your FSM without any of the crufty context you usually need to inject to get at those long if-then-elseif poopy methods.

Question: is this a new pattern? Or am I totally deluded? Do the departures from the usual State implementation seem valuable?

Here's the code:

public State(){}

protected const int ROLLOVER_THRESHOLD = 10000;

private int _trueSequenceNumber = 0;

protected bool _haveMissedMessages = false;

protected bool _isBadState = false;

public static State FindOutputState( bool isMessage, int intSeq, int extSeq )

{

State state = null;

if( isMessage )

state = MessageState.FindOutputState( isMessage, intSeq, extSeq );

else

state = HeartBeatState.FindOutputState( isMessage, intSeq, extSeq );

return state;

}

public bool HaveMissedMessages

public bool IsBadState

int TrueSequenceNumber

}

internal sealed class MessageState : State

{

public MessageState(){}

public new static State FindOutputState( bool isMessage, int intSeq, int extSeq )

{

// easy one first: all inputs that lead to InSynchState:

if( intSeq == extSeq )

return InSynchState.FindOutputState( isMessage, intSeq, extSeq );

// external sequence is internal plus one, basically OK

if( intSeq == ( extSeq - 1 ) )

return InSynchState.FindOutputState( isMessage, intSeq, extSeq );

// second easiest next; out of synch by more than one difference

if( intSeq < ( extSeq - 1 ) )

return OutOfSynchState.FindOutputState( isMessage, intSeq, extSeq );

// OK now for the rollovers;

// if internal is more than external pass off to rollover handler

if( intSeq > extSeq )

return MessageRolloverState.FindOutputState( isMessage, intSeq, extSeq );

return BadState.FindOutputState( isMessage, intSeq, extSeq );

}

/// <summary>

/// Leaf node. Internal was behind external by a recoverable amount.

/// </summary>

internal sealed class OutOfSynchState : State

{

public OutOfSynchState(){}

public new static State FindOutputState( bool isMessage, int intSeq, int extSeq )

{

// leaf node; we are out of synch but recoverable

State state = new OutOfSynchState();

state.HaveMissedMessages = true;

state.IsBadState = false;

state.TrueSequenceNumber = extSeq;

return state;

}

More on XBox Live

2004-12-01T01:52:00Z

Wow! Lots of comments/emails from that last post. Seems others have been trolled on Live too.

So there's a positive too: there are a host of decent, very nice people playing Halo 2 on Live. People who have the courtesy to say "good game" even after they trounce you...or get trounced.

It must be a difficult problem for the Live admins. (totally uninformed ranting to follow, which does not reflect Microsoft's position only my own, and does not contain any inside information)

The Live folks probably work under several constraints:

* they have to increase the # of subscriptions

* stupid people cause regular people to leave

* stupid people also pay for their subscriptions

* feedback is subjective; who knows what feedback I've got?

So they have to balance booting with subscription counts, keeping in mind that a certain number of judicious bootings will actually increase the number of Live subscriptions--by making it a more pleasant place to be.

They have to keep false positives low--unfairly booting a decent person due to negative feedback probably left by trolls is a bad thing.

Ultimately the most reliable boot-decider would be for Live admins to sit in and listen to games. I don't know how many Live accounts there are, but it's millions--so they can't all be monitored. Perhaps they listen to the accounts with the most negative feedback to decide; I don't know.

The constraints above sound like a system highly amenable to statistical analysis. I don't know if the Live team has taken a rational modeling approach to it and set it up as an optimization problem. Seems doable though; I'll have to ask my statistician PhD buddy John how hard/easy it would be. I'm just a dumb biochemist/doctor/developer :)

Or, how about a code solution? Perhaps a voice-recognition bot that caught racial slurs and other nastiness and flagged the account? It sounds Orwellian, but it _is_ a subscription service--not the town square. After a very few flaggings the abusive user would get a sysadmin message next time they logged into Live warning them...

Live Team--if ya'll are listening, I'm ready to code. I want Live to be the finest online gaming, period.

Stupid people on XBox Live

2004-11-30T23:33:00Z

DUDE! What is up with the troglodyte faction playing Halo 2 some nights?

Recently me and three of my friends were verbally assaulted by two separate groups. One group kept referring to us as "n******" (rude slang for African-American). I was flattered, but I realize they meant it as an invective; this really puzzled me though. Honestly, people, who talks like that? Where do these rock-crawling, slime-covered cretins breed?

Another group loudly yelled that we were of a certain religious heritage. Well, I'd be proud to be Jewish. But again, I think their intent was derogatory. I'll just take it as a complement, and keep trying to be as funny as Seinfeld.

These incidents repeat in various forms about once a night. I guess out of the dozen or so games we play every night that's not so bad, but I'd love to avoid the kind of person who spews nastiness. A little vulgarity in the heat of the moment is to be expected; after all, these ARE life-and-death struggles. But personal invective? I thought I'd left that behind in grade school.

My friend Chris has a good idea; he'd like Live to have "certified gamertags" much like good-seller ratings on eBay. This way, I could choose not to play with anyone with X amount of negative feedback.

Meantime, I encourage everyone on Live to use the feedback feature and leave negative feedback for trolls, troglodytes, orcs, and goblins whenever they appear. I believe the Live team does eventually ban those idiots...the faster, the better.

P.S. To the level 23, 24, and 25 people who start new clans every night and play at clan level 5--you are NOT cool. Now go do your homework or read a book tonight.

Software and the UniMog

2004-11-20T00:53:00Z

Coming soon...