Random Disconnected Diatribes of a p&p Documentation Engineer
Is it still a holiday when you never seem to stop travelling, you can't quite figure out what day it is, you're not really sure where you are, and you can't remember where you are supposed to be tomorrow? It sounds a bit like a mystery tour, which is somewhat worrying as I was actually doing the driving. And doing the planning (or, to be more accurate, the lack of planning). Still, we had a great time. And, yes, I just sneaked a few days away from my daily documentation duties to grab a brief respite from all things Windows Phone-oriented.
As we are due to fly off to Cyprus for a wedding (not ours) in a few weeks time, we decided to skip the airport hell and, instead, see some of the sights in our own country. OK, so we did kind of have a kind of plan, which included starting at the Eden Project in Cornwall and working our way home via a disparate selection of tourist-oriented delights and wonders. So I'd already booked one night in a hotel in St. Austell and bought tickets for the Eden Project online before we departed. And starting bright and not very early on Monday, we set off on the long and arduous 300 mile drive to Cornwall.
Except it wasn't either long or arduous. Even though we'd planned it for after the kids went back to school, I was amazed to find the roads nearly empty; so that, within a couple of hours, we were closing in on Gloucester on the M5 and decided to pull in at a service area for a coffee and a bun. So how come the place was packed so full we couldn't find a parking space? Where did everyone come from, and why were they here? Maybe that's why the roads were so quiet - everyone stopped at this particular service area and liked it so much they decided to stay for the week.
So, as we weren't due in Cornwall until late that evening, we detoured instead to visit a couple of my old haunts from when I lived in Gloucester a great many years ago. Cheddar Gorge (amazing), Wookey Hole (even more amazing), and Wells Cathedral (incredible). All well worth a visit, And, wow, three touristy things done and still a week left. Maybe we can go in for some kind of a record.
Next day, the Eden Project. Yes I'm a gardener, but only in the sense that someone who can boil an egg is a chef. But everyone says it is worth a visit, and they aren't wrong. It's also great exercise because they built it at the bottom of a very large hole, so you get a lot of practice at walking up and down hills. And while the open-air gardens are very nice, the real attractions are the bio-domes; of which the tropical rainforest one is fabulous. As well as sweating like a pig and getting bitten by ants, you really get the feeling you are in somewhere like the Amazon - except for the little signs on the trees that tell you what they are (I haven't been to the Amazon, but I assume their trees aren't labeled).
The next day we started the journey home via the remaining tourist attractions we'd planned to see. Except that my wife found the Tamar Otter Sanctuary in Launceston on the sat-nav so we only got thirty miles the first day. She loves otters, and there is the bonus of seeing deer, ducks, Scottish wild cat (there was only one), owls, and wallabies. The last of these was somewhat of a surprise - it seems an unusual mixture when it comes to selecting an animal theme, and I'm sure wallabies aren't native to Cornwall so they didn't just wander in by accident. But it's certainly a very nice laid-back and wonderfully relaxing place to visit.
My vague journey semi-plan indicated that the next stop was Slimbridge Wildfowl Trust, but somehow my wife persuaded me that we should drop in at Dawlish Warren on the way past - it was only a fifty mile detour. It seems that her friend's Mother goes there for a fortnight every year, so it had to be worth a visit. Hmmm, I guess if you enjoy a large field, small beach, and very limited facilities including a funfair and one shop it's great. Though I have to admit that it actually is a very quaint and attractive area. And there was even a railway there, so at least I was able to do a bit of train-spotting. And it would have been even better if any trains had actually decided to come through during our brief visit.
At last we got to the Slimbridge Wildfowl Trust. I really enjoyed this place, though I've probably now seen enough ducks and geese to keep me happy for a couple of years. And the way they follow you around is rather eerie. Every time you turn round, there's two following you and they won't go away until you take a photo of them. Two Nenes (Hawaiian Geese) followed us all day, despite me taking their photo several times. The reception desk people also give you a book with pictures of over a hundred ducks and geese so you can identify them. Shame then that the one duck that came and sat on our table while we were breaking for coffee wasn't in the book. Though it does say in the small print that it only shows the males in their summer plumage, so maybe this was a female or - as it was cold and raining - it'd already donned its winter attire.
After Slimbridge, the next day was Ironbridge Gorge Museum near Telford in Shropshire. Or, to be strictly accurate, ten museums. And all on the same single entrance and car park ticket. OK, so we wearied after about six and didn't do them all, though the tickets are valid for a year so no doubt we'll be back there again. But the ones we did visit are superb, especially the Blists Hill Victorian Town. A really amazing place, especially if you have an interest (as I do) in feats of Victorian engineering. There's canals, an inclined plane boat lift, iron casting, brick and tile making, Victorian shops and businesses, and more. I even consulted a Victorian doctor in his themed surgery about the pains in my weary legs from all that walking, but all he did was moan about the youth of today (circa 1890) and then charged me sixpence.
Ironbridge also offers you a chance to walk along a tunnel where bitumen drips from the walls (and you get to bang your head a great deal), see people making Coalport pottery, learn about the archaeology and history of the gorge, and - of course - see the actual iron bridge (the first of its kind, erected in 1779). Plan to be there for two days if you want to see it all.
Finally, my own choice, the Severn Valley Railway. It's nearly 25 years since I last went there, and it's changed quite a bit since. There's an extra few miles of track, a brand new (but built to an original old design using old materials) station, a new engine house museum, and loads more rolling stock and locomotives. Unfortunately, there's also more new restrictions on where you can go. Though I guess the active loco sheds and works are a little too dangerous in today's Elf and Safety society to be open outside of planned tours. But, if you have even the slightest passion for railways, you just have to pay a visit. It's by far the best of the many preserved railways in this country.
So that's it. Best part of a week travelling, sight-seeing, reminiscing, and walking up and down hills. But what a great way to get away from it all...
I'm sure my wife won't mind me publicizing the fact, but she is rubbish at packing stuff. When loading a car, instead of starting with the biggest items she starts with the nearest ones and then finds that there isn't room for the big items - and there are no small or medium sized bits left to fill the gaps either. Or, when we're going on holiday, waits until the suitcase is full and then decides to take four pairs of high-heeled shoes.
So what has this to do with guidance, p&p, and documentation (or, in fact, with anything)? Well, packing a book with content is very much like packing a car or a suitcase. There is only a predetermined amount of room for the necessary verbiage and schematics, and you have to put stuff into it in the correct order or it just looks like an embarrassing jumble when you get stopped at customs. The trouble is, writing a book by putting the big things in first and then filling the gaps round the edges doesn't seem like it will work. Especially if you don't actually know what the big things are when you start.
I don't know if it’s a universal truism, but I reckon that to get the makeup of a book right you must be able to see the complete finished work in your head before you start to write anything. It's all about figuring what the Table of Contents will look like before you commit finger to keyboard. OK, so maybe this approach doesn't work when you are writing fiction, though having a detective novel where you only decide who was the murderer when you start to write the last chapter seems unlikely. Discovering that Colonel Prune did it in the kitchen with a gas pipe when, by the end of chapter 2 you were convinced it was the butler, might be a good way to maintain suspense and add a twist to the story. But if Colonel Prune only makes an appearance three pages before the end, it doesn't seem like much of a mystery. Or just too much of a mystery...
Anyway, as we're talking computer and technology guidance here (where creating mysteries isn't usually approved practice) I should probably try and wend my way back to the topic that was in hand a few paragraphs ago. Such as guidance for Microsoft products like Windows Phone 7 - for which our upcoming guidance presented an interesting set of issues.
Thing is, you see, this guidance is part of a series of scenario-based books that cover cloud-based services and the corresponding client applications. When we did the Windows Azure book, it was pretty easy to figure out the TOC. We needed to introduce Azure, introduce the fictional company that was building cloud-based services, and then explain how they did it. Yes, we needed to talk about what Azure is, what it can do, and how you program for it. But these are closely related topics that can easily be covered in the chapters that discuss each part of the overall scenario; such as using tables and blobs, authenticating requests, building responses, and managing the data.
We didn't need to talk much at all about the operating system (Windows Server), the coding languages and framework (C#, Visual Basic, and .NET), and the communication protocols (WCF) because these are all well known and fully documented things. Yes, there are Azure-specific issues such as instrumentation, data storage, and other factors; but these differences are fairly limited in scope and impact, and are easy to include in the individual chapters of the overall scenario. The major bits such as the way applications run, the use of worker roles, and all of the considerations about designing your applications are pretty much the same as when you have a big lump of metal and plastic in an air-conditioned room next door to your office.
And then we come to Windows Phone 7 (WP7). One of the major strengths of WP7 is that lots of things are just the same as in any other .NET application. The platform you program on is Silverlight (or you can use XNA if you like), the languages are the same (we use C#), and the whole development paradigm (Visual Studio and Blend) is just like programming for the desktop or Silverlight in a web browser. There's even an emulator you can use to run and debug your applications, and so it's really just another automatically familiar environment for building applications.
Except it isn't. Once you start to dig, you discover all kinds of things you need to understand and program for. For example, in a desktop application it makes no difference to you if the user switches to another application such as Outlook when the familiar "ping" announces arrival of a new email message. Your application just blithely continues in the background doing stuff. On WP7, when the phone rings or the user decides to read an email that just arrived, your application stops. And the operating system might even decide to close it down altogether, and without giving it the chance to do that "You may lose any unsaved data" dialog thing. So you need to know about application lifecycles and tombstoning.
And what about performance issues? Unless you currently use a very old computer, the phone is somewhat of a poor relation in terms of CPU speed and storage space. Don't expect a quad core Xeon or 2 TB of disk space. Yes, there's a very smart graphics processor, but you still need to worry about frame refresh rates, graphics caching, transitions, and other display-related stuff if you are doing graphics-intensive things or else it could end up looking like a 1930's Charlie Chaplin movie. You need to know how to use the onboard graphics mechanism, the frame counter display, and the correct way to manage bitmaps and graphics objects.
And then there's all the other wondrous capabilities such as location services, the camera, sound recording, the accelerometer, media capabilities, messaging interaction, gesture detection, Windows Marketplace, push notifications, and even vibration alerts. You can (and probably will) use all these in your programs, so we need to explain them as well. In fact, there's a host of exciting capabilities that you can use in WP7 to make your applications attractive and interactive. Though they do need to conform to the standard UI design guidelines and themes for WP7; so we need to explain these as well.
And here lies the problem. Instead of starting with an introduction to the WP7 platform, a few useful links to well-known technologies, and some blurb to keep the marketing people happy, we have tons of stuff to explain that is likely to be new to a great majority of developers. Such as how you can apply common design patterns; how you manage storage and communication over different types of occasionally-connected network connections; how you fetch, store, and upload data; and more generally how you think about the design and architecture for your applications. You need to manage power consumption, think about how and when the phone will be used, and make the UI usable for people like me with fat fingers.
So a brief "About Windows Phone 7" chapter followed by diving into the scenario of the fictional company and their shiny new WP7 application is not an option. There's just too much to think about, and it will make the chapters describing each section of the overall scenario way too complex and bulky. Readers would lose the thread (and even lose the plot) trying to constantly switch from design guidelines and technical reference to scenario implementation details in each chapter.
OK, so we spent a while trying to shoehorn the content into a structure and TOC similar to the other guides in the series to preserve continuity, but we finally had to admit defeat. We just didn't have the right TOC - we'd created it before we could actually see what the completed book would look like, and then tried to mould the content into it. We'd put all the socks, underwear, and sun creams into the suitcase before trying to pack the kettle and the big box of teabags.
The answer was to throw away the TOC and build a new one that made sense for the entire content. An introduction chapter that has the required marketing blurb, explains what a wonderful device WP7 is, and then sets the scene for developers. A pragmatic and focused second chapter that does all the design advice things and explains the issues you need to consider before you fire up Visual Studio. And, to provide the rest of the required guidance without breaking the flow of the main chapters, a series of targeted appendices that cover the development environment, the phone platform, the device capabilities, and other peripheral stuff. Now the remaining chapters that form the bulk of the book can concentrate on the fictional company's scenario, and how they built their application.
Yes, an appropriate TOC does seem like an obvious prerequisite that should already have a big black tick next to it before you start. But it's easy to just dive in and start building content for the parts you already understand, and then add more as you discover other nuances and requirements. Without getting a tick for your TOC at the start, you'll end up (like we did) spending more time trying to sort it all out later. Probably just as the publisher is banging on your office window demanding the final manuscript...
As last week's babble seemed for some unaccountable reason to wander towards a science fictional theme, I thought I might as well follow up this week with something from my favorite (well, one of my favorite) sci-fi authors. I refer, of course, to the unforgettable Isaac Asimov. I got to thinking about his work while watching a TV program about how the Internet is shaping our lives, and how the future for young people will be influenced - and even (rather worryingly) - controlled by the social media sites and commercial content producers that inhabit it.
Asimov's famous character Hari Seldon, a mathematics professor at Streeling University on Trantor, developed the science of psychohistory, which effectively provides a way to accurately model the collective behavior of entire galactic populations over very long periods of time. Thus he could predict with unerring accuracy the major events that will befall each civilization; such as wars, plague, pestilence, and lack of Internet connectivity (OK, so I made that last one up).
But, anyway, until a rogue being appears on the scene, all is running pretty much according to plan and the regular appearances that his holographic automaton makes prove amazingly accurate. And there are some people who think that psychohistory could, in fact, be a reality and not just fiction. Adolphe Quetelet's "Social Physics" and John Xenakis's "Generational Dynamics" follow a similar theme, and there's even whole Web sites devoted to the science of "Cliodynamics".
Thing is, it seems that an increasing proportion of the population is influenced more and more by what they see on the Web. I'm personally not a twitterer or spacebooker, and I tend to avoid networking sites such as LinkedIn and the plethora of similar ones on the grounds that I'm too busy working with computers to spend my leisure time at the keyboard. I'm certainly not interested in what Stephen Fry had for breakfast this morning, and I'm quite content with having about 4,999,980 less friends than Lady Gaga.
Meanwhile, psychohistory relies on two axioms: the population whose behavior is modeled must be sufficiently large, and must be unaware that psychohistorical analysis is taking place. It's probably safe to say that the population of twitterers, spacebookers, diggers, stumblers, and the like is sufficiently large; and I'd imagine they're too busy twittering and spacebooking to notice that we're watching them. So the axioms should hold. Therefore, what can we actually predict? How about:
All I need now is my holograph likeness, a glass box, and a smoke machine...
If you're a Douglas Adams fan, you'll know all about the fabulously beautiful planet named Bethselamin. The ten billion tourists who visit it each year were causing so much erosion that they introduced a rule whereby any net imbalance between the amount you eat and the amount you excrete whilst on the planet is surgically removed from your bodyweight when you leave (and therefore, as Douglas mentioned in the book, it is vitally important to get a receipt every time you go to the lavatory).
Yes, it's a well-worn quote, but I'm beginning to get nervous that it's starting to come true here on our own little blue-green planet (located, of course, in the uncharted backwaters of the unfashionable end of the western spiral arm of the Milky Way galaxy). And all because I needed a couple of new batteries for my uninterruptable power supplies.
We have a wonderful battery supplier just a mile or three down the road from here who seems to stock every kind of battery that human endeavor has managed to create, including sealed batteries for every APC UPS made in the last 20 years. That just about covers all the various models I have scattered around my house and office. What's more, they are considerably cheaper than buying online, there's no shipping cost, and they take the old ones back for recycling.
OK, so the APC ones do come with a label so you can ship them back to APC, but the address on the label is New Jersey USA. The nice man at our local post office worked out that it would be cheaper to fly over myself and take them back rather than sending them by post - though, of course, I'm not allowed to take batteries on a plane any more...
So, anyway, there I am with a couple of dead RBC2 batteries and a bag of assorted other used AA and the like, when the young assistant asks me for my name, address, postal code, phone number, and where I obtained the original batteries. As he'd been very pleasant and helpful so far, telling him to mind his own business seemed a bit strong, so instead I politely inquired as to why he wanted to know. I had deposited on the counter a selection of coins and notes of the realm in way of payment, so surely all he needed was to grab the cash, stuff it in the till, and I'd be on my way?
No, it seems that he needs to give me a receipt for the old batteries that are going for recycling. Maybe if there's not quite enough lead in them when they get round to breaking them up, they'll send me a bill for the balance? Or perhaps I'm only allowed to ecologically dispose of a specific quota of murky sulphuric acid each year and he's worried I'm approaching the limit? Aha! - more likely it's in case I stole them so I could sell them for the scrap value (which seems to a little perverse when I'm giving them away free for recycling).
It's plainly all part of a secretive scheme connected with the move to weigh and analyse the contents of our dustbins, film and record the registration number of our cars when we go to the local rubbish tip, and remotely monitor our energy consumption every hour with the new smart electricity meters. It will all be fed into some huge computer that will send out letters once a month with demands for the requisite body parts to make up the imbalance between our resources input and recycled output.
So it's probably a good idea to make sure you do get a receipt every time...
According to the latest update bulletin from MessageLabs that lands in my inbox each month, around 90% of all emails passing over the Net are spam. And their global report says that around 120 billion spam emails are sent out from botnets every day. That means there's around one and a half million unwanted messages being launched onto and scooting around some part of the network every second. And that's without all the gunk required to accomplish the other types of malicious activity, such as the DNS Amplification attacks I'm being regularly subjected to at the moment.
Of course, the bulk of traffic on the Net is probably HTTP Web browsing, not email. But you have to wonder what percentage of the resources that make our interconnected technological experience work are required just to cope with stuff that nobody actually wants. How many power stations could we close down, how many degrees global warming could we avoid, and what savings could we make in cost, resources, and raw materials if we could just find a way to kill it off?
And how come the people who run the world are more concerned about making me use these awful new low energy light bulbs than applying their influence and capabilities (?) to an issue that should be easy to fix, and would have only positive outcomes for everybody? Not that our temporarily coalesced Government here in the UK have much idea about information technology anyway. They just announced that the previous plan to allow everybody to get "high speed" (2 MB) ADSL connections by 2015 was over-ambitious and will not now happen - even though they are wholeheartedly backing the introduction of multi-channel TV over the Internet in the next year or so. Yeah, that'll work...
And even the ISPs seem to be unable to do anything about it. My ISP, when asked about the recent DNS attacks, agreed that they were killing connection speed for many customers and affecting bandwidth availability across the network; and - yes - they know which ISPs and which IP addresses the attacks are coming from; but they are not allowed to block these addresses as part of some "international agreement". In the same way that they're not allowed to block the torrent of unsolicited fax and phone calls we get here that originate in various countries around the world.
I suppose the only saving grace is that I'm old enough to remember when networks were even less reliable and performant than the Internet is now. My first contact with digital electronic communication was though an acoustically-coupled (clamped onto a telephone handset with a rubber band) dial-up modem that ran at 1200/75. That's a sniffle less than 1.2 KB (not MB) down and 0.1 KB up. Somewhat slower than even the slowest dial-up today, and about one thousandth of the speed of my current (slow) ADSL connection.
Yet there was a real sense of adventure watching each 80 character wide by 25 line screen slowly fill up with text at a rate of about two screens per minute (even Twitter would have seemed slow then), and submitting a document back was often a "leave it running and come back the next day" scenario. Though usually the modem decided to drop the line when your neighbour's phone rang, or when the stroweger mechanical switch in one of the intermediate exchanges had some dust on its contacts. The typical procedure was to hit "send", go off and have dinner, then phone the recipient to see if it had arrived - and then read out the original so they could correct all the transmission errors.
That was the other problem, of course - no reliable error correction in the protocol or hardware. If you tried to communicate in real (slow) time, you had to contend with long waits before each word or part of a w or d a p p ear ed and be able to dec1p&er the stran%e char@£acte~s that ad0ed a cer/ain addi|ional piqu@ncy to the content. Or be prep@r3d to rec0n;ect when it sudden(y st*pp=d half~ay through a vit*lly |mp0~tant
So I guess I shouldn't really complain. I mean, I opened a Web browser and uploaded this blog post in only a few seconds, and was reading the result on screen in less than a minute. What would have been even more useful, of course, is if the content was actually worth all the resources it took to put it there...
As far as I know, nobody has yet been able to answer the long-standing question of what will happen if you spread butter on the back of a cat. Will it exhibit the buttered-toast effect, or will it still land on its feet? We know from the established principles described by Murphy's Law that toast always lands butter side down; but at least there is a thread of scientific explanation for this, which says it's because the buttered side is smoother and more "slippery" - thus offering less air resistance and causing that side to fall faster and hit the ground first.
Associated with Murphy's Law is the far more interesting Law of Unintended Consequences. I was reminded of this just last week while reading an article by a school teacher who was discussing the worrying trend in the increasing number of school children categorized as having Special Educational Needs (SEN). Of course, many people are applying the political football approach to this; citing the usual factors of poor housing, lack of parenting skills, broken families, and Government interference in the school syllabus.
However, the author of the article took a different approach. See if you can spot any factors that may contribute to the increasing trend in the following:
Of course, unintended consequences due to Government intervention are obvious all over the place. These days the police have to meet targets for the number of crimes they solve, so a constable who sees somebody running away from an illegally parked car they've broken into has to instantly decide whether to give chase on the off chance that he can catch the runner and find some case for arresting them, or instead fill out a ticket and slap it on the illegally parked vehicle.
Likewise, family doctors have to provide an appointment on request within two working days. So, if you decide to book ahead for a routine visit next week, you can't because that would reduce their "patient charter" average score. Meanwhile, as our local council strives to reduce incidences of fly-tipping, the local council-run waste reclamation facility only allows you to use it if you arrive in a car. If you decide to hire a van or truck when you finally get round to clearing out all the rubbish in your grandparent's garage, you need to apply a week ahead by filling in a four page form to apply for a "waste reclamation site access certificate".
But unintended consequences have a far wider area of incidence, especially in our own industry. For example, I decided to green up my server cabinet by virtualizing all of my servers (see blog entries passim). However, now the domain controller is a virtual machine that doesn't start up until the base O/S is running. As it's a member of the domain, it does lots of grumbling and sometimes has a complete hissy fit when it reboots because it can't find its domain controller. And my NAS can't authenticate with my servers because I upgraded them to Windows Server 2008 and the NAS has never heard of that.
However, when it comes to software, I guess the law really comes into its own. I've recently been working on a project that combines Windows Phone 7 with Azure cloud services. But the phone is not exactly a hugely powerful platform for applications. It would be nice to think you could buy one with a quad core Xeon processor and four gig of memory on board, but until they invent mobile power stations that seems unlikely. So I have to minimize my application's memory footprint, reduce power consumption when possible, be aware that garbage collection will interrupt execution, and limit the number of requests I send to remote services to minimize communication costs.
Yet my desires to follow best practice and implement well known design patterns suggest I should use data transfer objects and view models to store intermediate data, use dependency injection to decouple my components, and apply configuration-based composition of interfaces to maximize upgradability and minimize maintenance costs. But I'm not supposed to create lots of objects in memory in order to minimize memory footprint and garbage collection intervention; yet I'm also required to hang onto objects rather than recreating them repeatedly, and use less of them. Maybe I should take up tight-rope walking as a hobby to get some practice in phone application development?
And don't get me started on writing documentation. If I slavishly follow guidelines for minimizing content to only that which is directly applicable to a scenario, users will be left wondering how they accomplish something more real-life than my simple examples. But if I dive deep and cover every aspect, there's too much to read and it becomes overwhelming. And if I simply present the basic facts, it's hard to see how you could describe this as guidance. Yet, if I litter it with links to other resources, users just get lost in the plethora of semi-redundant and partially related information.
It's a good thing that my blog posts are concise and focused, and don't just wander aimlessly from one vague topic to another...
Well, perhaps last week I could have. It turns out that it was quite happily performing as an amplifier. And there was me thinking I understood this stuff. Another Decidedly Negative Scenario in terms of my network administrative abilities. But at least I've learnt some more things I didn't know that I didn't know. What follows is a gentle stroll through the intricacies of DNS and firewall management I encountered.
It seems that DNS recursive amplification attacks are back again (if they ever went away). I first noticed the constant blinking of the router lights last week and decided, even though everything seemed to be working fine, to investigate. I checked that I had all the latest updates and patches installed, so that's not the problem. And the Web server (a Windows Server 2008 Hyper-V virtual machine) has Windows Advanced Firewall and IP filtering enabled to allow inbound packets only to the Web server and the DNS server. After a little digging, by selectively disabling the firewall "allow" rules and watching the network connection status display, I discovered it was regular long bursts of requests over UDP on port 53 to the DNS server on my public Web/DNS server machine. But why? Who could be that interested in my DNS entries?
It didn't take long to realize that they were requests for a list of name servers configured in the DNS. This is, of course, the whole point of the attack. The attacker sends a small payload and gets back a ton of data. And the clever bit in terms of the attack is that the return address for the DNS lookup is spoofed, so the response from my DNS server is not sent back to the attacker, but is instead sent to someone else's machine as my free contribution to a DDoS attack on that sever. So what to do about it? My colleague and I run public DNS servers to host the entries for about twenty local Web site and blog domains that we manage (acting as primary and secondary for each other's domains). So I can't just disable DNS.
I'd assumed that just blocking packets inbound from the supposed sender would made no difference - the packets are not actually coming from that address. In fact, they're probably coming from lots of different addesses in some botnet. And, in addition, the attacked addresses are likely to change regularly. But the firewall also sees the spoofed address, and so blocking these addresses does work. While this is not the ideal way to solve the problem, at least it stops some of the flow over the Internet and removes me from the list of innocent attackers. You can add multiple remote IP addresses to a blocking rule, which makes managing the changing list of attack addresses easier. Perhaps, in real life, I should spend a few thousand dollars on a hardware firewall that detects and automatically blocks these kinds of attacks. It could soak up some more expensive electricity and further raise the ambient temperature inside my server cabinet.
I threw together a simple Windows Forms application that monitors the DNS log for recursive name server queries and notifies me so I can quickly detect new ones. You can download it here.
Another core issue is that a public DNS server should not allow public recursive address lookups. It should only resolve public IP address lookups for domains for which it is authoritative (in other words, public domain names that you manage), and should not be referenced from any machines on your internal network. They should have their own DNS server that does recursive lookups to your ISP, or they should send requests directly to your ISP if you don't have an internal DNS server. Even the DNS server machine itself should use the ISPs DNS server to do its own lookups. If you do have recursion enabled (it seems that 70% of DNS servers out there do), your DNS server will even go off and look up the IP addresses of all the name servers and return them - further adding to the load on your connection and the Internet as a whole. So, open the Advanced tab of the Properties dialog for the DNS server itself (in DNS Manager) and make sure that the Disable recursion setting is ticked. This also disables any forwarders defined in the Forwarders tab.
If yours wasn't ticked before, you'll probably now find that a browser and other applications on the server can't get to any external sites. This is because you accepted the defaults when you originally set up the server (i.e. no DNS server address assigned in the network properties dialog), which causes Windows to use the local DNS server (127.0.0.1). Without recursion and forwarders, your DNS server can only do lookups for the domains it hosts. You'll need to replace the 127.0.0.1 address in the Advanced | DNS tab of the Network Properties dialog for the IPv4 and IPv6 protocols with your ISP's DNS server addresses. Applications and services running on the Web/DNS server will now query the DNS servers that are configured in the network properties dialog, and will not use the local DNS server.
After you get everything working again, you'll probably see that an NSLOOKUP from another network into the DNS server returns a list of root server addresses. These are hints to the requesting server on where to go to find the address it is looking for (because your DNS server can no longer do a recursive lookup). And if you look in the DNS log at the contents of the packets returned by the DNS server, you'll see that they're still quite large. So I started wondering - do I actually need root hints in the DNS server if all it's ever going to do is respond to requests for authoritative domains that it hosts? It's never going to need to know where www.someothersite.com actually is, or tell someone else how to find it.
So I decided to experiment with root hints. If I delete or loose them all, I know I can get them back by copying the cache.dns file installed in the %System Files%\dns\backup folder. So I deleted them and then used the Copy from Server button in the Root Hints tab to load them from one of my ISPs servers. And they were completely different from my original set. So I wandered across to IANA to get the latest list - which is remarkably different from that in my ISP's DNS! And, digging deeper, they seem to have different lists in each of their DNS servers. No wonder DNS lookups are slow sometimes...
Anyway, getting back to the issue in hand, do I really need root hints? As a test, I removed all of them from my public DNS server, and removed the forwarders as well. Now a request for one of the authoritative domains still works fine, as do transfers to the secondary DNS server, but recursive requests prompt a very small return payload containing a "Server Fail" code. It's hard to tell from all the blogs and guidance I've read on DNS good practice if this is an approved approach but, to be honest, nobody should be querying my DNS server for non-authoritative domains anyway. So it's their own fault. Meanwhile my contribution to the DDoS attacks is very significantly reduced. Though now I get a Warning entry in Windows Event Log telling me that there are no forwarders or root hints each time the DNS service starts ... but I'm ignoring these.
Having now got my hands dirty, I thought it would be a good idea to see if I could harden the server a little more. Such as applying outbound filtering in the firewall. I do it in ISA Server on my private internal network, but its easy there because there are lots of appropriately pre-configured rules you can apply. Windows Firewall with Advanced Security (WFWAS) seems to have lots of predefined rules, but not for useful things like allowing Web browsing and sending email. Yet I must have read fifty articles on the Web looking for details of appropriate rule properties, and all I could find were articles and blog posts that said things like "Select the predefined SMTP Server rule ". Err, where is that? Only in ISA Server I suspect.
Mind you, it's not hard to use the New Outbound Rule Wizard create a Custom rule that allows Web browsing and services on the machine (such as Windows Update) to work. Basically it's any service/user using TCP through any outbound port to any remote server on ports 80 and 443 (and maybe 8080 or similar as well if you need to access sites you know are running on this or a non-default port. Visual Studio Team Foundation Server is a typical example). Make sure you create it in the Outbound Rules page, and be sure to specify the Public Profile, not Private or Domain. If you look at the Monitoring page in the WFWAS console, you should see that the Public Profile is active. If not, immediately dive into Network and Sharing Center, click "Customize", and set the connection type to Public!
To be more strict with Web access, you could configure the rule to allow only specific services or applications - but that's much more complicated. For example, what about your "alternative" Web browser, or Java Update Service, or the updates service for Adobe Reader? And remember Windows Update... As a mitigating factor, you should be running your browser in Enhanced Security mode on the server anyway, and only using it when absolutely necessary.
Allowing DNS queries from your server, and responses to other servers from your DNS server, out through the firewall is easier because there are preconfigured rules for these. I found that I needed to enable "All Outgoing (TCP)" and "All Outgoing (UDP)" in the "DNS Service" group, and "Core Networking - DNS (UDP Out)". Again, make sure you select the ones for the Public Profile, not Private or Domain. Finally, if you run an SMTP email server, you need to allow packets from this to escape out onto the Internet. I use the IIS 6.0 SMTP Service that is part of Windows Server 2008, carefully configured to prevent relaying. And as it does not receive email (the Reply To for all messages is my usual Web site administrator email address), it cannot be accessed from the Internet anyway because port 25 inbound is closed.
So, a simple Custom rule that allows only the SMTP server application (inetinfo.exe) to go out using TCP through any local port to only port 25 on remote servers should do it. Then flip over to the Properties dialog for the server itself (the root entry) in WFWAS Manager and set the Outbound connections drop-down to Block (the default is Allow) so that the only permitted outbound traffic is that defined in your enabled outbound rules.
Or so I thought. Being not a little naïve, I expected the SMTP service to use the local DNS Client (not the local DNS Server) to do the lookups required for delivering mail. I mean, everything else seems to use this - but not the SMTP Service. It obviously does its own lookups. Maybe this is something to do with last year's patch that changed the behavior of the SMTP Service and stopped the instance on my internal network from relaying essential email status messages - when it was quite happily doing so before. So, anyway, what you need is another rule that allows inetinfo.exe to go out using UDP through any local port to only port 53 on remote servers. And now (at least temporarily) everything started working properly again.
Of course, a couple of hours later I discovered the error messages in Windows Event Log telling me that the Time Service was broken. Ahh.. forgot that one, so add another Custom outbound rule to allow just the Windows Time Service (click the Customize button next to Services in the wizard) to use UDP on any local port to connect to any remote server on port 123. Want to check that you can ping the time servers you use? That's when you'll discover that PING doesn't work any more. And neither does NSLOOKUP or TRACERT. To allow pings out of your server, you can simply enable the preconfigured rule "File and Print Sharing (Echo Request ICMPv4 Out)" - a nice snappy name, though enabling file and print sharing sounds scary. But if you examine the rule (click Customize in the Protocols and Ports tab), you'll see it only allows ICMP Echo requests to escape.
Meanwhile, for TRACERT and NSLOOKUP, you need another Custom outbound rule that allows any service\user to use UDP through any local port to port 53 on any remote server. I imagined that the predefined rule "Core Networking - DNS (UDP Out)" would allow this, but it is limited to the svchost.exe program and so is no help for the other stuff like DOS utilities. If you want to use TRACERT and NSLOOKUP, and you create the rule for them, you can disable the "Core Networking - DNS (UDP Out)" rule. You can also remove the custom rule you created to allow DNS lookups for the SMTP service, as it also uses UDP to remote port 53. However, it's a good idea to leave it in place so the service will still work if you later decide to block TRACERT and NSLOOKUP.
Except, in my case, NSLOOKUP still wouldn't work - all I got was "UnKnown" (note the interesting letter case) for the DNS server name, and an IPv6 address instead of the usual xxx.xxx.xxx.xxx format one I was expecting. Typing "nslookup", then "set d2", and then a domain name produced the interesting response that the request was too long. In the end, my "large hammer" solution was to simply disable the IPv6 protocol in the properties for the network adapter and normal service was resumed. I guess this is something I'll need to come back and look at again.
What I can't help wondering, though, is when we'll finally solve the problems with spoofing that are already so widespread with email, and are obviously becoming just as common with DNS. I can (and do) use Sender Policy Framework (SPF) to advertise a list of valid IP addresses for email I send, but how do I do the same for DNS? And would it make any difference...?
A colleague pointed me at an interesting discussion the other day about whether geeks are actually "creative". It comes partly from a recent post by Ian Betteridge that rails against the claims that App Inventor, which is designed to encourage development of simple programs for Android, "enables people to be creative and not just passive consumers". However, what he doesn't explore is the real meaning of the word "creative" in today's terminology.
OK, so the dictionary defines "creative" as meaning "producing or using original and unusual ideas", with synonyms such as "original", "imaginative", "innovative", "artistic", and "inspired". These seem fine if you are talking about someone producing a stunning work of art in Corel Painter, or some spectacular new soundtrack using Roland Cakewalk. But nobody knocked up this software using a utility such as App Inventor. Instead it was created by a team of engineers working to fine tolerances and requiring a deep knowledge of the subject and technologies. Dare I say, "geeks"?
Looking at most Web sites, social network pages, online photo albums, TV programs, and magazine adverts, or listening to a large proportion of recently recorded music, it's hard to find much in the way of material you could even charitably describe as "original" or "innovative". Even computer applications seem mostly to be evolution rather than "inspired". How many word processors or virus scanners have you seen that you can actually say are "innovative", and when did you last install a program that was truly "original"? Maybe, as today's focus seems to be all about style (generally over content) you could perhaps ascribe the term "artistic" to many of these things - but even that rather stretches the imagination in the majority of cases.
However, according to my thesaurus, "creative" also spawns synonyms such as "inventive", "resourceful", "ingenious", and even "productive". So your modern word processor, spreadsheet, photo editing tools, and more - with their clever "automatic everything" and powerful Wizards for anything more complicated than typing a sentence - would fit well with this definition of creativity. Though, personally, I feel that equating creativity with productivity rather stretches the point. If I can type fast, I'm more productive. But the result often isn't creative in terms of the content.
I suppose, as a geek, I don't really want to be "artistic", or even "original", anyway. I'd like to be "productive", "inventive", and possibly "ingenious". I want what I build to be architecturally robust using tried and trusted techniques and proven technologies, and I'm happy to leave it to the UI designers to inject the artistic stuff. And, as I've never been on a creative writing course, I assume that what I do most days here at p&p is more about being technically accurate and informative rather than relying on artistic license (unlike my blogs posts).
Ian also suggests that we geeks no longer rule the universe, and that our era is over. Yet, without us, none of this creative stuff would exist - and the world would be very different. I can't see anyone using App Inventor or similar utilities to implement the software that controls a nuclear power station, or powers communications satellites. And I doubt that most corporate and financial data centers rely on programs written by a media analyst or a fashion designer. Was the O/S for the Android (or even App Inventor itself) written by a social communities coordinator or a society wedding planner? I don't think so. Geeks still do, and always will, shape our world. OK, so it doesn't look very pretty when we've finished with it, but we just hire in a creative artist to make the UI look nice afterwards.
But I guess where all this is going is that - today - the word "creative" actually has negative overtones in many scenarios. In a previous life as a salesman, I could almost guarantee that my manager's response to reading my monthly sales report would be to praise my creativity...