Random Disconnected Diatribes of a p&p Documentation Engineer
After spending part of the seasonal holiday break reorganizing my network and removing ISA Server, this week's task was reviewing the result to see if it fixed the problems, or if it just introduced more. And assessing what impact it has on the security and resilience of the network as a whole.
I always liked the fact that ISA Server sat between my internal domain network and the different subnet that hosted the router and modems. It felt like a warm blanket that would protect the internal servers and clients from anything nasty that crept in through the modems, and prevent anything untoward from escaping out onto the ‘Net.
The new configuration should, however, do much the same. OK, so the load-balancing router is now on the internal subnet, but its firewall contains all the outbound rules that were in ISA Server so nothing untoward should be leaking out through some nefarious open port. And all incoming requests are blocked. Beyond the router are two different subnets connecting it to the ADSL and cable modems, and both of those have their firewalls set to block all incoming packets. So I effectively have a perimeter network (we're not allowed to call it a DMZ any more) as well.
But there's no doubt that ISA Server does a lot more clever stuff than my router firewall. For example, it would occasionally tell me that a specific client had more than the safe number of concurrent connections open when I went on a mad spree of opening lots of new tabs in IE.
ISA Server also contained a custom deny rule for a set of domains that were identified as being doubtful or dangerous, using lists I downloaded from a malware domains service that I subscribe to. I can't easily replicate this in the router's firewall, so another solution was required. Which meant investigating some blocking solution that could be applied to the entire network.
Here in Britain, out deeply untechnical Government has responded to media-generated panic around the evils of the Internet by mandating that all ISPs introduce filtering for all subscribers. What would be really useful would be a system that blocked both illegal and malicious sites and content. Something like this could go a long way towards reducing the impact of viruses and Trojan propagation, and make the Web safer for everyone. But, of course, that doesn't get votes.
Instead, we have a half-baked scheme that is supposed to block "inappropriate content" to "protect children and vulnerable adults". That's a great idea, though some experts consider it to be totally unworkable. But it's better than nothing, I guess, even if nobody seems to know exactly what will be blocked. I asked my ISPs for more details of (a) how it worked – is it a safe DNS mechanism or URL filtering, or both; and (b) if it will block known phishing sites and sites containing malware.
The answer to both questions was, as you'd probably expect, "no comment". They either don't know, can't tell me (or they'd have to kill me), or won't reveal details in order to maintain the integrity of the mechanism. I suspect that they know it won't really be effective, especially against malware, and they're just doing it because not doing do would look bad.
So the next stage was to investigate the "safe DNS services" that are available on the ‘Net. Some companies that focus on identifying malicious sites offer DNS lookup services that automatically redirect requests for dangerous sites to a default "blocked" URL by returning a replacement IP address. The idea is that you simply point your own DNS to their DNS servers and you get a layer of protection against client computers accessing dangerous sites.
Previously I've used the DNS servers exposed by my ISPs, or public ones such as those exposed by Google and OpenNIC, which don't seem to do any of this clever stuff. But of the several safe DNS services I explored, some were less than ideal. At one of them the secondary DNS server was offline or failed. At another, every DNS lookup took five seconds. In the end the two candidates I identified were Norton ConnectSafe and OpenDNS. Both require sign-up, but as far as I can tell are free. In fact, you can see the DNS server addresses even without signing up.
Playing with nslookup against these DNS servers revealed that they seem fast and efficient. OpenDNS says it blocks malware and phishing sites, whereas Norton ConnectSafe has separate DNS server pairs for different levels of filtering. However, ConnectSafe seems to be in some transitional state between v1 and v2 at the moment, with conflicting messages when you try to test your setup. And neither it nor the OpenDNS test page showed that filtering was enabled, though the OpenDNS site contains some example URLs you can use to test that their DNS filtering is working.
The other issue I found with ConnectSafe is that the DNS Forwarders tab in Windows Server DNS Manager can't resolve their name servers (though they seem to work OK afterwards), whereas the OpenDNS servers can be resolved. Not that this should make any difference to the way DNS lookups work, but it was annoying enough to make me choose OpenDNS. Though I guess I could include both sets as Forwarders. It's likely that both of them keep their malware lists more up to date than I ever did.
So now I've removed all but the OpenDNS ones from my DNS Forwarders list for the time being while I see how well it works. Of course, what's actually going on is something equivalent to DNS poisoning, where the browser shows the URL you expect but you end up on a different site. But (hopefully) their redirection is done in a good way. I did read reports on the Web of these services hijacking Google searches and displaying annoying popups, but I'm not convinced that a reputable service would do that. Though I will be doubly vigilant for strange behaviour now.
Though I guess, at some point, you just have to trust somebody...
Having run out of ideas, and given up Binging a solution for my intermittent connectivity problems around Windows 8, IE 10, and Outlook, it was time to stop playing nicely. Time instead for a day with my head in the server cabinet, a handful of network cables, some sticky labels, and decisive action.
The problems documented over the past few weeks (see All I Want For Christmas) were still intermittently annoying, as well as being annoyingly intermittent. I was totally unable to track down any DNS problems, despite many hours experimenting with different forwarders, root entries, test scripts, clearing caches, and more.
I'd logged all dropped packets in ISA server for a day, but there were none related to the problem from the machines under test. Though ISA's performance monitor was consistently reporting an average dropped packet rate of 0.3 per second and I wasted half an hour tracing these back to my wireless access point. Even though all of its fancy USB, printer connection, and media sharing features are turned off it still insists on sending out a network discovery packet every three seconds. Highly annoying.
Then I read more on the ISA Server blog sites about how using the proxy client changes the behaviour of machines connected to ISA. Of course, I haven't actually installed the proxy client directly since XP days. I just assumed that some clever mechanism in Windows Vista, 7, and 8 used the Gateway/Router setting specified in DHCP to find the proxy server and set themselves up for it automatically.
What I read suggested that ISA itself is doing DNS lookups in response to requests from clients, whereas a ping or nslookup on the client uses the network DNS server or does its own DNS lookup. So trying to track faults with nslookup after a connection failure was a waste of time. By now I was rapidly tiring of trying to be a network administrator, and I didn't bother following this up any further to see if it really is the case.
All of which prompted the decision to perform some radical surgery in the server cabinet, and get rid of ISA Server altogether. It's a Hyper-V VM, so it won't reduce the physical server count - but it will simplify administration and backup tasks and, hopefully, resolve my connectivity problems. I replicated all the ISA outbound rules in the firewall of my load-balancing router, which sits between the ISA server and the two ISP modems. A day monitoring the router logs and fine-tuning rules suggested this would work fine.
Reconfiguring the network was deceptively easy. Simply power off the ISA VM, change the IP address of the router to the same as ISA (the address already specified in the DHCP scope options), and run some network penetration tests. Instantly everything seems to be faster, web page loads are snappier, and no sign of smoke or loud bangs. And if it all goes wrong, or turns out to be a mistake, I still have the ISA Server VM so I can easily revert to the previous configuration.
But will my simple load-balancing router be able to cope with all that extra firewalling and packet shifting load once I start hammering the network with my usual working-week vigour? It's only an old LinkSys RV042 with a 100 Mbs Ethernet port. Do I need to upgrade to something like the new RV320 instead? I guess I'll soon find out.
And, of course, the question now is what will I do next if my intermittently annoyingly connectivity problem is still annoyingly intermittent...?
So there's another New Year on the horizon and it's time to make some resolutions that will hopefully last for at least a few weeks into January. But at the moment I can think only one: find a new Internet provider.
As previously documented in these pages, I really do try hard to deal with my business cable broadband provider. But they seem to try even harder to make it difficult. I guess the only saving grace is that, on average so far, I've only had to actually contact them once every four years.
The trials and tribulations of it taking four months to get my account set up initially are long forgotten (except as an anecdote for long winter evening when geeks gather around a hot router discussing technology). And even the seven weeks waiting for an upgrade that simply involved changing the modem to a different model (where I did most of the configuration myself) are gradually fading into distant memory.
Of course, I joked at the time that it would probably take another four months to get the invoicing right, though after intervention from the local office manager it seemed for a while that I was being unduly pessimistic. After only a month, I had a correct invoice for the upgraded package. Amazing.
What I didn't realize was that I was still being billed for the old package as well. It was only when I checked the welter of paperwork dropping through the letterbox in more detail that I discovered two invoices with the same number. That's when I found that an "upgrade" is really a "brand new customer".
Yep, despite the difficulties in actually getting a line installed at all, or a modem replaced, I am now the proud owner of two different accounts - and I get the privilege of paying for both. I'm confidently expecting to be told there is a charge to have the old account closed, and a waiting list of five weeks to do so. Perhaps they'll send an engineer round again to check if I have two cables coming into the house.
It makes me laugh when I hear people say they will never deal with our ex-monopoly British Telecom ISP because they are "a pain in the neck" and "useless". BT are my secondary supplier and I cannot fault their service, be it technical or paperwork-related. The only problem is that their promised roll-out of high-speed fibre seems to have stalled before it got as far as me. I'd switch over to them tomorrow if I could get more than 1.5 Mbs.
Though, based on experience, I'll probably have half a dozen Virgin Cable connections by the time BT find a bit of fibre long enough to reach the cabinet on our street. It seems it's rather like Hotel California. You can cancel, but you can never leave…
UPDATE: According to the BT website today, the availability of its "Infinity" high speed upgrade that was due last September, morphed into October, drifted quietly into November, and was finally promised to be definitely here in December, is now advertised as "between January and March". Yet they still keep phoning me to ask why I haven't yet signed up for their broadband TV package.
So I continue to battle with Windows 8.1 and Outlook 2103 on my nice big Dell workstation. Our IT support department have given up on me, saying it's obviously a problem with my own network configuration. And it looks like they are correct. It's just a shame they can't tell me what the problem is.
For a long while Outlook has been doing strange things. It had a few days of keeping count of how many messages I sent during the day (see Downwardly Upgraded) but that problem seems to have gone away again. It also regularly loses its connection to the mail server and then restores it - sometimes immediately but at other times it takes several minutes. And, best of all, it waits about ten minutes before displaying the Windows 8 desktop notification of new emails. Usually I've read and deleted them by the time it pops up.
The same issues occur on other computers as well, both Windows 8 and Windows 7, but only when connected to my internal domain and going out through my ISA proxy server and load-balancing router to one of my ISPs. Bypassing all this, and plugging directly into the back of the ADSL modem, seems to solve the problem. So it's increasingly looking like an internal network issue.
I've checked all the DNS servers I use as Forwarders and they resolve fine. There are no event log warnings in any of the servers. The ISA proxy server log reveals no denied requests to my email host, and only one or two to anywhere else - certainly not enough dropped packets to justify the problems with Outlook. I turned on logging in Outlook and used the new Microsoft Message Analyser to read them, but I can't make any sense of the contents. I tried network packet sniffing, but that revealed nothing useful from the few bits I could decipher.
And then there's browser. Occasionally it has a spell of not being able to find sites. Today it couldn't find Bing for about five minutes yet other sites worked fine. Then it couldn't find the MSDB Blogs site. Other days it can't find anything for several minutes, then it all starts working again. Yet everything else seems to work just fine. My internet radio plays radio, Lync links, Team Foundation Server serves, and News has the up to date news.
I've tried disconnecting the modem for the cable ISP connection and just using ADSL to a different ISP, and vice versa. I've run network diagnostics and DNS validation checks. I've monitored the performance of the ISA 2006 server, and double-checked all the rules. I've played with the routing tables in the separate hardware load balancer. I tried specifying the proxy server settings manually in the browser. All to no avail.
Maybe my network is just too complicated. It's left over from the days when I was an IT consultant (well, jumped-up writer and occasional conference speaker actually) when I needed lots of infrastructure for developing and testing the few applications I built for customers. And, I guess, because I enjoyed playing with hardware. Perhaps it's time to review that decision. Do I actually need:
...just to use Word, Visio, and Visual Studio? Probably not. And all of a sudden I can see why my electricity bill is so high.
Perhaps my Christmas present to myself will be a nice hardware firewall that I can just plug everything else into and forget about it...
So it looks like my upgrade from Windows 7 to Windows 8.1 actually downgraded me to Windows 6.3. I know I don't keep up with all the latest whizz-bang O/S releases, but I can't say I've ever heard anyone eulogising Windows 6.3 as the ideal choice for today's modern computing environment.
My regular reader will know that I'm generally averse to changing anything that's not completely broken, and I was quite happy with my old big-iron Dell Precision 7500. It's got bags of disk space, lots of CPUs, two big screens, and is easy to use. But, as usual, I got overtaken by technology.
The problem in this case was Visual Studio. I was still on 2010 but the latest Windows Azure stuff needs at least 2012 to work. So if I have to upgrade on a box that's been gradually filling up with the usual effluent from years of operation and upgrades, why not go the whole hog and upgrade everything? Windows 8.1, Visual Studio 2013, and Office 2013. Who says I'm afraid to take risks?
I even ran the upgrade checker to make sure the box was capable of handling all this exciting new software, and it generally looked optimistic. So I hop over to our software distribution site and grab Windows 8.1 Enterprise, thinking I might as well have all the available goodies. Except that, after an hour or so, I discover that you can't install Media Center on this. Well you can, if you read some blog posts on nefarious sites, but it seems to involve lots of hacks that I probably want to avoid.
So do I accept that I have to give up watching the golf or snooker on TV in a window on the second screen while I work, and no longer enjoy some old Kate Bush videos to smooth the path through my daily Windows Azure documentation woes? No chance. Just grab Windows 8.1 Pro from our software site and install that over the top. Amazingly, it worked, and I quite happily paid my £6.99 to buy the Media Center add-on.
And so, onwards I go installing the rest of my daily working environment requirements. It all seems to just work, and even Corel Photo and a couple of other apps that are supposed to have issues with Windows 8.1 installed and ran. But here's the intriguing thing. When I send emails through Outlook 2013 on this box it seems to have gained a new memory feature. The send/receive bar starts off OK with "Sending message 1 of 1". But when I send another message, even long after the first has gone and the Outbox is empty, it says "Sending message 2 of 2". Then "Sending message 3 of 3", and so on.
I guess it's neat because I can tell how many emails I sent since the last time I opened Outlook. Though, according to our IT help desk, this isn't supposed to happen. Or Outlook getting fed up every now and then and locking up with the message "Send/Receive" in the status bar and nothing coming in or out. The technical term for this is, I'm reliably informed, "broken".
And the fun doesn't end there. According to the computer list in my Windows Software Update Service, I'm now running Windows 6.3. Have I actually downgraded from Windows 7? Though WSUS does seem to deliver the Windows 8.1 patches and updates to the computer without complaining. And at least, when I look at the System Info page, it's comforting to see that Windows 8.1 thinks it is 8.1.
And it's also kind of nice to reminisce about the last time we had a "point" upgrade in Windows. Though I doubt many people will remember Windows 3.1 now. It's interesting that, in those days, the biggest problems you had were trying to get devices such as printers, disks, and network cards to work at all. Usually it involved fighting with lots of different drivers, cables, connectors, and configuration files.
Now everything hardware-wise just works, and the biggest problem is figuring out how to make the increasingly complicated software do what you want. Or even be able to tell what it is doing. My corporate laptop insisted I upgrade from Windows 8 to Window 8.1 this week (I'm not even the boss in my own office, never mind in my own house). It sat there with just a green box in one corner of the screen for a whole day with no sniff of a status bar or any indication it was actually doing anything. I had to go down to the garage and look at the lights on the router to see if there was any sign of life. Please can I have my animated network icon back in Windows 8.2?
Though I'm still a little nervous that, when I upgrade next time, it will just take me back to Windows 7 again. Or maybe I have to work my way through 6.4, 6.5, 6.6, etc. first...?
Agile development is an important technique here at p&p; and throughout much of Microsoft. However, I'm yet to be convinced that it's a good way of creating user guidance and documentation. It seems to me that the process often gets in the way more than it helps to produce a great final product.
I've rambled on many times in the past about agile documentation. Most specifically in Can Writers Dance The Agile? and other posts here. Yet I keep thinking it needs deeper investigation - especially as the group I'm officially assigned to, CSI, insists we keep prodding it to see if it works.
Note: Here at Microsoft, CSI is "Content Services & International". It's probably a bit less exciting than doing clever stuff with fingerprints and DNA, but we do have fancy computers that nearly come close to those you see in the TV versions of CSI. Maybe we should call ourselves "CSI Microsoft," wear white coats, and walk around with a flashlight held over our head.
Anyway, coming back to the point, we've had another go at agile docs recently. Instead of a solid and agreed structure plan and detailed implementation notes of what we wanted to achieve, we started off with a "vision" (but see this post from a couple of weeks ago) and "brainstorming" to get a list of topics. Then we "asked the audience" with developer and advisory board surveys to see which of the topics they liked most. Next, we threw together some rough notes about each topic and then produced a first draft of each topic document.
The next stage was multiple reviews. After each review we restructured the content, and often the whole document, to get it closer to the ideal. But, because it's agile, we often changed our mind about what the doc was trying to achieve (remember, there's no detailed implementation notes to guide us here) and completely rewrote it. And then did this again after the next review pointed out the holes left by (or introduced by) the previous reworking. Some documents went through four or more complete restructurings, and several were rewritten twice.
The agile process also resulted in some of the "envisioned" topics being abandoned, often after they'd been rewritten several times, because - after investigation - they were too hard to define accurately. Or because they turned out to be not really relevant or practical. And as there is no overall structure plan, it was hard to see which topic should contain what section of the content, and how it related to other sections. It also meant that the focus could only ever be at the individual document level rather than the entire guide level, because that isn't defined yet.
But what we could be sure of was that each individual topic was precise, accurate to the nth degree, and compact with no irrelevant content. This is, of course, extremely important; especially if the content will be used as reference information. But is it still "guidance?" I guess that's the core of the problem. What exactly should "guidance" look like?
Typical equivalents of the word "guidance" include the obvious ones such as "help" and "advice." However, there are broadly two categories of meaning: "leadership" and "assistance." These almost seem like opposites - one leading from the front and the other pushing from the back. Yet the sub-meanings according to my thesaurus include "direction", "support", "management", and "control." Some of these seem more like they are related to aiding through understanding, whereas others are more related to despotic regulation. I'm going to take a guess that we're aiming for the first of these.
So did we end up with what we wanted, and does it aid through understanding? It's not completely finished yet and it will be a while before we see any user feedback. And there's no doubt that the content will be extremely useful for the specific users and use cases it addresses. But it still seems like we missed opportunities. The agile process narrowed the focus and transformed the content based on individual reviews of segments, and forced additional depth of detail. It also removed a lot of the general "understanding" content because it already was familiar to the experienced reviewers. Most of all, it resulted in huge amounts of extra work writing and repeatedly updating (and then sometimes discarding) the content.
Without a predefined (if flexible) structure and an overall feel for how it will all fit together and appear to the readers, there is nothing to prevent this wandering. As a writer, I'm lost if I can't see the finished thing in the back of my mind. It's like driving through a city in your car while blindfolded, and navigating by reversing and choosing a new direction at random every time you hit something.
Maybe agile is good thing that can help to focus guidance more accurately. Or maybe it just allows the guidance to wander away from the original vision and risk irrelevance. If the original plan was to provide guidance around X and you end up with fabulous documentation of Y instead, did you do a good job?
It's a bit scary when you turn on your computer and it's different from when you left it the day before. I don't mean it's a different computer (though sometimes that would be nice), but that something changed while you were in the land of nod.
This happened to me last week. As I sat yawning and stretching in front of the screen waiting for some sign that the technology was also waking up from sleep, I noticed that several of the shortcuts to network resources that used to be on my Windows 7 desktop had disappeared. The first immediate panic reaction is "do I have a virus?" But a scan with a virus checker, and exploring the event logs, found nothing untoward.
Next, try refreshing the desktop. No change. Then the trick of turning off display of icons (right-click the desktop, select View, and untick Show desktop icons) and then back on again. No luck. Next, look in the C:\Users\[user]\Desktop folder to see if the shortcuts are on the disk but not shown. Nope, not there either. Very odd...
Maybe it's something to do with File and Print Sharing, or some domain-related issue. So I wander down to the server cabinet and start poking the domain controller. Strange - at 5:00 AM precisely that morning it had started reporting errors that there was a fault in the Active Directory. It happened immediately an automated backup started. And now the network icon in the taskbar was reporting that it was connected to an "unidentified network" rather than its own domain network.
So I do the usual fix for that problem, disable and re-enable the network connection. Immediately it comes back with the correct network name. Except that I now have to reconnect to all the Hyper-V instances it hosts because they lost their connection when I disabled the network. And then, every six minutes, an error in the event log that Group Policy could not be processed because it can't find a domain controller. Even though it is one. The detail of the error is simply "Directory Error." Not exactly helpful.
For a minute or so I ponder a full reboot, but that means stopping all the Hyper-V VMs. One is the proxy server, and my wife is currently in the middle of her morning Facebooking session so that's not a relationship-friendly option. The next best guess is to restart Active Directory Domain Services, which automatically stops and restarts several other important sounding services. There's heart-in-the-mouth moment when the DNS Service takes almost a minute to start up, but thankfully it all comes back with no errors in the event logs. And, magically, several hours later no more Group Policy errors either. Amazingly, I seemed to have fixed it.
After that rather exciting start to the day, I do the research bit and discover that the shortcut issue is not my fault. According to KB 978980 on MSDN, being greedy and getting kicked for it is by design:
"The System Maintenance troubleshooter performs a weekly maintenance of the operating system [and] either fixes problems automatically or reports problems through Action Center. When there are more than four broken shortcuts on the desktop, the System Maintenance troubleshooter automatically removes all broken shortcuts from the desktop."
Furthermore, it says, "a broken shortcut is a shortcut to a file, folder or drive that may not always be available, for example, [...] a network folder that is currently not available due to the network not being available". I'll take a guess that being on a different network from the one the shortcut points to qualifies as "not always available." The workaround suggestion in the article is "keep the number of broken shortcuts on your desktop to four or less [or] create a folder on your desktop and move the shortcuts to that folder [which] will not be removed since they don't sit directly on the desktop."
Or you can just disable the System Maintenance utility in Control Panel | Troubleshooting | Change settings.
I can't help wondering how the meeting went where the Windows 7 developer team decided to include this feature:
"Have you seen Joe's desktop recently, it's covered in shortcut icons. I'm sure he has no idea what they're all for!"
"Yep, soaking up valuable resources and hiding that lovely picture of Niagara Falls that we went to so much trouble to include in our themes!"
"It shouldn't be allowed. There should be someone who checks user's computers regularly to make sure they aren't being untidy after we went to all that effort to make the background pretty!"
"Yes, but they might want to have a few shortcuts there for their favorite programs and regularly used files..."
"I suppose so. How many do you reckon we should allow? Twenty? Ten?"
"Why mess about. Let's just include a secret process that detects when a user is getting a bit slovenly and tidy up automatically. They'll never notice..."
Except that, as the KB article reveals, sometimes we do...
In the olden days, people with a vision changed the world. Scientists such as James Clerk Maxwell kicked off the entire revolution in harnessing electricity and magnetism to build our modern world. Bardeen, Brattain, and Shockley started the silicon revolution that gave us the microchip, and Tim Berners-Lee gave us the World Wide Web. But, sometimes, you have to wonder if being a visionary is going out of fashion.
OK, so there are plenty of people still inventing technological things, but mostly its evolution now. Some people even say that we've discovered all there is to know about physics and our planet. And lack of vision seems all-pervading when it comes to things such as politics. Where are the visionary leaders (for safety and impartiality, no names mentioned) of the past? It's pretty much an accepted fact that politics these days is a case of "going with the flow." Focus groups to tell you what policies are likely to get the most votes, and sound bites to keep the population satisfied.
So what about in the world of computing, user documentation, and guidance that I and so many others inhabit? Is vision still alive and well? And is it really important? When did you last hear of a new computing device/service/product/accessory that was really new and ground-breaking?
Wearable computers? I had a digital watch with a calculator in it twenty years ago. User input devices? Touch screens and motion detection have been around for ages. Mobile phones? Do you remember the eighties and brick-sized boxes? Internet TV? Windows 7 Media Center had that as a Silverlight-based add-on, and it was hardly a new concept then. Facebook and Twitter? Just evolution of CompuServe and bulletin boards. Online shopping? See How Much Computing Power Do You Need? Quantum computing? OK, so this one is relatively recent - but it's really just about moving particles around instead of electrons because we need to do things smaller, faster, and in parallel. Something we've been doing with CPUs for many years.
Maybe we have reached the point where there is nothing really new and visionary left to be invented in the world of computing. Though there's probably more chance of actually having a vision in our industry, and implementing it, than there is in the world of politics...