16.7ms in the life : Coding

Thoughts on software complexity

Peter Rosser — Sat, 03 Jun 2006 07:06:00 GMT

If you've ever tried to create a software application of any decent size, you have probably realized that what works with small tools more often than not does not scale well to larger sizes. There are whole books written about how to scale up and out, new languages and technologies invented to help manage it, and millions of dollars in educational budgets spent every year to achieve that elusive goal.

And yet, the landscape is still rife with examples of poorly designed software.

A significant part of the problem is not that engineers are not smart enough--it's that we outsmart ourselves. The conventional wisdom is that humans can store 7 ± 2 primitives (e.g. numbers) on the top stack of short-term memory. The number skews higher for strongly associative items, and vice versa. We can store fewer abstract items, often only 2 or 3 at a time. The issue is not one of comprehension, it's one of focused retension. When the number of data items in a class exceeds our capacity to retain them at once, the brain tends to move the perceived "less important" fields/properties into peripheral storage. This, I think, leads to not fully comprehending the interactions in the class. We engineers like to think of ourselves as intelligent, and that we can understand very complex ideas. This may be true, but it misses the point; the point of software engineering is to create reliable, performant, maintainable software. It is not to create complex applications.

Well, there's the problem, isn't it? In order to create the really cool applications, you must have complexity. It's simply unavoidable, especially given the advanced competition for consumer dollars. You can't very well create a next generation RPG that has realistic physics, with a "real" economy, kickass graphics and brutally smart AI without complexity.

Or can you?

Actually, I believe the answer is that you can, at least the bad kind of complexity that causes the pain. The key is differentiating between behavioral complexity and system complexity. Behavioral complexity is the aforementioned features like smart AI, or realistic physics. Those sorts of things are monstrously difficult to get right, but that's not what holds back the very large application development projects. The major cause for schedule slips, regressions, out-and-out bugs, and general angst is system complexity.

Before I go on, I'd like to define what I mean (yes, I do get to hijack words like this) by system complexity:

System complexity a property of a system that is directly proportional to the difficulty one has in comprehending the system at the level and detail necessary to make changes to the system without introducing instability or functional regressions.

In other words, the higher the system complexity, the harder it is to upgrade, maintain, and develop the system. Whether the system is a single class, an object library, an inheritance heirarchy, or an entire application, someone "schooled in the art" of software engineering should be able to understand it, given a reasonable time to examine the system. This ability should scale through the system. When one examines a particular object, it should be apparent without examining the private implementation details how to use it. From top to bottom, from bottom to top; the entire system should be accessible to the non-super-geniuses out there. What goes for good UI design also goes for good C++, C# and COM design: your target audience should be able to use it without extensive systems training.

System Complexity is what makes the Mythical Man Month true. It's what keeps that guy who wrote the accounting system from scratch in 1986 employed. It's what causes games, productivity applications, and even (some might say especially) large operating system projects to slip their schedules.

So what can you do about it? The first step to recovery is admitting you have a problem. Pedantic, but true. Getting back to the point I tried to make earlier about engineers--we like to think we are smart. Egos loom large in the software industry, and often with good reason. One of the smartest developers I have come across worked on a critical piece of the code for several years, and guarded his territory jealously. Make no mistake, this guy was brilliant--and he knew it. To make a long story short, when he left, the code he left behind was an engineering marvel [he says wryly] that no one else understood. Oh, we know what it does, but there's a crucial difference in understanding what something does, and grokking fully how it does it. Tracing execution paths through this code is difficult, and 1200-line methods named "Run" are common.

The author understood it all, or said he did, but my opinion is that he was fooling himself, too. Once you get a piece of the system working, after all, you don't need to touch that piece again unless the design changes, a bug is found, or something like that. Over time, though, people started to notice that even small fixes in this particular area would touch a dozen or more files, and churn lots of code. That's a symptom of high system complexity; he would go in to make the fix, then discover that 100 variables influenced the behavior, or the implementation in one area was strongly dependent on the implementation in another, or most often, both. But since the author had traveled this area so much, he was able to turn around the fixes in an acceptable timeframe, so he "got away" with it.

It's no surprise that when the author decided to move on to other things, as employees often do, that we had a tough decision to make. Do we maintain this code that's working? Do we refactor? In the end it came down to pragmatism and schedule: the problem was recognized, and we chose to refactor the system in measured stages, minimizing the risk to the product and getting back to a good, maintainable state.

That's all I have time for today... my build is done, so back to work I go. Windows Vista waits for no man!

Using exported DLL functions

Peter Rosser — Thu, 23 Feb 2006 01:11:00 GMT

Now and then it's necessary to use "private" or "internal-only" functions in DLLs that you did not write, don't have the source for, and/or cannot get public interfaces for. I recently had to do this again for a utility application, so I thought it would be nice to document the process a little more clearly for others.

A starting assumption is that the reader is familiar with GetProcAddress() in its "main" form, which is when you know the full signature of the function you need to call. For example, if you have a copy of foo.dll that exports Bar(LPWSTR, DWORD*):

// library function is: Bar(LPWSTR lpParam, DWORD *pdwParam)
typedef void (__stdcall *BAR_PROC)(LPWSTR, DWORD*);

HANDLE hFoo = LoadLibrary("foo.dll");
BAR_PROC proc = (BAR_PROC)GetProcAddress(hFoo, "Bar");

// call the library function
DWORD dwParam = 0x4;
(proc)(L"string", &dwParam);

But what if you need to access a function that is not included in the named exports of the library? 90% of the time the functions will be exported with a name, but sometimes not, and for various reasons. The most common reason is for "security through obscurity", because using the function is supposed to be something that you, the user, "should never have to do". Unfortunately, there are scenarios when you need to do just that, for reasons the original publisher never considered, such as if you need to tweak the way something behaves in your mission-critical server process, and the company that made the original custom component has gone out of business.

Enter the "ordinal" function access method. If you look at the MSDN docs, they even allude to using a function's ordinal with GetProcAddress, but it's a bit of a mystery how to a) get the ordinal, and b) how to use it even if you get it. Once you know the ordinal and prototype of a function, it's simple to use it--getting the function prototypes is the fun part.

Say you got your hands on a .DEF file that lists the function ordinals, or have used link.exe and through sufficiently wizardly use of a hex editor have examined the [NONAME] functions and determined what their prototypes are. Let's pretend that the Bar function above was actually a [NONAME] function, and you got a listing like this from link.exe /dump /exports foo.dll:

Microsoft (R) COFF/PE Dumper Version 7.10.4035 
Copyright (C) Microsoft Corporation. All rights reserved. 


Dump of file c:\myprojects\foo.dll 

File Type: DLL

  Section contains the following exports for foo.dll

    00000000 characteristics 
    41107692 time date stamp Tue Aug 03 22:39:30 2005 
        0.00 version 
           1 ordinal base 
           6 number of functions 
           4 number of names 

    ordinal hint RVA      name 
          3    0 00006948 Connect 
          4    1 00007798 Close 
          5    2 00012000 GetFoo 
          6    3 00014D12 SetFoo 
          1      0000F3A0 [NONAME] 
          2      0000D20A [NONAME]

  Summary

        6000 .data
        2000 .reloc
        9000 .rsrc
       18000 .text

The depends.exe tool provides similar data in a GUI fashion. You can examine the binary at offsets F3A0 and D20A to determine what the function signature is (well, what the base pointer sig is), and match them up to the expected signature of Bar(LPWSTR, DWORD*) manually to get the proper ordinal. Or more easily, if you know that one of the methods is the droid you are looking for, you can just code up a quick test app and poke at the method as if it were the one you wanted. If it works without throwing an exception... ;-)

Getting back to the main point, to load the exported-by-ordinal function, there's a handy macro pre-defined in the Visual Studio (v6 or later, IIRC) build environment that can package up that ordinal into the "expected" LPSTR for GetProcAddress. In my example, assuming Bar was actually at ordinal 1, you would do this:

// library function is: Bar(LPWSTR lpParam, DWORD *pdwParam)
typedef void (__stdcall *BAR_PROC)(LPWSTR, DWORD*);

HANDLE hFoo = LoadLibrary("foo.dll");
BAR_PROC proc = (BAR_PROC)GetProcAddress(hFoo, MAKEINTRESOURCEA(1));

// call the library function
DWORD dwParam = 0x4;
(proc)(L"string", &dwParam);

The only difference is that you use the MAKEINTRESOURCEA macro to convert that ordinal into something that GetProcAddress can understand, rather than the string name of the function.

If you really want to get tricky, you don't actually need GetProcAddress at all to do this magic. If you can guarantee that the DLL won't change underneath you, or you don't mind breaking if it does, or you have smart scanning code that can "find" the offsets again, you can plug the RVA directly into the FARPROC variable. It's all just numbers under the hood, after all. Using the example output of link.exe from above, you can do this:

// library function is: Bar(LPWSTR lpParam, DWORD *pdwParam)
typedef void (__stdcall *BAR_PROC)(LPWSTR, DWORD*);
#define PROC_OFFSET 0x0000F3A0

HANDLE hFoo = LoadLibrary("foo.dll");
BAR_PROC proc = (BAR_PROC)(hFoo + PROC_OFFSET);

// call the library function
DWORD dwParam = 0x4;
(proc)(L"string", &dwParam);

Why does this work? LoadLibrary(dll) loads the specified library into the process' address space, and returns a HANDLE value. This HANDLE is the starting address of the library, and since the RVA values for a DLL are offsets from the starting address, you can perform simple addition to the base to get a function's entry point. Pretty cool, huh? All GetProcAddress does is read the DLL's export table into a struct and performs the math for you, returning the result. I'm a big fan of not reinventing the wheel, so I use GetProcAddress when there's an exports entry for what I need.

What do you do if the function you need is not exported at all? Nine times out of ten it's much easier to just get it exported, but if that's not possible, well, that's a topic for another day.

DVB Radio "No TV Signal" Workaround

Peter Rosser — Mon, 23 Jan 2006 07:50:00 GMT

I am currently investigating a fix for two DVB Radio issues identified a workaround for those affected. If you are getting a "No TV Signal" blue overlay when tuning to a DVB Radio station, particularly a low-bitrate station like BBC World Service, or you are seeing 15+ second channel change times for changing between DVB Radio and DVB Video channels, this workaround may apply to you.

This involves editing your registry, so if you are not comfortable doing that, please wait for the binary fix to be released.

The key is:

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Media Center\Service\Video\Tuners\DVR

Set NearUnderflowCheckMillis=4000 (0x00000fa0).

If you are not experiencing problems with DVB Radio, do not change this value. If you are in the U.S., for example, it can cause instability with ATSC tuning. Once the binary fix is released, you should return this value to its original value of 2000 (decimal).

This is a non-supported, non-official workaround and is provided AS-IS, with no warranty.

Windows Insomnia: Why Won't My Computer Go To Sleep?

Peter Rosser — Thu, 05 Jan 2006 08:18:00 GMT

One of the issues I am investigating for a fix relates to the auto-suspend feature in Windows. The way it's supposed to work is you set a time in the Power control panel (Control Panel --> Performance and Maintenance --> Power Options) that controls how long the computer waits until it goes into "standby" or "suspend". Whether that is S2 or S3 depends on your BIOS settings. What's going on with the bug I'm looking at is sometimes Windows will not suspend when it should, or indeed, ever, unless you close Media Center or manually select Standby.

This is pretty annoying, especially since my Media Center consumes about 200W while fully active (250-275W if engaged in HDTV playback), and only about 15W in S3 suspend. With electricity around $0.084 / kWh, that works out to $0.40 each day left unchecked. If I let it go all month, that's another $12, just because of this stupid bug (well, and because I'm lazy and forget to manually suspend... but I should not have to). Lame, lame, lame. Extrapolating to all of the customers affected by this issue, I can see it adding up to a big problem. Say only 250,000 users hit this (a low figure, I bet), that's $3,000,000 per month in wasted energy costs, based on the rates I pay. Yikes!

While I can fix this problem, I cannot fix all of the things that prevent computers from going into standby automatically, because they should really not be fixed, or because Windows software is not causing the problem in the first place. Several things can cause a computer to fail to go into standby, and figuring out exactly what is causing the problem can be very difficult.

So, what are the most common causes of Windows insomnia? In rough order of likelihood, here is my SWAG:

There is user activity. This usually means keyboard or mouse activity, but can be "simulated" by applications via SendKeys(), etc.
Windows is not configured to go into Standby automatically. Since it's not the default for most computers, this makes the list. Easy to fix.
An application tells Windows not to go into Standby automatically. Different from user activity simulation because this is what applications are supposed to do to prevent Standby. [ SetThreadExecutionState(state) ]
An application generates enough system activity (e.g. disk access, network access, CPU usage) that Windows does not consider the machine to be "idle" enough.
A hardware device driver prevents it.

Case 1 is very unlikely, since really the only applications that do this sort of thing are hacks/trainers/evil programs (and testing programs, if you want a legitimate example). Unless your computer is infected by malware, this is probably not the cause. Run a quality scanner to tell. For case 2, just set Control Panel setting.

For cases 3 and 4, troubleshooting gets trickier. For the average end-user who does not know how to use a Kernel Debugger like WinDbg or SoftICE, and does not have easy access to kernel symbols, figuring out who is keeping the computer up all night takes a little wanton process termination. Before you open up taskmgr and start gunning processes, though, there are a couple of things you can check:

Are you sharing any folders on the network? If a remote computer is connected to your machine and refreshes the connection (i.e. sends a command), the Server service will call KeInhibitSystemShutdown(), which does exactly what it sounds like it does: resets the system's idle timer. How do you tell if this is happening? An easy way is to pull up a command window (Start --> Run --> cmd) and type:

nbtstat -s

If you see any "IN" connections, they can keep you awake. Most well-behaved Windows applications will not needlessly refresh connections over the network, but others don't play that nicely. A ubiquitous example would be WMP or iTunes--both will scan network folders for changes if you add the path to your "watched folders" list. The best way to solve this problem is to fix the client side (the other computer), but if that's not possible, you can just deny access to that computer/user. In the worst case, you can stop the Server service (net stop server), but that prevents sharing altogether.
Check your Application and System event logs for errors or warnings. Often, exceptional situations will be reported there and can provide a clue to why Windows is not going into Standby.
Hardware (or more appropriately, drivers for hardware) can prevent Windows from entering standby, too. If you used to be able to go into standby, but cannot now, have you added or replaced hardware, or updated drivers? This can be tricky, too, since most people set their Standby timers for longer than is convenient to sit around and wait on, so failure to go into Standby can be missed for days or weeks.
To eliminate any remote networking cause, isolate your computer from the network (unplug the network cable!). Low-tech, but effective.

If you have eliminated network shares, suspicious event log entries, and hardware from causing the problem, the next thing to do is start killing processes and see which one (or if you're unlucky, which ones) are doing it. You need to first exclude those processes that are part of Windows itself, since terminating them is not very wise. An easy way to get a fairly complete list of the Windows processes is to boot into Safe Mode, press Start --> Run, and type:

tasklist >"%SystemDrive%\SysProcesses.txt"

A new file will be created on your system drive (usually C:) that contains the list of processes you shouldn't touch. Reboot back into normal Windows mode, log in, and:

Open the SysProcesses.txt file
Start taskmgr (right click on the taskbar and select Task Manager) and click the Processes tab.
Set your Standby timer to 1 minute (in the Power Options control panel)
Let 1 minute elapse. If the computer does not suspend, proceed to the next step.
Select a process not listed in the SysProcesses.txt file in taskmgr and kill it.
Go to step 4.

If you run out of processes to kill and the computer still won't go into Standby, it's time to start shutting down services in the Services console (Start --> Run --> services.msc). If you have stopped all of the services you possibly can, it's either a piece of hardware or one of the core Windows processes, and things are a bit tougher. To troubleshoot those, you need the help of a kernel-mode debugger.

Visual Studio 2005 Shipping on my Birthday

Peter Rosser — Tue, 01 Nov 2005 08:14:00 GMT

You know I'm a geek when something like that gives me even a little bit of a thrill. November 7th... not just an election day or my birthday, but also the VS2005 public ship day. ^_^. I'm installing it now on my primary and backup dev boxes at work now... sometimes the perks of working here are pretty nice. Mmm... generics.

One of my coworkers suggested I take another look at Managed C++ (during our conversation I expressed my "dismay" at the prior version's foibles), so I'll take a peek at that as soon as I have a spare cycle or two. It's pretty hard to get any free time with the painting at home and the heavy influx of QFE issues we always get post release, but I'm sure I'll manage.

Getting a QWORD from the registry using C# and P/Invoke

Peter Rosser — Fri, 21 Oct 2005 07:43:00 GMT

From time to time you need to use really, really big numbers, like when describing, say, the time of day you need to download a certain set of data from remote servers. Well, you probably don't need that much granularity in the time, but that's what MCE uses to schedule EPG downloads. <g>

So, we store this thing off in the registry at in a value named dlRegTime. I can see why the dev did this: it's easy to convert a DateTime to an Int64 by way of DateTime.ToFileTime(), which makes for a clean, unambiguous value to write. I'd almost bet that he was patting himself on the back... until he had to actually write the value to the registry.

The problem comes when you need that value from C# (or, I guess, VB.NET). The 1.1 Framework designers apparently did not think that anyone could ever possibly need a 64-bit value from the registry, and if they did, they could P/Invoke. That's exactly what you have to do. Here's an example where I convert the FILETIME stored as a QWORD into a DateTime:

[DllImport("Advapi32.dll")]
static extern uint RegOpenKeyEx(UIntPtr hKey,
    string lpSubKey,
    uint ulOptions,
    int samDesired,
    out IntPtr phkResult);

[DllImport("Advapi32.dll", EntryPoint="RegQueryValueEx")]
static extern uint RegQueryValueEx_QWORD(
    IntPtr hKey,
    string lpValueName,
    uint lpReserved,
    ref uint lpType,
    ref long lpData,
    ref int lpcbData
    );

[DllImport("Advapi32.dll")]
static extern uint RegCloseKey(IntPtr hKey);

const int KEY_QUERY_VALUE = 0x1;
const string EPG_REGKEY = 
    @"SOFTWARE\Microsoft\Windows\CurrentVersion\Media Center\Service\EPG";
const string DLTIME_REGVALUE = "dlRegTime";

DateTime GetNextDownloadTime()
{
    UIntPtr HKEY_LOCAL_MACHINE = (UIntPtr)0x80000002;
    IntPtr hkey = IntPtr.Zero;
    DateTime date = DateTime.MinValue;

    try
    {
        uint lResult = RegOpenKeyEx(
            HKEY_LOCAL_MACHINE, 
            EPG_REGKEY,
            0, KEY_QUERY_VALUE, out hkey);

        if (0 == lResult)
        {
            long qword = 0;
            uint uType = 0;
            int cbData = 8;
            
            lResult = RegQueryValueEx_QWORD(hkey, 
                                            "dlRegTime", 
                                            0, 
                                            ref uType, 
                                            ref qword, 
                                            ref cbData);

            date = DateTime.FromFileTime(qword);
        }
        else
        {
            throw new ApplicationException("The registry key " + 
                EPG_REGKEY + 
                " could not be opened for reading.  The error code was 0x" + 
                lResult.ToString("x"));
        }
    }
    finally
    {
        if (IntPtr.Zero != hkey)
        {
            RegCloseKey(hkey);
        }
    }

    return date;
}