- Change of Topic
-
Well, it's been a long time since September.
I've decided to learn from my inaction; while I work hard on perf, I'm still in the learning curve and I don't particularly enjoy blogging about what I don't know.
So, I'll change the subject.
Back about seven years ago, we released a control called the DHTML Editing Control; dhtmled.ocx. It was a simple ActiveX control hosting Trident, the IE rendering engine, plus TriEdit, which implemented most of the designer features released in Visual InterDev (remember VID?) such as Table Editing and Source Code Preservation. There's some interesting history and a few good stories associated with the life of this component which I'll post under this topic.
We recently came to realize that we are not in touch with the community of our users. I'm interested in rectifying that. Let's begin.
- Fix vs Change
-
It has been a painful week for checkins.
We have a complicated system for checking in our code. There are hundreds of devs in our division and tens of thousands of source files in our product. Different groups have different cultures; one may have a "death before dishonor" ethic in which they wouldn't tolerate a member of their group checking in a change which would break the daily build, others may have more of a "let's try this..." attitude. Thus, we have policies for code reviews, verification, and checkin.
This week we lost some hardware used for buddy testing and a couple of obscure bugs got checked in which didn't break the suites on one machine but did on another, making the normally tedious task unusually painful. And so the suggestion arose to change our process to improve efficiency. A noble ambition, but with its drawbacks.
How do you go about improving a system's efficiency when you don't have any baseline data, or any tests to determine the effects of your changes? We have several anecdotes in circulation this week, but what use is that for evaluation? Worse, it seems to me that the metrics that we'd need would be painful to collect. If we wanted to compare time spent coding to time spent checking in it would require substantial effort on each dev's part. I keep three computers busy most of the time. Seven, occasionally. What part of that is actual checkin time, and who wants management scrutinizing your hours spent coding if you're not already so blessed. (This is a valuable metric by itself, but streamlining the checkin process does not justify the cultural changes it takes to acquire this kind of info.)
It occurred to me that sampling might be helpful, but still involves near spyware on each dev's computer. While trust runs pretty high here, these kinds of systems are sensitive to the slightest abuse, or rumor of an abuse. Non-relative information would be simpler to collect, but it's still difficult to measure the actual amount of human engagement in a process which might consist of a significant period of sync'ing and building, then a script running a batch of suites, and then investigating and correcting build or suite failures. As the build and run phases are time consuming, devs tend to fork off and check back before and after lunch; raw numbers are useless. True, there could be buttons to press indicating "I've begun dealing with non-routine problems" and "I'm returning to doing routine tasks" but I'd imagine such buttons would get pressed very little. The benefits and burdens don't match up, and I've found that such systems are no more reliable than self-selected survey samples.
It's important to keep the infrastructure running efficiently, and increasing the productivity and reducing the level of frustration of the team is a wonderful goal. At the moment, however, I'm stumped when it comes to collecting useful metrics.
- Code Complete
-
After an unexpectedly long process, I'm finally code complete on the Beta 2 VWD performance suites. We identified our performance needs a couple of months ago, set our goals early in August, and now we've finally got the infrastructure in place to monitor these test cases from here to Beta 2 release, and on to RTM.
Monday I'll be busy getting the 486 files involved code reviewed (twice) and checked in. I'm glad to have time to take a break first.
- Suite Frameworks
-
It's been a busy week of writing test cases.
We have so many test suite frameworks I can't believe it. Our checkin suites are written against one framework (but other teams use other frameworks.) Our QA tests use another framework, and the perf test cases I'm writing use yet another framework. None are complete or robust; if you use a new function of feature of an existing method you better check the source code for the infamous "TBD." For example, I needed to open a local host web instead of a file system web. The function existed in the header file, but not the .cpp file. I fleshed it out, but another function which pushed the appropriate button for local host as opposed to file system button was implemented only as TBD.
My last test case involves modifying an aspx file and viewing it in the browser. How do you terminate the test? We usually use CodeMarkers, which are simply window messages with a few tests to see if they're enabled or not, to terminate timings. The alternative was to poll and search, so I decided to write an ActiveX control which fires a CodeMarker when rendered. But it turns out that XPSP2 may disable displaying ActiveX controls! You have to click on an "action bar" to enable the control to be loaded. But I've found a way to save the registry setting, change it, and restore it after the test, as long as the test is running with admin privileges.
I'm hoping to finish up five test cases tomorrow, after six weeks of committee meetings defining the test cases, experimentation to set performance goals, and wresting with the test framework and security precautions to implement the tests. Party time!
- An unexpected side effect of storing short directory paths
-
This is a bit off the topic, too, but I'm running a time consuming process on two other machines right now...
I run a web server in my basement on DSL. Over the weekend I decided to finally upgrade the 80 GB hard disk that had been spinning practically non-stop for 18 months with a shiny new 160 GB disk. Seagate provides a copy utility that runs under windows, so I rebooted in Safe Mode (assuring that virtually all my services were shut down), ran the copy, shut down, pulled the old drive, strapped the new one as a master, and rebooted.
Everything came right up. Except for SQL Server. My website is a derivative of I Buy Spy, so I basically had nothing showing. I tried to manually start the SQL Server service and was ever so pleased to see the error message "Cannot start service. Error 3: File not found." Just which file is left as an exercise for the reader. My Event Log displayed the exact same error.
I searched Google for "SQL Server" and "File Not Found" and got plenty of hits, but no help. I opened the Service manage, stared at the properties of the SQL Service, and noted the name of the exe it launched. I thought if I ran it manually I might get more info. I located it, double clicked it, and it opened a command window and announced that it was opening files, starting this and that, and trying to log in as the disabled "guest" account. It seemed to run fine. Hmmm.
Then I looked more closely at the path to the exe. It pointed to binn in the Microsoft SQL Server directory in Program Files, but the pat used the short versions; .../PROGRA~1/MICROS~2/... An inspiration struck. I used "dir /ad /x" to look at the short names of my program file directories, and indeed, Microsoft SQL Server's was different!
I used RegEdit to locate and change the nasty string, and SQL Server came right up! A happy ending at last. And the moral of this story is: don't store paths in short name format. All your testing may not uncover a single problem, but you can create some hideous problems for your users.
- Code Folder References
-
I got email that I got feedback from Karl, but I don't see where it ever showed up on the Blog. He says:
Ok, this is a little off topic, but how the heck do you reference things in the Code folder such as HttpModules, HttpHandlers and ConfigurationHandlers? You don't know what the assembly name is...I was told you could use the actual word "Code"..but I think that was an alpha thing, and I don't think it works in the beta.
<httpModules>
<add name="Test" type="TestModule,ASSEMBLYNAMEHERE"/>
</httpModules>
I was concerned about exactly the same thing last week, and I was told that "Code" would do, or the form <add name="Test" type="TestModule"/>. In the particular case I was interested in, there wasn't any indication that either of these were working. With bits just a little earlier than I'm using, the assembly name "__code" would work (that is, in fact, what I'm trying to replace in a testbed web) as the name of the assembly generated from the Code folder was __code.dll. This is definately going away, however.
I'll run this by the appropriate devs and add my findings here later.
There are a lot of devs on vacation today, but one of our migration testers, Rachel, replies:
I know that for beta 1, migration needed to use __code for assemblies in the HttpHandlers and HttpModules sections like this:
<httpModules>
<add name="CommunitiesModule" type="ASPNET.StarterKit.Communities.CommunitiesModule,__code"/>
</httpModules>
Note that there are two underscores before "code."
- Cardiac Performance Testing
-
I intended to post yesterday, but just before lunch my chest started hurting. Then my left shoulder. Then my left arm. Hmmm...
There's a hospital right around the corner, so I asked my boss, Brad Bartz, to drive me over so I could have it checked out. I thought it would take 30 minutes, but it took 24 hours. My description was pretty convincing, and it turns out the enzyme test needs to be taken about six hours after onset. That put it just about the time that the treadmill test was no longer available for the day. So they preferred that I waited around until the next day when I could take the test perhaps as early as 8:00 AM.
It turned out to be 11:00 before I got on the treadmill. That was an interesting setup; the computer was running an app which was pretty clearly written in Visual Basic. I thought back to bug triages I'd been through in the VB group and crossed my fingers. All worked well, however, including my cardio-vascular system. No heart attack. A day well wasted. At least (for some reason) I got a lot of sleep, 12 hours or more. There wasn't much else to do but read, and I was a little too distracted to read much.
The treadmill test was a great example of performance testing. First they get a baseline, then ramp up usage gradually and do blood pressure tests periodically as well as continuous EKG as you walk and eventually run. Then they monitor ramp-down, which was the only big difference I saw between testing software and physiological performance. Maybe something from the experience will suggest an idea in future performance tests design. I can't think of other non-software perf tests I've witnessed. I've participated in system automation testing in high-rise buildings and hospitals, but this is strictly functional testing and has nothing to do with performance. I've participated in emergency communications exercises with ARES (Amateur Radio Emergency Service) and these are sensitive to throughput, but are a little different; we spend a lot more time analyzing the quality of the throughput than the quantity; were the messages accurate and intelligible? Are the procedures sound? Does the protocol work as expected? Etc...
I know there must be other types of performance tests out there. If anyone knows of some interesting examples, please post a comment.
- Other things that have gotten slower by design
-
Another feature of Visual Web Developer which is certain to seem slower than VS 2002 or 2003 is the opening of "miscellaneous files." These are files which (should) sport Intellisense but are not part of a web or a project. In earlier versions of VS these simply didn't have Intellisense hooked up. So, for example, if you typed "Response.", nothing would happen.
VWD 2005 creates a hidden, "miscellaneous" project to wrap around the miscellaneous file. It searches upward, looking for an actual web the file may be a part of, looking for a /Code folder, web.config file, or other clues (I can dig these up and enumerate them if anyone's vitally interested) and figures out where to "root" the web. If a web is discovered and it's not from a previous version of VS, the project system does what it would for a normal web; /Code Intellisense is generated, global.asax is honored, everything.
If you open two miscellaneous files, you may get one or two miscellaneous projects. If you opened c:\foo\defalut.aspx and c:\foo\info.aspx they'd both share the same hidden project. If you opened c:\foo\aspx and c:\bar\data.aspx there'd be two different miscellaneous projects. Most of the spin-up is asynchronous but the memory footprint of the Intellisense and type resolution services is noticeable.
Has anyone out there run into problems with this? Is double-clicking on a file on the desktop to make a tweak too slow? Do you get unexpected Intellisense (or none at all) with these types of files sometimes? Does the Intellisense make you happy to leave notepad behind at last for this type of operation, or does the startup time make you right-click and "Open With..."?
- What's Different About the VWD Build System?
-
There have already been several posts about build being slower in VWD than in Visual Studio 2003. This is true.
With Visual Studio 7.0 and 2003 there was a huge gulf between the SDK version of examples, starter kits, etc., and the Visual Studio versions. The organization of the source code differed mostly in single-file SDK versions and code-behind VS versions. But there was an even more substantial difference in deployed webs. SDK versions seemed to simply and magically compile themselves, while the VS versions needed to be built and prop'd to the website. One big assembly was generated in the /bin directory. The SDK versions could be modified right on the web server using only notepad; the VS versions had to be modified in the VS Solution, built, and prop'd. Strangest of all, you could make some changes in the running VS webs using notepad, like modifying some text on an aspx page, and it would show up the next time you loaded the page in your browser, but other changes (like modifying the code) simply wouldn't "take."
This really didn't delight everybody. Modifying a sample meant fixing the C# & VB version for both the SDK & VS versions. It was difficult to exchange code. It was confusing. It looked broken. And having to resurrect a project to make a tweak and replace every executable bit to implement a one-line change was simply not a decent model for many kinds of web development.
Early in 8.0 we decided to rip the extensive build system out of the web project system, and instead rely on the managed Client Build Manager, an ASP.Net class wrapping the Build Manager, the ASP.Net piece that compiled the SDK-style webs. The advantages seemed pretty significant: excellent fidelity between development and deployment code generation, any-language support, no-solution-file web projects (my favorite) and simplicity. What the build system used to do was to spin up its own compilers and keep most code built to the state just before writing the actual assembly, so doing a build actually meant just emitting the assembly file.
Now we rely on the Build Manager itself, which isn't really aware that Visual Studio is even running. We've still got our own compilers building the symbol tables used by Intellisense, but they are not used to emit the executable code. And thus we incur some substantial costs in latency and working set.
Was it a good decision? Only time and your comments will tell. The new web project model is a delight to work with; all you have to do is select a folder and it behaves like a project; no files are added to it at all. Could we have accomplished this without duplicating compilation costs? Will version independence be a substantial win going forward? Will the cumulative compilation hit make old and bitter people out of us all? I'm interested in your feedback.
Here's a hint. For "consistency" we currently do a full build internally when you hit F5 before we request the target page, which will trigger a Build Manager build. If the delay is bothering you, try this:
- Right click on your project node in the Solution Explorer.
- Select Preferences.
- Select the Build tab.
- Uncheck "Build web site as part of solution."
Do this per web; there's no global setting.
If this makes a noticeable difference in your day-to-day experience, let me know.
Do you feel that this setting should be off by default?
- Digital Diabetes
-
I've posted an article on a metaphor for working set bloat; Digital Diabetes.
It's not intended to be dark humor; the point is that methods for successfully managing diabetes may be transferable to managing applications with out-of-control working sets as well. I don't intend to offend anyone; I'm diabetic myself.
- Visual Web Developer (Whidbey) Performance
-
Hello! I've been a dev at Microsoft for eight years now, and have been working on Visual Studio for the last five or six of them.
Recently I've changed roles. I've always worked on the web development part of Visual Studio (from Visual InterDev through Visual Web Developer) but I'm moving off the feature area and into performance.
Until recently this responsibility was spread over four people on our team, and it was one of many responsibilities for each of them. Now virtually all the performance testing, monitoring and investigation are my responsibility. I'm new at this; I've had a few narrow performance responsibilities and a little training in the past, but I've got a huge amount to learn and a thousand tasks to complete. It's like heaven. It's hard to beat getting paid to learn new skills, particularly interesting ones. My old friend John Pence and I used to say "Always get paid for learning." It was the company slogan of MacMan, Inc.
Let me clarify my role: my responsibilities are specific to Visual Web Developer, the development environment. If you've downloaded VWD Express then the Project System (represented by the Solution Explorer window and a vast expanse of invisible plumbing behind it) and the ASPX document windows cover a lot of the area I'm primarily concerned with, along with a few additional areas which are harder to visualize; the Cassini-based web server which starts up when you press F5, the Build command, Publishing, Copy Web, the acts of opening and closing web sites, and basically any function which pertains to the website as a whole or the web pages within that site. Intellisense hookup, too, although the Intellisense itself is implemented by language teams. The Shell is the framework of Visual Studio that’s wrapped around our package; it’s the part that’s the same for C# Express and VWD Express.
I'm not involved with performance of the Shell, the ASP.Net runtime, the .Net class library, C#, VB, or SQL, but I am getting to know most of these people, so if you have comments on any of these areas I might be able to pass them on but probably won't be able to follow up on them unless they're major issues which impact the entire product. If you're a Visual Web Developer beta user, I'd love to hear about your performance-related experiences with the beta.
For you precious beta testers (I am one of you) I've got some questions. Please let me know:
- Have you used other Visual Studio products for web development before? VID, VS7, VS7.1? How does VWD stack up performance-wise?
- How much RAM are you using? Are you memory bound? What do you think the minimum RAM requirement should be?
- Have you run into limits where performance breaks down? Say, one or two web forms are OK, but once you hit 317...
- Are there spots where you reach for the reset button, thinking the system is hung? Where are those spots?
- Most important (if you have one of these please drop what you're doing and post a reply) have you ever hit a painfully long pause right in the middle of a thought, that is, have you lost your train of thought while your hard disk flashes and you CPU moans? Where was that point? What do I need to know to reproduce that experience?
If you've got performance complaints with Visual Web Developer, you've found the right guy to notify. I can't guarantee I can solve every problem of course, but I can get your problem areas on the map. I can make sure test cases cover them. I can assure that fixing them or shipping them becomes a conscious decision. I know it's not always easy to tell, but these decisions aren't made lightly. If I don't hear from our users, my focus will be directed solely by my best guesses and suggestions of our team and division. Your input could make a big difference in the product we ship.
