Mac Mojo : Testing

Longest remote home folder test … 600 miles…

ExCntx — Fri, 02 Nov 2007 11:13:00 GMT

Sorry it’s been so long since my last post, but I’ve been quite busy fixing bugs which are either old, new, borrowed and quite far away. In fact some are so far away it takes driving all the way to Burbank, CA to verify the fix.

I’ve spent a good number of weeks working on fixing a variety of Remote Home Folder bugs. If you are not sure what I am talking about you can read about it here: Apple Server Desktop Management. The basic idea is that all user directories live on a remote server. This allows anyone to log into any Apple Macintosh connected to the business or university network and have access to their home directory. The idea has been around since UNIX was developed. Unfortunately, when the age and design of your application pre-dates UNIX … okay, not really, but when the application wasn’t designed for this functionality you are guaranteed to run into a few problems along the way.

So why send me? Because I worked on the fix, silly, why else would you send such a shy and demure sort of guy who really doesn’t like people. (If you believe that I’ve got this bridge in Brooklyn that I just put on the market.) Seriously though, sending me all the way to Burbank to test a fix, and make me drive there to boot? What’s up with that?

The reason I went was because one of the known issues for enterprise and schools was the performance impact of running the Office applications when a user had a Remote Home Folder (RHF). Having spent my time working on emulators, video games and lots of other stuff that I can’t talk about, performance is my middle name. I was tasked, as well as a few other people with focusing on Remote Home Folder issues for this next version and getting things put right.

Reproducing the RHF bugs in our lab proved quite the challenge. The configurations we were trying to test with either differed in network bandwidth, network hardware, Mac OS software versions or even computer hardware versions. On top of that, some customers had a few users or 10,000 users. Trying to narrow down the configuration so we could reproduce the problem involved a great deal of creativity on our part. So when we have a customer close enough to our team that can try out our fixes it’s a great benefit to us and them!

So, back to the bug. Everyone out there that remembers Get1Resource and PBGetCatInfo, go get yourself a cookie … on me. Everyone who doesn’t know what those are, well I am about to explain why these two functions caused us a great deal of pain. Older applications for the Mac OS, think pre-OS X, used resource forks and resource files. These resource forks/files contained icons, data, code, window layouts, strings, etc. The way you could scan these files is by using the combination of these two old Apple APIs.

When Office launches, we scan all of the office applications, reading resource forks and collecting information to shove in the User Preferences for the suite, essentially building up caches. When you install Mac Office 2004, there are over 3000 files. On your local hard drive, this scan can take anywhere from 25 seconds to over a minute (depending completely on hardware). See where I’m going with this? Remember what I said about RHF and application design. Networks are not as fast as hard drives, no matter what anyone tells you, reading/scanning 3000+ files over the network is a bit of a slow thing to do. Some customers were reporting it would take up to 10 minutes to launch our application using RHF.

Now in Mac Office 2008, we have done a great deal of modernization across the board. No longer are we using resource forks and files. We have created application bundles, library frameworks and are completely Mach-O. Unfortunately with doing this, the number of files is now upwards of 20,000. That might seem like a lot, but considering that Safari and iTunes install about 4700 & 5800 files respectively, 20,000 files is not too bad for something as large as Office. (If you are interested in checking, try the following: Open Terminal; type cd /Applications; type find ./iTunes.app –name “*” | wc –l)

The reason behind this is that resource forks and files are now broken up into a single file per resource. Needless to say, if launching Office 2004 with RHF was slow it got much worse in Office 2008. If you were reading carefully, you would have noticed I said, “When Office launches.” Yeah, we discovered every time Office is launched it scans all the files, to verify the integrity of the caches it has build. That problem on top of the fact that we now had 20,000 files caused our performance to drop even further. This is where the team I work with started attacking the problem. It took a lot of digging, using Shark, our own internal debugging tools and really trying to understand what was going on, since the people who worked on this code I think left the company when ADA was created. After a great deal of work, many hours spent with stop watches and testers configuring lab machines, we were able to get the launch times down to 20 seconds or so depending on your machine, no matter if you are using RHF or not. Of course that was in our lab, which is why we needed to make a trip to a customer to test out all our hard work.

But why take a road trip? Burbank is a quick flight from our offices. Simply put, pregnant women don’t fly. My wife and I were expecting our first child and she had to be in San Diego for the week. Being the diligent husband, and actually really wanting to check out the San Diego office (which by the way doesn’t suck), I decided to drive down with her and along the way; I could stop by our valuable customer and verify our fix. This leads me back to why I drove all the way down to Burbank.

Watching the fix work in front of customers is a great feeling. I could say more about the visit, but there’s that implant in the back of my neck that hurts when I talk/type too much about things I’m not allowed to … ouch! They were quite happy with the results, and even happier that I was able to get another serious remote home folder bug fixed after I saw a problem they were having because we hadn’t seen it in our lab. That’s the great benefit of testing in a real world environment.

It was quite satisfying for me and the team: Stephen Shaw, Steven Splinter and Kirk Engelmeier to get this fixed for our customers.

Update on Office 2008 progress!

Geoff Price — Tue, 09 Oct 2007 02:08:00 GMT

Last Wednesday was our general “ZBB” target for the Office 2008 project, a major milestone on the road to release and something we’ve been pushing hard for over the past several months. We saw some fantastic surging by the development team to clear out backlogged product issues late into Wednesday night. Thursday, we sat down to review project status and metrics in depth, and to hear all teams report out on their plans and readiness to lock down for release. The bottom line takeaway that you may be most interested in: all teams have confirmed readiness to ship on the current schedule. As previously announced, this means RTM in December and general availability starting on January 15th (by region).

ZBB what? This stands for Zero Bug Bounce, or as alternately phrased, Zero Bug Backlog. At this point we have been logging, tracking and verifying all changes to the product in great detail for some time – collectively we refer to this list of logged issues as the “bug list”, though in reality it includes a variety of issues including bug/defect reports, tracking records for artwork or content, usability improvements, numerous suggestions (submitted by team members or beta testers, or representing customer or partner requests), and various other categories of issues. The ZBB milestone is defined as the date across which we will no longer carry logged product issues that are more than one week old. Hence, the “backlog” of issues has been cleared out, and all older pending decisions on what we are or are not going to change before we ship have been made. It also means that the developers have “caught up” or “outpaced” the incoming find rate of our test efforts.

From here on out, the focus is increasingly on lock down. We are testing for defects literally around the clock, using a variety of methods ranging from automated efforts such as “massive file testing” to targeted manual test passes and “ad hoc” free testing. We also continue to log and fix significant issues found in private beta testing. At this point we remain busy fixing bugs as they are found (any one of our millions of lines of code could be the source of a defect,) but there will now be increased scrutiny on making changes – i.e., is there sufficient customer need to justify a proposed code change given the associated risk of regression (the outside chance that making a given change causes a new and potentially more severe problem.) Stabilizing the product for release requires conscious commitment to slowing and eventually stopping the changes we’re making to it, and as a result we now start reviewing all proposed changes daily in “triage”.

I’d like to congratulate and thank everyone on the product team for an amazing effort on ZBB – our developers, testers and program managers have all done incredible work. I’d also like to thank every other person who has helped in the creation of Office 2008: planners, researchers, designers, artists, writers, editors, builders, product managers, recruiters, HR generalists, international teams, MVPs, beta testers, partners, vendors, contractors, our many friends at Apple, partners across Microsoft, and many others have all helped us get this far. Finally, a very sincere special thanks to all of our families and friends for their support during this push.

Collectively we’ve made nearly four years worth of heavy investment in modernization, improvements and features in Mac Office. We look forward to continuing to share more about the new suite here on the blog and out in the world straight through RTM and into next year. But, as folks like to say, at this stage of the project the most important feature is shipping.

We’re getting more excited about that feature every day.

Speaking of numbers...

joeleblanc — Thu, 16 Nov 2006 02:35:00 GMT

One of the features in Excel I'm in charge of testing is "Calculation, Functions, and Formulas", which (surprise!) is exactly what it sounds like. Being a big math and numbers geek, I was very excited to learn that I would inherit these features when I started. The way that we test these this area is a nice snapshot of how we utilize AppleScript and our automation system.

Functions are the heart of Excel, even the most novice Excel users are familiar with things like SUM() and AVERAGE(). Excel has somewhere around 320 functions all told, many of which are very specialized and not used frequently. In order to make sure we regularly test every single functions, we've designed a system in which a single script and test file exists for each one. These files consist of functionality, boundary, error condition, and other tests, usually about one per row. Each row (this is row 11 in the SUM.XLS test file) looks something like this:

A	B	C	D	E	F	G	H	I
Manual Example	=SUM(E11:I11)	55	TRUE	10	7	9	27	2

Column A is a description of the test

Column B contains the formula itself

Column C is a hard-coded expected result

Column D is set to "=C11=B11", which returns TRUE if the values equal each other, and FALSE if they do not.

Columns E-I are the arguments used for the formula in Column B

At the top of each file, cell $D$1 is set to “=AND(StatusArea)” with StatusArea being a name defined as whatever part of Column D is filled with tests. It all boils down to one cell on one sheet; If any of the results are incorrect we will know right away.

For each test file, we have a little piece of AppleScript that opens the file with help from our automation framework, forces a re-calc by replacing “=” with “=”, logs the value of $D$1, and closes the file.

set myusedrange to active sheet's used range
replace myusedrange what "=" replacement "=" look at part search order by rows without match case
Log (LocalizedTrueString, range ("$D$1")'s string value, "String Value of $D$1 (expect: TRUE/(pass)) ")
close every workbook saving no

Lather, rinse, repeat, 321 times.

Each of these scripts runs every time there is a new build, on both CPU platforms. Failures appear right away, and we can compare the results over time to make sure that PPC and Intel are doing the same math. We can also quickly fire off a test run if we want to check out the latest OS patch, or critical check-in to Excel’s code. Scripts like this and our automation system enable us to test across a huge part of the product almost immediately. The simplicity of it all means that anyone can check on the state of our calc engine, and update the test files with new cases they find at any time.

Living complexity

Geoff Price — Fri, 10 Nov 2006 23:16:00 GMT

Brad's Carl Sagan-style post on code size stirs up some familiar questions about complexity management here at MacBU. With every release of Office, we remove, rewrite and add new code. Often we can remove significant amounts of code without losing functionality – say, remove use of a custom HTML renderer when the OS offers a built-in one instead. I recall how many of us took it as a badge of pride during the Carbon transition of Office v.X that after a year of intensive coding our net contribution in terms of new lines of code was significantly negative. Those were the days, hacking and rewriting. But of course, as the total functionality of the suite increases over time, the total volume of code used to build it tends to inevitably follow. There has been a lot of functionality added in twenty years.

Is it necessary? It's pretty easy to find people who agree that Office has more functionality than they personally need. Where the agreement tends to fall apart is when you ask what the first feature to be cut should be, at which point one person's "bloat" is another's "only feature I use every day". It's also nice to know the functionality and compatibility is there, as you find the need or the advantages of it. I use all of our apps every day, not to test but just to get work done – Exchange-based mail and calendaring in Entourage, analyzing data and lists in Excel, IMing over corp chat – it's an interesting job being able to use the software every day to talk about the job of making the software. My Office skills were pretty minimal when I got here, but (and some coworkers might smirk at this) they've come a long way. You may not have heard of (say) pivot tables, but once you start using them for certain kinds of analysis you can't imagine how you ever lived without them.

Of course, our goal isn't to just pile features on top of features, but to harness the increasing power in the suite with an effective and focused user experience that puts functionality in reach and in context. In some cases that can be pretty easy, but generally it's actually pretty hard, and something we have to think about with everything we do.

Is it slow? Generally, there's no linear relationship, where adding 10% code means everything is 10% slower. There's quite a lot of engineering happening down at the metal to swap in the currently executing code as the application runs. The problem is in how much work the application is trying to do – if it's trying to do a lot, it's necessarily going to take longer to do. When we work on performance bottleneck areas (such as boot speed) that is generally what we are looking at, i.e. do we really need to be doing all this now? For this release, running native on Intel is orders of magnitude more important for speed than trying to trim raw code size, for example.

Is it sustainable? It's important to understand that while the code has been written and expanded over the past twenty plus years, it has also been tested and validated for twenty plus years. Insane amounts of testing. Stabilized or "baked" code provides a baseline for correctness and quality – as new changes introduce unintended problems (regressions), we know we just need to hunt down which new change is the culprit and understand how it needs to be redone. The real impact is on productivity: while correct, that baked code can vary wildly in how well factored it is (well factored being the opposite of "spaghetti"), and therefore how sensitive to regression. We definitely have some pasta mixed in as part of our overall diet, and we have to develop strategies for dealing with that. Sometimes it's best to simply say "do not disturb the water", but that's not always an option (say, for example, you decide to blow out the Excel cell table.)

Shouldn't we just rewrite it all from scratch, and "do it right" so that *all* of the code is perfectly well factored according to modern sensibility? JoelOnSoftware made some great observations on that topic in this article: Things You Should Never Do, Part I. (I believe link-master Brian sent that to me originally. Pyramid was before my time here so I'll leave it to others to comment on whether Joel's observations on that project are fair and accurate or not.)

Some people keep souvenir copies of tech editorials from ten years ago predicting that Microsoft will be unable to release any further versions of Excel, because the code base has simply gotten too big and Microsoft will have to rewrite it from scratch. That was quite a few releases of Excel ago.

Joel points out that code doesn't rust, but the complexity problem is real because massive change is inevitable if you are trying to keep your software on the leading edge – especially on the Mac. We plan to continue shipping Mac software into the foreseeable future, so it is easy for us to make major architectural decisions with the very long term in mind. Where it gets hard is balancing this against short term customer need – what resources can we apply to deep rearchitecture or local refactoring, given that these resources will not be available to solve the most pressing problems that are actually visible to customers? While we made significant investments in future capability earlier in the current product cycle, right now that customer need is pretty extreme and we're 100% focused on getting this release out into the market.

Finding this sort of balance overall, and dealing with this level of complexity in general, is clearly our defining challenge with Office. It takes some time to get your head wrapped around it all. We might go days without seeing a particular new developer, and we'll have to send a veteran search and rescue squad into their area of the code to pull them out. Of course, it doesn't just impact developers, it's in the density of the test cases, the maintenance of the automation, the range of functionality and interactions our designers have to understand, the scope of documentation, the pace of change. Managing complexity is where we live. It's not for everyone, but fortunately I find it all pretty fun at the end of the day... gives my head something to do.

On the power of a small team -or- Why MacBU testing rocks

joeleblanc — Sat, 30 Sep 2006 03:14:00 GMT

Geez, now I have to follow up David's awesomely funny entry about the food alias with my dry post on testing.

One of the first classes I took after starting here was called “Testing at Microsoft for New SDETs.” This class was full of about 25 people who had either just started with the company, or just started a new job as an SDET. (SDET here means “Software Development Engineer in Test) A good portion of the class was very applicable to what I would end up doing, although a lot of it was focused on Windows technology. Some of the more applicable parts related to test planning, test cases, and more of the theory of testing that applies to any platform you’re working on. During lunch I was talking to the instructor about what I would be working on, the MacBU, and how we might do things differently than the rest of the company. When I told him how big the team was, he was shocked. “How can you test with such a small group and ensure a quality release?”I told him, honestly, that I had no idea! I had just started, and I really didn’t know how we managed to do such a good job with the (seemingly) small group. After working here for a while longer and thinking about how we do things compared with other teams I’ve worked for, I’ve come up with a few thoughts on why our small team is successful.

Simplicity: Office (on both platforms) is sometimes referred to as “bloated” or “complicated.” While we do have a very complex feature set, I think that our program managers (the people that design features) have done a great job of keeping the product simple and straightforward. There are many areas that have the potential to be needlessly complex, and we’ve managed to keep things smart. This simplicity is not only the result of good testing and design, but it also serves to help testers own more areas completely, and leads to better...

Knowledge: Each person on the team knows their product very well, as well as the other products in the Office suite. When you have dozens of people testing an application, no individual person is necessarily going to look outside of their areas to see the big picture. I think understanding how features relate to, and interact with, each other helps us deliver much more consistent applications, something that is key for Mac users to accept them. When you have too many people working to ensure the quality of something, you lose sight of the fact that you’re not just delivering 100 individual things, you’re delivering a whole product. Individual testers and not just leads/managers knowing more about the product leads to better...

Prioritization: Anyone who works in software knows that not every feature gets finished, and not every bug gets fixed. If we waited for that to happen, we would never release a product. The fact that we know what needs to work, how it should work, and when it needs to be done lets us make much better decisions about where our priorities lie. This kind of prioritization happens in any product, but usually at a higher level. As a part of a small team, I feel like I have the ability to affect real change, and make a difference by speaking up about our products’ quality. Empowerment of "the little guy" leads to the most important factor in our success in the MacBU...

Passion: The people I work with are committed to the team and to the products. Very few people end up working in the MacBU because it’s an easier job than the rest of Microsoft or the quickest way up the corporate ladder. People are drawn to this group because they care about the product, enjoy working on the platform, and are committed to it. There is no substitute for a someone enjoying their job, it makes you much more productive and improves the entire process. So I guess to sum up why I think the MacBU is as efficient as we are: Knowledge of what you need to do, and your personal ownership of it makes for a very committed team that is able to accomplish things that would normally take a much larger group of people.

I hope I don't come off like some kind of management training handbook or motivational speaker. At least I didn't arrange those into a cheesy acronym! The new Joe LeBlanc SKPP small team testing methodology!@# :rolleyes: I'm just making an outsider's first impression of how things work around here, and why I think we're as good as we are.

Have a great weekend!