Software Test Engineering @ Microsoft

Differences between Encarta Premium 2006 and Student 2006

The back of the box for Encarta Premium 2006 has a nifty feature chart that I've been unable to find online. Here's a posting of that chart, containing the differences between the North American versions of Encarta Premium 2006 and Microsoft Student:

 -Greg

I Love the MVP Program

Encarta's MVP, Stephen Stakey, has just launched the Encarta SupportZone- you can find it here. As he says, it's a place to "find how-to guides, videos, and links to other Encarta-related websites".  He's already spent a lot of time in the newsgroup, dug up a lot of the pertinent information that's come over time, and conglomerated it into a centralized location.

This is a great example of how cool the MVP program can really work. Stephen's active on the Encarta newsgroup, has been involved in Encarta Betas for *at least* as long as myself, and does a great job keeping in touch with the product team. It's really enlightening to have a user on the outside who is a great contributor to the product and is a big fan of the app that we spend so much time working on.

Love the site, Stephen!

-Greg

Encarta Blog is up and running!

The Encarta Team blog is now up and running with it's very first post. Take a look here!

The IE Team blog is currently the best team blog I've read, but I hope our Encarta blog is ultimately of similar quality.

BTW- Reading the IE blog it really came in handy today. I got to teach myself how to use the Fiddler powertool during an internal MSN Encarta bugbash of the MSN Encarta Beta website. It's one of the best testing tools I've ever used- I was able to throw a host of bad inputs into some of our Edit boxes, watch how the inputs are handled by the server, and then "fiddle" with the data at various times during a communication to see how extensive error handling is implemented.

I didn't find any bugs using the Fiddler in my first attempts, but I'm sure it'll help me find some issues in the future.

Greg

Reasons to be a Test Developer

Steve Rowe has an excellent blog post on Three Reasons To Consider Being a Test Developer (aka SDET). Nihit Kaul also follows up with some interesting comments here.

One piece I'd like to add about the advantage of being a Tester (whatever the job title : ) is that you'll learn your underlying product literally *better than anyone*. It's true that PMs have put their sweat and blood into creating detailed Specs, and Devs have literally built the application from the ground up and have an extensive knowledge of underlying algorithms- it's their own code, after all! Ultimately, I would have to argue that the Tester of a feature is going to have the most knowledge of the in's, out's, upsides, and downsides of the features.

Why?

It's our job! We've spent such a long time hacking away and learning our feature, that we've inadvertantly picked up full expertise from a true power user's perspective. When we think of a user scenario that's outlandish but possible and should be supported, a Tester is the only member of a product team with the freedom to go ahead and try it. PMs have their hands tied, and Devs are already busy off writing more code.  An SDET is free to write automation to learn exactly where the product works, and exactly where it doesn't. And, they're probably happily doing some exploratory testing when they become inspired to search for a potential bug that may have been hiding.

From my own experience, when a question comes up in a meeting like "How do we handle environment condition X when executing feature Y?" - all eyes at the table will drift to the Tester, and they'll usually know the answer. :-)

Greg

Why does Software Crash #2 – The Access Violation (continued)

In our last post (here), we took a look at an Access Violation of a user mode application attempting to access memory in kernel mode address space.

It's interesting to note that we can't just go reading and writing memory in the kernel- if we try to blindly access another application's memory, we'll get the same AV error (on an NT OS, anyhow). Let's take a look:

   int foo;

   int * ip = (int*)16777216; //Address 0x01000000,  in another process' address space 

   foo = *ip; //Crash!

This will crash with the following error:

Unhandled exception at 0x004173c8 in cpractice.exe: 0xC0000005: Access violation reading location 0x01000000.

A few notes:

• On most machines, you'll see a Watson dialog instead of this unfriendly crash dialog. I've used the `unfriendly' crash dialog here for demonstration purposes.
• AT posted some interesting comments in the other post here. The win32 API has tools for sharing memory between processes, and the above code was legal on DOS, which was without a protected memory space.

Why does Software Crash #1 – The Access Violation

Pop quiz: what does this line of code do when executed?

            int foo = (*((int*)0));

If you’re an astute reader, you can solve the answer just by reading the title of the blog post. But, more interestingly, let’s forget about that buzzword and analyze exactly what’s happening. The access violation is probably the most common crash in unmanaged software, so let’s break it down piece by piece to discover what’s happening.

      int foo;

      int * ip = NULL;

      foo = *ip; //crash!

On the first line, we declare an integer variable named “foo”.

The second line declares a pointer to an integer named “ip”, and initializes this value to NULL (which is simply a fancy term for 0). Pointer variables “point” to a location in memory. In this case, we initialize it to point to the address NULL, or 0, of memory. Location “0” is in area of memory reserved by our operating system.

The third line attempts to grab the value of address 0, which is prohibited, and assign it to our variable “foo”. Our operating system smartly catches this and says “No way- you can’t see my reserved memory location!”. Our operating system shuts down this badly behaving application with an error like the following:

Unhandled exception at 0x004173c8 in cpractice.exe: 0xC0000005: Access violation reading location 0x00000000.

This error message is a bit cryptic, but it's understandable if broken down. 0xC0000005 is the error code designation for an Access Violation. 0x00000000 is the location that we tried to read (this is our NULL value!). And, 0x004173c8 is the arbitrary memory location where our application happened to be running at the time of the crash.

AV's can happen for both reading and writing, which is part of the reason that they are so common. Pointers in general are one of the most challenging topics in computer science, which is yet another reason this crash is so commonly seen.

-Greg

“Why does Software Crash” – The series!

Starting this week, I’ll be starting a series on the nuts and bolts of software crashes. Complex software has a seemingly infinite number of ways that can force a dreaded crash dialog, and I’m up for the task of enumerating why many of these things happen.

I plan on starting from examining short C programs that crash, and working my way up to broader environment issues that can cause instability on a user’s machine. As always, feel free to ping me as I go along and I’ll address any questions as they come up (and I’m sure I’ll have some questions myself as I write this series!)

-Greg 

White Box security practice

Mike Howard has posted some examples of buffer overflows by incorrect use of "safe" string functions. It's an enjoyable exercise- take a look here!

Greg

Difference between STE and SDET roles at Microsoft

What’s the difference between a Software Test Engineer (STE) and a Software Design Engineer in Test (SDET)?

The Microsoft career site has two excellent definitions for the Software Test Engineer and Software Design Engineer in Test roles at Microsoft. In my experience, these clearly defined role definitions have very vague applications here at Microsoft. After all, aren’t we all just assuring that the software that we ship is the highest quality possible?

Historically, it seems to have made a lot of sense that these two positions had distinct roles. Years ago, Software Test Engineers were testers who had limited programming experience. I’ve heard particularly frightening stories from Microsoft old-timers where Software Test Engineers were used as monkey labor- “Run this test fifty times, and call me if this breakpoint gets hit”. STEs didn’t need a strong CS background to get their jobs done- they just needed a lot of patience, and basic understanding of *using* computers.

The Software Design Engineer in Test role was reserved for Testers who were capable of writing tools and/or automation. I’ve also heard stories from other old-timers about STE hires that write tools in their first half year on the job having their job title switched by management to SDE/T.

It’s easy to understand the difference between STE and SDET with a black and white example. If you’re testing an API, then yes, you’ll surely be an SDET (how can you do any testing without writing a line of code?) Conversely, if you exclusively black box test a user interface (without tools/automation), then you’d have the job title of STE. The gray area between these two extreme examples is huge, though. I don’t know any STEs who don’t end up writing at least *some* code. In my personal experience, I’ve spent the past three months on the job writing automated test cases. Like many of my fellow STEs, this doesn’t strike me as uncommon. I'm sure there are plenty of STEs that in practice write more code than SDETs! This difference is particularly pronounced across different product teams at Microsoft.

Management seems to be pushing for a unification of these two roles, probably for this reason. Both roles dedicate themselves to product quality, both are reliant on strong fundamental understanding and application of CS principles.  A quick look on the career website seems to back this up- there are currently 326 active postings for SDETs, compared with 44 for the job title of STE. This doesn’t mean software testing is going away- I only see it as a reflection of the increasingly technical nature of testing software at Microsoft. It looks like the job title that the company is shifting to is SDET, which is more a mouthful but also more impressive sounding.

In short, don't be surprised if the name of this blog becomes "Software Design Engineering in Test at Microsoft" to match the way that the management wind seems to be blowing. If it comes to that, hopefully Chris and I can come up with less unwieldy blog title! :-)

-Greg

A Great Use For Blogs

I'm going to head off today and talk about something in the blog world, and not at all about testing.

Every now and then I hear of a new use for blogs that I really like. Most of us are used to the standard paradigm, I write in a blog so my family and friends can keep up with me easily, I write in a blog to share my ideas about the discipline I work in. But I was talking to a friend of mine who came up with a great use for the technology, and thought I'd share it here.

He's the computer teacher at a private school down in California, and for this year he decided it would be great to set every teacher up with a blog. They can use this to post homework assignments, communicate status updates, and update parents on what they'll be covering in class over the next week; all kinds of stuff.

One of the teachers even took it further. She enabled comments on the blog and the first day had her kids go out and do some online tests and post the results as comments. This got all the kids involved and interested in the blog. It was a great success.

I was talking to him this week and he said this has been a huge hit with everyone. He used Blogger to set up all the pages, and raved about their interface (my personal blog uses Blogger, I'm also a big fan.) Most notably, all the teachers have to do is send an email to a specific address and it gets posted up on the blog, complete with formatting.

It always makes me happy when blogs make roads into non-technical sectors. I think they're a wonderful communication tool and have lots of great uses ahead of them. Stories like this remind me they have a lot of places to grow to. I look forward to the day when people stop having to ask what a blog is, then give a puzzled look as I try to explain it. I fully expect it to be just like the web, email, instant messaging, and plenty of other communication tools - soon enough blogs will just be another tool we use to keep in touch and share information.

Recruiting Cold Calls

I think there's some recruiters making the rounds of Microsoft Bloggers. Gretchen and Zoe, as well as Heather have talked a fair amount about how a blog can help your career, and just last week I think I saw a bit of that.

I got cold called by a recruiter on my office phone. I didn't think much of it, but then I read that Eric Lippert got called for the exact same opportunity (Eric changed the names to protect the innocent in his story, but it's clearly the same place.) For me the conversation went a little differently though:

Recruiter: (Description of company that exactly matches the description Eric related) is looking to hire Engineers, including some leads. Is that something you'd be interested in?

Me: What kinds of Engineers are we talking about?

Her: Software Engineers.

Me: Yeah, see I don't really write code (this is a slight mistruth, more on that later.)

Her: Really, when did you stop?

Me: I never did.

Her: So then what do you do?

Me: Test work.

Her (with great disappointment in her voice): Oh...  Do you konw anyone that might be interested in this? It's a great opportunity.

There are a couple things I'd like to note at this point, although some of these are just wild speculation.

1.) Saying I don't really write code was a mistruth. I actually write code most days. But the kind of code I write and the kind of code developers write are very different. I don't like writing code all the time, I like testing - and using the code I write as a tool to be more efficient and effective in my testing. I'll write more about this in a post in the very near future, but during my recruiting conversation I figured the safe path was to avoid that whole issue entirely and say I don't write code, since I don't write the kind of code she was looking for.

2.) While it was flattering to get called by a recruiter. That flatterly quickly faded as I realized there had been absolutely no background work to see if I was actually anywhere close to a fit for the position. It's not like it takes much, pop my name into google and it's clear very quickly that my field is not Software Development.

3.) I understand that cold calling people just plain sucks. It can't be fun. And searching for names of people to call has got to be tough. But I'm thinking that mining the list of Microsoft Bloggers might not be your best bet (if in fact, that's what was happening.) All the Microsoft Bloggers I know are really passionate about Microsoft and love it here. That's the main reason we blog - we want to share our passion and experience with the rest of the world. Microsoft has, shall we say, a bad reputation in the industry. People think we're all cogs in a big machine of evil here. I whole-heartedly disagree with this and a big part of this blog is outreach trying to dispell that opinion.

4.) I'm not a recruiter, but it seems to me like it would be easier and less emotionally draining to do a very small amount of research into people before calling them. For example, a google search on me will tell you very quickly that I'm not a good fit for these developer positions. That would have taken less time and been easier for all parties then calling me. If any recruiters disagree with that sentiment I'm really curious to hear your thoughts.

All that said, I think it's great that the economy is picking back up - it's wonderful news. And I eagerly await the day I get called about a test position. Not because I'm itching to leave, but because the other big part of this blog is trying to get the industry to respect testing more and treat it more seriously. The first step in that process is taking the role seriously enough to find and pay for good talent.

Chris

Meet Greg @ Syracuse

Ever wanted to meet real, breathing, Microsoft employees and you're located around Syracuse NY?   

• If Yes: Good News! I'll be at the Microsoft booth at the Syracuse Career Fair on Sept 30th. We'll be taking resumes, answering questions, and asking some of our own. Please come out and say hello!
• If No: Consider changing your mind, and then see the instructions at "If Yes". :-)

Now that we're entering the Fall Career Fair season, you may want to check out my previous post on Career Fair Tips. The advice listed there will help you chat with any company, but should be especially useful for Microsoft.

-Greg

Testing Complex Software

It's really hard to wrap your head around how complex software like Office or Windows is. These programs are huge, and have hundreds of millions of users. All of those users have different configurations and different settings and different ways they like to use their software.

It's only natural that with all those differences some problems will be unearthed.

This is one of the key problems of testing big software like this. You can test and test and test, but it's a bit frustrating to know that even with all your work someone, somewhere will have a problem with your stuff. I had a great conversation with two developers at lunch today about this. Once your software stabilizes testing becomes a low-yield game (tangent: at this point one of the developers cheerfully said “not when you're testing my stuff!”) You test a lot of surface area, and don't find many problems. This is one of the hardest times for testing, the temptation is to say “yep, it's done!”

This is a key dark art for shipping successful software. Everyone wants to be done, but someone has to have a good idea of when done actually happens. Ship too soon and you've got a buggy product out there. Ship too late and diminishing returns has kicked in so your quality isn't much better, but you just lost a lot of money by not being out in the marketplace. Every time software ships someone, or hopefully a group of someone's, thought about these issues an made a trade-off.

A lot of people (I read about this a lot on slashdot...) claim Microsoft errors too far on the side of buggyness when we make this trade off. When I read things like this I typically get a littel upset. I know when I sign my name saying software can ship I mean it. I really, truly believe the software is ready to go out the door. I understand that some people will have some problems, but I know I've worked very hard to find all the problems I could. Most of the testers I know feel the same way, we get personally upset when a customer finds a bug in our area - we feel bad for not catching it first.

Anyway, what reminded me of all this is the fixed list for the recently released Windows Service Pack 2. It's sort of humbling, the number of obscure and random problems something like Windows can cause. If anyone doubts the complexity of testing something like Windows, go take a look at it. I know I'm going to go back and read it when we get close to shipping - this is the kind of thing that provides motivation to keep looking for problems, even when you think you've looked everywhere there is to look.

Chris

Where to live- Seattle or the East Side?

I remember listening to a young Microsoft Intern about what had been recommended to him about living arrangments. To summarize what he said: “Redmond is the best place to live, because the rush hour traffic to Seattle is terrible!“

Every Microsoft employee has a strong opinion about the best location to live. It's one that I had been analyzing recently, and pre-empted my move from the Redmond suburbs over to Alki Beach in West Seattle. Here's my opinion on the Pro's and Con's of the two living arrangments- and yes, I am a 24 year old single male which influences my answers!

Pros of Redmond/East Side:

• Short commute to work
• More free time spent at home (important if you have a family)
• Condensed Intern population in the summer
• Possibility of going home for lunch
• Closer to the Microsoft sports fields
• More flexible work hours
• Marymoor Park in Redmond
• PRO Club in Bellevue

Cons of Redmond/East Side:

• Limited Night life (though Kirkland has a small selection)
• Minimal live music
• Non-short commute times if you live in the Eastside perimeter
• Long commute to Seattle
• Poor public transportation on Friday and Saturday nights ($25 cab fare from Downtown Seattle to Redmond)
• Fewer college age people

Pros of Seattle:

• All the amenities of a great city!
• Better nightlife, live music, etc.
• Professional sports teams, etc.
• University of Washington's campus 
• Better access to non-Microsoft activity partners (check here)
• Single bus routes connecting Downtown, Capital Hill, Greenlake to Microsoft Campus
• Vanpool and Carpool options available for commuting in those H. O. V. lanes!
• Great hang out areas, like Greenlake, Alki, Belltown, etc.

Cons of Seattle:

• Traffic is a crapshoot, and often severe, especially across the 520 bridge
• Less flexible work hours (7:30-9:30AM, 4:30-6:45PM travel times will leave you stuck in traffic)
• Longer commute
• Less likely to hang out with East-Side friends
• PRO Club Seattle is inferior to the Redmond location

As a young person who likes to take advantage of the great offerings in Seattle, I've decided that it's the place for me. If you're not going to do “city“ activities several times a week, then you'll probably be happier on the East Side. But, for me, I'm going to happily be commuting to Seattle tonight!

PS: I'm sure I missed several important points, and I've omitted some prominent exceptions to all these rules. Feel strongly about something said here? Leave a comment!

-Greg

Beta Testing to be `Bleeding Edge'

There's an interesting New York Times article about Beta Testing for the sake of `bleeding edge' status here. It mentions:

Software testing used to be done by serious computer users. Now, testers are motivated less by service than by the status of being among the first to try the newest software.

This is a trend that I've noticed increase as well. Various pieces of software seem to be more and more anticipated all the time. Microsoft Employees were recently eligible to participate in the Halo 2 Beta, which has spurred some bragging from those involved. The cutting edge of the technology curve is a tough place to get to, and an even tougher placer to stay. Beta Testing is a great way to enable being the first kid on your block with the latest toy, and the joy of searching for bugs can be even more fun that using the software like a `normal' user.

Want to try out Beta testing for Microsoft products? Visit Betaplace.

-Greg