Testing

Arjan`s World » LINKBLOG for May 31, 2008

Arjan`s World » LINKBLOG for May 31, 2008 — Sun, 01 Jun 2008 00:19:59 GMT

PingBack from http://www.arjansworld.com/2008/05/31/linkblog-for-may-31-2008/

re: Testing

pkirchner — Mon, 02 Jun 2008 15:16:32 GMT

Hi Kirill,

I would be interested in how do you check for expected behaviors in the GUI testings. E.g. how do you test that the editor does the correct syntax highlighting or that the correct file opened after you selected it in the Solution Explorer.

Do you use OCR or someting similar?

Thx,

Peter

re: Testing

Horacio N. Hdez. — Wed, 04 Jun 2008 18:35:35 GMT

hi kirill

the backend and frontend team also uses unit-test?

thanks

hnh

re: Testing

Kirill Osenkov — Fri, 06 Jun 2008 11:33:29 GMT

Peter, GUI verifications are interesting. We don't do pixel-by-pixel baseline comparison, because this is too low-level - depending on a color scheme, font-size or dialog position on the screen the picture might be totally different.

Instead, we use Active Accessibility and UI Automation a lot - Windows technologies that allow screen-readers and accessibility devices to enumerate windows and controls. For a given window, you have a list of all child windows that are contained in it and their names. You just search for a name, and once you've found a window, you issue windows messages to emulate keyboard and mouse. We have a pretty powerful framework with several abstraction layers that provides this functionality for us. For every UI control (like a listbox) it provides an automation peer - this way I can enumerate over all items present in a listbox and know what's the currently selected item. I can also programmatically manipulate UI controls - push buttons, clear lists, type text in textboxes, open and close windows, etc.

Horacio, I suppose by the frontend and backend you mean the compiler team. C# compiler has a tremendous amount of test .cs files that is being run through the compiler to make sure it compiles them correctly. This huge set of tests is called the C# conformance tests. I'm not sure if the compiler devs use unit-tests - you better ask them yourself :) Almost all of the C# compiler devs are blogging:

Ed Maurer is the dev lead on the C# compiler team: http://blogs.msdn.com/ed_maurer/

Other developers:

Eric:

http://blogs.msdn.com/ericlippert/

Sam:

http://blogs.msdn.com/samng/

Sreekar:

http://blogs.msdn.com/sreekarc/

Chris:

http://blogs.msdn.com/cburrows/

Thanks,

Kirill

re: Testing

Kirill Osenkov — Sat, 07 Jun 2008 21:16:24 GMT

Horacio,

I just checked with the C# compiler team - they use unit-tests too.

re: Testing

Imran Qureshi — Thu, 10 Jul 2008 16:30:58 GMT

You mentioned:

Many of our tests use a new data-driven test framework where our test scripts are written in XML and are interpreted by a script-like engine.

Is this available for people outside Microsoft to use?

re: Testing

Kirill Osenkov — Thu, 10 Jul 2008 21:22:37 GMT

Not yet. But we're definitely thinking about making this public.

What are you more interested in: generic data-driven XML based test framework or its particular application to automated testing of Visual Studio?

re: Testing

Jose Antonio Arias — Thu, 17 Jul 2008 11:22:17 GMT

Hello Kirill.

I'm interested in generic data-driven XML based test framework.

re: Testing

Malcolm — Thu, 18 Dec 2008 10:27:36 GMT

I am interested in the generic data-driven XML based test framework itself. I can see how that could serve in a lot of environments. Can you go into more detail about the tools you use for the UI-automation with Active Accessibility?

Thanks!

re: Testing

Kirill Osenkov — Sat, 27 Dec 2008 04:32:05 GMT

Thanks everyone for your interest. I've asked my management and it turns out that the test frameworks we use currently are not public yet, so I can't talk about them just now. As soon as I can tell you more, I'll post an article about this on my blog. Thanks again.