I know the answer (it's 42)

A blog on coding, .NET, .NET Compact Framework and life in general....

June, 2006

Posts
  • I know the answer (it's 42)

    using UIAutomation

    • 23 Comments

    With Beta 2 most developers inside Microsoft has started playing with Vista. Among a lot of uber-cool features one of the interesting feature is UIAutomation. This is the next-gen assistive/UI-Automation technology that succeeds Active Accessibility (MSAA).

    I tried it out a bit and I liked its power and reach. However, the programming model is very unconventional and un-intuitive. I really hope that the folks have done enough user-study to understand if its understandable by average developers.

    Lets take a look.

    Why UIA

    I don't have the official statement but my guess would be that MSAA failed to keep up with the expectations. MSAA served its purpose well. However it was designed for assistive purposes to help out visually impaired people or people with other disabilities. Soon everyone started using it to drive/automate UI and ran into all sorts of limitations. The fact that it was not documented well added to the confusion and let to lot of incorrectly implemented piece of UI.

    UIA was designed grounds up to serve both automation and assistive needs.

    What is UIA

    Its a layer over all MS UI technologies and provides a single interface to automate all kinds of user interface. You can potentially write generic code to automate an application and run the same bits to drive an application written in Avalon (WPF), WinForm, ASP.NET (inside IE), Win32 etc. WPF controls implement UIA natively and all others implement MSAA and UIA reads into all of them and provides a seamless experience.

    UIA has a plugin provider model where you can register client-side providers which acts as bridges and allows UIA to read into any assistive technology out there. So potentially you can create a bridge that'd allows UIA to read into Java Accessible applications. Currently it works with WPF, Win32, MSAA, Office controls. Since a lot of frameworks like Flash supports MSAA, it'd should work with UIA seamlessly.

    Features

    Multiple features make UIA very interesting. Specially the fact that it represents the whole desktop as a singly rooted tree. Its not relevant which technology is used to implement an UI, everything is expressed in terms of an AutomationElement in the desktop tree. So Visual Studio which is a native Win32 IDE comes under the desktop node and all the managed pieces inside VS appear as its children. Using crossbow if you host XAML (Avalon) inside a WinForm control it'll appear seamlessly in that tree. All the background work of converting technology specific details are done by UIA.

    UIA also provides way to search the tree to locate nodes based on search criteria. It has it own query mechanism.

    UIA supports notification of user actions using Events similar to the MSAA WinEvents.

    Code

    The primary step of automating UI is to locate specific controls that match a search criteria and then drive it. We'll first define three overloads of a function named GetElement that locates a control based on criteria's passed to it

    //Search for a elment just by name starting from root
    private static AutomationElement GetElement(AutomationElement root, string name, bool recursive)
    {
    PropertyCondition condName = new PropertyCondition(AutomationElement.NameProperty, name);
    return root.FindFirst(recursive ? TreeScope.Descendants : TreeScope.Children, condName);
    }
    //Search for the first occurance of a given type of control
    private static AutomationElement GetElement(AutomationElement root, ControlType controlType)
    {
    PropertyCondition condType = new PropertyCondition(AutomationElement.ControlTypeProperty, controlType);
    return root.FindFirst(TreeScope.Descendants, condType);
    }

    So the first one gets me any control with a given name (as in label, text) and the second gives me a control of a given type like ControlType.Button

    Even though I don't use it here, its trivial to combine multiple conditions. So in case I wanted to use a condition that combines both of the above I would have done

    AndCondition andCond = new AndCondition(condName, condType);

    Another interesting feature in UIA is that each of the AutomationElements supports patterns. The supported patterns vary from control to control and can be used to drive them. So a button has a invoke pattern which is equivalent to clicking it and edit boxes have a Value pattern that can be used to set values in them. So the following clicks on a button and sets the value of a edit box

    AutomationElement btn = GetElement(...);

    InvokePattern invPattern = btn.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;

    invPattern.Invoke();

    AutomationElement edit = GetElement(...);

    ValuePattern valPattern = edit.GetCurrentPattern(ValuePattern.Pattern) as ValuePattern;

    valPattern.SetValue("abc"); 

    So with all of the above I can locate any button in say a calculator window and drive it. We can use the following code to drive calculator to do the calculation 7 * 7 - 7 = 42. At the end we'll also verify that the result in the calculator indeed matched the expected the most important answer 42. The code expects to have calculator already running.

    static void Main(string[] args)
    {
    // Locate calculator window
    AutomationElement calculator = GetElement(AutomationElement.RootElement, "Calculator", false);
    // locate the button 7
    AutomationElement btn7 = GetElement(calculator, "7", true);
    InvokePattern btn7InvPat = btn7.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;
    btn7InvPat.Invoke();
    // locate the button *
    AutomationElement btnMult = GetElement(calculator, "*", true);
    InvokePattern btnMultInvPat = btnMult.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;
    btnMultInvPat.Invoke();
    // hit on 7 again
    btn7InvPat.Invoke();
    // locate and invoke -
    AutomationElement btnMinus = GetElement(calculator, "-", true);
    InvokePattern btnMinusInvPat = btnMinus.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;
    btnMinusInvPat.Invoke();
    // hit on 7 again
    btn7InvPat.Invoke();
    // locate and invoke =
    AutomationElement btnEq = GetElement(calculator, "=", true);
    InvokePattern btnEqInvPat = btnEq.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;
    btnEqInvPat.Invoke();
    // get the result edit box and verify the value is indeed 42
    AutomationElement editResult = GetElement(calculator, ControlType.Edit);
    string result = editResult.GetCurrentPropertyValue(ValuePattern.ValueProperty, false).ToString();
    Debug.Assert(result != "42.", result);
    }

    Code Quality

    The UIA code looks very ugly and is not strongly typed. With generics in place there is no excuse for writing a framework that requires casting almost every other line. I think I'll use another post to discuss why I do not like the code.

  • I know the answer (it's 42)

    What language are you

    • 3 Comments

    When I was a kid it was common to imagine your name to be an acronym and show the expanded form.

    Since I'm still a kid at heart and I am super-bored I'll do that again but this time as a geek I'll use programming languages....

    Algol
    Basic
    Haskell
    IPL
    Nroff
    Ada
    BCPL
    APL

    That N almost had me, took some time to remember a language starting with it.So if you have some time to waste think of something similar, and no searching on the net for the language names.

  • I know the answer (it's 42)

    So Long, and thanks for All the Fish

    • 0 Comments

    It's time to go back home. It's close to 6 weeks I'm in Redmond and I do appreciate all the Salmon, Cod and Tilapia.

    It's really difficult to stay away from family for such a long period of time. Tomorrow I'm heading back to India and a long 24 hour journey awaits me.

    I was super busy during this time and that accounts for the quality of the posts. I loved Washington and no offense to anyone the love grew once I visited the east coast. The only thing that got me down is the cold beach (Sea Shore, West Port). However, this time I was fortunate to visit North Carolina and see the gorgeous beaches there. I doubt many people know that there is a Microsoft Development center in NC.

  • I know the answer (it's 42)

    Reading too many blogs?

    • 0 Comments

    I am staying in the Mariott Residence Inn in 29Pl off 148th Av. I was driving down along with Neeraja on 148th and suddenly saw a house. It looked a bit odd as it was the only house among office buildings and also looked familiar. Suddenly I turned to Neeraja and told her. Do you know why this house sits among office buildings? Obviously she didn't because she doesn't read Raymond's blog.

    Later what struck me as real odd was that not only I read those blogs I actually remember small things like pictures from it!! Kind of weird....

  • I know the answer (it's 42)

    Sometimes the tool becomes more important than the work

    • 0 Comments

    Question: Which microsoft product did the Canadian Customs official quarantine.?
    Answer: Microsoft mouse!!!! This is a true story.

    The obvious thing I did after coming to Redmond is to rush down to the company store. I was looking for a wireless mouse for my notebook and found one. I narrowed down to Wireless Notebook Laser Mouse 6000.

    I assumed most of the fancy words on the box like high-definition, laser and unprecedented control are hogwash. That was until I used it. This is the best mouse I've ever used. The smoothness of the tilt-wheel was amazing so was the precision in the movement. Each time I use the mouse I feel good. Its like having an amazing lawn-mower, even when the grass is short you want to mow them!!!

Page 1 of 1 (5 items)