Hi, over the last six months I’ve been in contact with a developer who builds Assistive Technology (AT) apps which use the Windows 8 UI Automation (UIA) client API. It’s been really exciting for me to see the results of his great work, as those results will be extremely helpful to his customers. It’s also been very interesting for me to see him focus so much on the performance of his app, as that’s an area where UIA’s caching functionality can be a big help.

Given that some of the discussions we’ve had during this time might be of interest to others building AT apps with UIA, with the dev’s permissions I’ve included some of the more general interaction below.

I hope this can help others build lots of exciting UIA client apps too! 

Guy

 

Building a cache

Q. I'm using the unmanaged COM API and building an element array using FindAllBuildCache. I trigger the build whenever the active window changes using SetWindowEventHook implemented in C#.  It works well but builds the element array immediately the new window takes focus, I've been using a delay before the build of the elements array to make sure everything is painted on the window but was trying to find a more reliable window loaded type of event.  Just wonder what technique you would recommend to do this?  So as an example if Windows Explorer becomes the new active window how do I ensure all the UI elements are loaded before building my element array?

 

A. I would say that it's good that you're using the COM UIA API, rather than the managed API that comes with the .NET Framework. The latter can be handy in some situations, (particularly for devs who aren't familiar with COM), but it can't access some of the data that is accessible through the COM API. It's also good that you're using UIA's caching, given that caching provides a better performance.

The thing that complicates your scenario is that different features will behave differently. They'll present all sorts of different UI, and will use different UI frameworks, eg Win32, WinForms, WPF, and Windows Store apps add more to the mix. There's no single event that you can react to which will indicate that all the UI you're interested in is now on the screen. Browsers will make this even more complicated, because when you navigate to a different page or place on a page in a browser, you might only get a bunch of UI restructuring for the hosted HTML, and no window-related events.

In this situation, I find it interesting to use the AccEvent tool that comes with the Windows SDK. It's not very intuitive to use, but it can be helpful. It can be set up to report UIA events coming from all UI, or just from a particular window. So as a test, I pointed it to an existing (but minimized) Windows Explorer window, and then restored the window such that it was the active window. (AccEvent has no Select All and Clear All when picking which events you want to listen for, which is frustrating.)

This was the output: 

UIA:PropertyEvent [Name = "Address: Computer"] Sender: ControlType:UIA_ToolBarControlTypeId (0xC365), Name:"Address: Computer"

UIA:StructureEvent [StructureChangeType_ChildrenInvalidated = TBI<rtid>] Sender: ControlType:UIA_ProgressBarControlTypeId (0xC35C), Name:""

UIA:StructureEvent [StructureChangeType_ChildrenInvalidated = TBI<rtid>] Sender: ControlType:UIA_ProgressBarControlTypeId (0xC35C), Name:""

(I've removed lots of structure changed events and bounding rect changed events here…)

UIA:PropertyEvent [BoundingRectangle = {l:141 t:682 r:1119 b:1091}] Sender: ControlType:UIA_WindowControlTypeId (0xC370), Name:"Computer"

UIA:FocusEvent Sender: ControlType:UIA_ListItemControlTypeId (0xC357), Name:"OS (C:)"

 

So in this case, the Window Explorer window raised lots of events relating to its contents being restructured and moving around, (which is what you'd expect when the window's unminimized,) and followed all that with a single focus changed event. There's no event after that saying the window UI has stabilized, and there's no guarantee that at the actual point you get the focus changed event, a feature has really completed the redisplay of its UI. When I tried a similar thing in Internet Explorer, I again got lots of structure changed and bounding rect changed events, with a focus changed event near the end of the list. (I also got events for the text changing in a search field, and a selection changing on the browsers tabs, but that doesn't help us here.)

There are a couple of documented UIA events that sound like they may be of interest, (ie UIA_LayoutInvalidatedEventId and UIA_AsyncContentLoadedEventId), but in practice these aren't often raised, so aren't useful to you.

So all in all, I don't think there's much more you can do than use the delay you already have. If it's very quick to gather the data you need, perhaps you could reduce the delay if it feels too long, and if you find you get more structured changed and bounding rect changed events after you've gathered the data, gather it all up again to account for the changes and re-present it.

 

Q. I used the COM API because I was having speed issues with the managed API although some of it may have been down to threading in the early days of playing with the API.  I am building the cache on change of active window again for speed issues so that when the user command is called I can display the UI pretty instantly.  If I don't build the cache until the user command event then it can, for example with Internet Explorer, take a couple of seconds to respond although with most UIs is pretty instant. 

You mentioned updating the Cache on bounding rectangle changes and the likes and I notice there is a BuildUpdatedCache method which you need to run on each element.  So if with my elements array I wanted to update my cacheRequest then I have to iterate through the array and update each element presuming it still exists on the window?  I can’t quite work that one out so I have basically just been running FindAllBuildCache on the various window changing events. 

 

A. Hi, as it happens, It don’t think I’ve ever called BuildUpdatedCache(). I wasn’t avoiding it, but I think whenever I wanted to refresh the data which I was presenting to the user, I called FindAllBuildCache() again, as you’re doing. In the case of an array of elements as you have, I’ll bet that’s definitely the best thing to do. When it comes to performance in things like this, it’s all about avoiding cross-process calls. So if you can make a call which involves (say) one cross-proc call, gathers a load of data once it’s in the other process, and then returns , that’s going to be a lot faster than making a load of cross-proc calls which gather some data in each call. BuildUpdatedCache() is going to involve at least one cross-proc call each time, so if you call it for each element in an array, that’ll be slow. Calling FindAllBuildCache() will make far fewer cross-proc calls, (sometimes only one.) So I would imagine BuildUpdatedCache()  is handy when you’re interested in a particular element. (By the way, if you’re making repeated calls to FindAllBuildCache(), you’ll want to release objects from earlier calls that you’re finished with. I seem to remember it not being obvious as to when some of these objects would be released by UIA and when you needed to explicitly release them, but my samples will demonstrate what I thought was the required action.)

 

Events

Q. I am trying to implement a bounding rectangle property changed event for any of the descendant elements on a relevant window, e.g. Internet Explorer.  If I build my cache using TreeScope.Descendants I can display my rectangles as shown below.  I am then trying to handle a bounding rectangle property changed event for any of the descendants and seem to be getting sporadic results of when it is triggered when it is not.  Forgetting about threading for now, code is something like this, should this record a change to any of the bounding rectangles of the descendants are am I, as is most likely the case, not implementing correctly? 

The "activeWindow" is just the element for the current active window, in this case:

Internet Explorer (activeWindow = uia.ElementFromHandleBuildCache(Win32API.GetForegroundWindow(), myCache);

  

int [] propArray = {UIA_PropertyIds.UIA_BoundingRectanglePropertyId};

uia.AddPropertyChangedEventHandler(activeWindow, TreeScope.TreeScope_Descendants,

      myCache, new PropertyChangedEventHandler(activeWindow, win), propArray);

class PropertyChangedEventHandler : IUIAutomationPropertyChangedEventHandler
{
    numberedWindow myWindow;

    public PropertyChangedEventHandler(IUIAutomationElement sender, numberedWindow win)
    {
        myWindow = win;
    }

    public void HandlePropertyChangedEvent(IUIAutomationElement elem, int propertyID, object obj)
    {
        if (propertyID == UIA_PropertyIds.UIA_BoundingRectanglePropertyId)
            MessageBox.Show("Property changed", "", MessageBoxButton.OK, MessageBoxImage.Error);
    }
}

 

When I say behaving sporadically, as an example if I zoomed in on the window below the event would be triggered but with other webpages it isn't and if I scroll down the page it isn't.  I am presuming if I scroll down the page that the bounding rectangle property should change.  I basically think the problem is that the event is not being fired for all the bounding rectangle changes of the descendants.

 

A. I wouldn't assume there's anything not correct in your implementation. The Windows UIA API is a powerful tool, but every now and again some unexpected behavior is unearthed. It's always possible that UIA has some issue, and all we can do is try to find a workaround.

In the case you've described, it is interesting that it seems that sometimes you get the events and sometimes you don't. As you point out, one very relevant question is whether it's possible that the UI isn't firing the events you'd expect. The first thing I do in this situation is run the AccEvent SDK tool. That tool takes some getting used to, but it can be really helpful. If you have the Windows SDK on your machine, you'll have accevent.exe somewhere.

If you run that tool, go to the Mode menu and verify that the "UIA events" item is checked. (We're only ever interested in UIA events now.) Then click to the Settings item and see the AccEvent settings. In the "Event Type" list clear everything but the "Bounding Rectangle" property changed event. (There's no Clear All and Select All button here, so changing the options can be a real pain.) Then in the "Scope" list, click the top level browser window, (assuming the browser's already running.) Then in the "Report properties" list, have a small number of useful things selected, (such and Name, Control Type and BoundingRectangle.) Then click OK. Once back in the main AccEvent UI, go to the Events menu, and click "Start Listening".

AccEvent will now listen for UIA events, and report what it gets. It doesn't using caching, (ie it'll go back to the source process to get the properties of the element,) but it's pretty reliable. If it doesn't report some events, I always take that to mean the events aren't being raised. I just tried this with IE, and when a page was loaded, I saw some BoundingRectangle property changed events. However, when I scroll down, I see no events raised. On another page however, I did see some events generated when I scroll. (Actually, I think events those events might be useless in this case.) So you might want to give this a go, and see if the events you expect are really being generated. At the moment, I'm rather assuming you won't see those events.

If the source UI isn't generated the events you want, there's no way you can force the events to appear. Whether events are generated is entirely up to the source app or the framework that the UI's built in. Neither UIA nor your client app can affect that. So it's then a case of figuring out whether there's some workaround you can find, perhaps in the form of reacting to other events that get generated in the particular scenario. So what I then did in this situation is have AccEvent look for many more events, in case there's anything that I can use. I just gave this a go, and found that I still not got useful events.

This makes things more complicated, as it looks like there are no useful events being generated in the scroll scenarios. I know what I'm about to say sounds ugly, but one way to account for this would be to use a low level keyboard hook and mouse hook to detect certain key and mouse action. For example, if after certain key presses, (eg arrows, page up/down, home/end), or left mouse button down, assume that the UI shown on the screen might have changed and after a slight delay to allow the UI to settle down, refresh your own UI which sits on top of everything. Far from ideal, but sometimes if apps you care about don't generate the events you want, you need some fairly brute force workarounds.

 

A mix of perf and inaccessible UI

Q. I was trying to work out the best way of dealing with the situation:
 
1. The active window has no UIAutomation API

2. It simply takes too long to build the hierarchy of elements.

With the first option it's not too difficult because effectively I just get an empty array of elements pretty quick when running ElementFromHandleBuildCache.

However with the other scenario I'm not too sure how to abandon the cache build after a specific amount of time other than using some pretty crude devices. I also expect there will be some other applications where building the element tree is not possible in a realistic amount of time.

 

A. When you say that the active window has no UIA support, I assume that you can get a UIA element from the main hwnd for the app? UIA should always provide you with that. But I expect the problem is that various UI shown inside the hwnd doesn't seem to have UIA elements associated with it. If so, what happens if you point the Inspect SDK tool at the UI for which you're not getting UIA elements? If Inspect also can't see it, maybe this is some custom UI and the people who built the UI didn't add an accessibility support into it, (either in the form of UIA or MSAA.)

I don't have a great solution for the UIA calls simply taking too long. (On a side note, the Windows 8 UIA has some support for timing out long calls into UIA. That was really added because if the target app has hung for good, then a Windows 7 UIA call could block forever.)

I expect you've already constrained your cache to be as slimmed-down at it can be. Eg only requesting things that support the InvokePattern and are not offscreen. It's possible that some apps are very slow to respond regardless of what's in the cache requests.

If you point Inspect to some of the slow apps you encounter, I wonder what it typically reports. Eg is it because the UIA tree is really, really deep or wide? Or maybe the tree is corrupt.

 

Q. Yes I can always get the main window as a UIA element:

rootWindow = uia.ElementFromHandleBuildCache(Win32API.GetForegroundWindow(), myCache);

 

I am already restricting it to invoke pattern elements that are visible and when using Inspect it clearly shows the applications with no UIA support (i.e. no elements are picked up on the UI) and using it with the particularly slow app shows a fairly detailed element tree. I was more concerned with applications that I don't have and haven't come across that users of my application might come across and I was trying to find a generic way to timeout if any sort of issue was hanging the cache build or making it take an excessively long time.

 

A. For apps that support MSAA rather than UIA natively, UIA uses its own "MSAA Proxy" to talk to the app. The MSAA proxy takes the MSAA data provided by the app and converts it as best it can to UIA data, and returns that to the UIA client. Converting things like Name properties is straightforward, but converting the MSAA element hierarchy to UIA such that the navigation calls work is pretty complicated. Inspect will report what the provider is when you point it to an element, and if you point it to Internet Explorer, you'll see "MSAA Proxy". Internet Explorer isn't a native UIA provider.

So in general if the source app is either an MSAA or UIA provider, a client can in theory get to the UI. But if the app has not included some piece of the UI in the accessible element tree, there's really nothing UIA can do about it. UIA's a channel through which an app can expose its UI to Assistive Technology (AT) clients. In the past I've effectively been asked how a client app can use UIA to make an inaccessible app accessible, (and the answer is - it can't). In some cases if the client really wants to, it can make serious assumptions about specific inaccessible apps, like where UI is relative to the top left corner of the app window say. Eg I remember some popular app which could occasionally slap some completely inaccessible UI in the middle of the app window when it started. There's no way to programmatically invoke that UI unless you assume that you know where it is. While doing this sort of thing is undesirable, if a customer of yours relies on some inaccessible app, there may be no alternative. (Obviously you know all this already, having built client apps for so long.)

 

Q. After you mentioned the invoke pattern and visible elements in your last email I actually went back through my various property conditions that I had built the cache with and found that one of the edit property conditions I had set wasn't restricted to on-screen or enabled for that matter. That particular setting was causing much of the delay with the slow app. Anyway at that point I decided to go back to scratch and build the cache properties independently for each of the popular applications/groups of application I would be using. My application now checks the process name of the active window and uses an appropriate property structure for each of the different applications/group of applications. When I say appropriate I mean a set of properties which allow access as fast as I can get it without missing out any elements. I started with a simple AND property condition which was invoke pattern available, enabled and on-screen and took it from there.

As a simple example searching on those three conditions brings up pretty much all of the elements with Shell based UI screens (Windows Explorer, programs and features, control panel etc) but with the likes of Internet Explorer misses out quite a few elements. Combo boxes, edit-based elements like search boxes etc. Anyway I managed to get much more appropriate property searches for the different groups of apps and have got pretty much instantaneous response on most interfaces where Internet Explorer is probably the slowest but it's down to something like 1.5 to 2 seconds maximum for the most complicated HTML pages which is more than acceptable. With most webpages it is pretty instantaneous. I can also build the cache on change of active window and a few other tricks to speed up even more.

I find it best to change the cache request properties with different applications not only for quicker searches in some instances but to stop duplicate rectangles on various items, particularly Internet Explorer. Quite often as an example hyperlinks can have text control links as children leading to small rectangles inside large rectangles duplicating functionality making for a messy display. I could write methods to remove child rectangles inside parent rectangles and other ancestors but it's easier just to set a particular set of properties with different applications I find.

A couple of other questions if I may as it will probably save me some time when I start doing the clicking of rectangles. I can obviously invoke the elements with invoke patterns but I also see there is the method getClickablePoint () and I can also simply work out mouse coordinates from the bounding rectangles. After a quick search I can’t actually see any way of doing right clicks so I presume I can just use the Windows API mouse events for right clicking using whatever method to find the best coordinates.


 
A. You're right that UIA does not support telling an element to perform its right-click action, (which often would result in a context menu appearing.) I think the only way you can do this is to do exactly what you're considering. Get a point which you feel lies over the element, and use SendInput() to simulate a right click. Note that this could be more complicated if you're presenting your own UI over the element at the time. If you have some area of your own window that's not 100% transparent beneath the right click, your UI will get the click, not the target window. For some reason I seem to feel that not all UI does such a good job at implementing getClickablePoint as they do the BoundingRect property. (But I don't remember why I think that, so I might be wrong.) Some point inside the BoundingRect should be fine to click on, so long at the element is rectangular.

 

Response. It seems to work fine with either a midpoint on the bounding rectangle or using getClickablePoint (). Interestingly getClickablePoint() adds a little bit of latency to creating the window with the rectangles so I am going with the bounding rectangle midpoint.

 

Context Menus

Q. Guy have you dealt with UI automation in terms of context menus and other menus for that matter? So as an example if I was running Internet Explorer and opened the Favourites menu how would I know it had been opened and how can I then show UI over the menu items?

In fact most menus are accessible but for some reason with Internet Explorer the menus, e.g. Favourites, seem to be descendants of the desktop rather than the Internet explorer window.

 

A. I've just pointed the Inspect SDK tool at an IE context menu, and as you say, its parent is the desktop element. This isn't something you can influence, and so it's a case of working with that UI as it is. I assume the issue here is that you're specifically listening for events being raised by the foreground window or its descendant elements, and in the case of the content window, those events aren't being detected.

As a test I ran the Accevent SDK tool. I went to the settings UI, and said that I want to see events raised by the direct children, (not all descendants), of the desktop element. And I only want to detect the MenuOpened event, (and no other events). When I did that, and went to IE and right-clicked, I could see a MenuOpened event. I could also see that the accessible name of the element raising the event was "Context". I then went back to the Accevent tool, and changed the settings to report the process id of the element that raised the event, and I could see that the process was one of the iexplore.exe processes that I could see listed in Task Manager.

So if you want to be able to detect a context menu being opened while IE is the app that the user's in, perhaps when you register for the events you're listening to now, you also need to register for MenuOpened events from elements which are direct children of the desktop element. 

I notice that when invoke the Favourites menu in IE, I didn't get the MenuOpened event. But I did get a FocusChanged event, so maybe that would be one way of detecting that the Favourites menu has been invoked. 

 

Caching and properties

Q. Guy can you please explain the following. If I set up my cache as shown below you will see that one of the conditions included in the simplified overall condition (overallCondition) for FindAllBuildCache () is a ClassNamePropertyID (nameMessagePanelCondition) and I have not added those type of UIA_PropertyIds to the cache request yet it still gives me a CachedBoundingRectangle for the element that meets the nameMessagePanelCondition. I would have thought I would have had to include ClassNamePropertyIDs to the cache request as in:

myCache.AddProperty(UIA_PropertyIds.UIA_ClassNamePropertyId);

 

However I have not had to do this. Also any comments on this structure generally as seen below. To simplify the question I am only using a tree item condition and the class name condition in the overall condition.

//create CacheRequest
myCache = uia.CreateCacheRequest();                                             
myCache.AddProperty(UIA_PropertyIds.UIA_ControlTypePropertyId);
myCache.AddProperty(UIA_PropertyIds.UIA_BoundingRectanglePropertyId);

//all elements must be enabled and on screen
enabledCondition = uia.CreatePropertyCondition(UIA_PropertyIds.UIA_IsEnabledPropertyId, true);                                    
onScreenCondition = uia.CreatePropertyCondition(UIA_PropertyIds.UIA_IsOffscreenPropertyId, false);
enabledAndOnScreen = uia.CreateAndCondition(enabledCondition, onScreenCondition);

// Add a Property Condition which is a Control Type Property, included in cache
treeItemCondition = uia.CreatePropertyCondition(UIA_PropertyIds.UIA_ControlTypePropertyId, UIA_ControlTypeIds.UIA_TreeItemControlTypeId);

treeItemCondition = uia.CreateAndCondition(treeItemCondition, enabledAndOnScreen);

// Add a Property Condition which is a Class Name Property, NOT included in cache
//condition for Outlook 2007 message panel           

nameMessagePanelCondition = uia.CreatePropertyCondition(UIA_PropertyIds.UIA_ClassNamePropertyId, "SUPERGRID");      

// Create overall condition
overallCondition = uia.CreateAndCondition(treeItemCondition, nameMessageReadCondition);  

rootWindow = uia.ElementFromHandleBuildCache(Win32API.GetForegroundWindow(), myCache);

//set scope of cache                             
myCache.TreeScope = TreeScope.TreeScope_Element | TreeScope.TreeScope_Children;          

//populate the element array
elementArray = rootWindow.FindAllBuildCache(TreeScope.TreeScope_Descendants, overallCondition, myCache);  

//so this piece of pseudocode is where i happens to be the element that meets the nameMessageReadCondition

elementArray.GetElement (i).CachedBoundingRectangle   … This is still a valid cache condition, how come??

 

Although the element array is built using FindAllBuildCache any element from the array still gives me access to the "current" conditions as shown below. So does this mean that all the current properties are populated regardless of the element array being built with the CacheRequest?

elementArray.GetElement (i).CurrentName  

 

A. So as I read the code…

The myCache object is set up for the ControlType and the BoundingRect. This is used in two places later. The first is in the call to ElementFromHandleBuildCache() to get the element for the foreground window. And the second is in the call to FindAllBuildCache(), when certain descendants of the foreground window element are accessed.

Almost all the rest of the code relates to building up the condition to be supplied to the FindAllBuildCache() call. The condition is that the element’s enabled, not offscreen, a treeitem control, and has a “SUPERGRID” classname.

You question is, why were the properties involved with the condition not required to be set in the cache request? This is what I think the situation is…

The condition and the cache are independent of each other. The condition only relates to decisions around which elements are to be returned from the function call. You could pass in your overall condition with no cache at all, and still get the same elements back.

The cache is used to allow faster access to the data you want to access after you’ve made the call which got you the elements. Say some call to FindAllBuildCache() got you 100 elements, and you then need the bounding rects of all those elements. If you didn’t use a cache, you’d get the current bounding rects for all the elements in turn, and that would mean you’d be making 100 cross-process calls. Cross-process calls are slow, and so that hurts the perf of your app. So you create a cache request for the bounding rects. This means while UIA is gathering all the elements the FindAllBuildCache() call, it’s also collecting all the bounding rects. It’s doing that during the one cross-proc call where the elements are being collected, and so the work to get the bounding rects results in no more cross-proc calls. Later when you get the cache bounding rect element, it’s returning the cached value. Technically that cache value might be stale, but often a client app would prefer to live with that issue rather than incurring slow cross-proc calls.

Now, your other question is also an interesting one. That is, how come the CurrentName property works even though a cache request was used when getting the elements? You’d set up a cache request for the ControlType and BoundingRect, and that means you’re not making a cross-proc call when getting those cached properties. But by default, UIA still allows you to get any current property you want on the element. You can get the current name, or even the current bounding rect, (not that you’d do that, given that it’s already cached). But if you do get a current property, you’re incurring the perf hit of a cross proc call, and only you can decide whether that’s acceptable. If you think it’s pretty rare for you to want to get a current name, but once in a while it happens, you could get the current name at that time. But if you feel that it’s likely that you’ll need the current name for a few items, perhaps you’d add the name property to your cache request.

To get the current property, all you need is a full reference to the element. (That is, a reference that still connects to some element in the provider process.) The cache request contains something called AutomationElementMode. By default that’s set to AutomationElementMode_Full, which means you get a full reference. If you’d set the mode to AutomationElementMode_None in the cache request, then all you get is a way to get the cached properties that you’d requested. If you try to get current properties in that case, I’d expect the calls to fail.

By the way, the cache request itself also has a scope and condition. I think this can get pretty confusing, and so wouldn't necessarily change your code. If your code works, is understandable and maintainable, and perf is acceptable, then that’s good. But technically I wonder if the following would work…

1. Set the cache’s TreeFilter to the overallCondition.
2. Set the cache’s TreeScope to descendants.
3. Make only one call to ElementFromHandleBuildCache().
4. Call GetCachedChildren() to get the descendants that meet the condition.
5. Access the cached properties from those children.

I did something like this as an “alternate approach” at the end of LinkProcess.cs up at http://code.msdn.microsoft.com/Windows-7-UI-Automation-0625f55e/sourcecode?fileId=21468&pathId=534397530. But like I say, there may be no real advantage for doing this over what you already have, (and could have behaviours I don’t know about), so I’d stick with whatever works.

 

Timeouts interacting with unresponsive providers

Q. When I capture the elements on the HTML document within a particular browser I am getting a 4 seconds delay with some densely populated HTML pages but this only happens with Windows 7 and not Windows 8 where it is virtually instant. 

 

A. Regarding the difference in performance of the UIA API between Windows 7 and Windows 8, one possible reason for this relates to timeout functionality that was added to UIA in Windows 8. In Windows 7, when you made a call to a UIA client API, the call could block until the provider process finished responding to the call. So if the provider was slow, then your client app might be unresponsive for a while. Worse, if the provider hung permanently, then so might your client app. Obviously that's a big problem for your customer.

So in Windows 8, UIA was enhanced such that the client APIs automatically have timeouts associated with them. The intent is that if a provider process does not respond to some call within a certain time, then UIA will stop waiting and return control to the client. In that case, the client API should return an error code of UIA_E_TIMEOUT, but I don't know if that's always guaranteed.

There are two timeout settings. The first relates to establishing contact with a provider. A provider should be able to respond to this very quickly indeed, and so if there's even a small delay in this happening, that suggests that the provider is unresponsive. So the default timeout for that action is only two seconds. The other timeout relates to gathering information from the provider after contact with the provider has been established. Some operations will involve processing many hundreds of UI elements, (and this is likely the case in your situation when you're working with a browser), so it could take a while for the provider to gather all the data that's being requested. So by default the timeout for that action is 20 seconds.

The client app can configure these timeouts. If the defaults aren't optimized for the apps you're working with or the operations you do, the properties you'd set for configuring the timeouts are IUIAutomation2::ConnectionTimeout and IUIAutomation2::TransactionTimeout. (All the new functionality added for Windows 8 is listed up at http://msdn.microsoft.com/en-us/library/windows/desktop/hh437316(v=VS.85).aspx.)

Now having said all that, sometimes the actual timeouts your client app experiences may not be exactly 2 seconds or 20 seconds. Depending on the action being performed, sometimes the timeouts can get compounded internally because UIA might have to make multiple interactions with the provider beneath a single call to a client API. That can result in the client experiencing a longer timeout than the published default values.

But despite there sometimes being some variation in the actual timeout experienced, (and I believe it being possible for UIA_E_TIMEOUT to not be returned after a timeout has occurred,) the addition of the timeout functionality for the client APIs is a very important enhancement to UIA in Windows 8. If a provider process is hung, the client app must be able to continue to serve its customer.

I don't know if this can explain the differences you're seeing. I would have thought it might be relevant if you'd seen your Windows 8 client API call return after 2 seconds, but you've said that the call is virtually instant. Also, if the timeout had occurred, I wouldn't have expected you to get all the data you're after in that case. A timeout means that the provider didn’t supply you with the data you need in a responsive manner. I'm assuming that you did get the data you need. But I'm afraid I can’t really think of another reason for this difference in behavior between Windows 7 and Windows 8.

 

 

Addendum

Following the discussion below around finding items in a context menu, I pointed Inspect to the context menu that appears when I right-click on the desktop. After I'd arrowed through the menu, I pressed Ctrl+Shift+5 to refresh the Inspect UI. Once I'd done that, I could see the menu items in Inspect, as shown here...