I recently built a very simple C# app which uses the Windows 7 native-code UI Automation (UIA) API to gather text from a Windows Live Mail window, as part of having that text spoken. Sometimes having the text spoken can make text in a mail being read easier to understand, or to detect issues with mail being composed. The app used the UIA TextPattern to gather the text, but did so with a single call to collect all the text in the mail. The app might be useful as it is in some situations, but I've just updated it to leverage some more of the TextPattern and TextRange features.

As it happens, with UIA's TextPattern and TextRange, you can still do a lot more to manipulate and traverse text that my updated app does, but the updated sample introduces the use of methods such as:

  • GetBoundingRectangles() - Get all the bounding rectangles for the text in a range.
  • GetSelection() - Get only the text that's selected in the mail.
  • Move() - Move an entire range forward or background by some unit, (for example, one paragraph).
  • MoveEndpointByRange() - Move one end of a range to be equal to the start or end of another range (or even the same range.)
  • MoveEndpointByUnit() - Move one end of a range by some amount, (for example, by one paragraph).
  • ScrollIntoView() - Tell the provider app to scroll its view, such that text is actually visible on the screen.

 

The full set of functionality for TextPattern and TextRange is available at http://msdn.microsoft.com/en-us/library/windows/desktop/ee696214(v=vs.85).aspx and http://msdn.microsoft.com/en-us/library/windows/desktop/ee696221(v=vs.85).aspx.

The comments in the sample app show why and how the above calls are made. I should say that I've not done a great deal of testing with this app, but I'm sure the UIA related code is working great, which is what the sample's really all about. It's worth pointing out that when working with TextPattern and TextRange, your client app can only use what the provider app supports. If the provider has chosen to only partially implement TextPattern, then there's nothing you or UIA can do to add the missing functionality. Rather, you make the best of what the provider does support.

This is what you can do with the app once you’ve opened a Windows Live Mail window to read or compose a mail…

  • To read the whole mail, just hit the Play button.
  • Whenever text is being spoken, pressing the Stop button will restore the app to its initial state.
  • Whenever text is being spoken, pressing the Play button again will stop the speech in such a way that pressing Play again will start speaking the same text again.
  • To start reading from the current selection, (or where the caret is), check the "Start reading the mail from the selection" radio button, and press the Play button.
  • To have some visual highlight of the paragraph being read, check the "Highlight the text being read paragraph by paragraph" checkbox. For this sample app, I reused a magnification class I created for another sample, but you could choose to highlight the text in some other way. (The sample app assumes the Windows Live Mail window is not obscured by anything, and the magnified text fits on the screen. I've not looked into a black flash that can sometimes appear in the magnification lens when the magnification is refreshed.)
  • If you’ve checked the "Highlight the text being read paragraph by paragraph" checkbox, then when the text is being read, you can jump to the next or previous paragraph by clicking the Next and Previous button respectively.

 

 

So while I've only leveraged a subset of what the TextPattern and TextRange can do, the updated app still can do some very useful things.

The updated sample's out at http://code.msdn.microsoft.com/Windows-7-UI-Automation-9ce18fd5.

Finally - if you happen to know of someone who might find the sample app a useful tool in practice for working with e-mail, but it needs a few tweaks before it'll really work well for them, let me know. I'd be happy to try to help you use UIA to extend the functionality further in order to turn the app into a working tool for someone.

Thanks,

Guy