Joe Calev's WebLog

So what's happening with this blog!?

Over the past several months I have come to realize that I needed to change the way I blog.  The primary issue is that everything that comes from this blog is often interpreted as coming from Microsoft itself.  This has created a number of challenges and problems recently.

Back in January, I created a new blog called "Photographing the Earth. One millimeter at a time..." that covers my interests in photography.  I often felt with this blog that I had two audiences - those interested in photography and those interested in Office Communications Server and Speech Server.  The two audiences cared little for the others' posts.  This new blog has been very active and I have posted there every day.

This leaves me now with the issue of what to do with this blog.  For the most part, I would like to keep it on topic with OCS material, but most of that material I cannot blog about right now.  When we do release something public, you can expect to find more information here, but in the meantime I don't expect to have the time to write much more that is OCS specific.

Currently in my spare time I have been learning more of the technical aspects of photography.  This is beyond reading photography books and more towards understanding "what is light?" and learning the algorithms behind image processing.  I am still at the beginning stage here, but do intend to blog about this more as my knowledge increases.  As this is quite technical material, I may place it here or at my other blog - I haven't decided yet.

Therefore I just wanted to let everyone know that this blog has not falled off the face of the Earth, but it will be awhile before anything much shows up here again.

Relatively, things aren't all that bad

It is pretty much impossible right now to not see the news and get a little depressed.  While Microsoft is a healthy company and work life has been good here, all is not rosy in the economy in general.  I now feel like a pauper when I travel to Europe as the dollar picture isn't looking good, the real estate market is hurting (which hurts us directly because my wife is a real estate agent), and the latest news is the economy is heading for a recession.  However, things are still not that bad.

Yesterday my wife came home and announced that the nephew of one of the agents in her office died.  He was four years old and was killed in a sledding accident of all things.  You can read more details about it here.  As a father of a four year old and a three year old who desperately want to go sledding my heart went out to the parents.  I simply cannot imagine such a horrible thing happening and I don't know how I could manage to live day to day if something happened to one of our boys.

So, although some people already have or will lose their homes, jobs, and money in the next several months, things really aren't all that bad.  I'm sure these poor parents would glady have traded their home, jobs, and entire life savings right now.

I spent some time researching sled safety, as my kids still want to go sledding.  The following are general safety tips you should stick to when sledding.

1) Do not sled where there are any obstacles.  The hill should not have a road  or body of water below and there should be no buildings or trees that you have a remote possibility of hitting.

2) At least if the child is young, have him or her wear a helmet.

3) Give the child instruction on how to turn and stop the sled.  Make sure the child understands how to do this.

4) Avoid inner tube, saucer, and disk sleds which have no control.  Ideally, use a sled with a steering mechanism.

5) Supervise your children at all times while they are sledding.

6) Do not sled in areas where there are protruding rocks or where the area is rocky in general.

7) Never sled in the dark.

8) Never pull a sled with a motor vehicle.

9) Never sled head first.  Always sit face forward.

10) Do not use plastic sheets to sled as they can be punctured from objects under the snow.

Office Communications Server Resource Kit ships

OK, this may be old news to many of you out there, but the Office Communications Server 2007 Resource Kit just shipped.  I haven't had a chance to look at it yet but you can be sure I will.  I hope to receive a free copy of it at an internal event next month, at which time I also hope to sign up to contribute to the next resource kit.  As there still aren't a lot of resources out there on OCS 2007 yet, I expect this to be on a lot of bookshelves.  Actually, after looking at Amazon.com there is only one other book out there - though I suspect several more are on the way.

Personally, I think it's about time that a number of books be released on different aspects of OCS 2007.  For instance, there could be some books on setup and deployment and other books on developing applications with UCMA and UCCA.  Time will only tell.

All types of fun with marshaling!

There are several golden rules that I have learned over the last week while spending some time PInvoking Win32 functions from C#.

1) The content at PInvoke.net can often not be trusted.

2) The overwhelming precentage of articles on the net explaining how to PInvoke certain methods can also not be trusted.

The following are some helpful tricks and tips that I have learned.

Pay attention to whether the Win32 method returns BOOL or BOOLEAN.

The Win32 BOOL is 4 bytes, while a BOOLEAN is only one byte.  Why do we care?  We care because the .Net Boolean only maps cleanly to BOOL, not to BOOLEAN.  The following is an example of how to marshal the TranslateName function.

[return:MarshalAs(UnmanagedType.U1)]

[DllImport("Secur32.dll", CharSet = CharSet.Unicode, SetLastError = true)]

private static extern Boolean TranslateName(

String lpAccountName,

EXTENDED_NAME_FORMAT AccountNameFormat,

EXTENDED_NAME_FORMAT DesiredNameFormat,

StringBuilder lpTranslatedName,

ref uint nSize);

Notice in particular the return: attribute specified above.  This allows us to marshal back the returned BOOLEAN to the .Net Boolean.  If you do not do this, I have noticed that the function will actually fail but will still to appear to return true.   This can be very frustrating to debug.

When using EntryPoint, make sure the specify the ‘W’ version.

[DllImport("advapi32.dll", SetLastError = true, EntryPoint = "ChangeServiceConfig2W")]

private static extern bool ChangeServiceConfigDescription(

      IntPtr hService,

      uint dwInfoLevel,

      ref SERVICE_DESCRIPTION lpInfo);

This makes sense, but it is easy to forget.  If I do not use the W version above, I will have only one character in the description.

If a string must be passed in a passed structure, it must be passed using an IntPtr.

See the definition of SERVICE_DESCRIPTION used above.

[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Unicode)]

private struct SERVICE_DESCRIPTION

{

    public IntPtr lpDescription;

}

To copy your managed string to the structure, use Marshal.StringToHGlobalUni.  Just make sure to call Marshal.FreeHGlobal to free the memory once you no longer need it.  The same applies when retrieving a string from a structure, except there you must use Marshal.PtrToStringUni.

Overengineering overly simplistic solutions

I have noticed through working both at Microsoft and in other places a tendency for the two following practices to occur.  At first, it would seem they contradict each other but in truth I very often see them together.

1) The product is made to fit one kind of ideal user, most often the user who is the least tech savy.  While the team may use multiple personas, almost every feature must still be accessible and understandable to this user.  While this may help in some everyday programs, I find it particularly dangerous in APIs - where a class is designed for only simple uses and when confronted with a more advanced use case becomes completely useless.

2) The underlying mechanisms that make up the product are overly complicated.  In some ways, it's almost like the developers were bummed that the product itself isn't that complicated, so they intentionally make the code that runs it complicated.

Perhaps these are two different issues, but I suspect they are heavily intertwined.  For instance, it is unfortunately a common practice here at Microsoft to not use our own APIs to accomplish things.  While this has improved recently, the main reason developers give is the API doesn't do exactly what they want or doesn't perform as they need.  Here are some nightmare cases that I have actually seen (note: none of these occurred in my current group and to my knowledge this product no longer ships).

a) A development team did not trust the implementation of ATL, so they created their own version from the ground up.  Predictably, it suffered from numerous hard to debug issues.

b) A developer decided to learn more about multiple inheritance in C++, so he implemented the entire feature with it.  When the bugs mounted up, the developer left the team because he was sick of fixing them.  The feature was eventually removed from the product shortly before shipping due to instability.

c) To save a few MB of download, several developers decided to create their own image format to replace the PNG format.  The effort took several months, crippled the product until late in the design cycle on a legacy OS, caused the product to take 10% more memory, and resulted in a 5% performance degredation of the product.

These are plain examples of the code behind a product being unnecessarily complex.  Had the product team used ATL, avoided multiple inheritance, and stuck with the PNG format the code would have been shorter, customers would have seen fewer crashes and other bugs, and the product team would have finished sooner or have been able to add more useful features.  While these examples are extremes, I have seen the same types of issues occur many other times.

Very often, however, more advanced users of our products become very frustrated when they try to do something more difficult.  For example, recently I became very frustrated with the ServiceBase and ServiceInstaller classes that ship in .Net.  First, I needed to be able to install a service myself without the user getting involved.  This is possible, though rather tricky, to do with the ServiceInstaller class.  Eventually I wound up having to PInvoke all of the code because a number of things I needed to set on the service weren't available in ServiceInstaller.  ServiceBase soon suffered a similar fate, as I was playing around with shared services which ServiceBase had poor support for.

I won't delve much more into APIs though because it is truly very hard to get things right there.  From a UI point, I cannot count how many times I have needed to do something in Word and Excel that was hidden in some strange place.  Of course, that's when I was lucky enough to find it.

Note that this should not be construed as Microsoft bashing.  Every company I have ever worked for had an element like this.  I once worked for a now defunct dot com that was creating a web site for the customer.  The web site had to be as simplistic to use as possible - so users would need minimal training.  However the implementation consisted of the team trying to get several different products to work together.  I asked why they had such a strange colloboration of products and the answer was - "we wanted to learn more about them and also see if we can do it".  Unfortunately for them, the products didn't work and their solution never shipped.

Some tricks with service restart logic

Today I will venture outside the safe confines of Office Communications Server and discuss some quirks I have noticed with Windows services.  (Disclaimer: I am not in the Windows org so these are just my observations after some experimentation) Some of you may be aware that in the services control panel, you can right click on a service and on the "Recovery" tab you can manage what occurs when a service fails.

The following are the options.

First failure: what should occur the first time the service fails.  Valid options are "Take No Action", "Restart the Service", "Run a Program", and "Restart the Computer".

Second failure: same options the second time a service fails

Subsequent failures: same options for any subsequent failure

Reset fail count after: the number of days the service must be running before the failure count is reset

Restart service after: the amount of time in minutes to wait to restart the service

This is very nice, but it is very easy to misunderstand what these values actually do.  I have seen a number of services (and I tried this myself) set these values to 0 days and 0 minutes.  The problem is your service will continually restart if you set the failure count to reset after 0 days, if the service at least started correctly.  The result is only the first option ("first failure") will ever be run. 

To fix this, set the failure count to reset after one day.  The drawback to this approach is your service may stay stopped after failing several times but this likely means something is toast anyways.

One thing also to take into account is not all services will work with the reset logic - or in other words just setting the recovery options on any service does not guarantee that it will restart.  In order for the service to restart, it must exit abnormally.  This generally means the service must exist with a non-zero exit code and the service status must not be stopped (note: this has changed for Vista - it is possible to set the service status to stopped and provide an exit code to trigger the restart logic).

How to track down the WMI error in OCS

There are some product architectures created here at Microsoft that I am marvelled at with their simplicity, ease of use, and power.  Unfortunately, I do not find WMI to be one of those architectures.  With hope, some day we will have a sane replacement for WMI but in the meantime we still need to deal with it and the fact that we use it for most settings in Office Communications Server.

Perhaps the most annoying thing I find is that I'm never told exactly what I did wrong when I attempt to set a value or write a query.  Instead, I get a long HRESULT number that, at first glance, tells me absolutely nothing.  In this post, I hope to shine a bit of light on this.

The first thing you should do when confronted with this number is try to look it up under the common WBEM error codes.  A list of the error codes with descriptions can be found here.  You should be able to find your error code there along with a description that may tell you what happened.  Of course, if the error code is WBEM_E_FAILED or is WBEM_E_UNEXPECTED this will not help you very much, except to confirm that there is indeed a problem.  You may also be puzzled if one of your parameters cannot be null but you do not know which one.

From here the best solution is to use OCSLogger.  You can find OCSLogger in \Program Files\Common Files\Office Communications Server 2007\Tracing.  You will need to run it on your front end or home server (depending on the edition).  Under "Logging Options" you should choose "LcsWmi".  Under level, choose "All" and under Flags choose "All Flags".  Leave all other options at their defaults.  Then click start logging and rerun the WMI operation.

While most of the data generated is only readable by Microsoft, you will be able to read the errors and get enough information to likely determine what the problem is.  If you need to follow up with Microsoft on the issue because the logs are not obvious, you can use the same logs to send to Microsoft.

Doing good things while working at MS

Working at Microsoft is really interesting in that I often see my work being used throughout the world.  It was nice seeing the press about the Unified Communications launch, knowing that I helped participate.  Executives here like to mention that we are changing the world and affecting people's daily lives - but the truth is I don't see this very often. 

Sure, Office Communications Server is a ground breaking product.  We are not just changing communications from POTS to VOIP, we are changing the way you communicate.  It is the integration with Office and the APIs we provide that set us apart from competitors like Cisco.  However, let's be realistic here.  I have yet to hear reports of something whose life has improved due to enhanced presence.

Recently though, I have had the honor of being part of a project that did improve others' lives.  Sure, I played only a small part but I hope I still had an impact.  This was through helping one of our TAP program customers - 4PatientCare.  For those of you unfamiliar with our TAP program, many MS products choose specific customers who have interesting product scenarios and partner with them.  The partner receives direct access to the product team and receives early drops of the product.  By the time the product ships, the partner has a fully functioning application running on top of our product.  For Microsoft, this provides important feedback and helps validate the utility of the product.

Anyways, in short, 4PatientCare created a remote monitoring application that regularly calls at risk patients.  If you are interested in more details I suggest you contact them directly, but the important result of the application is that people who would have ordinarily not called a doctor due to unusual symptoms now get the treatment they need.  The application calls these patients at specific times and asks them a series of questions.  Based on the answers to the questions, the system may transfer the patient to a nurse. 

I will not reproduce the statistics here (because I do not remember them), but the numbers are staggering.  The simple fact is, thanks to this system, there will be people who are still alive who otherwise would not be.  This impacts not only these individuals, but everyone around them.

I had the privilege of advising 4PatientCare on how to best make use of Speech Server for this application and it is nice to know that, though I only played a small part, my work will dramatically improve the lives of others for years to come.

Announcing the new blogs!

Several days ago I mentioned that I will be splitting my blogs so that I can focus directly on Microsoft related discussions here and photography on a different blog.  This weekend I have finally started this effort and you can now find my new blog on photography at http://www.calevphoto.com.  This blog will cover my passions of macro photography and travel photography and I hope you will enjoy the pictures I post there and the photography posts.

I have also started up a real estate blog for my wife - http://www.nelyasblog.com.  I believe the information provided there will be very helpful to those looking to buy or sell a home and to those interested in real estate.  Some of the blogs here will be written by my wife, Nelya, while others I will write.

I think over the long run this will be a very good choice because it enables me to do things with my photography blog that were not possible with this blog.  Of course, I still have a huge task in front of me to advertise the new blogs and get views up.

So, for those who enjoyed the photography discussions and pictures I used to post on this blog, please take a look at http://www.calevphoto.com.  For those who keep track of this blog for Microsoft purposes, I hope this change will enable me to keep more on topic.

Where OCS finds data for WMI classes

In yesterday's post, I discussed how knowledge of where the WMI provider retrieves its data is important when querying it.  Today I thought I would provide a list of the WMI classes and where the data comes from for each.  If you perform a lot of WMI querying in OCS, this list may be helpful.

Note that you may also see the following terms used.

Global = Active Directory
Pool = SQL server
Machine = WMI

MSFT_SIPAddressBookURLData - SQL
MSFT_SIPApplicationElement - WMI
MSFT_SIPApplicationPriorityList - WMI
MSFT_SIPApplicationSetting - WMI
MSFT_SIPArchivingServerSetting - Active Directory
MSFT_SIPDomainData - Active Directory
MSTF_SIPEdgeProxySetting - Active Directory
MSFT_SIPESEmSetting - SQL
MSFT_SIPESGlobalRegistrarSetting - Active Directory
MSFT_SIPESGlobalSearchSetting - Active Directory
MSFT_SIPESServerSetting - Active Directory
MSFT_SIPESTrustedServerSetting - Active Directory
MSFT_SIPESUserACEData - DCOM
MSFT_SIPESUserContactData - DCOM
MSFT_SIPESUserContactGroupData - DCOM
MSFT_SIPESUserSetting - Active Directory
MSFT_SIPFederationAuthorizedForwardingProxyData - WMI
MSFT_SIPFederationAutoFedAllowedDomainData - WMI
MSFT_SIPFederationDeniedDomainSetting - WMI
MSFT_SIPFederationDirectPartnerTable - WMI
MSFT_SIPFederationExternalEdgeListeningAddressSetting - WMI
MSFT_SIPFederationExternalEdgeListeningPortData - WMI
MSFT_SIPFederationExternalEdgeSetting - WMI
MSFT_SIPFederationInternalDomainData - WMI
MSFT_SIPFederationInternalEdgeListeningAddressSetting - WMI
MSFT_SIPFederationInternalEdgeSetting - WMI
MSFT_SIPFederationInternalServerData - WMI
MSFT_SIPFederationListeningAddressSetting - WMI
MSFT_SIPFederationNetworkProviderTable - WMI
MSFT_SIPGlobalArchivingSetting - Active Directory
MSFT_SIPGlobalFederationSetting - Active Directory
MSFT_SIPListeningAddressData - WMI
MSFT_SIPLoadedExtensionModuleElement - WMI
MSFT_SIPLogOptions - WMI
MSFT_SIPLogServiceSetting - WMI
MSFT_SIPLogSetting - SQL
MSFT_SIPLogVersion - WMI
MSFT_SIPPoolSetting - Active Directory
MSFT_SIPProxySecuritySetting - SQL
MSFT_SIPProxySetting - SQL
MSFT_SIPRemoteAddressData - SQL
MSFT_SIPRoutingSetting - WMI
MSFT_SIPRoutingTableData - SQL
MSFT_SIPTroubleShootingSetting - WMI
MSFT_SIPUserReplicatorSetting - SQL
MSFT_SIPVersion - WMI

Office Communications Server and WMI

As many of you are likely aware, Office Communications Server (and previous versions) make use of WMI for storing various pieces of data.  You can find details of the WMI classes that currently ship with OCS 2007 here.  While this gives helpful information about what types of classes and properties are available, there are some fine points that I believe would be helpful to know.

First, our WMI provider is quite a bit more complicated than your traditional WMI provider.  In a simplistic view, it provides seemless access to data from three separate sources.

1) Active directory

2) WMI

3) SQL server

In theory, when you query these WMI classes you do not need to know from where the WMI provider is getting its data.  However, in reality this is not true.  In order to query a WMI class that uses active directory, the account you are running under must have permissions to AD.  If this is not the case, your query will fail.

In a somewhat finer point, say you wish to query MSFT_SIPRoutingTableData.  Say you know the MatchURI and wish to perform a query.  Your first inclination would be to probably create a query similar to the following.

SELECT * FROM MSFT_SIPRoutingTableData WHERE MatchURI='[match Uri]'

This may give you the results you want, but may also be extremely slow.  Why is this the case?  Say you have several pools in your forest, and they are spread over a wide geographic area.  For instance, one pool is located in Singapore.  When you perform this query, the WMI provider will retrieve the results from all pools.  This has two very bad consequences.

a) If the WMI provider must query SQL servers that are located very far away, this query will be very slow and may waste valuable network bandwidth.

b) This will return results from all pools.  Therefore, in some cases you may only expect a single result but will actually receive multiple results.

The solution is to only query the backend when performing a WMI query on a SQL backed class.  Therefore, instead of the above query you would do the following.

SELECT * FROM MSFT_SIPRoutingTableData WHERE Backend='[backend]'

If this is an SE pool, then you can use (local)\\\\rtc (extra slashes for WMI).  If this is Enterprise Edition, you may or may not know the backend.  In a later blog I will discuss you to retrieve this.

You will then need to iterate through the results and check the MatchURI property for the one you are looking for.  Admittedly, this is not an elegant solution but in most cases it will be the better performing one - unless you only have one pool in your enterprise.

You may wonder why you cannot simply create a query checking for both results.  Unfortunately our WMI provider does not support multiple query search terms and therefore this will not work.

So how can you tell where the WMI provider retrieves its data for each class?  For SQL server backed classes, this is easy.  If the class contains a backend property, then it is SQL backed.

Back to blogging

OK, so one of my new year's resolutions has been to finally keep this blog going.  I thought I would start with a quick status blog and explain what I have in store for the near future.

As much as I would love to blog about the cool new feature I am working on, I am unable to give much light on this at the moment.  I can say that this will be a very important component of Office Communications Server and, once it is public, I will have a series of blogs detailing it.  In the meantime, we are working very hard to finish it and get it to a highly stable and robust component.

On the photography end, I was lucky enough to get a few toys in December and bought the Canon 70-200 4.0L IS.  I really love this lens - particularly the IS.  I thought long and hard about what lens to get but ultimately I needed a primary lens for my camera (I had sold my Tamron 24-135) and the 70-200 2.8 IS was simply too heavy.  Also reports that the 4.0 lens is sharper and has better IS pushed it over the edge.  In other toys I received the CP-E4 flash battery pack which I now use constantly with my 580EX and MT-24EX flashes and I received the angle viewfinder C which I am still learning how to use.

Blog-wise, the big news is that I will be splitting up my blogs in the very near future.  The blog that you are currently reading will stay focused on Office Communications Server and microsoft related topics.  I may occasionally go off topic with thoughts about travelling, but for the most part I want this blog to be a source of information for OCS.

The second blog, which will be on a new site, will cover my passion for photography.  I have always felt guilty about discussing photography here and this blogging software is not very well geared for photography anyways.  I will of course announce the site here when it is ready and I will keep permanent links between the different blog sites for those who care.

The final blog will be a joint project between my wife and I and will cover real estate.  This will primarily be my wife's blog (she is a real estate agent), but at times I will add my own views.  I have avoided discussing my opinions on real estate and the real estate market here because I felt that they were a conflict of interest, given that I obviously will promote my wife.  I will likely announce this blog here when it is ready, but I will not provided a permanent link between the sites as I suspect they will have few readers in common and also due to conflict of interest.

I hope everyone had a great New Years and I promise that you will soon have some useful new blogs to read!

Turning off contact me for now

I apologize that I have not had the time to add much to this blog recently.  With dev schedules and drop dates I have not had much of a chance to create something useful, and I figured most of the readers here are not interested in reading about vacation plans and photography. :)

I have had to turn the 'contact' feature of this blog off for now.  In the future I may turn it on again but the fact is I was receiving too many e-mails.  In previous blogs I detailed using UC forums and I simply do not have the time to answer these questions.  I apologize if this seems rude but I would rather spend the little free time I have working on quality blogs that will help everyone rather than answering individual questions.

Testing while coding

Wow! It's been a long time since I wrote a blog!  Things have been very busy around here.  I am busy coding a new integral piece of the next version of OCS.  Unfortunately that is about all I can say about it at this time. 

When I was a tester, my code was the tests for the developer's code.  Therefore, I did not have code that tested my own code.  Developers would create simple unit tests that would call methods directly in their code.

Since I have moved to dev, I now write unit tests for everything I check in.  However I have taken a slightly different approach than some other devs in that I don't just write 'unit tests' but instead write any test that I can easily code.  The goal of many of the tests are to break the product.

While it has taken me some time to write these tests (112 at current count) and maintenance has taken some time, I find the results of the tests worth their weight in gold.  Now, whenever I make any type of change, I run the tests and in about three minutes I can determine whether I broke anything.  I also have found several bugs that would have been found much later.  In the end I suspect that product quality will be much higher because of these tests.

The tests also save me a lot of worrying.  Whenever I make a change that I think could be problematic I just run all of the tests.  Because of the breadth and depth of the tests, I know that if they all pass my change is relatively safe.  This also saves me the embarrassment of a tester reporting that I broke something that previously worked.

Thoughts on testing speech applications

Personally, I feel one area that we did not address overly well in Speech Server is allowing developers to test their speech applications.  Granted, this seems to be missing from many platforms these days but I think in the case of speech applications this is especially important because they can be tricky to test.

First, "testing" is a very broad topic.  There are many types of testing that you can (and should) do on a speech application not limited to the following.

Functionality - This testing basically makes sure your application does not crash and works as designed.  This is what most people think of as "testing" and there are a number of publicly available test frameworks (such as NUnit) that can help you here.

Performance - For speech applications, this means your applications respond in an appropriate amount of time such that users do not feel that the application is slow to respond.  Our tuning tools can help you determine whether this is an issue, but we provide no automated way to do this.

Load - This involves running as many instances of your application as possible and verifying that they work reasonably under high server load.

Long haul - This involves running your application over a long period of time.  We performed long haul testing on Speech Server itself, but you want to make sure your application can handle running 24/7 for a long period of time.

Usability - Are your users able to use the system?  Actually we do provide a mechanism to test this - using the debugger that ships with the Speech Server tools.  The debugger was designed with specific usability testing features.

Globalization - Does your application work properly in other languages and locale?

There are other forms of testing, but I think for speech applications the above are the most useful.  For most companies, performance, load, and long haul testing are performed in pilot runs.  Functionality testing is often performed by the engineers themselves, though larger companies may have test teams.  Globalization testing is often molded into functionality testing.

What would be nice is to have a way to automatically determine whether your changes broke another feature.  This is the way the majority of testing occurs at Microsoft - with automated tests that run regularly.  This automated testing occurred not just for the platform itself, but also for the tools and sample applications that we shipped.

Within our test and development teams, we had several platforms we could use to test the platform and applications.  Unfortunately none of these will be publically released but I have a pet project to create something separate that I can release here (don't get your hopes up though - chances are I won't be able to finish it).

When we were shipping Speech Server, we tested our sample applications by automating the debugger.  As we had an extensive test framework that automated and verified the debugger, it was easy to expose this to the applications testers.  This worked fine for testing our applications, but breaks down with larger applications that make use of external resources such as databases.  For external users, this is also a difficult approach because it requires automating Visual Studio and the debugger, which are no simple tasks.

The two methods remaining are using UCMA and UCCA. 

UCMA would be very ideal here because we can create multiple connections to the application and therefore test both functionality and stress and performance tests.  Another nice thing about using UCMA to test is that a framework built this way could also test non-speech apps also built using UCMA.  Unfortunately UCMA does not currently have a media stack.  This makes verifying responses sent from the application and sending audio to it exceedingly difficult.

The other option is UCCA.  UCCA does allow us to send and receive media, but we cannot create peer to peer connections.  For Speech Server, we shouldn't need to register our client.  I have not experimented with UCCA in depth, but it would also seem difficult to perform stress testing with it.  Still, when I have time to investigate UCCA I may look into the feasability of this approach.