Where are we going, and what's with the handbasket? : platform builder

Introduction to the Catalog Object Model

dcook — Wed, 19 Dec 2007 00:03:00 GMT

Today's post will provide a basic introduction to the Windows Embedded CE Platform Builder Catalog. If you don't know what that is, consider yourself lucky. If you do have to deal with the catalog data, today's post might be helpful. We'll write a quick C# utility that demonstrates the catalog object model and also provides some useful diagnostic information that might be helpful to catalog file authors. The tool will display any errors or warnings that PB encounters while loading the catalog. It will also list all of the catalog files that it was able to load.

I don't really know how it happened, but somehow about 5 years ago, through no fault of my own, I ended up owning the PB "catalog". For those who don't know, the catalog is a repository for information about the CE operating system, and PB uses this information to help developers customize their CE builds. For example, if you have the environment variable "SYSGEN_AUDIO" set when you build a CE operating system, your operating system will include sound system support. The catalog has a list of these "features", along with a friendly title, description, and some metadata that helps PB ensure that the environment variable is properly exposed to the developer.

There were some significant issues with the catalog's design, and maintaining the catalog data was not a fun experience. CEC files were not Maintaining the code that worked with the catalog was also problematic. For PB 6.0, I finally had the opportunity to start from scratch with a new catalog format. While a certain amount of backwards compatibility was required (the new catalog needed to essentially provide the same data as the old one, and there would need to be an upgrade path). So I designed a new catalog system, written in C# and based on XML files. While it isn't perfect, I think it is an improvement.

From a development perspective, one thing that is nice about the new catalog is that it has a pretty clean object model (an interface for developers who want to use the catalog). While we've never published the interface for the object model, we don't support it for external use, and everything I say here comes without warrantee or guarrantee of future compatibility, it isn't too hard to get started with basic tasks using the catalog object model. The reflection and metadata included in every managed assembly make this possible.

Before I go too far, I want to be clear that the design for the catalog object model emphasized the most common use of the catalog as a read-only repository. Loading and reading data from the catalog is (hopefully) reasonably simple and efficient. Using the object model to create or edit catalog files is a bit more of a challenge and not quite as efficient.

Loading a catalog means parsing about 2 megabytes of XML and loading the data into memory. While this happens in less than 1/10 of a second, we don't want to do this too often. In addition, we don't usually need (or want) multiple copies of the catalog in memory. To help with this, the catalog object model provides a cache of read-only loaded catalogs. When loading a catalog via the object model, you can choose to get your own copy of the catalog or you can ask for a read-only copy, which may come from the cache. (The cache is based on the .NET garbage collector via weak references. As long as anybody is using a catalog, it won't be evicted from the cache. As soon as nobody is using it, it becomes eligible for garbage collection. If it re-enters use before it is collected, it is no longer eligible for collection.)

The first step in using the catalog from your own code is to reference the catalog library. This code is in the Microsoft.PlatformBuilder.Catalog assembly, which (assuming you've installed Platform Builder 6 into C:\Program Files) can be found in C:\Program Files\Microsoft Platform Builder\6.00\cepb\idevs\Microsoft.PlatformBuilder.Catalog.dll. In Visual Studio, add a reference to this file. For convenience, you can add a "using" declaration to the top of your code to import the Catalog's namespace:

using Microsoft.PlatformBuilder.Catalog; 

Next, to load a catalog, you use the CatalogManager class. The CatalogManager is a collection of catalog files. It handles searching through a winceroot for catalog files to load, loading all of the catalog files into memory, and maintaining the organization of the data in the catalog. It "flattens" the contents of the files, merging the data from all of the catalog files into a single collection of catalog elements. It also groups the data by type (so I can get a list of BSPs without going through all elements in the whole catalog), and generates indexes (so I can quickly look up a BSP by name).

When creating a CatalogManager, you can either create your own (writable) copy or you can load a read-only cached copy that might be shared by other components in the same process. To create a writable copy, just use the CatalogManager's constructor:

CatalogManager catalog = new CatalogManager();

The default constructor for the CatalogManager will load only the "Global" catalog files. These are the catalog files in C:\Program Files\Microsoft Platform Builder\6.00\Catalog, and need to be available to PB whether or not it is working with a CE OS tree. To load the catalog files from an OS tree as well as the global catalog files, just add the winceroot path to the constructor call:

CatalogManager catalog = new CatalogManager(@"C:\wince600");

If you don't need to make changes to the catalog manager's data and don't need precise control over the load sequence, you can use the CatalogManager factory method. This will return a cached catalog if one is available, so it is usually more efficient for repeated use than creating and loading a new catalog each time.

CatalogManager catalog = CatalogManager.GetCatalogManager(@"C:\wince600");

Next, we want to know if the catalog manager had any trouble while loading the catalog. For normal operation, developers generally don't want a defective catalog file to prevent PB from working correctly, so the catalog manager will silently catch catalog load errors. However, when working on catalog files, it is helpful to know what PB really thinks about your catalog. The catalog manager keeps track of warnings and errors generated during catalog load. You can access the errors via the catalog manager's Messages property, which is a read-only collection of CatalogMessage objects.

The catalog manager also provides a utility method to format the error or warning messages. It will then write the messages to any TextWriter (i.e. a StreamWriter or StringWriter). For example, to print all error or warning messages to stdout, you could do this:

CatalogManager.WriteCatalogMessages(catalog.Messages, Console.Out); 

Finally, we want to access some of the data in the catalog. Let's print out a list of all of the catalog files that were successfully loaded:

foreach (CatalogFile file in catalog.Files)
{
    Console.WriteLine("CatalogFile: {0} - \"{0}\"",
        file.FileInformation.Title, file.FullName);
}

Similar collections exist for the BSPs, CPUs, and "Items" in the catalog.

Here is a simple command-line program that pulls this all together. You can build this via a Visual Studio C# command-line app project, or you can compile this at the command line with the csc.exe C# compiler. The only trick is to add a reference to the Microsoft.PlatformBuilder.Catalog.dll assembly (from C:\Program Files\Microsoft Platform Builder\6.00\cepb\idevs).

Have fun!

using System;
using System.IO;
using Microsoft.Win32;
using Microsoft.PlatformBuilder.Catalog;
namespace CatalogErrors
{
    public class Program
    {
        public static void Main(string[] args)
        {
            try
            {
                string winceroot = null;
                if (args.Length > 0)
                {
                    string arg0 = args[0].ToLowerInvariant();
                    if (arg0 != "/?" &&
                        arg0 != "-?" &&
                        arg0 != "/h" &&
                        arg0 != "-h" &&
                        arg0 != "--help")
                    {
                        winceroot = args[0];
                    }
                }
                else
                {
                    winceroot = GetDefaultWinceroot();
                }

                if (winceroot != null &&
                    !Directory.Exists(winceroot))
                {
                    Console.Error.WriteLine(
                        "ERROR: Winceroot \"{0}\" not found.",
                        winceroot);
                    winceroot = null;
                }

                if (winceroot == null)
                {
                    Console.Error.WriteLine(
                        "Usage: CatalogErrors [winceroot]");
                    Console.Error.WriteLine(
                        "Displays any CE catalog load warnings and errors.");
                    Console.Error.WriteLine(
                        "If winceroot is not specified, uses default from registry.");
                }
                else
                {
                    Run(winceroot);
                }
            }
            catch (Exception ex)
            {
                Console.Error.WriteLine(
                    "ERROR: {0}: {1}{2}",
                    ex.GetType().FullName,
                    ex.Message,
                    ex.StackTrace);
                ex = ex.InnerException;
                while (ex != null)
                {
                    Console.Error.WriteLine(
                        "     : {0}: {1}{2}",
                        ex.GetType().FullName,
                        ex.Message,
                        ex.StackTrace);
                }
            }
        }

        private static string GetDefaultWinceroot()
        {
            string winceroot = null;
            using (RegistryKey localDir = Registry.CurrentUser.OpenSubKey(
                @"Software\Microsoft\Platform Builder\6.00\Directories"))
            {
                if (localDir != null)
                {
                    winceroot = localDir.GetValue("OS Install Dir") as string;
                }
            }

            if (winceroot == null)
            {
                using (RegistryKey globalDir = Registry.CurrentUser.OpenSubKey(
                    @"Software\Microsoft\Platform Builder\6.00\Directories"))
                {
                    if (globalDir != null)
                    {
                        winceroot = globalDir.GetValue("OS Install Dir") as string;
                    }
                }
            }

            return winceroot;
        }

        private static void Run(string winceroot)
        {
            CatalogManager catalog =
                CatalogManager.GetCatalogManager(@"C:\wince600");
            CatalogManager.WriteCatalogMessages(catalog.Messages, Console.Out);

            foreach (CatalogFile file in catalog.Files)
            {
                Console.WriteLine("CatalogFile: {0} - \"{0}\"",
                    file.FileInformation.Title, file.FullName);
            }
        }
    }
}

Missing OS Design View tab in PB 5.0

dcook — Wed, 12 Sep 2007 01:41:00 GMT

A common complaint about PB 5.0 is that the OS Design View tab will sometimes mysteriously disappear. The best answer I have is that people should be using PB 6.0. Unfortunately, that answer tends to make people want to punch me. In the interest of my own safety, here is what I know about alternative methods of resolving the problem.

The OS Design View tab is one of the places where the old PB shell code (VS 6.0 vintage 1998) meets up with the new PB object model (totally re-written and designed for integration with the VS 2005 shell). So this is more evidence that you can't put new wine in old bottles -- you'll almost always end up with leaks.

In the case of the OS Design View, most of the issues I've resolved were due to an exception leaking. Exception handling is a great thing, but it has the good/bad quality that it is often easy to forget that you're calling a function that can throw an exception (good when you don't need to handle it, bad when you do...).

Ideally, the exception will be caught by PB and PB will properly recover and/or display a useful error message, but this doesn't always happen, in which case PB just silently fails or crashes. Luckily, the exception type we use does a little bit of logging. The exception constructor will send a quick message to the system debug stream. Sometimes this message is useless, but every once in a while, the exception has enough information to help you track down the problem.

There are several utilities that can monitor the debug stream. DBMon.exe is a simple console tool that is part of the Microsoft Platform SDK. DebugView can be downloaded from the SysInternals website. If you run cepb.exe under a debugger such as WinDbg or Visual Studio, you'll also see the messages from the debug stream.

In the case of Missing OS Design View tabs, I have occasionally been able to track down the problem by monitoring the system debug stream while opening an OS Design (pbxml). For example, I saw an "Access Denied" exception that occurred while trying to open a file in an OS Design. The exception message included the file name. Removing the "Read Only" attribute from the file allowed me to successfully open the OS Design.

Obviously, this is a problem with PB. PB should probably be able to do its job with read-only access to the file. And if it fails, it should at least pop up a useful error message indicating what the problem was. Hopefully, we'll do better in the future. In the meantime, you can try to identify the file that is causing trouble for PB and remove the read-only attribute. While this might cause trouble for your source control system, you'll hopefully at least be able to get back to work.

Start a debug stream monitor utility such as DbMon.exe or DebugView, then start cepb.exe. Alternatively, launch cepb.exe under a debugger such as Visual Studio or WinDbg.
Recreate the scenario that is causing trouble. For example, if the OS Design View tab is missing when you open a particular OS Design, open that OS Design.
Examine the messages logged to the debug output stream.
If you find any messages that look interesting, try to resolve the problem they indicate. For example, if you see that one of the messages refers to an access denied failure when opening a particular file, verify that the file exists and that you can access the file. If the file is read-only, try removing the read-only attribute from the file using Explorer (right-click on the file and select "Properties") or the DOS Attrib command.
Leave a comment below or contact me to let me know if this does or does not work for you!

PbcXml: Metadata in PB

dcook — Sun, 09 Sep 2007 02:57:00 GMT

When Platform Builder interacts with a CE OS build tree ("winceroot", i.e. C:\WINCE600), it needs to know a little bit about the contents of that build tree. For example, it needs to know what SYSGEN variables are available and what they mean. It needs to know what BSPs are available and what options they provide. It needs to know what CPUs have been installed. We refer to this information as the winceroot metadata.

This data is either hardcoded into PB, or encoded into the "catalog". The catalog was historically used when data changed so often that code changes for the data got to be annoying. In any case, the data was still essentially hardcoded into PB since the catalog lived in PB's Program Files directory.

As PB has matured, we've tried to move in the direction of decoupling PB from the winceroot. One consequence of this is that we try to move as much metadata out of PB and into the winceroot itself. The first step was to move the catalog from PB's Program Files directory into the winceroot. Then we moved some of the hardcoded information into various metadata files in the winceroot or figured out some way to dynamically infer that metadata from existing files in the winceroot. Finally, in PB 6.0, we rewrote the "catalog" system using XML instead of "CEC" files. One of the goals was to make it usable with all kinds of new metadata. We consolidated a few of the other metadata files into the catalog as well as including all of the metadata from the old catalog. The next version of PB will have many additional metadata types in the catalog.

One immediate disadvantage of this is that while we want PB to be decoupled from the winceroot, we have immediately created a hard break between PB 5.0's metadata format and PB 6.0's metadata format. This is why PB 6.0 doesn't work with the CE 5.0 build tree as shipped. Since PB n had never really worked well with the CE n-1 build tree before, we didn't consider this to be a particularly significant problem. (PB 6.0 includes a tool called "CecImport.exe" that can be used to extract the bare-bones 5.0-format metadata from a CE 5.0 build tree and write it in the 6.0 format. This is unsupported, but it may work if you really want to use PB 6.0 tools with a CE 5.0 tree. Additional tweaking beyond the automatic conversion may be required for best results.)

Another disadvantage is that PB n will choke on the metadata from PB n+1. This probably could have been avoided had we used a somewhat more relaxed mechanism for parsing the metadata files. We decided that we wanted the schema to be as tight as possible so that catalog authoring errors would show up quickly. Instead of ignoring unrecognized metadata, we raise an error.

The new format brings several advantages. As XML, it is a lot easier to manipulate with general development tools (Visual Studio's XML editor, XmlSpy, or even Notepad) than the old catalog database. The schema (PbcXml600.xsd, which may or may not be renamed for future versions) contains documentation for each element and attribute type (aka "what in the world was Doug thinking when he invented this element?"). Adding new data to the catalog is as easy as dropping a new XML file into the appropriate Catalog directory in the winceroot.

The manipulation of the catalog is managed by a C# assembly named "Microsoft.PlatformBuilder.Catalog.dll". This includes the object model for the catalog (classes corresponding to each XML element), helper functions for loading and querying the data, a system for determining where a catalog element should show up if the catalog "tree" is shown, a "catalog control" displaying the catalog as a tree view, and classes that know how to parse some of the CE 5.0 metadata and turn it into PB 6.0 catalog data. (While we don't publish this object model, we might do so in the future if there is sufficient demand. In the meantime, you are welcome to examine it with ILdasm or Reflector.)

We also included a rudimentary catalog editor with PB. This editor does not allow editing of the full range of metadata supported by the catalog, but it hopefully handles the most common tasks for component and BSP developers. To be honest, I never use it -- I just open the catalog files in Visual Studio's XML editor. Visual Studio finds the PbcXml600.xsd schema and that provides all the editing help I need.

When loading the catalog, the catalog's client generally requests "please give me a catalog for this winceroot". The catalog loader then enumerates all *.PbcXml files from the following locations and all subdirectories:

PB's Program Files "catalog" folder. This folder contains metadata about the current PB installation that is specific to PB and not winceroot-dependent.
winceroot\PUBLIC\*\Catalog
winceroot\PLATFORM\*\Catalog

I think there are actually one or two more places that the system checks for catalog files, but the exact list escapes me at the moment, and the locations listed above are the ones most commonly containing catalog files.

All subdirectories of the listed location are checked. If there is a "Catalog" directory, then all *.PbcXml files from that directory and all subdirectories are loaded. For example, if there is a C:\WINCE600\public\ie\catalog folder with subdirectories 1033 and 1041, then the loader will load catalog\*.pbcxml, catalog\1033\*.pbcxml, and catalog\1041\*.pbcxml.

The CE 6.0 catalog includes metadata for the following (as far as I remember -- I don't have access to the schema right now):

WinceRoot. Basic metadata about the current winceroot, such as the version of CE it contains.
Core OSes. There is one CoreOS element for each Core OS (_TGTPLAT) supported by the winceroot.
BSPs. There is one Bsp element for each BSP installed in the winceroot. The PB catalog editor contains support for editing BSPs.
CPUs. There is once Cpu element for each CPU supported by PB. There is one CpuInstalled element for each CPU for which support has been installed.
OS Design Templates. There is one element for each OS Design Template. The OS Design Template is what drives the "New OS Design Wizard". It contains information about what features should be enabled by default for the new OS Design and what features should be shown in the series of option screens in the wizard.
Catalog Items. There is one Item element for each independent "feature" or "driver" in the catalog. The PB catalog editor contains support for editing catalog items.

Let me know if you have specific questions about the catalog or if there are areas you would like me to blog about.