Where are we going, and what's with the handbasket? : ce

Introduction to the Catalog Object Model

dcook — Wed, 19 Dec 2007 00:03:00 GMT

Today's post will provide a basic introduction to the Windows Embedded CE Platform Builder Catalog. If you don't know what that is, consider yourself lucky. If you do have to deal with the catalog data, today's post might be helpful. We'll write a quick C# utility that demonstrates the catalog object model and also provides some useful diagnostic information that might be helpful to catalog file authors. The tool will display any errors or warnings that PB encounters while loading the catalog. It will also list all of the catalog files that it was able to load.

I don't really know how it happened, but somehow about 5 years ago, through no fault of my own, I ended up owning the PB "catalog". For those who don't know, the catalog is a repository for information about the CE operating system, and PB uses this information to help developers customize their CE builds. For example, if you have the environment variable "SYSGEN_AUDIO" set when you build a CE operating system, your operating system will include sound system support. The catalog has a list of these "features", along with a friendly title, description, and some metadata that helps PB ensure that the environment variable is properly exposed to the developer.

There were some significant issues with the catalog's design, and maintaining the catalog data was not a fun experience. CEC files were not Maintaining the code that worked with the catalog was also problematic. For PB 6.0, I finally had the opportunity to start from scratch with a new catalog format. While a certain amount of backwards compatibility was required (the new catalog needed to essentially provide the same data as the old one, and there would need to be an upgrade path). So I designed a new catalog system, written in C# and based on XML files. While it isn't perfect, I think it is an improvement.

From a development perspective, one thing that is nice about the new catalog is that it has a pretty clean object model (an interface for developers who want to use the catalog). While we've never published the interface for the object model, we don't support it for external use, and everything I say here comes without warrantee or guarrantee of future compatibility, it isn't too hard to get started with basic tasks using the catalog object model. The reflection and metadata included in every managed assembly make this possible.

Before I go too far, I want to be clear that the design for the catalog object model emphasized the most common use of the catalog as a read-only repository. Loading and reading data from the catalog is (hopefully) reasonably simple and efficient. Using the object model to create or edit catalog files is a bit more of a challenge and not quite as efficient.

Loading a catalog means parsing about 2 megabytes of XML and loading the data into memory. While this happens in less than 1/10 of a second, we don't want to do this too often. In addition, we don't usually need (or want) multiple copies of the catalog in memory. To help with this, the catalog object model provides a cache of read-only loaded catalogs. When loading a catalog via the object model, you can choose to get your own copy of the catalog or you can ask for a read-only copy, which may come from the cache. (The cache is based on the .NET garbage collector via weak references. As long as anybody is using a catalog, it won't be evicted from the cache. As soon as nobody is using it, it becomes eligible for garbage collection. If it re-enters use before it is collected, it is no longer eligible for collection.)

The first step in using the catalog from your own code is to reference the catalog library. This code is in the Microsoft.PlatformBuilder.Catalog assembly, which (assuming you've installed Platform Builder 6 into C:\Program Files) can be found in C:\Program Files\Microsoft Platform Builder\6.00\cepb\idevs\Microsoft.PlatformBuilder.Catalog.dll. In Visual Studio, add a reference to this file. For convenience, you can add a "using" declaration to the top of your code to import the Catalog's namespace:

using Microsoft.PlatformBuilder.Catalog; 

Next, to load a catalog, you use the CatalogManager class. The CatalogManager is a collection of catalog files. It handles searching through a winceroot for catalog files to load, loading all of the catalog files into memory, and maintaining the organization of the data in the catalog. It "flattens" the contents of the files, merging the data from all of the catalog files into a single collection of catalog elements. It also groups the data by type (so I can get a list of BSPs without going through all elements in the whole catalog), and generates indexes (so I can quickly look up a BSP by name).

When creating a CatalogManager, you can either create your own (writable) copy or you can load a read-only cached copy that might be shared by other components in the same process. To create a writable copy, just use the CatalogManager's constructor:

CatalogManager catalog = new CatalogManager();

The default constructor for the CatalogManager will load only the "Global" catalog files. These are the catalog files in C:\Program Files\Microsoft Platform Builder\6.00\Catalog, and need to be available to PB whether or not it is working with a CE OS tree. To load the catalog files from an OS tree as well as the global catalog files, just add the winceroot path to the constructor call:

CatalogManager catalog = new CatalogManager(@"C:\wince600");

If you don't need to make changes to the catalog manager's data and don't need precise control over the load sequence, you can use the CatalogManager factory method. This will return a cached catalog if one is available, so it is usually more efficient for repeated use than creating and loading a new catalog each time.

CatalogManager catalog = CatalogManager.GetCatalogManager(@"C:\wince600");

Next, we want to know if the catalog manager had any trouble while loading the catalog. For normal operation, developers generally don't want a defective catalog file to prevent PB from working correctly, so the catalog manager will silently catch catalog load errors. However, when working on catalog files, it is helpful to know what PB really thinks about your catalog. The catalog manager keeps track of warnings and errors generated during catalog load. You can access the errors via the catalog manager's Messages property, which is a read-only collection of CatalogMessage objects.

The catalog manager also provides a utility method to format the error or warning messages. It will then write the messages to any TextWriter (i.e. a StreamWriter or StringWriter). For example, to print all error or warning messages to stdout, you could do this:

CatalogManager.WriteCatalogMessages(catalog.Messages, Console.Out); 

Finally, we want to access some of the data in the catalog. Let's print out a list of all of the catalog files that were successfully loaded:

foreach (CatalogFile file in catalog.Files)
{
    Console.WriteLine("CatalogFile: {0} - \"{0}\"",
        file.FileInformation.Title, file.FullName);
}

Similar collections exist for the BSPs, CPUs, and "Items" in the catalog.

Here is a simple command-line program that pulls this all together. You can build this via a Visual Studio C# command-line app project, or you can compile this at the command line with the csc.exe C# compiler. The only trick is to add a reference to the Microsoft.PlatformBuilder.Catalog.dll assembly (from C:\Program Files\Microsoft Platform Builder\6.00\cepb\idevs).

Have fun!

using System;
using System.IO;
using Microsoft.Win32;
using Microsoft.PlatformBuilder.Catalog;
namespace CatalogErrors
{
    public class Program
    {
        public static void Main(string[] args)
        {
            try
            {
                string winceroot = null;
                if (args.Length > 0)
                {
                    string arg0 = args[0].ToLowerInvariant();
                    if (arg0 != "/?" &&
                        arg0 != "-?" &&
                        arg0 != "/h" &&
                        arg0 != "-h" &&
                        arg0 != "--help")
                    {
                        winceroot = args[0];
                    }
                }
                else
                {
                    winceroot = GetDefaultWinceroot();
                }

                if (winceroot != null &&
                    !Directory.Exists(winceroot))
                {
                    Console.Error.WriteLine(
                        "ERROR: Winceroot \"{0}\" not found.",
                        winceroot);
                    winceroot = null;
                }

                if (winceroot == null)
                {
                    Console.Error.WriteLine(
                        "Usage: CatalogErrors [winceroot]");
                    Console.Error.WriteLine(
                        "Displays any CE catalog load warnings and errors.");
                    Console.Error.WriteLine(
                        "If winceroot is not specified, uses default from registry.");
                }
                else
                {
                    Run(winceroot);
                }
            }
            catch (Exception ex)
            {
                Console.Error.WriteLine(
                    "ERROR: {0}: {1}{2}",
                    ex.GetType().FullName,
                    ex.Message,
                    ex.StackTrace);
                ex = ex.InnerException;
                while (ex != null)
                {
                    Console.Error.WriteLine(
                        "     : {0}: {1}{2}",
                        ex.GetType().FullName,
                        ex.Message,
                        ex.StackTrace);
                }
            }
        }

        private static string GetDefaultWinceroot()
        {
            string winceroot = null;
            using (RegistryKey localDir = Registry.CurrentUser.OpenSubKey(
                @"Software\Microsoft\Platform Builder\6.00\Directories"))
            {
                if (localDir != null)
                {
                    winceroot = localDir.GetValue("OS Install Dir") as string;
                }
            }

            if (winceroot == null)
            {
                using (RegistryKey globalDir = Registry.CurrentUser.OpenSubKey(
                    @"Software\Microsoft\Platform Builder\6.00\Directories"))
                {
                    if (globalDir != null)
                    {
                        winceroot = globalDir.GetValue("OS Install Dir") as string;
                    }
                }
            }

            return winceroot;
        }

        private static void Run(string winceroot)
        {
            CatalogManager catalog =
                CatalogManager.GetCatalogManager(@"C:\wince600");
            CatalogManager.WriteCatalogMessages(catalog.Messages, Console.Out);

            foreach (CatalogFile file in catalog.Files)
            {
                Console.WriteLine("CatalogFile: {0} - \"{0}\"",
                    file.FileInformation.Title, file.FullName);
            }
        }
    }
}

Missing OS Design View tab in PB 5.0

dcook — Wed, 12 Sep 2007 01:41:00 GMT

A common complaint about PB 5.0 is that the OS Design View tab will sometimes mysteriously disappear. The best answer I have is that people should be using PB 6.0. Unfortunately, that answer tends to make people want to punch me. In the interest of my own safety, here is what I know about alternative methods of resolving the problem.

The OS Design View tab is one of the places where the old PB shell code (VS 6.0 vintage 1998) meets up with the new PB object model (totally re-written and designed for integration with the VS 2005 shell). So this is more evidence that you can't put new wine in old bottles -- you'll almost always end up with leaks.

In the case of the OS Design View, most of the issues I've resolved were due to an exception leaking. Exception handling is a great thing, but it has the good/bad quality that it is often easy to forget that you're calling a function that can throw an exception (good when you don't need to handle it, bad when you do...).

Ideally, the exception will be caught by PB and PB will properly recover and/or display a useful error message, but this doesn't always happen, in which case PB just silently fails or crashes. Luckily, the exception type we use does a little bit of logging. The exception constructor will send a quick message to the system debug stream. Sometimes this message is useless, but every once in a while, the exception has enough information to help you track down the problem.

There are several utilities that can monitor the debug stream. DBMon.exe is a simple console tool that is part of the Microsoft Platform SDK. DebugView can be downloaded from the SysInternals website. If you run cepb.exe under a debugger such as WinDbg or Visual Studio, you'll also see the messages from the debug stream.

In the case of Missing OS Design View tabs, I have occasionally been able to track down the problem by monitoring the system debug stream while opening an OS Design (pbxml). For example, I saw an "Access Denied" exception that occurred while trying to open a file in an OS Design. The exception message included the file name. Removing the "Read Only" attribute from the file allowed me to successfully open the OS Design.

Obviously, this is a problem with PB. PB should probably be able to do its job with read-only access to the file. And if it fails, it should at least pop up a useful error message indicating what the problem was. Hopefully, we'll do better in the future. In the meantime, you can try to identify the file that is causing trouble for PB and remove the read-only attribute. While this might cause trouble for your source control system, you'll hopefully at least be able to get back to work.

Start a debug stream monitor utility such as DbMon.exe or DebugView, then start cepb.exe. Alternatively, launch cepb.exe under a debugger such as Visual Studio or WinDbg.
Recreate the scenario that is causing trouble. For example, if the OS Design View tab is missing when you open a particular OS Design, open that OS Design.
Examine the messages logged to the debug output stream.
If you find any messages that look interesting, try to resolve the problem they indicate. For example, if you see that one of the messages refers to an access denied failure when opening a particular file, verify that the file exists and that you can access the file. If the file is read-only, try removing the read-only attribute from the file using Explorer (right-click on the file and select "Properties") or the DOS Attrib command.
Leave a comment below or contact me to let me know if this does or does not work for you!

Does PB 5.0 work side-by-side with PB 6.0?

dcook — Mon, 14 May 2007 02:27:00 GMT

Hey, PB expert guy, I have a question.

Shoot.

Can I run PB 5.0 side-by-side with PB 6.0?

Short answer or long answer?

Um, short answer, please.

Kind-of.

What kind of answer is that? This is a yes-or-no question!

Ok. Then the answer is no.

That’s not what I wanted to hear.

Oh, sorry. The customer is always right, I forgot. Yes, yes, of course you can.

Are you lying to me?

Technically, no.

I want the truth.

You can’t handle the truth.

Ha ha. Very funny.

Well, you can run them side by side, but you may or may not be happy with the results.

Hmm. You’re weaseling out of the question, aren’t you?

Yup. Weaseling is what separates us from the animals. (Except the weasel.)

Ok, then the long answer please?

Sure. But first, recognize that the following information is offered without any warrantee or guarantee. You follow these instructions at your own risk. Neither Microsoft nor dcook accepts any responsibility for the use or misuse of these instructions. Use at your own risk. Back up your data first. I’m sharing this information because it has helped others, but it may or may not help you.

Yes, I accept. (click.)

Platform Builder 5.0 and 6.0 tools do not always work perfectly when both are installed on the same computer. (Note that this refers to the PB IDE and tools, not to the OS build tree. The build tree is fairly self-contained.) Most things work, but there are a few challenges. There are also a few steps you can take to minimize the challenges.

The first thing to know is that you must install PB 5.0 first. If you’ve already installed PB 6.0, I’m sorry, but you'll have to uninstall it. PB 5.0 broke some installation rules and will clobber some PB 6.0 settings. And there are a few things that PB 6.0 had to do that aren’t pretty, so a repair of PB 6.0 might not work either. If you’ve already installed PB 6.0 and want to install PB 5.0, your best bet is to uninstall PB 6.0 first. Here is the recommended order of installation:

Install PB 5.0.
Make a backup copy of all files in C:\Program Files\Common Files\Microsoft Shared\Windows CE Tools\Platman\bin. (Only the files in the bin folder need to be backed up. The subfolders do not need to be saved.)
Install Visual Studio 2005 and all associated service packs (as of this writing, SP1, and if you’re running Vista, the SP1-Vista-GDR).
Install PB 6.0 and all associated service packs (as of this writing, SP1).
Compare the current set of files in the Platman\bin folder with the set of files you backed up. Restore any missing files. (Do NOT overwrite any existing files.)
If you have any Connection Manager device settings that were created with PB 5.0, you’ll need to delete them and recreate them.

At this point, PB 5.0 and PB 6.0 should both be mostly working – or at least as well as can be expected.

What’s that supposed to mean?

Well, the above steps worked for me. But there are all kinds of things that could go wrong, and I may not have tested some aspect of PB that is important to you, so this may not work perfectly for everybody.

The largest trouble spots are with the x86 Emulator, the Remote Tools (especially PerfMon and CeCap), and with CoreCon (download and KITL transports).

Why is this such a problem in the first place? Why don’t you just fix this mess?

I would love nothing better than to fix this mess. Unfortunately, as long as we support cemgr.exe, we’re going to have some side-by-side troubles with the Platman\bin folder. Back in PB 4.0, when the best practices for MSI development were not nearly as well-known as they are today (and they still aren't very well-known), a few mistakes were made in the development of the PB 4.0 Platman install. The result is that if a file was part of the PB 4.0 Platman install in the Platman\bin folder, and we want that file to still exist after PB 6.0 is installed, we have to include that file in PB 6.0. Due to various security and legal issues, there are a few files that were part of PB 4.0 that we can no longer ship in PB 6.0. As a result, installing PB 6.0 causes these files to mysteriously disappear.

Any other issues?

Yeah. If you uninstall either PB 5.0 or PB 6.0, you’ll probably have to repair the remaining version of PB before it runs correctly (Add/Remove Programs, select PB, then select Repair). Certain registry settings for COM interfaces can't really be properly shared between versions of PB.

Also, the CoreCon datastore is a bit more fragile than I would have liked. Sometimes things go wrong with the datastore resulting in problems when you launch Connection Settings from PB or when you try to create a Visual Studio for Devices project. (You may also run into problems while installing PB 6.0 if your datastore is corrupt.) If you have trouble with the CoreCon datastore, try the following:

1. You can safely delete your local (per-user) CoreCon datastore settings. (See http://msdn2.microsoft.com/en-us/library/ms184403(VS.80).aspx for details.) This sometimes fixes issues with launching Connection Settings or with creating VSD projects. Go to your user’s application data directory, navigate to the Microsoft folder, and delete the CoreCon directory. For example, 'rd /s "C:\Documents and Settings\YourUserName\Local Settings\Application Data\Microsoft\CoreCon" '. (Warning: DO NOT delete the CoreCon settings under All Users!) The next time you activate CoreCon, it will rebuild your per-user datastore. You will lose any devices that you may have configured, but it shouldn't be too hard to recreate them.
2. If the above does not help, you may have a problem with your global (all users) CoreCon datastore settings. This is a hard problem to solve. These files are in the All Users application data area (i.e. C:\Documents and Settings\All Users\Application Data\Microsoft\CoreCon\1.0, or C:\ProgramData\Microsoft\CoreCon\1.0 on Vista). You can’t just delete these files and hope they come back (because they won’t). This might work:

Make a backup copy of the data in the CoreCon\1.0 folder.
No, seriously. Make a backup copy. If you lose the data in this folder, it may be quite difficult to get it back.
Rename the 1.0 folder to something else, like “1.0_original”.
If a friend or co-worker has a working installation with a similar set of programs (i.e. they successfully installed PB 5.0, VS 2005, and PB 6.0), try copying the data from the other computer to your computer. Ensure that you’ve deleted the per-user CoreCon data (step 1 above), then launch PB 5.0 or VS 2005 and try to open Connection Settings.
If this doesn’t work, rename this folder out of the way (i.e. rename it to 1.0_from_other_computer), then try a repair install of PB 5.0, then VS 2005, then PB 6.0.

Yikes!

Yeah. Tell me about it.

Some people have had better luck just installing PB 5.0 and 6.0 on separate machines, dual-booting, or even installing PB 5.0 into a Virtual PC image.

There are also ways to make PB 6.0 work with OS trees from CE 5.0, but that is a more complicated subject (and while it generally works, it is definitely unsupported by Microsoft).

Changes in Nmake 8.0

dcook — Wed, 21 Mar 2007 05:07:00 GMT

I've run into this issue twice now (in different forms) after upgrading build systems from old versions of Nmake to Nmake 8.0 (the version from Visual Studio 2005), so I think that means it's time to blog about it.

Scenario 1:

Your stuff builds ok, but the output is totally wrong. Characters are missing from the output, and during the build, your computer keeps beeping at you.

Compiling .._koho.c ..\gaimem.c ..\k_area.c ..\k_clearn.c ..\k_grbg.c ..\k_hnkn.c ..\k_moji.c ..\k_regist.c . evconv.c ..♂lbmgr.c ..nvdbg.c ..\k_frame.c ..\k_kenti.c ..\k_disp.c ..\k_atwid.c ..\k_crsr.c ..\k_kkti.c .. ngmgr.c ..\k_koho.c ..\k_bun mk.c ..comment.c ..utotune.c ..

Investigating, you find that all of the missing characters are C-style escape sequences - "\a", "\r", "\n", etc., and the escape sequences have been replaced by the corresponding control code - beep, CR, LF, etc. So why did nmake suddenly decide to start processing escape codes?

Scenario 2:

The following is something of an idiom for makefiles. It shows the command line that is about to be executed, then executes it. (This is a bit of a simplification. As presented, this serves no purpose, but the pattern becomes useful when inline response files are involved.)

.cpp.obj:
  @type <<
  $(CL_COMMAND_LINE)
<<NOKEEP
  $(CL_COMMAND_LINE)

After upgrading to Nmake 8.0, this starts causing an error. Type complains that it can't find the temporary inline file. Replacing "type" with "cmd /c type" fixes the problem.

Both issues have essentially the same root cause. Tune in next time for the solution.

What is Platform Builder?

dcook — Sun, 11 Mar 2007 12:21:00 GMT

I work in the Windows Embedded CE group on the "PB IDE tools" team. That might take a bit of explaining. (Sometimes, I'm not entirely clear myself.)

CE is the "miniature" version of Windows. (The big version, referred to as "NT" on this blog, is familiar to most people -- nearly everybody knows what I mean when I talk about Windows 2000, XP, or Vista.) When talking to family members and friends, I get the most success by explaining it as "the operating system that runs Pocket PCs and SmartPhones". That's accurate (assuming they don't confuse it with PalmOS or their Nokia cell phone), but not a complete description. The CE operating system is actually used in a lot of other things such as industrial automation, GPS navigation systems, and routers. While skiing with my sister's husband, I was trying to explain this and I noticed that the little barcode scanners they were using to scan our lift tickets were running CE, so that made it a bit easier to explain. In any case, CE tends to be used in computerized components that are used as "devices" or "appliances" and not "computers".

I am a member of the "IDE Tools" team. Officially, this means that my team is responsible for the Platform Builder shell and project system used to develop for CE, but unofficially it means "if it runs on the desktop and it isn't owned by any other team, it belongs to the IDE Tools team". As a result, I get to stick my fingers in nearly every aspect of development for the desktop - compilers, debuggers, build systems, setup, C#, C++, scripting, you name it. If it runs on a Windows NT platform (Windows 2000, XP, Vista, etc.), I'm supposed to know all about it. It's my job.

Platform Builder ("PB") is the IDE for developing the CE operating system. This also takes a bit of explaining.

CE is different from NT in a lot of ways. You don't go out, buy a copy of CE, and install it on your phone/PDA/router/refrigerator/toaster. You buy a phone or a PDA with CE already built into it. That copy of CE was customized and tweaked to make it a (hopefully) perfect fit for the phone or PDA onto which it was installed. Typically HP, Samsung, HTC, or some other hardware vendor will design and develop some hardware to meet a need, then use the PB tools to tweak the CE operating system to work well with their hardware. The hardware and the CE operating system are then sold as a single item. So you won't see a copy of "Windows CE" on the shelf at your local software retailer.

PB has a catalog that allows the vendor to choose the parts of the operating system they need. A minimal CE operating system with just a kernel and basic device drivers is about 300K. A fully featured CE OS with a web browser, web server, solitaire, .NET Compact Framework, media player, etc. can be 64 MB or more. Somewhere in the middle is an OS that (hopefully) has all of the features you need for your device. This is important because each additional megabyte of flash or RAM adds to the cost of the hardware, reduces battery life, and increases device size and weight. These costs become especially significant if plan to ship 1 million CE-based talking toasters. (Are you sure you don't want a nice English muffin right now?)

Most devices running CE are proprietary hardware designs with a unique combination of microchips designed to fill a specific niche in the market. This usually means that the CE operating system needs to be customized somewhat to run on the specific hardware. Often, device manufacturers have to write their own device drivers since they're using proprietary hardware. PB provides many tools to help with this process. PB provides an integrated development environment (IDE) to allow developers to edit and compile their code (driver, application, etc.), copy it to the device ("download"), run their code and control its execution (remote shell), and debug their code (debugger).

Four different teams work on PB - debugger/connectivity, remote tools, build, and shell/project system (that's my team!). The responsibilities of the other teams are fairly self-explanatory and well-defined. The debugger/connectivity team works on the PB kernel debugger, the KITL connectivity layer, and the platform manager/connection manager transport layers. The remote tools team works on tools that run on the desktop to provide data about what is going on within a device (remote registry editor, remote file viewer, kernel tracker, etc.) The build team (currently one person, recently split off from the shell/project system team) is responsible for improving the CE build system. The responsibilities of my team (shell/project system) are a bit more nebulous, and we tend to wind up doing anything not covered by any other team (this drives my manager nuts).

The exact boundaries are sometimes hazy, but since the goal is "get the product working well", not "make my team look good", the boundaries don't have to be rock-solid.