barrybo

My "hush-hush" project is finally opening up a little! 

After delving deep into the world of mobile and embedded applications as part of the DeviceEmulator project, I decided to learn a very different style of software... video games.  Instead of squeezing bytes and milliwatts out of code, I now worry about framerates, real-time audio data decompression and of course, piracy.

As we move close to tech-transfer, we are growing the team.  Here is the job description text: 

Join an incubation team in MSR is working on a project aimed at high-end PC video games.

We are looking for someone with excellent kernel and driver development experience and debugging expertise, ready to take ownership of multiple product features. Areas of responsibility may include: game content protection, 3D graphics acceleration, OS/driver development, hardware prototyping, device firmware development, and compilers. Strong assembly, C, C++, and debugging skills are required along with a proven track record of shipping product. DirectX experience and PC video game development experience are not required, but would be a plus.

The project is expected to transfer into a high-profile product group and rapidly accelerate to a V1 release. This position would follow the project into the product group.

To apply, visit http://members.microsoft.com/careers/search/details.aspx?JobID=A1E89CF7-20D7-464F-91E5-692D9EAF7B1B&start=1&interval=10&SortCol=DatePosted, or search for job code # 204427 at http://www.microsoft.com/career.

DeviceEmulator 2.0 adds several new peripheral devices, including notification LED and vibrate.

If the emulator is running without a skin, or with a skin that predates the LED feature, then the emulator displays the LED as a blinking box in its title bar.  However, you can integrate it into the skin and make it look like a real LED.  The .zip file attached to this post contains a sample skin I put together - it is a PocketPC 2003 skin, updated so the power button blinks.

Behind the scenes, the LED is represented in the skin XML as a button, just like the buttons used to send keystrokes into the emulator.  The difference is that the LED contains an extra XML attribute:

isNotificationLED="true"

The "up" skin bitmap is used to display the LED in the ON setting.  The "down" skin bitmap is used to display the LED in the ON setting.  Note that skin buttons with isNotificationLED="true" are no longer buttons that can send keystrokes, so the onPressAndHold and onClick attributes are ignored.

Back in October, I left the DeviceEmulator team for a new position in Microsoft Research.  The project's details are hush-hush, even inside Microsoft, so I've been keeping a low profile in the blogsphere.

It isn't time to announce the project yet, but it is time to hire another developer.  The job description is up on microsoft.com at http://members.microsoft.com/careers/search/details.aspx?JobID=A1E89CF7-20D7-464F-91E5-692D9EAF7B1B&start=1&interval=10&SortCol=DatePosted.  If that link doesn't work, visit http://www.microsoft.com/careers/ and enter 204427 as the job code value.

So if you're qualified, and are itching to know just what it is I'm up to, here's a chance to come join the team!

There are some mechanisms built into the DeviceEmulator's JIT which can make debugging and tracing both the JIT and guest code fairly straightfoward.

 1)  Define LOGGING_ENABLED to 1 in include\emulator.h, to compile in debug logging in the JIT.  Several variables control logging:

- LogInstructionStart - begin logging once 'n' guest instructions have been executed

- the LogIf() macro specifies the conditions under which logging happens.  The expression "(Cpu.CPSR.Bits.Mode == UserModeValue && InstructionCount >= LogInstructionStart)" logs only instructions executed in usermode, and only after LogInstructionStart instructions have executed.  This is a handy trigger, as it filters out interrupt handlers.  JIT logging will dump the disassembled ARM opcodes and register state, showing you a single-step trace of your code as it runs.

2)  In cpus\arm\armcpu.cpp, edit g_fOptimizeCode to be 0/false instead of 1/true.  This disables cross-instruction optimizations such as the code that recognizes ARMFlushICacheLines and converts it to a no-op.  It also inserts a call to SingleStepHelper() between guest instructions, to poll for single-step requests from the hardware debugger interface.

3)  Write your own low-level debugger - create an instance of IDeviceEmulatorDebugger from another process, and you can read/write memory, read/write registers, break in, single-step, set breakpoints, etc.  This is the interface that the Platform Builder eXDI hardware debugger connects to.  See cpus\arm\debugger.cpp for the implementation of this interface - it's quite straightforward.

4)  If you are changing the JIT for some reason, and it stops working, open cpus\arm\armcpu.cpp and check out the comment block above "//#define GOOD_EMULATOR 1" - this feature allows you to run a known-good DeviceEmulator.exe and a known-bad DeviceEmulator.exe in lock-step with each other.  Each one steps forward by one guest instruction, then they compare results and if there is a difference, assert.  Please note that the two emulators quickly diverge if you enable guest hardware interrupts, as interrupt delivery is not slaved between the two emulators - timer interrupts will arrive at different times.

Barry

Windows Embedded CE 6.0 Platform Builder 6.0 SP1 is now available at http://www.microsoft.com/downloads/details.aspx?FamilyId=BF0DC0E3-8575-4860-A8E3-290ADF242678&displaylang=en.

The service pack adds eXDI hardware-level debugging into the Platform Builder 6.0 debugger, which is great news!  Even better, is that the eXDI driver for the DeviceEmulator shipped as sample source code.  Check out %ProgramFiles%\Microsoft Platform Builder\6.00\cepb\EXDI2\SAMPLEDRIVER\KdStubDe.  This is an updated version of the eXDI driver I demoed at PDC last year using PB 5.0.

The DEComInterfaces.idl declares the COM interface to the DeviceEmulator, and lurking inside it is IDeviceEmulatorDebugger.  The KdStubDe translates eXDI method calls into IDeviceEmulatorDebugger calls, giving you the ability to debug OS images even if no KITL/KdStub is present, and to debug all of the code in the image, including interrupt handlers and boot code.

Please do take time to read the README.TXT to get familiar with the limitations of DeviceEmulatorEXDI... there are a few gotchas where the emulator's implementation makes hardware-level debugging a bit challenging.  ;-)

Back in October, I said farewell to the DeviceEmulator, and joined a new project in Microsoft Research.  That project is still secret - I joke that at next TechFest, we'll have a booth, but it'll just be 4 black-curtained walls, with no computers and no people.  But if you've seen me around the campus, you'll know by my smile that it is that great combination of a very tough problem and a strong team to solve it.  I'm having a ridiculous amount of fun! 

Along the way, I have been learning about GPUs and 3D graphics programming.  This is a new area for me - I have worked in the VB runtime, WOW64 (which was focused on server scenarios) and DeviceEmulator (mobile scenarios), and none of those had much need for DirectX and D3D.

The Accelerator team have been very helpful in my crash course in GPU programming.  If you haven't heard of Accelerator, please do check it out - it is a set of libraries for .NET that makes it straightfoward to write data-parallel code that downloads and runs on the GPU hardware.  You need to know nothing about GPUs, only managed arrays.  Their original paper is: 

David Tarditi, Sidd Puri, Jose Oglesby. "Accelerator: simplified programming of graphics-processing units for general-purpose uses via data-parallelism",  Technical Report MSR-TR-2004-184, Microsoft Corporation, December, 2005.

You can download Accelerator here.  They're hiring, by the way, if you have a passion for concurrent programming and experience in this field.  I'm looking forward to multi-core CPU version - Accelerator gives you a new/different way of efficiently spreading computation across your available computing resources.

The DeviceEmulator team has just released DeviceEmulator 2.0 in all languages that Visual Studio 2005  supports.  Here are the URLs:

ENU: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=dd567053-f231-4a64-a648-fea5e7061303

CHS: http://www.microsoft.com/downloads/details.aspx?displaylang=zh-cn&FamilyID=dd567053-f231-4a64-a648-fea5e7061303

CHT: http://www.microsoft.com/downloads/details.aspx?displaylang=zh-tw&FamilyID=dd567053-f231-4a64-a648-fea5e7061303

DEU: http://www.microsoft.com/downloads/details.aspx?displaylang=de&FamilyID=dd567053-f231-4a64-a648-fea5e7061303

ESN: http://www.microsoft.com/downloads/details.aspx?displaylang=es&FamilyID=dd567053-f231-4a64-a648-fea5e7061303

FRA: http://www.microsoft.com/downloads/details.aspx?displaylang=fr&FamilyID=dd567053-f231-4a64-a648-fea5e7061303

ITA: http://www.microsoft.com/downloads/details.aspx?displaylang=it&FamilyID=dd567053-f231-4a64-a648-fea5e7061303

JPN: http://www.microsoft.com/downloads/details.aspx?displaylang=ja&FamilyID=dd567053-f231-4a64-a648-fea5e7061303

KOR: http://www.microsoft.com/downloads/details.aspx?displaylang=ko&FamilyID=dd567053-f231-4a64-a648-fea5e7061303

Congrats to the DeviceEmulator team!

Anand just posted that Device Emulator V2 is now available as a standalone download (http://blogs.msdn.com/anandba/archive/2007/03/13/device-emulator-v2-is-live.aspx).

Congrats to the emulator team!

Two things made V2 particularly interesting to work on.  First, is the number of code contributions from across Microsoft.  I've lost track of all of the names, but Windows Mobile, Windows CE, and even Customer Support Services (CSS, formerly PSS) all made emulator and BSP changes.  Second, we started coding before V1 was even out the door (in VS2005), and ran a double-speed product cycle, to align with Windows Embedded CE 6.0.   While V1 targetted the Visual Studio user, V2 targetted both the CE 6.0 Platform Builder user (with KITL-over-DMA and the in-the-box BSP) and Windows Mobile 6.0 (with battery, backlight, contrast, expanded NOR flash, headset/carkit/speakerphone).  We also took time to work on things that everyone cares about:  raw performance in the JIT, with a wide range of performance features (http://blogs.msdn.com/barrybo/archive/2006/05/23/605314.aspx), Windows Vista support, and a bit more user-friendly config dialog.  Of course, we fixed plenty of bugs from customer reports and from Dr. Watson logs you so kindly uploaded.

Please do opt into the Customer Experience Improvement Program when the emulator launches for the first time.  The data collected directly feeds into the planning process for future emulator versions.  It helps us understand simple things like how frequently configuration options like the COM ports are used, how often the config dialog fails to reconfigure, CPU speed, etc.

Enjoy!

Barry

I use a pretty extensive suite of tools as a developer - two IDEs, debuggers, code analysis tools, profilers, etc.  But one tool stands out as the glue between all of my tools - OneNote 2007.

Here's a rundown of a few of the tasks where Outlook is invaluable to me:

1. As my short-term memory.  I paste output from my debuggers and other tools, to keep track of them.  Many tools I use are GUI and don't have a built-in "scrollback" mechanism where I can see previous state.  For example, when I debug, I often want to see the contents of the memory window from some point in the past, not the current contents of that memory.  OneNote is a quick and easy place to group together this "short term" content in a single page.
2. As long-term memory.  It seems I'm always doing code-reviews of unfamiliar code, looking for bugs, or to understand how a subsystem works.  OneNote's free-form note-taking and rich copy/paste let me quickly jot down what I learn about the codebase, and keep the notes with me at all times.
3. As a research tool.  If I copy/paste content from my web browser into OneNote, the new content is automatically tagged with the page URL.  So it's easy to return to the content later, and revisit the URL.
4. New in 2007:  Content syncs between my many machines.  My Windows Vista laptop and Windows XP 64-bit desktop sync their OneNote content automatically and transparently.  So notes I make on my desktop are available when I bring my laptop home at night.  OneNote Mobile on my PocketPC device also syncs a subset of my OneNote content.
5. As a bug reporting tool.  It's easy to log repro steps for a bug as I create them, then I can use OneNote's screen capture to record precisely what I saw on my screen.
6. As a note taker during meetings.  Notes are automatically data and time stamped.  Outlook and OneNote have some great interactions, where OneNote can create new Tasks, and Outlook can associate OneNote content with contacts and calendar items.

OneNote's UI seems to work exactly the way I work - I almost never use the menus or toolbars, as it's in the right state already.  Just right-click on the right part of my notebook, pick "New Page" or "New Subpage" and start typing.  It doesn't interrupt my train of thought with modal dialogs or any other up-front decision-making. 

It's also very easy to reorganize content - since it all sits in OneNote, it's easy to drag pages and subpages around, shuffle portions of pages around, etc.  When I kept notes with notepad, I had content scattered across many files in many directories and bookkeeping had gotten out of control.

I'm still exploring OneNote.  I haven't had a chance yet to look at:

• Live sharing - several people can work simultaneously on a shared page
• Hyperlinks to OneNote notebooks, sections, pages or paragraphs, to link content together
• Annotate PowerPoint slides - use "Insert slides as printouts" then annotate overtop of the slides in OneNote
• Audio and video recording (you can even search the audio recording!)

If you're interested in experimenting with OneNote 2007, a trial edition is free to download, from http://us1.trymicrosoftoffice.com/product.aspx?family=onenote&culture=en-US.  There is also an online test drive at http://www.microsoft.com/testdrive  if your browser supports the Citrix Web Client control and you have a Passport account.

Device Emulator V2.0 is finished, and RTMs along with Windows CE 6.0.  It's going to be a great release, building on the V2.0 CTP, with improved features and performance (and quite a few bug fixes).  The V3.0 emulator is already under development, and I can't give away details, but I'm sure that the Device Emulator Manager will be leaps and bounds more user-friendly and useful in V3. 

So I decided this is a good time for me to switch gears, leaving the DeviceEmulator in the capable hands of Anand, Vijay, Mohit and the rest of the crew.  I'll continue to lurk on the embedded newsgroups and Device Emulator General forum on MSDN.

Beginning Monday October 2nd, I started my new job in MSR.  The project hasn't been made public, but it was one of those once-in-a-lifetime opportunities that I just couldn't pass up.  I believe some of the work we do can be applied to future DeviceEmulator releases, so I'll be keeping in close contact with the DeviceEmulator team.  :-)

My Channel9 video is up!  http://channel9.msdn.com/Showpost.aspx?postid=224510.  It's a talk about the DeviceEmulator in general, with some highlights of the Shared Source DeviceEmulator.

Very small!  The DeviceEmulator nk.bin can be as small as 1.3mb!  With that 1.3mb OS image, you can deploy and debug native applications using Visual Studio for Devices.  Here's how...

In Platform Builder

Install the Device Emulator BSP from http://www.microsoft.com/downloads/details.aspx?FamilyID=474b03a6-e87d-455d-bc16-b8cf18ef39b4&displaylang=en.  Pay special attention to the non-skin configuration options in the SDK XML file.

Create a new platform using the wizard...

• Step 4 - choose custom device
• Step 5 (Applications and Services Development) - check "C Libraries and Runtimes" (which checks everything underneath it)
• Step 6 (Applications - End User) - leave everything unchecked
• Step 7 (Core OS) - check "Power Management (Minimal)", "Device Manager", and "Debugging Tools \ Toolhelp API"
• Step 8 (Networking Features) - check "Networking Features \ Winsock Support"
• Step 9 (Device Management) - leave everything unchecked
• Step 10 (File Systems and Data Store) - check "File System - Internal (Choose 1) \ RAM and ROM File System" and "Registry Storage (Choose 1) \ Hive-based Registry"
• Step 11 (Fonts) - leave everything unchecked
• Step 12 (International) - check "Locale Services (Choose 1) \ National Language Support (NLS)"
• Step 13 (Internet Client Services) - leave everything unchecked
• Step 14 (Grahics and Multimedia Technologies) - leave everything unchecked
• Step 15 (Security) - leave everything unchecked
• Step 16 (Shell and User Interface) - check "Graphics, Windowing and Events \ Minimal Input Configuration"
• Step 17 (Windows CE Error Reporting) - leave everything unchecked
• Step 18 (Voice over IP Phone Services) - leave everything unchecked
• Finish!

Now, go to the Platform\Settings menu, and for the release configuration, switch to the Build Options tab and uncheck "Enable CE Target Control Support (SYSGEN_SHELL=1)" and "Enable KITL (no IMGKITL=1)".

Then switch to the Environment tab and add "BSP_NOTOUCH" with value "1"

Now build it using the Platform Builder "Build OS \ Sysgen" menu.  The retail nk.bin image will be about 1.3mb in size.

Next, create an SDK for it.  Go to Platform Builder's "Platform \ SDK \ New SDK..." menu, fill in the details on the first page, then leave the defaults for the rest.

Finally, build the SDK using the Platform Builder "Platform \ SDK \ Build SDK" menu.

Install the SDK

Install the new SDK on your Visual Studio 2005 machine.

Now comes the hacky part of the process you need to hack your datastore a little.

• Navigate to C:\Documents and Settings\All Users\Application Data\Microsoft\corecon\1.0\addons and look for the newest .xsl file whose name is a GUID.  Open that file in notepad or your favorite XML editor
• Edit the RemoteCcClientFile, RemoteCcShutdownFile, RemoteCcTransportLoaderFile and RemoteCcCMAcceptFile properties, removing the "%CSIDL_WINDOWS%" text and just deploying to the root directory.  There are two instances of each property, one for emulator targets and one for hardware device targets.  Edit both instances of each property.
• Edit the RemoteTransportFile property, again, removing the %CSIDL_% constant, so that the dmatrans.dll deploys to the root directory.
   

 ie.

<PROPERTY ID="RemoteCcCMAcceptFile" Protected="true">%CSIDL_WINDOWS%\CoreCon%CcVersion%\CMAccept.exe</PROPERTY>

becomes
<PROPERTY ID="RemoteCcCMAcceptFile" Protected="true">\CoreCon%CcVersion%\CMAccept.exe</PROPERTY>

If something goes wrong, delete the XML file, uninstall and reinstall your SDK, and it will be recreated.

Use the SDK from Visual Studio 2005

Launch Visual Studio, and do a File/New Project, to create a new C++ SmartDevice project.  Pick your SDK from the list of targets, and select "Console" as the application type, with no ATL or MFC support.

Once the wizard has finished, there are a few work items left:

• Remove the #include of commdlg.h from the top of the wizard-generated .cpp file
• Remove the #include of atlcecrt.h from the bottom of stdafx.h
• In the project properties, make the following edits in the "All Configurations" configuration:
• Under Configuration Properties \ Linker, click on Additional Dependencies then type in this text:  coredll.lib corelibc.lib $(NOINHERIT)
• Under Configuration Properties \ Deployment, edit the "Remote Directory" entry, removing the "%CSIDL_PROGRAM_FILES%" text.  Deploy your app into a subdirectory under the root directory.
• Make the following edit to your project properties under both the Debug and Retail configurations:
• Under Configuration Properties \ Debugging, remove the "%CSIDL_PRGORAM_FILES%" text.  The remote executable is in a subdirectory under the root directory.

Now, F5 should build your app, launch the emulator, deploy and debug.

Diagnosing Problems

• If the emulator launches but deployment isn't happening, you might have left a %CSIDL_*% constant in a path somewhere. 
• Go to VS's Tools/Options dialog and navigate to "Device Tools \ Devices", click on your emulator entry and click Properties.  In the emulator properties dialog, click "Emulator Options" then on the Peripherals tab, check the "Create text console for serial port 1" checkbox.  This will cause the emulator to open a console window as it launches, that will show you diagnostics output from the Windows CE kernel.
  

What's with the CSIDLs? 

Visual Studio for Devices must resolve %CSIDL_% symbols in pathnames by executing code on the device/emulator, that calls SHGetSpecialFolderPath().  The OS image size grows substantially if we pull that API in, as it only comes in as part of the GUI shell.

We must painstaking remove all %CSIDL_% symbols from the datastore and project properties.  Otherwise, VSD's device-side code will go into a long polling loop waiting for the shell APIs to come online, and when the timeout eventually expires, the VSD operation will fail.

One of the great things about the DeviceEmulator is that it runs inside VirtualPC and VirtualServer.

However, there is one gotcha about using the Virtual Switch driver (download from http://www.microsoft.com/downloads/details.aspx?FamilyID=dc8332d6-565f-4a57-be8c-1d4718d3af65&displaylang=en) inside a VirtualPC.

For security reasons, VirtualPC/VirtualServer blocks the Virtual Switch running inside the VM from receiving broadcast network packets.  This in turn means that the DeviceEmulator running inside a VirtualPC cannot acquire an IP address from your LAN's DHCP server... the DHCP negotiation is done via broadcast packets.

If you are running the emulator inside a VirtualPC and want to communicate with other server apps inside the VirtualPC, such as IIS and Exchange, then the best solution is to do the following:

1. Install the microsoft Loopback Adapter into the version of Windows running inside the VPC.  Use "Start/Help And Support" and search for "install loopback adapter' and pick the first hit in the Microsoft Knowledge Base.
2. Configure the DeviceEmulator's NE2000 to bind to the "Microsoft Loopback Adapter"

Now both the Device Emulator and Windows inside the VirtualPC will use auto-ip addresses over the private LAN created by the loopback adapter.

Get it here:  http://www.microsoft.com/downloads/details.aspx?FamilyID=faa8c81d-7316-4461-a0ed-6c95b261ddcd&DisplayLang=en.  The license text is available here:  http://msdn.microsoft.com/vstudio/license/de_academic.aspx.

This release is the full source to V1 DeviceEmulator.exe, which you can compile yourself using Visual Studio 2005.

We created this release to enable experimentation with the emulator:

- create extensibility points to "plug in" new kinds of hardware

- extend or modify the ARM-to-x86 JIT (hint:  my blog on V2 performance at http://blogs.msdn.com/barrybo/archive/2006/05/23/605314.aspx has some "homework assignment" tasks).

- create emulators for whole new CPUs and motherboards

- instrument the emulator to collect performance data on your application or OS image

Question:  would you like to see this posted up to CodePlex (http://www.codeplex.com) so the sources reside in a source control system with bug-tracking etc.  This would enable collaboration on projects.

Enjoy!

Barry

The Adapt Your App Site (http://msdn.microsoft.com/mobility/windowsmobile/howto/AdaptYourApp) is a great resource for learning about display sizes and orientations in Windows Mobile.

The site has links to the Device Emulator V1.0 standalone release plus V2 CTP, as well as  pointers to all of the OS images available on microsoft.com - 2003 SE, WM 5.0, WM 5.0 MSFP, and localized WM 5.0 images.

Barry