For The Developers! : TxF

TxF Managed Wrapper on Code Gallery

jolson — Wed, 23 Apr 2008 07:40:59 GMT

Hello, everybody, I have released the first version of a rudimentary managed wrapper around Transactional NTFS to Code Gallery on MSDN: http://code.msdn.microsoft.com/txfmanaged.

The wrapper in its current form is very basic but should give developers a good idea on how simple it is to use Transactional NTFS from managed code. Feel free to go grab it, play around with it (there are some sample screencasts using the wrapper on the Developer Meet Server Show), and feel free to use it any way you wish.

Enjoy!

Tech Ed Developer (Europe)

jolson — Mon, 05 Nov 2007 14:03:07 GMT

Howdy all, I will be at Tech Ed Developer this week doing several things. First of all, I will be giving two talks: the first on Application Compatibility in Windows Server 2008 outlining the top issues developers need to be aware of when developing for it, and second, a talk on Transactional NTFS as it pertains to Windows Server 2008.

Also, I will be "manning" the Ask The Experts booth for Windows Server 2008. So if you want to chat about Windows Server 2008 (or just say hello), make sure to drop on by (I will have the hours I plan on working the booth posted there at the booth).

Here's a rundown:

Monday, 3pm - [INF203] WS08 For Developers - App Compat, Tent 1
Thursday, 4:30pm - [INF307] WS08 For Developers - Transactional NTFS

I'm also hoping to attend some sessions myself. These will mostly be around C++ (TLA323, TLA302, TLA404, etc.).

So, if you're going to be at the conference, please, drop on by the Ask The Experts booth for Windows Server 2008 and say hello!

Listing Running Transactions In A Resource Manager

jolson — Fri, 20 Jul 2007 00:34:00 GMT

The more I work with TxF, the more I realize that our (Microsoft's) coverage of certain KTM topics is sorely lacking. One of those areas is around documentation and samples of managing resource managers (important if you are wanting to investigate programatically the running status of a resource manager).

Sometimes, being an evangelist can be quite fun (or frustrating) because the documentation leaves something to be desired and you can't run to the internet because the only results you get back or to the documentation on MSDN. And of course, since you're on the cutting searching for the control codes online provides you with two results, both pointing back to MSDN. It becomes doubly "interesting" if you are a managed developer because a large number of management-oriented information for KTM is all done through DeviceIoControl (an API that is NOT fun to try to P/Invoke into, especially using certain control codes).

Anyways, that's all beside the point... moving right along.

If you are writing an application to monitor TxF (or even building it into your own application), there are some core "things" that you need to do. One of the first ones (and the one I will cover in this post) is to get a list of transactions in a given resource manager. This is done using the FSCTL_TXFS_LIST_TRANSACTIONS control code.

What structure is DeviceIoControl passing back to us in this instance? A TXFS_LIST_TRANSACTIONS structure. Looking at this structure, we see two fields: NumberOfTransactions and BufferSizeRequired.
That's nice and all, but where is the information about those transactions? The information is stored in a TXFS_LIST_TRANSACTIONS_ENTRY structure.

The trick is that the TXFS_LIST_TRANSACTIONS_ENTRY structure is mentioned nowhere in the documentation for FSCTL_TXFS_LIST_TRANSACTIONS. So how do you get to it? An array of TXFS_LIST_TRANSACTIONS_ENTRY structures are stored in memory after your TXFS_LIST_TRANSACTIONS structure. So assuming you have two running transactions in the given resource manager, you would pass in a pointer to the TXFS_LIST_TRANSACTIONS structure, and here is what the memory would actually look like coming back from DeviceIoControl:

The impact of this is that you need to allocate enough memory for not only your TXFS_LIST_TRANSACTIONS structure, but the array of TXFS_LIST_TRANSACTIONS_ENTRY structures as well. You might notice a little bit of a "chicken and egg" problem here. If I'm using FSCTL_TXFS_LIST_TRANSACTIONS to retrieve the transactions, how could I know how many entry structures to allocate when I don't know how many transactions there are yet?

Well, assuming there are one or more running transactions in the resource manager, you have to call DeviceIoControl with FSCTL_TXFS_LIST_TRANSACTIONS at least twice: once to find out how big of a buffer you need to allocate (read: to calculate how many ENTRY structures to accomodate for), and then another time after allocating the extra memory to have it fill the actual ENTRY structures.

Why do I say "at least" twice? Because, between the time we make our first call and the time we make our second call, more transactions could have appeared and we will have to allocate even more memory. A little bit of a "cat and mouse game," I must say.

If you are a managed developer, this may sound a bit scary. Don't be scared though, it's not too bad (and I know I at least have fun writing this kind of lower-level code :P).

So let's dive right in and look at the code to make this work!

In pseudo-code, this is what we have to do:

------------------------
while (true)
{
// Deallocate memory from last loop through this

// Allocate memory for transaction list buffer

// Call DeviceIoControl with FSCTL_TXFS_LIST_TRANSACTIONS

// If DeviceIoControl Succeeded

// Get list of transaction entries after TXFS_LIST_TRANSACTIONS structure
}

// Deallocate any remaining memory
------------------------

Not that bad really. So here it is implemented in native code:

#include 
#include 
#include 
#include "Winioctl.h"

int _tmain(int argc, LPTSTR argv)
{
    DWORD lastError, bytesReturned;
    HANDLE rmDirectory;
    TXFS_LIST_TRANSACTIONS *txList = NULL;

    // Get handle to directory where resource manager resides
    rmDirectory = CreateFile(TEXT("C:\\"), 
        GENERIC_READ,
        FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
        NULL,
        OPEN_EXISTING,
        FILE_FLAG_BACKUP_SEMANTICS,
        NULL);

    if (rmDirectory != INVALID_HANDLE_VALUE)
    {
        // Get a list of transactions within the given
        // resource manager
        DWORD neededBufferSize = sizeof(TXFS_LIST_TRANSACTIONS);
        while (true)
        {
            // Deallocate previously allocated memory if we are back
            // here after getting back ERROR_MORE_DATA
            if (NULL != txList)
            {
                delete[] (char*)txList;
                txList = NULL;
            }

            // Get the list of running transactions in the RM
            txList = (TXFS_LIST_TRANSACTIONS*)new char[neededBufferSize];
            if (!DeviceIoControl(rmDirectory,
                FSCTL_TXFS_LIST_TRANSACTIONS,
                NULL,
                0,
                txList,
                neededBufferSize,
                &bytesReturned,
                NULL))
            {
                // How did we fail?
                lastError = GetLastError();
                if (lastError == ERROR_MORE_DATA)
                {
                    // We need to enlarge the buffer and try again
                    neededBufferSize = txList->BufferSizeRequired;
                    continue;
                }
                else
                {
                    _tprintf(TEXT("Unhandled exception: %u\n"), lastError);
                    break;
                }
            }
            else
            {
                _tprintf(TEXT("Transaction Count: %u\n\n"), txList->NumberOfTransactions);

                // List Tx details
                TXFS_LIST_TRANSACTIONS_ENTRY *txEntry;
                txEntry = (TXFS_LIST_TRANSACTIONS_ENTRY *)((char *)txList 
                        + sizeof(TXFS_LIST_TRANSACTIONS));
                for (DWORD i = 0; i < txList->NumberOfTransactions; i++)
                {
                    txEntry++;

                    // Get GUID string
                    WCHAR szGuid[40];
                    StringFromGUID2(txEntry->TransactionId, (WCHAR *)&szGuid, 40);

                    // Print Tx info
                    _tprintf(TEXT("TX %s: %u\n"), 
                        szGuid,
                        txEntry->TransactionState);
                }

                break;
            }
        }

        // Any memory left to deallocate?
        if (NULL != txList)
        {
            delete[] (char*)txList;
            txList = NULL;
        }
    }
    else
    {
        _tprintf(TEXT("Unable to get valid handle to Resource Manager\n"));
        return 1;
    }

    _tprintf(TEXT("\nPress any key to exit...\n"));
    getchar();

    return 0;
}

And that's how you do it. It's not the most fun getting in to DeviceIoControl (unless you're a geek like me and enjoying this kind of stuff), but hopefully with this post out there to help it will be easier to find out how to do this since there is a lack of documentation around this area.

In closing, there is some work that needs to be done around making management of kernel-level transactions a lot easier than it is now. In some ways, it is pretty clear that this is still a "1.0" technology. So while you could use the KTM directly for your transaction coordination, I would highly recommend sticking with using DTC for your transaction coordination as it is much more mature from a management perspective (and there are quite a few enhancements with DTC in Vista and Windows Server 2008).

Windows Server 2008 Developer Training Kit

jolson — Fri, 20 Jul 2007 00:03:51 GMT

James Conard announces over on his blog that our team has just released the Windows Server 2008 Developer Training Kit for Beta 3. Mosey on over and check it out. You can find resources on everything from WCF, to Cardspace, to WF, to Windows Eventing, to PowerShell, to IIS 7, to Transactional NTFS.

So if you are a developer interested in checking out Windows Server 2008, make sure to grab it.

Top Ways To Light Up Your Apps On Windows Server "Longhorn"

jolson — Thu, 26 Apr 2007 03:41:00 GMT

Whether you are a developer building web applications, to a developer writing core server based system services, Windows Server “Longhorn” provides many new features that you can leverage to build more secure, flexible, and innovative applications.

1) Build More Flexible Web Applications

For developers wanting to build flexible web applications, IIS7 provides a very powerful platform in which to build upon. With IIS7, you can take control of the web server footprint to increase security and decrease patching, more quickly resolve faulty applications, go to market faster and reduce support costs with simplified deployment and application configuration, as well as rapidly leverage new technologies with IIS7’s extensible framework. IIS7 is also more efficient to manage with the introduction of new management APIs, a powerful new UI, a suite of command line tools, as well as the capability to manage IIS7 directly from Windows PowerShell.

More Information:

· The .NET Show: Bill Staples and Scott Guthrie on IIS7

· End-To-End Extensibility Example

· Downloads on IIS.net

2) Build Connected Systems

If you are writing enterprise grade distributed applications, Windows Server Codename “Longhorn” is the best platform you can target. Every aspect of the application life cycle enjoys state of the art support: service development, hosting, process modeling, manageability, performance, interoperability out of the box, extensibility, protection of the current investments… all those facets of distributed software development and management are natively implemented by Windows Server codename “Longhorn” features. Many of these features are provided through the Application Server role and the integration of the .NET Framework 3.0 within Windows Server Codename “Longhorn”.

Even if you are simply porting your Windows DNA based application, you will enjoy the great benefits brought by various key enhancements in the core services such as a completely redesigned TCP stack, enhanced transaction capabilities, MSMQ evolutions, and much more.

More Information:

· .NET Framework Developer Center

· Microsoft .NET Framework 3.0 Community

· Introducing the .NET Framework 3.0

3) Develop Federation-Aware Applications

As the business need for ubiquitous user access has evolved, the demand upon Web applications is to no longer only support users inside the firewall. Potentially, these same applications, or portions of them, need to now be accessed by vendors, partners or customers as well. How do you provide access for these constituencies to applications built around the domain-based identity model? At the minimum, it infers additional user account management. Each constituent must have a domain-based identity inside the firewall as well. If this is hundreds or potentially thousands of additional user accounts, it’s a further burden on IT and help desk staff to manage provisioning/de-provisioning of accounts and password resets for those loosely affiliated with the organization.

With Windows Server “Longhorn”, technologies like Active Directory Federation Services (ADFS), Active Directory Lightweight Directory Services (AD LDS), and Windows Authorization Manager (AzMan) provide you, the developer, with the capabilities to develop “Single Sign-On”, Federation-Aware applications.

More Information:

· .NET Developers Guide To Identity

· Active Directory Federation Services (ADFS) Resources

· Identity and Access Management Developer Center

4) Design Highly-Manageable Applications

In today’s rapidly changing IT environment, it is critical that application errors be diagnosed and solved quickly by IT Operations, without the need for costly escalation. This requires datacenters and applications to be designed to be highly-manageable. Microsoft’s strategy for enabling these highly-manageable systems is called the Dynamic Systems Initiative.

The Dynamic Systems Initiative (DSI) is Microsoft's technology strategy for products and solutions that help businesses enhance the dynamic capability of its people, process, and IT infrastructure using technology. To enable applications to be managed within the DSI world, they must be “designed for operations”.

With technologies like Windows Eventing 6.0, Windows Installer 4.0, Windows PowerShell, Microsoft Management Console 3.0 (MMC 3.0), and Task Scheduler 2.0, Windows Server “Longhorn” enables you to develop highly-manageable applications.

More Information:

· Windows PowerShell

· Design for Operations

· Dynamic Systems Initiative

5) Develop More Reliable Applications

Building truly reliable applications that can handle everything thrown at them, including everything from power outages to hardware failures, has always been notoriously difficult. With the addition of several new technologies in Windows Server “Longhorn”, building reliable applications has become easier than before.

By leveraging the new Application Recovery and Restart APIs, Restart Manager, Transactional NTFS, Transactional Registry, or even the new Kernel Transaction Manager, you can let the platform handle a large number of details and build more reliable applications at the same time.

More Information:

· About Transactional NTFS

· Transaction Resources List

· Kernel Transaction Manager and Friends

6) Build for Scalability

Windows Server Codename “Longhorn” provides new scalability enhancements that enable you to build, deploy, host, and manage even the most demanding applications. The businesses of today and tomorrow Windows Server Codename “Longhorn” provides new scalability enhancements that enable you to build, deploy, host, and manage even the most demanding applications. The businesses of today and tomorrow demand more scalability from their servers, and Windows Server Codename “Longhorn” delivers by providing developers new ways to control the concurrent processing behaviors of their applications.

With the new Thread Pool API enhancements, developers have more control over how worker threads execute asynchronous callbacks within their application. By efficiently managing these callbacks and the creation of new threads, these enhancements provide a means to reduce the number of application threads needed by an application. The Thread Ordering Service in Windows Server Codename “Longhorn” controls the execution of one or more client threads. It ensures that each client thread runs once during the specified period and in relative order. Even enhancements to debugging these scalable application have been made with technologies like Wait Chain Traversal, enabling debuggers to diagnose application hangs and deadlocks.

More Information:

· Thread Pooling

· Using Wait Chain Traversal

7) Go Virtual

Virtualization is a very powerful technology that enables consolidation of existing servers, business continuance through flexible data recovery capabilities, lowers the requirements (and total cost of ownership) of testing and development environments, as well as providing unique capabilities to make the management of branch offices easier than before.

Windows Server Virtualization is Microsoft’s new hypervisor-based Virtualization product for Windows Server “Longhorn”. There are many virtualization enhancements coming with Windows Server Virtualization, like:

· 32-bit and 64-bit support for hosts and guests

· Leverages new hardware-based virtualization technologies

· Capable of supporting large memory (>4 GB)

· Supports “memory over-commit” (assigning more virtual memory to a virtual machine than physical memory on the server)

· Symmetric Multi-Processing capabilities for virtual machines (up to 8way)

· Pass-through disk access for virtual machines

· New Virtual Service Provider / Virtual Service Client architecture (allowing requests to shared hardware like disk devices, networking devices, etc.)

· Runs on Server Core installation, minimizing footprint and increasing security of host

More Information:

· Microsoft Virtualization Home

· Windows Server Virtualization - An Overview

Developer, Meet Server - Transactional NTFS and WCF

jolson — Thu, 05 Apr 2007 03:18:35 GMT

I recently posted the second screencast in my new "Developer, Meet Server" screencast series. This latest screencast covers using Transactional NTFS in a service-oriented environment via WCF.

Enjoy!

Transactional NTFS and Free Disk Space

jolson — Thu, 05 Apr 2007 03:14:50 GMT

Andrew recently watched my screencasts and had some great questions that came up around how Transactional NTFS behaves with the file system. So, I figured I would answer them here. So, if you want to see the context around his questions, check out his post.

Here's the paragraph with the questions:

I wonder what happens to files that never get committed - ala a long running transaction. They have to be taking up file space. Jason says not even Windows Explorer knows about them, but they have to exist somewhere. What if they overflow? Is there a way to clean house? If the poop is invisible, does it still stink?

Great observations and questions Andrew! In the case of a long running, open transaction, the files will exist on the disk. Hence, they are certainly taking up space on the disk. If you have 256 MB of free space on your disk, and try to write 257 MB of data to the disk within a transaction, you will get an "out of space" error like you would outside of a transaction.

Is there a way to clean house? Yes, rollback or commit the transaction. On rollback, any changes made within the transaction are rolled back, and then consequently removed form the disk.

But what if my server crashes in the middle of a transaction. Would I have turds floating around the file system? No, you would not. When an NTFS-formatted volume is mounted in Vista or Windows Server "Longhorn", any transactions that are "in-doubt" (the transaction manager is no longer around and we have no clue on the status of the transaction) will be rolled back automatically, effectively removing all the changes from disk. So, in that sense, you would not have any file turds you have to clean up after a system crash (like you would have if you were doing a "temp file -> rename" type of methodology.

Now, as is usual in transaction systems, you have to be _very_ aware and _very_ cautious of long-running transactions as it could cause some long-term problems that you don't want to deal with. So, there certainly are scenarios where Transactional NTFS just is not a good fit. As tends to be the case with all technology, make sure you are using the right tool for the right job :).

Announcing: New "Developer, Meet Server" Screencast Series

jolson — Sat, 10 Mar 2007 01:27:41 GMT

Welcome everybody to the inaugural, first ever, grand opening, “opening the doors to the public” celebration day kick-off for the new “Developer, Meet Server” screencast series. I am your sincere, humble (as ever), gracious and wonderful host/chef/tour guide Jason Olson.

In this series of screencasts we will be diving through all the various cool developer-oriented enhancements on the Windows Server "Longhorn" platform that developers can leverage to build robust and powerful applications. Specifically, in this first screencast, we will take a quick peek at Transactional NTFS. In future screencasts, we will go into a more detailed look at Transactional NTFS, so think of this as a quick introduction.

The screencast can be found in one of the main tubes on the Intarw3b known as the ever-lovely Channel 9: http://channel9.msdn.com/Showpost.aspx?postid=289816.

So sit back, grab a cup of coffee/tea/InsertYourFavoriteLocalTastyBeverageHere, and enjoy this screencast.

LHS Feature Of The Week #11 - Transactional NTFS + USB Flash Drive

jolson — Sat, 02 Dec 2006 01:02:46 GMT

Hello everyone, and welcome back to the “Feature Of The Week” newsletter. This week, we will revisit a prior Fear Of The Week technology that we discussed in order to show off a cool capability of that technology. If you are a developer wanting to mess around with TxF (Transactional NTFS), you should especially read on :).

Did You Know? Did you know that on Vista, Transactional NTFS will work with an NTFS-formatted USB Flash Drive? It’s true! If you want to play around with Transactional NTFS and how it can interact with SQL, for instance, but you only have one hard drive, you don’t have to worry about Secondary Resource Managers and all that hubbub, gizmos, doodads, and what-cha-ma-call-its. Just pop in that NTFS-formatted USB Flash Drive, and play around until your heart’s content.

Since the USB Flash Drive is not the system drive (well, we hope it’s not the system drive in this case :P), you can circumvent the need for secondary Resource Managers when wanting to test out interoperability scenarios with Transactional NTFS (like TxF + WCF, TxF + SQL, TxF + MSMQ, etc.).

Happy Coding :).

Generic Transactional NTFS Links...

Documentation

Windows SDK

Videos/Webcasts

[Video] Vista Transactional File System
[Webcast] The Revolutionary Vista TxF Infrastructure
[Webcast] Developing Apps For Vista With TxF

Blogs

Transaction Resources List

jolson — Fri, 08 Sep 2006 21:05:41 GMT

I'll be updating this post over time to reflect new resources that are made available regarding the transactions platform in Windows.

Websites:

[Transactions] Transactions Programming Forum - MSDN
[Transactions] Transaction Management in Windows - Microsoft.com
[TxF] TxF Documentation - MSDN

Blogs:

... (Read More)

Other Transaction Resources

jolson — Wed, 06 Sep 2006 22:50:59 GMT

While I'm on the topic of Transaction resources out there, I might as well give a more detailed list. Just off the top of my head, here's the resources you should definitely be checking out if you want to get into transactions on the Windows platform (or even if you are already using transactions on the Windows platform):

Microsoft Sites:

Blogs:

Jim Johnson
Florin Lazar
Well, and I hope, me :).

If I have left out any obvious ones, please let me know so I can update my list :).

Understanding Transactions in .NET

jolson — Wed, 06 Sep 2006 22:42:30 GMT

I came across this article today by Sahil Malik. If you are wanting to get up to speed with System.Transactions in general, as well as how it integrates with ADO.NET, make sure to check it out. He also covers some general concepts regarding transactions as well. So, if you are wanting to understand and be able to explain concepts like Transaction Managers, Resource Managers, Durable Enlistments, etc., also make sure to read it :).

Enjoy!

Using TxF in Vista RC1

jolson — Tue, 05 Sep 2006 22:28:13 GMT

Jim Johnson has a new post about how to use TxF with the new model in Vista RC1. You will obviously need the RC1 bits to mess around with it.

I'm hoping to expand on this topic in the future to show some code samples of using it not only by itself, but also using it with SQL and the like. I'll also have some "context" posts around that describing the architecture and relationships around TxF so you can better understand the articles.

Until then, go check out Jim's post :).

What is TxF and why do I care?

jolson — Thu, 17 Aug 2006 07:54:16 GMT

A friend of mine made a good comment on my last post about TxF: "what the heck does TxF do and why should developers care?" I hope to answer that question in this post. If you've never heard of TxF, read on. If you have and still don't "grok it", read on. If you just happen to love the wonderful language that is expelled from my tranquil appendages as they dance over the keyboard, then by all means, read on as well :).

So, what is TxF? TxF is the "nick name" for a new technology coming with Vista and Longhorn Server: Transactional NTFS. You may have read about or used logging file systems in the past, and you may have used journaling file systems in the past as well. But TxF is a fully-transactional file system, all built on top of NTFS. And when I say "fully-transactional", I mean it supports full ACID transactions on the file system itself.

Why should you care about TxF? You should care about TxF because... (Read More)

The State Of TxF

jolson — Wed, 09 Aug 2006 21:33:20 GMT

[Usual Disclaimer: my opinions in no way reflect or represent the opinions of my employer]

You'll probably hear me start cheering for TxF after a couple of posts because it is a great technology, but at this point in the ballgame, let's take a step back and take a sobering look at the state of TxF as it exists in post-Beta 2 Vista.

As I've been getting up to speed with TxF, I've realized that there is very little up-to-date information on TxF available (except for an internal presentation that was given at an internal conference). Most likely, any blog posts you'll find on the internet regarding TxF and how to use it are no longer accurate because of the newly revamped API... (Read More)