Florin Lazar - Consistency Checkpoint : Programming

Phase0 in .Net System.Transactions

florinlazar — Sun, 09 Apr 2006 23:20:00 GMT

In a previous post (http://blogs.msdn.com/florinlazar/archive/2006/01/29/518956.aspx) I talked about Phase0 and some useful things it allows you to do. Although not referred as Phase0, phase 0 features are available in .Net System.Transactions in two forms. Actually in three, but the third is a derivate. I will make many references here to the previous post on Phase0 and thus I recommend reading it first.

The first form where Phase 0 shows up is through EnlistmentOptions.EnlistDuringPrepareRequired that you can pass to EnlistVolatile or EnlistDurable methods. This allows you to create new enlistments during Prepare phase (and after Commit was called) and thus it allows you to create the caching resource manager (RM) discussed in the earlier post. To create a cache RM, you will first call EnlistVolatile with EnlistmentOptions.EnlistDuringPrepareRequired, and then, during Prepare you will connect to the database to write the updates. The connection to the database will usually call EnlistDurable on the transaction. Similar to Phase0 in the native DTC API, during the Prepare phase, one can enlist again passing EnlistmentOptions.EnlistDuringPrepareRequired which will issue another round of Prepare on that enlistment and so on.

The second form comes with DependentTransaction. You get a DependentTransaction by calling the DependentClone method on a transaction object. You can get the other two scenarios enabled by Phase0, “early commit protection” and “async programming” by tweaking the DependentCloneOption to RollbackIfNotComplete and BlockCommitUntilComplete respectively. Then, you signal that the work is done by calling the Complete method of this dependent transaction.

Thus you will write the DoTransfer method in .Net as follows:

int DoTransfer(int amount, Transaction tx)
{
    // create the dependent clone
    DependentTransaction dtx = tx.DependentClone(DependentCloneOption. RollbackIfNotComplete);
    DB1.OpenConnection(tx);
    DB1.DoDebit(amount);
    DB2.OpenConnection(tx);
    DB2.DoCredit(amount);

// signal work complete on the “phase0” enlistment
dtx.Complete();
}

If the DependentTransaction is not Complete-ed before a Commit is issued by the client, the transaction will be aborted.

The “third” derivate form is TransactionScope itself. TransactionScope protects you against early commits by automatically handling under the covers a DependentTransaction with RollbackIfNotComplete. That is why TransactionScope is recommended for most of the transactions scenarios. Given this you can write with the same confidence the DoTransfer method as follows:

int DoTransfer(int amount, Transaction tx)
{
    // creating a TransactionScope by passing in the received transaction
    // this will set tx to be the ambient transaction and create a "rollback" dependent transaction
   using(TransactionScope ts = new TransactionScope(tx))
   {
       DB1.OpenConnection(); // note that now OpenConnection is picking the transaction from the ambient context instead of passing it as a parameter
       DB1.DoDebit(amount);
       DB2.OpenConnection();
       DB2.DoCredit(amount);

       ts.Complete();
   }
}

Other Scenarios enabled and/or made easier by DependentTransaction

The “Worker Threads Scenario”
This is explained both at http://pluralsight.com/blogs/jimjohn/archive/2005/05/01/7923.aspx and http://msdn2.microsoft.com/en-US/library/ms229976(VS.80).aspx

Passing transaction ownership in a sequence of threads
Described at http://pluralsight.com/blogs/jimjohn/archive/2006/01/01/17769.aspx.

Async Programming
This is the scenario described in my previous post where the client asks asynchronously a service to do some work as part of the transaction, but doesn’t want/need to wait until the service completes the work. The code will look like the following in .Net:

Client Code
void MainFunction()
{
    using(TransactionScope ts = new TransactionScope())
    {
        // dtx is a member of the current class of type DependentTransaction
        // its lifetime needs to be at least until the ack from the service is received
        this.dtx = Transaction.Current.DependentClone(DependentCloneOption.BlockCommitUntilComplete);
        AsyncDoTransfer(tx, AckFromDoTransfer);
        ts.Complete();
     }
}
void AckFromDoTransfer()
{
     this.dtx.Complete();
}

Server Code
void AsyncDoTransfer(tx, AckFromDoTransfer)
{
DependentTransaction dtx = tx.DependentClone(DependentCloneOption. BlockCommitUntilComplete);
AckFromDoTransfer(); // the client can Commit safely from now on since I have dtx created on the service side

    // the dependent clone auto-created by TransactionScope doesn’t help in this scenario because it rollbacks on early commit while we need a blocking clone
    using(TransactionScope ts = new TransactionScope(tx))
    {
        // doing the work that takes a lot of time
        DB1.OpenConnection();
        DB1.DoDebit(amount);
        DB2.OpenConnection();
        DB2.DoCredit(amount);
        ts.Complete();
    }

// work if finished on the service so we can safely let the transaction continue
dtx.Complete();
}

Our Own Forum is Now Live! Please Join Us at the "Transactions Programming Forum"

florinlazar — Sat, 11 Mar 2006 02:34:00 GMT

I'm really happy to announce that our own forum, dedicated to transactions in Windows and .Net is now live at http://forums.microsoft.com/MSDN/ShowForum.aspx?ForumID=388&SiteID=1

The RSS feed for the forum is http://forums.microsoft.com/MSDN/rss.aspx?ForumID=388&Mode=0&SiteID=1

I hope that by having a forum that is dedicated to Transactions technologies we can build a better channel for our customers to get quick and accurate answers to their questions. We encourage everyone in the community to participate in this forum.

I look forward to us working together to make this forum a valuable resource for Transactions discussions.

MSDTC: The Magic of Phase Zero (Phase0) – Or – When Using 2PC Transactions Is Not Enough

florinlazar — Mon, 30 Jan 2006 01:24:00 GMT

The most known technique of implementing distributed transaction is the "two-phase commit" (2PC). Here is a quick summary of how this technique works:
   - Active phase: After the transaction is created and before the "commit" message is issued by the creator, the transaction is in the "active" state. During this phase, resource managers can "enlist" and become part of the transaction. When "commit" is issued, the transaction moves to Phase1.
   - Phase1 (preparing phase): in which the superior transaction manager (TM), after receiving the Commit message from the creator of the transaction, "asks" the enlisted resource managers or the subordinate transaction managers to "prepare" for commit; if all respond with a successful "prepared" answer then the transaction moves to Phase 2, otherwise the transaction is aborted
   - Phase 2 (committing phase) the superior TM writes a durable log entry about the status of the transaction and then starts sending "commit" messages to all the resource managers and subordinate TMs part of the transaction. In the case of any type of error (communication, computer crash etc), the superior TM will continue to send the "commit" messages until a successful "committed" message is received from all the participants. When all the participants responded "committed", the log entry corresponding to the current transaction is removed from the log and the transaction ends.

In addition to the "two-phase commit" technique, MSDTC supports another phase called Phase Zero (Phase0) which occurs after the creator of the transaction calls Commit and before Phase1 starts. To participate in Phase0 you need to enlist as a Phase0 participant: http://msdn.microsoft.com/library/?url=/library/en-us/cossdk/htm/transactionphase0enlistment_6j04.asp?frame=true and http://msdn.microsoft.com/library/?url=/library/en-us/cossdk/htm/pgdtc_dev_8ldf.asp?frame=true. MSDTC will always start with Phase0 and remain there until all the Phase0 participants responded.

The great benefit of Phase0 is that during this phase, any type of enlistment (including Phase0) in the transaction it is still allowed. Once the transaction moves to Phase1, the new enlistments will be denied. Since you can create a new Phase0 enlistment during Phase0, MSDTC executes this phase in waves. After "commit" is issued, the TM will first send "Phase0Request" messages (http://msdn.microsoft.com/library/en-us/cossdk/htm/itransactionphase0notifyasync_8ywk.asp) to all known Phase0 enlistments. If all of them successfully replied Phase0Done (http://msdn.microsoft.com/library/en-us/cossdk/htm/itransactionphase0enlistmentasync_5pt1.asp), the TM will look for new Phase0 enlistments that occurred meanwhile and it will start another Phase0 wave if any is found and so on. Phase0 will end when no other new Phase0 enlistment is found after a wave. After Phase0 ends, the transaction moves to Phase1.

Phase0 allows the following three scenarios: caching resource managers, protection against "early" commits and "safe" propagation of transactions in async programming.

Caching Resource Managers
Using a Phase0 enlistment, a middle-tier component can delay the connection to the database(s) as long as possible, acting as a cache to the requests from the client and thus reducing network traffic and database locks. This works great for those scenarios in which the client does many changes to the same data before it finally decides to end and save the work. When the client "ends" the work, the transaction is committed, and the TM issues a Phase0Request to the caching component. Before replying Phase0Done, the component will open all the necessary connections to the databases that need to be involved and it will persist the final data. These databases will now be "durably" enlisted in the transaction. After Phase0Done is received, the TM will continue with the 2PC and commit the transaction. See also http://msdn.microsoft.com/library/?url=/library/en-us/cossdk/htm/pgdtc_dev_0y2b.asp?frame=true

Protection Against "Early" Commits
In general, 2PC transactions help to ensure integrity of data during different types of system errors. Using Phase0 you can also protect from badly written software.

Let me give an example: let's say a transaction is created on the client side and then propagated to a middle tier component that is asked to do work as part of that transaction. The work that this component needs to do involves two databases, DB1 and DB2, as follows:

int DoTransfer(int amount, transaction tx)
{
DB1.OpenConnection(tx);
DB1.DoDebit(amount);
DB2.OpenConnection(tx);
DB2.DoCredit(amount);
}

The component is using the received transaction to communicate with the two databases to ensure the integrity of its data. Both Debit and Credit operations need to be successful or none should occur. This looks great until now. But what if the client doesn't wait for DoTransfer to finish and calls Commit right after the DB1.DoDebit returned and before DB2.OpenConnection started? A badly written client can do that, maybe the programmer wanted to use multi-threading but didn't quite get it right. What will happen is that the component is left with inconsistent data in the databases, a debit operation that occurred without a credit operation. There is nothing wrong with the transaction itself; from the point of view of the 2PC transaction, only one enlistment occurred and that one responded successfully to both "prepare" and "commit" phases. It is the application logic that is wrong here.

Can you protect against this type of situation? Yes, you can. One way is to write bug-free code, but mistakes happen. If you don't own the client code, the problem is even harder. The best solution is to use a Phase0 enlistment and modify the middle tier code as follows:

int DoTransfer(int amount, transaction tx)
{
EnlistPhase0(tx); //Creates a Phase0 enlistment with the transactions

DB1.OpenConnection(tx);
DB1.DoDebit(amount);
DB2.OpenConnection(tx);
DB2.DoCredit(amount);

SignalWorkCompletedToPhase0Enlistment(tx); // signals the Phase0 enlistment that the work was completed
}

Your Phase0 enlistment should start with a "WorkCompleted" flag set to false. The method SignalWorkCompletedToPhase0Enlistment mentioned above will set this flag to true. If a Phase0Request is received and the flag is still false, it means that the Commit was issued before the DoTransfer finished its work. At this point, you have two options:
- one option (the recommended one when "early commits are not expected) is to abort the transaction and log some error; if you own the client code you might want to catch and fix these "early" commits
- the other option is to hold the Phase0Request, received while DoTransfer is still doing work, until the flag becomes true and only then let the transaction continue by replying with Phase0Done; use this option when you expect to receive "early" commits as part of your system logic and flow.

Asynchronous Programming
This is very similar to the early commits scenarios mentioned above, but this time, the early commits are "by design". One can encounter scenarios where a piece of the application starts a transaction and delegates the work asynchronously to other parts of the application (a different thread of execution, a remote location) and without waiting for a response continues with committing the transaction. In these scenarios, we will choose the second option from above, the one that blocks the commit until work is finished. The code is similar:
Client code:
  Transaction tx = StartTransaction();
  AsyncDoTransfer(tx); // like create a separate thread and let it do the transfer operation etc
  SomeOtherAsyncWork(tx);
  CommitTransaction(tx);

Server code:
  // inside AsyncDoTransfer that receives the transaction from the client
  EnlistPhase0AndBlock(tx); // if we get a Phase0Request we will block and wait until SignalWorkCompletedToPhase0Enlistment is called and only then reply with Phase0Done

DB1.OpenConnection(tx);
DB1.DoDebit(amount);
DB2.OpenConnection(tx);
DB2.DoCredit(amount);

SignalWorkCompletedToPhase0Enlistment(tx); // signals the Phase0 enlistment that the work was completed
return;

Looks nice and safe... Well, there is a problem even with this code. What if, Commit is called even before we are able to enlist in Phase0 in the AsyncDoTransfer method? We can end up again with some inconsistent data. The solution is to create some sort of acknowledgment from the server to the client saying that at least it got to the phase0 enlistment: "hey client, I know I'm slow and I will do your requested work later but here is my ack that you can Commit your transaction safely any time from now on". To accomplish this we will have to use phase0 enlistments on both the client and server side as follows:

Client Code:
  Void MainFunction()
  {
    Transaction tx = StartTransaction();

    EnlistPhase0AndBlock(tx); // if we get a Phase0Request we will block and wait until SignalWorkCompletedToPhase0Enlistment is called and only then reply with Phase0Done
    AsyncDoTransfer(tx, AckFromDoTransfer); // like sending an async message to a server and let it do the transfer operation etc

    CommitTransaction(tx);
}
void AckFromDoTransfer()
{
    SignalWorkCompletedToPhase0Enlistment(tx); // signals the Phase0 enlistment that the work was completed
}

Server Code:
  void AsyncDoTransfer(tx, AckFromDoTransfer)
  {
   EnlistPhase0AndBlock(tx); // if we get a Phase0Request we will block and wait until SignalWorkCompletedToPhase0Enlistment is called and only then reply with Phase0Done

    AckFromDoTransfer(); // the client can Commit safely the transaction from now, since I'm protected by phase 0

    // doing the work that takes a lot of time
    DB1.OpenConnection(tx);
   DB1.DoDebit(amount);
    DB2.OpenConnection(tx);
    DB2.DoCredit(amount);

SignalWorkCompletedToPhase0Enlistment(tx); // signals the Phase0 enlistment that the work was completed
}

The phase0 enlistment on the client side will make sure that we keep the transaction in phase0 until we get all the acks from our servers. Look at this as a method to pass phase0 "ownership" from client to servers.

All these work due to the "magic" of phase0 that allows "infinite" number of enlistments, until no new phase0 enlistments are being created and the existing ones replied with a successful Phase0Done. I compare the Phase0 enlistment to the AddRef method from the good old COM lifetime management. A Phase0 enlistment will call "AddRef" on the "active phase" of the transaction, while a Phase0Done will call "Release" on the counter. While the counter is higher than zero, the transaction will not be allowed to enter Phase1.

Why and When to use transactions?

florinlazar — Tue, 04 Oct 2005 11:07:00 GMT

Many people ask - do I need transactions? Why do I need them? When to use them?

The answer is simple: use them all the time, unless you have a very good reason not to (for instance, don't use atomic transactions for "long running activities" between businesses). The default should always be yes. You are in doubt? - use transactions.

Why are transactions beneficial? They help you deal with crashes, failures, data consistency, error handling, they help you write simpler code etc. And the list of benefits will continue to grow with time.

Don't think in terms of transactions only when you talk to two or more databases. Starting with technologies like LTM, transactions are made available to be used for any multi-action operation. "X = A+B; Y = C*D;" can be transactional or atomic, as some prefer to name it, and can be seen as a single unit of work.

That is also why support for transactions is available almost everywhere. In database systems, in COM+, in ASP.NET, in .Net Framework with System.Transactions and EntepriseServices, in Indigo, in Biztalk etc

You might be saying now: "but my friend X doesn't use transactions - why it is important that I use them?". It is mostly a problem on the education side. The industry didn't talk a lot about transactions and their usage until recently. The original scope of transactions was limited to database scenarios. But technologies around transactions evolved and continue to evolve, in terms on capabilities, ease of use, flexibility, performance etc.

We used to talk about composable objects (as in OOP). We currently talk a lot about composable web services (as in SOA). The technologies are already here to start talking about composable atomic units of work.

How about the future? One avenue is STM or "software transactional memory". Maybe another one is transactional support at hardware level.

Transactions Sessions at PDC 2005

florinlazar — Wed, 14 Sep 2005 20:31:00 GMT

If you are at PDC this year, don't forget to attend the sessions related to transactions. An easy way to find these sessions is to go to http://commnet1.microsoftpdc.com/content/sessions.aspx and search by Keyword="transaction".

I especially recommend the session called "FUN320 - Windows Vista & "Longhorn" Server: Improving Reliability Using System.Transactions and the Transactional NTFS and Registry". Jim Johnson, our architect for transactions is presenting this session together with Dana Groff, the program manager in Core File Services focusing on transaction technologies.

Here is the full list:

FUN320 Windows Vista & "Longhorn" Server: Improving Reliability Using System.Transactions and the Transactional NTFS and Registry

Day/Time: Thursday, September 15 5:15 PM- 6:30 PM Room: 406 AB

Speaker(s): Dana Groff, Jim Johnson

Session Type(s): Breakout

Session Level(s): 300

Track(s): Fundamentals

You can build significantly more reliable applications by using the expanded and enhanced classes in the System.Transactions namespace. This session covers how to use new transaction-oriented programming techniques for application stability and robustness. Learn about the new systems in Windows Vista and "Longhorn" Server for supporting Transacted Files (TxF) and Transacted Registry (TxR) operations in both native and managed code.

FUN323 Microsoft Research: Future Possibilities in Concurrency

Day/Time: Friday, September 16 8:30 AM- 9:45 AM Room: 406 AB

Speaker(s): Tim Harris

Session Type(s): Breakout

Session Level(s): 300

Track(s): Fundamentals

Learn about the investment Microsoft Research is making in future languages and tools to help write software that will gain the maximum benefit from tomorrow's multi-processor/multi-core hardware. In this session, we start by introducing some of the tools and techniques we've developed for tracking down and preventing bugs in multi-threaded software. We then turn to the language features that we're prototyping in research: our work on transactional memory is developing a programming model where data can be shared directly between threads without needing to worry about locking or low-level deadlocks; our work on synchronization is leading to a unified abstraction for co-ordination between threads and between processes. We'll demo some of our prototype systems, showing how these techniques can lead to software which is not only easier to develop, but which can scale from uniprocessors up to highly parallel systems.

Distributed Atomic Transactions - Wishful Thinking?

Day/Time: Tuesday, September 13 9:00 PM- 10:00 PM Room: 511 C

Session Type(s): Birds of a Feather

Session Level(s): 300

Web Services, via SOAP, is the preferred choice of doing wide-reaching distributed communications today. However, one of the few factors that has hindered its adoption in the enterprise today is its lack of ability to handle transactions between remote un-trusted (independent) boundaries. There are two schools of thought today with regards to transactions --- The Atomic Way or The Compensating Way which pretty much takes out the atomicity of ACID. Indigo supports WS-AT via Microsoft DTC. Will this be widely adopted in the enterprise collaboration space? Or are we living in Fool's Paradise?

COM307 Windows Communications Foundation ("Indigo"): Writing Reliable and Transacted Distributed Applications

Day/Time: Wednesday, September 14 3:15 PM- 4:30 PM Room: 403 AB

Speaker(s): Shy Cohen

Session Type(s): Breakout

Session Level(s): 300

Track(s): Communications

What do you do when your network connection breaks half-way through a request? How do you synchronize state changes across different Web services? How do you overcome a system crash without losing important messages? Windows Communications Foundation (formerly codename "Indigo") provides simple and powerful reliability mechanisms that allow you to easily address these types of network and application issues. Take an in-depth look at reliable sessions, queues, and distributed transactions, and how these technologies are used to achieve reliable, transacted communication.

COM202 Windows Communications Foundation ("Indigo"): A Lap around the Windows Communications Foundation

Day/Time: Tuesday, September 13 1:00 PM- 2:15 PM Room: Halls C & D (Petree Hall)

Speaker(s): Omri Gazitt

Session Type(s): Breakout

Session Level(s): 200

Track(s): Communications

Windows Communications Foundation (formerly codename "Indigo") is Microsoft's new platform technology for building distributed applications, bringing the best of ASP.NET Web services, .NET Remoting, Enterprise Services, WSE, and System.Messaging into one unified framework. This session introduces the core concepts in the Windows Communications Foundation programming model-addresses, bindings, contracts, and behaviors-and presents the Windows Communications Foundation feature set through the lens of these concepts. This session covers security, reliable messaging, transactions, queues, hosting, transports, interop/integration, and more. If you have never seen an introduction to Windows Communication Framework, this session is for you.

The slides from these sessions are currently available at http://commnet.microsoftpdc.com/content/downloads.aspx

What do you want to know about transactions?

florinlazar — Sun, 08 May 2005 00:02:00 GMT

Please don't hesitate to send me your suggestions or questions or topics you would like to be discussed in this blog. You can post a comment to this post or use http://blogs.msdn.com/florinlazar/contact.aspx to let me know. I'll prioritize the list and cover as much as I can.

ADO.NET/System.Data blog

florinlazar — Tue, 21 Sep 2004 01:45:00 GMT

Today, in most of the cases, transactions (distributed or not) are used in combination with ADO.NET (System.Data). Angel Saenz-Badillos talks about ADO.NET and Whidbey in his blog at http://blogs.msdn.com/angelsb/ Consequently he talks about our System.Transactions as well, so make sure to take a look there too.

Of course, my dream and my team's dream is to make transactions be used everywhere and anytime (not only in combination with ADO.NET) to ensure correctness and reliability in software. And System.Transactions is a really big step towards making this dream come true.

Developing software in a non-privileged environment

florinlazar — Sat, 10 Jan 2004 02:49:00 GMT

You (i.e. your login account) don’t have to be a member of local Administrators group to do your daily tasks on your computer, including developing software. It might take a while to get used to it, but with time it will be as normal as using the seat belt in your car. The benefit is not only in ensuring a more secure environment, but also the software you produce is more likely to be available for use in a non-privileged environment. The software should not require elevated privileges unless it really needs them. Here are some resources about developing software as a non-admin: Developing Software in Visual Studio .NET with Non-Administrative Privileges and How to develop code as a non-admin.