ESE/ESENT Database Stuff

Version 1.5 of ManagedEsent released. PersistentDictionary goodness

2009-09-13T05:02:00Z

There is a new release of ManagedEsent on Codeplex. There are some new APIs, constants for the new Windows 7 ESENT features, bugfixes and performance improvements.

There is also a new Esent.Collections.dll which contains a PersistentDictionary generic class. This is a drop-in compatible replacement for the generic Dictionary or SortedDictionary classes for applications that need very simple data persistence. The database is stored in a location specified in the dictionary constructor.


var dict = new PersistentDictionary<string, int>("DataDirectory");
dict["foo"] = 17;
dict["bar"] = 18;

The dictionary entries are persisted automatically and updates are transacted providing data consistency and crash resilience.

6 New ESENT features in Windows 7

2009-08-19T00:35:00Z

A quick look at some of the new features that are available in the Windows 7 version of ESENT. You'll need esent.h from the Windows 7 SDK to see these definitions:

1. Column Compression

Columns of type JET_coltypLongBinary and JET_coltypLongText can now be compressed. There are two ways of enabling compression.

Create the column with the JET_bitColumnCompressed flag. This can only be used with long-value columns, otherwise ESENT will return JET_errColumnCannotBeCompressed.
Set the column with JET_bitSetCompressed. This flag can be used with any column type, we just ignore it for non-compressable columns. Alternatively you can use JET_bitSetUncompressed when updating a compressed column with data that you don't want compressed.

How does compression work?

For 'short' columns (less than 1kb in size) we look to see if all the data is 7-bit ASCII characters. If so we compress each character to its 7-bit value. This is the only compression done for short values. It was added to help with the fact that Exchange stores a lot of strings which are often just 7-bit ASCII values, e-mail headers for example. Turning a 16-bit Unicode character to a 7-bit ASCII value gives a large savings and is cheap to do.

For 'long' columns (greater than 1kb) we apply Microsoft's XPRESS compression, which is similar to ZIP or RAR compression. We only store the compressed data if it is smaller than the uncompressed data. Of course determining that data doesn't compress well costs CPU cycles so if you are storing already compressed data you shouldn't use compressed columns.

Unlike SQL Server's page compression this feature is designed to compress large column values, not remove duplication when multiple records have similar values. This is done because this feature was added for Microsoft Exchange, where most of the data is stored in relatively large BLOB columns (e.g. e-mail bodies).

What else do I need to know?

Nothing, really. A compressed column looks just like a normal column and can be indexed, set and retrieved as normal. There is no way to retrieve the compressed data, you will always get uncompressed data when calling JetRetrieveColumn or a similar API. If you want to know how well a record compressed use the new JetGetRecordSize2 API which gives both the logical (uncompressed) and physical (compressed) size of a record.

2. 32kb and 16kb page support

ESENT now supports 32kb, 16kb and 2kb pages. Changing the page size affects b-tree performance and you may find your application runs faster with bigger pages. Bigger pages also allow more columns to be set in one record before getting JET_errRecordTooLarge.

3. JetPrereadKeys

If you want to retrieve non-contiguous records quickly you can use JetMakeKey to calculate their keys and then call JetPrereadKeys. Prereading the records will issue the I/O as efficiently as possible, which makes retrieving the records a LOT faster if I/O is involved. Records must be preread in either ascending or descending key order. JetPrereadKeys may not be able to preread all the records so it tells you how many keys were preread so that you can issue a preread call for the rest after processing the initial batch.

4. Space Hints

In older versions of ESENT you could specify the initial size of a table/index, but didn't have much control over how the table size grew. Space hints allow you to give the initial size of a table/index and then indicate how much space the table should use as it grows. This feature is designed to allow databases with large numbers of tables where the tables start out small (perhaps one page) and then grow their allocation size so that they have contiguous space.

5. JET_paramWaypointLatency and JET_bitReplayIgnoreLostLogs

Write-ahead logging depends on the underlying storage being durable, which isn't really true for inexpensive desktop-class machines. To improve database recoverability we have added a 'Lost Log Resiliency' (LLR) feature to the Windows 7 version of ESENT. Setting the JET_paramWaypointLatency parameter to a non-zero value will enable the LLR feature, causing ESENT to delay writes to the database until N logfiles have been generated, where N is the waypoint depth.

For example, if the waypoint is set to 1 and the current log generation is 28 then ESENT will not write any pages to the database that have been modified by log records in generation 28. This means that the database can be recovered without the most recent log but there will be data loss! If you are comfortable with some potential data loss and want to emphasise database recovery then set JET_paramWaypointLatency to 1 (or a small value, there is a performance penalty for maintaining the waypoint). You have to use the JET_bitReplayIgnoreLostLogs option with JetInit2 to tell ESENT that it is OK to perform a lossy recovery.

This will allow you to recover your database even if a system crash loses the most recent log. In our power-down testing this improves the chances of a sucessful database recovery after a machine with non-battery-backed write caching is powered down.

6. JET_bitTermDirty

If you want to shutdown REALLY quickly you can use JET_bitTermDirty, which won't even flush the database cache. Recovery will take longer the next time but no data will be lost.

I know this was a very quick summary. If you have specific questions, leave them in the comments or use the 'contact me' area and I'll try to answer them or provide samples.

Multimethods in C# 4.0 with 'dynamic'

2009-08-14T01:01:00Z

(Continuing from the post on implementing the Visitor pattern in C# 4.0)

A virtual method is polymorphic in one dimension (the runtime class of the object the method is attached to). Multimethods (multiple dispatch) are polymorphic in multiple dimensions, which can be very useful in some cases. The dynamic keyword in C# 4.0 allows for method selection that depends on the runtime class of the arguments, not just the attached object. This allows true multimethods. (Of course the lookup rules for dynamic are complex so extremely complex cases might not resolve the way you want).

The basic idea is to create method overloads which take all the (subclass1, subclass2) combinations that you are interested in and then have the (superclass, superclass) method cast its arguments to dynamic and then call the method again and let runtime resolution do its thing. In the spirit of the Wikipedia article on multimethods, here is a spaceship/asteroid collision example:


using System;
namespace MultiMethods
{
    class Program
    {
        class Thing { }
        class Asteroid : Thing { }
        class Spaceship : Thing { }
        static void CollideWith(Asteroid x, Asteroid y) 
        {
            Console.WriteLine("Asteroid hits an Asteroid");
        }
        static void CollideWith(Asteroid x, Spaceship y) 
        {
            Console.WriteLine("Asteroid hits a Spaceship");
        }
        static void CollideWith(Spaceship x, Asteroid y) 
        {
            Console.WriteLine("Spaceship hits an Asteroid");
        }
        static void CollideWith(Spaceship x, Spaceship y) 
        {
            Console.WriteLine("Spaceship hits a Spaceship");
        }
        static void CollideWith(Thing x, Thing y)
        {
            dynamic a = x;
            dynamic b = y;
            CollideWith(a, b);
        }
        static void Main(string[] args)
        {
            var asteroid = new Asteroid();
            var spaceship = new Spaceship();
            CollideWith(asteroid, spaceship);
            CollideWith(spaceship, spaceship);
        }
    }
}

Implementing the Visitor pattern with the 'dynamic' feature of C# 4.0

2009-05-29T02:05:00Z

A lot of the talk around the new dynamic keyword in C# 4.0 (available in the beta of Visual Studio 2010) is around interaction with external components (dynamic languages, HTML DOM, COM) but I'm most excited about being able to stragetically add dynamic lookup to an ordinary C# program. Normally, when an object is passed to a method the method lookup depends on the compile-time type of the object. This means that code like this:


class MyClass
{
 public void Foo(Stream s) { Console.WriteLine("Stream"); }
 public void Foo(MemoryStream s) { Console.WriteLine("MemoryStream"); }
}
class Program
{
 static void DoSomething(Stream s)
 {
  MyClass obj = new MyClass();
  obj.Foo(s);
 }
 static void Main(string[] args)
 {
  DoSomething(new MemoryStream());
 }

Will print out 'Stream', not 'MemoryStream'. Method resolution for dynamic objects has to use the runtime type of the object, so simply by changing DoSomething() to take a dynamic instead of a Stream we get a program that prints out 'MemoryStream'. Basically, dynamic lets us implement multi-methods in C#! (Which I really do find very exciting).

A good use of this feature is when implementing the Visitor pattern. C# code that visits all the nodes in a Linq expression tree tends to be quite ugly, with a huge switch statement and a lot of casts to traverse the tree. By casting the Expression class to dynamic we can call a Visit() method which depends on the exact subclass of Expression. The code below prints an expression tree without needing any explicit conditional or loop statements (using dynamic makes the runtime do all the boring lookup work for us). It is also quite a bit shorter than the C# 3.0 equivalent. Note that if there isn't an implementation for a specific expression type then the Visit method for the parent class will be called.


using System;
using System.CodeDom.Compiler;
using System.IO;
using System.Linq.Expressions;
namespace MultiMethods
{
    class ExpressionPrinter
    {
        private readonly IndentedTextWriter output;
        public ExpressionPrinter(TextWriter output)
        {
            this.output = new IndentedTextWriter(output);
        }
        public void Print(Expression exp)
        {
            dynamic d = exp;
            this.Visit(d);
        }
        private void Visit(Expression exp)
        {
            output.WriteLine("Expression");
        }
        private void Visit(BinaryExpression exp)
        {
            output.WriteLine("BinaryExpression: {0}", exp.NodeType);
            this.VisitSubExpression(exp.Left);
            this.VisitSubExpression(exp.Right);
        }
        private void Visit(ConditionalExpression exp)
        {
            output.WriteLine("ConditionalExpression");
            this.VisitSubExpression(exp.Test);
            this.VisitSubExpression(exp.IfTrue);
            this.VisitSubExpression(exp.IfFalse);
        }
        private void Visit(ConstantExpression exp)
        {
            output.WriteLine("ConstantExpression: {0}", exp.Value);
        }
        private void Visit(MethodCallExpression exp)
        {
            output.WriteLine("MethodCall: {0}", exp.Method.Name);
        }
        private void Visit(ParameterExpression exp)
        {
            output.WriteLine("ParameterExpression: {0}", exp.Name);
        }
        public void Visit(UnaryExpression exp)
        {
            output.WriteLine("UnaryExpression: {0}", exp.NodeType);
            this.VisitSubExpression(exp.Operand);
        }
        private void VisitSubExpression(Expression exp)
        {
            output.Indent++;
            dynamic d = exp;
            this.Visit(d);
            output.Indent--;
        }
    }
    class Program
    {
        static void Main(string[] args)
        {
            var p = new ExpressionPrinter(Console.Out);
            Expression<Func<int, int>> e = i => (0 == i%2) ? -i * i : Math.Abs(i);
            p.Print(e.Body);
        }
    }
}

The output is:


ConditionalExpression
    BinaryExpression: Equal
        ConstantExpression: 0
        BinaryExpression: Modulo
            ParameterExpression: i
            ConstantExpression: 2
    BinaryExpression: Multiply
        UnaryExpression: Negate
            ParameterExpression: i
        ParameterExpression: i
    MethodCall: Abs

Kobe - the Web 2.0 application that wasn't

2009-04-17T22:22:00Z

Microsoft has released Kobe, "a getting started resource kit for planning, architecting, and implementing Web 2.0 applications and services using the Microsoft Platform" and a lot of the response has been quite critical (Ayende, Karl Seguin). While the code, architecture and lack of tests have all been thoroughly criticized I think there is an even higher-level point that has been missed -- Kobe, a Web 2.0 sample, was released and managed in a very Web 1.0 way.

Although the Kobe Project Overview video talks about a Web 2.0 application as having an online community and fostering "active and social user engagement" Kobe itself is available though an inactive, non-social MSDN page with no hint of community. The response on the other hand was far more Web 2.0, with people discussing the project through blogs and on Twitter. At this point the web page is updated to say "This sample application is being updated to incorporate community feedback. An updated version will be released shortly.", but that level of interaction is too little, too late.

I think the biggest mistake here is not releasing a bad Web 2.0 sample, but being unprepared to engage with the online community discussing your bad Web 2.0 sample. Authentic participation on twitter, informative blogging and frequent updates would go a long way here.

iiobo C++ ESE library

2009-04-07T06:54:00Z

(I saw a link to this from the Wikipedia entry on esent)

A company called iiobo has released a C++ toolkit for using esent (I'm guessing they developed it for internal use). I haven't looked at the code, but their website says that they have:

"C++ classes that provide a higher level abstraction of Extensible Storage Engine e.g DbCursor, DbTable, DbSchema etc. These well designed abstractions allow you to seamlessly work with transactions and other database considerations inside object-oriented C++ code."

Their code is released under the BSD license.

http://www.iiobo.com/ExtensibleStorageEngine.aspx

Version 1.2 of ManagedEsent available on CodePlex

2009-04-07T06:07:00Z

Version 1.2 of the ManagedEsent interop layer for the ESENT database engine is now available on CodePlex. The three most useful additions here are:

JetEscrowUpdate
JetIntersectIndexes
Helper functions to set/retrieve columns as DateTime

JetEscrowUpdate works like Interlocked.Add, allowing a column to be updated in an atomic fashion by mutliple sessions without conflicts. There are several restrictions -- the column must have a default value, has to be JET_coltypLong and cannot be indexed.

JetIntersectIndexes takes a set of index ranges and returns the records which are found in all the index ranges. This can be used to select records that meet multiple criteria. The StockSample.cs sample application shows how to use JetIntersectIndexes.

The release is up on Codeplex.

The source code now contains the source code for Pixie which is a 'work in progress' managed object layer for ESENT. It is nowhere near complete and will be changing a lot over the coming months.

Why does JET_coltypBit return 0xFF when set to true?

2009-04-01T21:28:00Z

On our internal mailing list a user asked "When I retrieve a column of type JET_coltypBit that is set to true, why do I get 0xFF and not 0x1"?

My co-worker Ketan had to reach deep into his memory for the answer :-)

In the very old days, the Jet API was designed as a replacement for the BC7 ISAM. BC = Microsoft Basic Compiler.

http://msdn.microsoft.com/en-us/library/wts33hb3(VS.80).aspx

"When Visual Basic converts Boolean values to numeric types, False becomes 0 and True becomes -1."

Now why would do they do this? Because Basic treated Boolean/Bitwise as being the "same" operation. What does "NOT" do? It flips all the bits; hence

TRUE = NOT FALSE

-1 = ~0

Unsigned considered harmful

2009-03-27T03:22:00Z

(or is "xxx considered harmful" completely worn out as a meme?)

I believe that, in general, people should avoid unsigned variables, even when dealing with quantities which should only be positive. I have three major problems with unsigned variables:

Subtraction doesn't always make sense

Unsigned numbers are used to model positive integers and positive integers aren't closed under subtraction.While subtracting two integers always gives another integer, subtracting two positive integers doesn't always give a positive integer. Integer subtraction is much more useful than positive integer subtraction!

Of course with wraparound the subtraction gives an unsigned integer, but that doesn't make arithmetic sense. Signed integers have the same problem, but only at the extremes where wraparound occurs. Even with idealized infinite-bit-length numbers subtraction of unsigned integers is problematic.

Lack of Error-checking on underflow

If subtraction does underflow most systems won't generate a runtime exception if they go negative (which would be useful), instead they silently wrap-around (which is not useful). In the end unsigned variables simply alias invalid negative values to positive ones.

Lack of Invalid Values

It is extremely useful when a type has support for a clearly invalid/unused value. This can be used both for error-checking and loop termination. Take a loop that iterates through a string backwards:

int IchLastCommaInString(const char * const sz) {

int cch = strlen(sz);

for(int i = cch-1; i >= 0; --i) {

if(',' == sz[i]) {

return i;

}

return -1;

}

If we change the code to use an unsigned variable ("Array indexes can never be negative!") then we lose the ability to easily detect the termination condition:

int IchLastCommaInString(const char * const sz) {

int cch = strlen(sz);

for(unsigned int i = cch-1; i >= 0; --i) { <= BAD! unsigned ints are always >= 0

if(',' == sz[i]) {

return i;

}

return -1;

}

To use an unsigned variable we often end up on the dark and dismal path of "if(x > BIGNUM)" where BIGNUM is chosen to be a value so large that we 'know' that x is really a negative number in disguise.

Just use signed variables.

JET_COLUMNID scope

2009-03-17T23:47:00Z

A JET_COLUMNID identifies a column inside of a table (it is actually an index into an internal per-table array of column definition structures). The database meta-data provides a mapping from column name to columnid. The mapping of column names to columnids can change when a database is defragmented by esentutl (esentutl /d). This means that each time you open a database you should use JetGetTableColumnInfo or JetGetColumnInfo to get the ids of the columns you are going to use. This only has to be done once per table -- different sessions that are using the same table will see the name name->columnid mapping unless you do something crazy like rename your columns with JetRenameColumn (something that I do not recommend)!

What does a failing Assert() mean?

2009-03-02T00:52:00Z

When an assert goes off what does it mean? In the codebases I work in, a failing assert doesn't always indicate that the code is about to fail.

There actually are several conditions that we want to check with asserts. The first three are from Meyer's Design By Contract:

Preconditions: conditions that a method requires
Postconditions: conditions that a piece of method guarantees to be true after executing
Invariants: class invariants that a method must preserve

I also see two other real-world uses of asserts:

Catch probable client misbehaviour: These are cases where the method will work, but the arguments are probably incorrect (e.g. salary greater than $1,000,000,000).
Untested cases: in a situation where not all code-paths have been tested (legacy code) we can put asserts in paths that should work, but aren't currently tested.

The second set of uses are good, but can make code hard to change because making changes can set off a lot of asserts.

It is useful to distinguish between the cases above. Eiffel does the first three with REQUIRE, ENSURE and INVARIANT. Unfortunately, in plain C++ or C# we just have Assert() which doesn't give any context. We are moving to using different Assert-like macros in our codebase:

Assert(): this condition must be true for the code to work. Assert( NULL != pv );
Expected(): a condition that we expect to be true, but isn't required. These probably indicate client misbehavior. Expect( cbToCopy < 0x40000000);
Untested(): Document untested code paths. These can be used to expand test coverage, or indicate that changes are extra-risky.

Immutable objects in Fluent Interfaces: a lesson from Functional Programming

2009-02-16T13:17:00Z

Fluent interfaces are a nice way of building up an API which reads very nicely. Chaining methods together is often a big part of fluent interfaces, but doing so isn't always as straightforward as it looks, especially for configuration objects. Suppose I'm building up some database meta-data, and I'm defining a new column using method chaining:


 var firstColumn = DefinedAs.TextColumn("name");

Now suppose I have another table which also happens to have a text column called 'name', except it also has a default value. To save typing I decide to build on top of the column definition I already have:


 var secondColumn = firstColumn.WithDefaultValue("Jane Doe");

If the WithDefaultValue() method is implemented the wrong way it will directly modify the firstColumn structure, which is a very unfluent thing as it doesn't fit with the name of the method (a method called 'SetDefaultValue' might be expected to modify the existing object but 'WithDefaultValue' sounds like it is creating a new object). Instead, for an interface like this, I think that an existing configuration should never be modified. Instead, applying an option to a configuration object should always return a new configuration object. Doing this means that firstColumn and secondColumn will refer to different objects and the API won't surprise the user.

(Of course the functional programming people knew this all along)

The expense of creating the extra objects is something to consider, but in the vast majority of cases creating configuration objects like this shouldn't be a big part of application runtime.

Cheaply estimating the number of records in a table

2009-02-10T08:22:00Z

Counting the number of records in a table can be extremely expensive, as it requires reading all the database pages off disk. It is possible to get a reasonably accurate estimate the number of records in a table by using the JetGetRecordPosition API. JetGetRecordPosition should be called when the cursor is on a record and it fills in a JET_RECPOS structure with two numbers, the estimated position of the record table (centriesLT) and the estimated number of records (centriesTotal). Together these two numbers form a fraction indicating how far into the table the record is.

We calculate centriesTotal and centriesLT by looking at the fanout of the pages in the b-tree that lead to the current record so the numbers produced by JetGetRecordPosition aren't terribly accurate unless all records in the b-tree are the same size and are packed to the same density. The estimate can be improved greatly by taking multiple samples from different places in the b-tree. We can easily go to different places in the b-tree with the JetGotoPosition API, which tries to position the cursor to a fractional location in the table. This has the same problems as JetGetRecordPosition (it isn't very accurate) but it is perfect for this purpose.

So, we can simply use JetGotoPosition to position the cursor at different locations in the table and then call JetGetRecordPosition to get an estimate of the number of records in the b-tree. Averaging out the numbers should give a reasonable estimate. The code looks like this:


JET_ERR ErrGetApproximateRecordCount(const JET_SESID sesid, const JET_TABLEID tableid, const int cSamples, int * const pcRecords)
{
 *pcRecords = 0;
 
 JET_ERR err;
 
 int cRecordsTotal = 0;
 for(int i = 0; i < cSamples; ++i)
 {
  JET_RECPOS recpos = {0};
  recpos.cbStruct = sizeof(recpos);
  recpos.centriesTotal = cSamples + 1;
  recpos.centriesLT = i+1;
  
  if (JET_errSuccess != (err = JetGotoPosition(sesid, tableid, &recpos)))
   return err;
   
  if (JET_errSuccess != (err = JetGetRecordPosition(sesid, tableid, &recpos, sizeof(recpos))))
   return err;
   
  cRecordsTotal += recpos.centriesTotal;   
 }
 
 *pcRecords = cRecordsTotal / cSamples;
 
 return JET_errSuccess; 
}

To see how accurate this is I wrote a test program that inserted between 1 and 100,000 records into a table, each record having between 1 and 1024 bytes of data. Then I called ErrGetApproximateRecordCount with cSamples ranging from 1 to 64. I ran a total of 1,000 trials of the test program and graphed the results. It looks like somewhere around 15 samples gives a reasonable accurate number (< 5% error). Assuming a non-cached table the 15 samples will cost 1 or 2 page reads each so it should take about 1/3 of a second, which is very cheap compared to the cost of reading in the entire table.

A dbm-like interface for Esent on IronPython

2009-01-12T11:53:00Z

I've wanted to be able to use Esent as a simple, fast, reliable and persistent key/value store. The Python standard libraries contain that kind of functionality in the *dbm modules. I've been working on providing identical functionality layered on top of Esent. I've uploaded the esedb module to the release page for ManagedEsent on Codeplex. To use it you need Esent.Interop.dll and esedb.py. The module exports an open function which returns a dictionary object that is backed by an Esent database. The dictionary keys and values are restricted to strings.

>>> db = open('test.db')
>>> db['a'] = 'somedata'
>>> db['a']
'somedata'

The module is still under development, but the current functionality should be quite stable.

A C++ Refactoring Trick for "Encapsulate Field"

2009-01-08T13:21:00Z

I was working on some old code that had a structure that looked like this

struct Foo { bool fOptionEnabled; };

Tons of places in the code would check to see if the option was enabled, and do something based on it. The change I needed to make required that I do some extra work whenever the option was enabled. A good way to do this is to use the "Encapsulate Field" refactoring to reroute access to fOptionEnabled through accessors and then change the set method to be a more domain-specific "EnableOption()" and do all the required work in it.

The problem is that the amount of code affected to do this was very large and I didn't want to do it all at once. Instead I used an intermediate way of turning off just the set access to the variable (at the cost of making the structure a few bytes larger):

struct Foo { private: bool m_fOptionEnabled; public: const bool& fOptionEnabled; Foo() : fOptionEnabled(m_fOptionEnabled) {} };

This change meant that code that read fOptionEnabled continued to compile and work, while code that tried to set it started failing. This let me change all the places that turned the option on to use Foo::EnableOption(). After that I actually went ahead and encapsulated read access to the variable as well, but using this trick means you can work in smaller steps, with fewer changes between passing tests. That is always a good thing when you have to manually refactor code.

Doing refactorings like this definitely makes me appreciate the argument behind Meyer's Uniform Access Principle. In C# this change can be made quite easily by using properties (of course, hiding the implementation like that will upset some people -- they want to see the () so that they know what is going on).