I’m not sure how many people know that Windows comes with an embeddable, transactional database engine which is available to developers through the Windows SDK. The ESENT database engine can be used whenever an application wants high-performance, low-overhead storage of structured or semi-structured data. This can range from something as simple as a hash table which is too large to store in memory to a complex application with many tables, columns and indexes. ESENT is used by the Active Directory, Windows Desktop Search, Windows Mail and several other Windows services and a slightly modified version of the code is used by Microsoft Exchange to store all its mailbox data. The ESENT API is available through the SDK and can be used on all versions of Windows from Windows Server 2000 on up.

The significant technical features of ESENT include:

·         ACID transactions with savepoints, lazy commits and robust crash recovery.

·         Snapshot isolation.

·         Record-level locking — multi-versioning provides non-blocking reads.

·         Highly concurrent database access.

·         Flexible meta-data (tens of thousands of columns, tables and indexes are possible).

·         Indexing support for integer, floating point, ASCII, Unicode and binary columns.

·         Sophisticated index types including conditional, tuple and multi-valued.

·         Individual columns can be up to 2GB in size. A database can be up to 16TB in size.

·         Can be configured for high performance or low resource usage.

·         No administration required (even the database cache size can adjust itself automatically).

·         No download. Your application uses the esent.dll which comes with the operating system.

Caveats: ESENT should only be used for applications which have simple, predefined queries; applications that want to do ad-hoc queries should investigate a storage solution that provides a query layer. The database file cannot be shared between multiple processes simultaneously.

To use ESENT you just need to link with esent.lib and include esent.h, which is part of the Windows SDK. Here is a sample program that creates a database with one table, inserts a record and then retrieves the data from the record:

// We need to define JET_VERSION to be at least 0x0501 to get access to the

// JET_bitDbOverwriteExisting option and JetCreateInstance. That means this

// program will run on Windows XP and up (Windows Server 2003 and up).

#undef JET_VERSION

#define JET_VERSION 0x0501

 

#include <stdio.h>

#include <string.h>

#include <esent.h>

 

// One possible error-handling strategy is to jump to an error-handling

// label whenever an ESENT call fails.

#define Call(func) { \

       err = (func); \

       if(err < JET_errSuccess) { \

       goto HandleError; \

       } \

} \

 

int main(int argc, char * argv[]) {

       JET_ERR err;

       JET_INSTANCE instance;

       JET_SESID sesid;

       JET_DBID dbid;

       JET_TABLEID tableid;

 

       JET_COLUMNDEF columndef = {0};

       JET_COLUMNID columnid;

 

       // Initialize ESENT. Setting JET_paramCircularLog to 1 means ESENT will automatically

       // delete unneeded logfiles. JetInit will inspect the logfiles to see if the last

       // shutdown was clean. If it wasn't (e.g. the application crashed) recovery will be

       // run automatically bringing the database to a consistent state.

       Call(JetCreateInstance(&instance, "instance"));

       Call(JetSetSystemParameter(&instance, JET_sesidNil, JET_paramCircularLog, 1, NULL));

       Call(JetInit(&instance));

       Call(JetBeginSession(instance, &sesid, 0, 0));

 

       // Create the database. To open an existing database use the JetAttachDatabase and

       // JetOpenDatabase APIs.

       Call(JetCreateDatabase(sesid, "edbtest.db", 0, &dbid, JET_bitDbOverwriteExisting));

 

       // Create the table. Meta-data operations are transacted and can be performed concurrently.

       // For example, one session can add a column to a table while another session is reading

       // or updating records in the same table.

       // This table has no indexes defined, so it will use the default sequential index. Indexes

       // can be defined with the JetCreateIndex API.

       Call(JetBeginTransaction(sesid));

       Call(JetCreateTable(sesid, dbid, "table", 0, 100, &tableid));

       columndef.cbStruct = sizeof(columndef);

       columndef.coltyp = JET_coltypLongText;

       columndef.cp = 1252;

       Call(JetAddColumn(sesid, tableid, "column1", &columndef, NULL, 0, &columnid));

       Call(JetCommitTransaction(sesid, JET_bitCommitLazyFlush));

 

       // Insert a record. This table only has one column but a table can have slightly over 64,000

       // columns defined. Unless a column is declared as fixed or variable it won't take any space

       // in the record unless set. An individual record can have several hundred columns set at one

       // time, the exact number depends on the database page size and the contents of the columns.

       Call(JetBeginTransaction(sesid));

       Call(JetPrepareUpdate(sesid, tableid, JET_prepInsert));

       char * message = "Hello world";

       Call(JetSetColumn(sesid, tableid, columnid, message, strlen(message)+1, 0, NULL));

       Call(JetUpdate(sesid, tableid, NULL, 0, NULL));

       Call(JetCommitTransaction(sesid, 0));    // Use JetRollback() to abort the transaction

      

       // Retrieve a column from the record. Here we move to the first record with JetMove. By using

       // JetMoveNext it is possible to iterate through all records in a table. Use JetMakeKey and

       // JetSeek to move to a particular record.

       Call(JetMove(sesid, tableid, JET_MoveFirst, 0));

       char buffer[1024];

       Call(JetRetrieveColumn(sesid, tableid, columnid, buffer, sizeof(buffer), NULL, 0, NULL));

       printf("%s", buffer);

 

       // Terminate ESENT. This performs a clean shutdown.

       JetCloseTable(sesid, tableid);

       JetEndSession(sesid, 0);

       JetTerm(instance);

       return 0;

 

HandleError:

       printf("ESENT error %d\n", err);

       return 1;

}

 There is a lot of information about ESENT, including complete documentation for all the APIs, on MSDN http://msdn.microsoft.com/en-us/library/ms684493(VS.85).aspx. You can get an overview of the underlying technologies by reading the “Using Extensible Storage Engine” topics http://msdn.microsoft.com/en-us/library/aa964813(VS.85).aspx.

Laurion Burchall
Software Design Engineer, ESENT Team