Eugenio Pace - Software as a Service Architecture Guidance

You don't have a beta account with SSDS, but you'd like to use the latest LitwareHR anyway

Well, too bad...No, seriously now, as I mentioned in a previous post, we developed a "mock" (but fully functional for LitwareHR's purposes) offline SSDS proxy to enable development independently from the live SSDS service. The offline proxy is included in LitwareHR latest release, but by default, online SSDS is configured. The offline database is also installed albeit it is empty.

Here's what you need to do:

1. Insert LitwareHR metadata into the offline database (named OfflineSitka):
1. For convenience you can use this file or alternatively you can:
2. Run the ShouldPerformDeploymentProvisioning test in DeploymentProvisioningFixture (you will find this in LitwareHR.Portfolio.Datamodel.Sitka.Test project). You need to change the config file for this to work (see below).
2. Configure the web services hosts to use the offline proxy:
1. Open the web.config files on each of the Host projects under [Install Folder]\bin:

Comment out the live proxy with the offline (the setting key is "SitkaProxy"):

If you want to run the tests against the offline database, you need to do the same with the tests app.config files.

When you get your beta SSDS account, you can switch back to the live proxy and you should be ready to go.

Feedback is most welcome!

LitwareHR on SSDS available for download

The latest version of LitwareHR is now available for download here.

From our CodePlex site:

This new enhanced version of LitwareHR includes the following new features:

• Upgraded codebase for SQL Server Data Services support
  • New "DataModel"
  • New infrastructure for data access to SSDS (including caching, cross-container search, offline development, etc.)
• New UX
  • Better graphics and layout
  • Helper links for streamlining demos (demo "auto-complete")
• Various bug fixes
• New "Dependency Checker" tool to verify and install pre-requisites
• New configuration tool to setup LitwareHR (SSDS config, etc)

The MSI contains all the tools, source code and some binary dependencies (e.g. EntLib) to build LitwareHR. You will need Visual Studio 2008 to build all. This version of LitwareHR is meant to use SQL Server Data Services (SSDS) as the on-line storage provider. You can register for a beta account with this service http://www.microsoft.com/sql/dataservices/default.mspx.

As usual, any feedback is greatly welcome!

Paging in SSDS & Parallel Queries

Tim Jarvis raised a good point in my post on cross-container queries which is: how to handle paging? SSDS currently supports a very simple paging pattern that uses the entityId. By design, the first 500 entities will be returned, but the entities will be returned in entityId order. So, you can get the next 500 instances with a query like:

from e in entities where e["FieldName"]== "Field Value" && e.Id > "LastId" select e";

You can then iterate over the returned list if entities, check if you got less than 500 entities (that means that there are no more), or continue querying if you got 500 until you get less than that. In each iteration you update LastId of course.

Now, back to parallel queries, you could add an extra parameter to specify the maximum amount of total entities you are expecting as a result, monitor the amount retrieved by each thread and cancel any outstanding (parallel) queries if the limit is reached.

If you want to implement a "get me the next 1000" across containers, then of course you need to implement a way to keep the list of LastId's on each container. Probably returning this as a "query state object" to be passed back to the method afterwards.

In any case, in my experience, in most interactive scenarios where you are searching, result sets greater than 2 or 3 pages means you are not filtering adequately and maybe another approach is better: "hey, we've found lots of entities with that search criteria, maybe you should refine it?...". 

End to end demo of LitwareHR on SSDS

Here's an end-to-end demo of LitwareHR using SSDS. The total demo lasts about 11 min and you will see:

1. Tenant Provisioning and customization (takes the first 5 min approximately). I show the initial tenant provisioning (creation of a new tenant in LitwareHR, initial configuration, etc) and then a basic customization (e.g. look & feel, position entity shape, etc). You'll notice that I switch back and forth between www.litware.com and SSDS to show how entities look like in the store.
2. Using the new created instance (this takes the last 6 min). I show a hypothetical recruiter logging-in, opening new positions, then an applicant browsing the open positions and submitting a Resume and finally the recruiter browsing the posted applications. Again, you'll see how do these entities look like in SSDS.

If you want to see the video on a different window, click here.

LitwareHR on SSDS - Part VI - Unit of Work support

For years now, we have been using transactions in our systems. Years ago, there's been a democratization of the more complex concepts underlying transaction management and we've been quite happy. On the Microsoft side there's been technologies like: MTS, COM+, Enterprise Services, DTC and of course SQL Server; all of them providing great abstractions for single or distributed transaction management.

In its current incarnation, SSDS is not aware of any in-progress client side transaction. That means that if you are writing code that is hosted in COM+ for example, and you are interacting with "transaction aware" resources (e.g.: on-premise databases, MSMQ, etc) and SSDS; then SSDS will be immune and agnostic of any commits or rollbacks. Any successful operation on SSDS will be final, durable. It is up to you to "undo" anything in SSDS in case of an exception and a rollback.

This holds true, even if you only interact with SSDS. There are no "begin transaction"/"commit"/"rollback" semantics in SSDS. Each successful create, delete, update is final and consistency between operations is an application level responsibility.

In LitwareHR we did not have the requirement of interacting with multiple, heterogeneous resources. All our storage is in SSDS, so we needn't support for distributed transactions. However, we needed support for atomicity: that is treating a set of operations as a unit (e.g. during provisioning, where multiple entities are created in LitwareHR's own container: a Tenant entity, permission sets, defaults, styles, etc.)

If something failed, we wanted to be able to undo anything that might have been done in a simple way. In other words we wanted a "Unit of Work" (UoW) for a set of operations against SSDS, without pushing too many details to the application developer.

Once again, the Repository class was extended to support the concept of a UoW. This allows details to be hidden to higher layers of the application. As people might want to implement more or less sophisticated versions of UoW, Repository simply understands the semantics of this interface:

public delegate void Operation();

public interface IUoWManager
{
    Guid UoWId { get; }
    void Begin();
    void Commit();
    void Rollback();
    void RecordOnRollback( Operation Undo, IEntity entity );
    void RecordOnCommit( Operation Confirm, IEntity entity );  
}

The "UoW-aware" Repository operations now have two versions. For example Insert signatures are:

public T Insert(T entity, IUoWManager UoWManager );

public T Insert(T entity); 

In this way, UoWManager is essentially a context that can be passed across multiple operations for final confirmation or deletion.

Here are two tests that demonstrate how to use the new API:

[TestMethod]
public void Insert2EntitesCommit()
{
    SimpleUoWManager uow = new SimpleUoWManager();

    using (Repository<MockEntity> r = new Repository<MockEntity>(tenantId))
    {
        Guid id1 = Guid.NewGuid();
        MockEntity e1 = new MockEntity(id1);

        Guid id2 = Guid.NewGuid();
        MockEntity e2 = new MockEntity(id2);

        uow.Begin();

           r.Insert(e1, uow);
           r.Insert(e2, uow);

        uow.Commit();

        Assert.IsNotNull(r.GetById(id1));
        Assert.IsNotNull(r.GetById(id2));
    }
}

[TestMethod]
public void Insert2EntitesRollback()
{
    SimpleUoWManager uow = new SimpleUoWManager();

    using (Repository<MockEntity> r = new Repository<MockEntity>(tenantId))
    {
        Guid id1 = Guid.NewGuid();
        MockEntity e1 = new MockEntity(id1);

        Guid id2 = Guid.NewGuid();
        MockEntity e2 = new MockEntity(id2);

        uow.Begin();

           r.Insert(e1, uow);
           r.Insert(e2, uow);

        uow.Rollback();

        Assert.IsNull(r.GetById(id1));
        Assert.IsNull(r.GetById(id2));
    }
}

SimpleUoWManager is well...simple; so it doesn't take into account things like machine shutdown, etc. If the process this code is running on dies unexpectedly in the middle of a UoW, then data stored in SSDS might be inconsistent from an application perspective.

In this implementation SimpleUoWManager simply stores a collection of operations (callbacks) to be called when the confirmation is received (Commit is called) or a list of "undo" operations to call if there's a rollback. The "undo" of an insert is a delete, the undo of an update is an update with a previous version, etc.

Repository is responsible for defining what "undo" or "confirm" really mean. For example, Insert with a UoWManager looks like this:

public T Insert(T entity, IUoWManager mgr)
{
    mgr.RecordOnRollback( () => Delete( entity.Id ), entity );
    mgr.RecordOnCommit(() => ConfirmInsert(entity), entity);

    InsertTentative( entity, mgr.UoWId );

    return entity;
} 

and InsertTentative simply adds a UOW_ID property to the entity to signal that it is part of a UOW:

private void InsertTentative(T entity, Guid uow)
{
    entity.Fields.Add( "UOW_ID", uow.ToString());
    Insert(entity);
}

ConfirmInsert removes that property.

Obviously this doesn't work with Enterprise Services aware resources (like SQL Server, MSMQ, etc). For that, you should look at implementing a Resource Manager that works with COM+.

More on parallel queries across containers in SSDS

Our current implementation of cross-Container queries follows a very common pattern and roughly looks like this (no exception handling shown for simplicity):

public List<T> CrossContainerSearch( string[] containerIds, 
                                     string query, 
                                     SearchDelegate searchDelegate )
{

    Event[] events = new Event[ containerIds.Count() ];
    Results<T> results = new Results<T>();
    for( int x = 0; x< containerIds.Count(); x++ )
    {
        State s = new State();
        Event[x] = s.Event = new Event();
        s.Query = Query;
        s.Results = results;
        s.ContainerId = containerIds[ x ];
        s.SearchDelegate = searchDelegate;
        ThreadPool.Queue( new WaitCallBack( worker ), s );
    }

    WaitHandle.WaitAll( events );
    return results;
}

The State object is used to pass parameters to the worker (e.g. the query to execute, the delegate that the worker method will call to actually perform the search, the event to signal when the query is done, etc.).

The search workers are queued in the ThreadPool, eventually a thread will pick them up and execute. WaitHandle.WaitAll simply blocks until all events are signaled by all scheduled callbacks. When this unblocks, it means that all queries have completed. This is equivalent to a WaitForMultipleObjects API call.

The worker method looks like this:

void Worker( object s ) 
{ 
   State state = (State) s; 
   state.Results.AddRange( state.searchDelegate( state.ContainerIdstate.Query ) ); 
   state.Event.Set(); 
} 

First we recover the state, the search delegate is called with the query, the results are stored, and the event is set to signal completion.

My friend Arvindra suggested using CCR (Concurrency and Coordination Runtime) for this. I'll definitely take a look at that.

In the meantime, I re-implemented the same code using .NET parallel extensions which simplifies things greatly. A lot of plumbing goes away, and it's much simpler and easier to read. It roughly looks like this:

public List CrossContainerSearch(string[] containerIds, 
                                 string query, 
                                 SearchDelegate searchDelegate )
{
   Results results = new Results();   

   Parallel.For(0, containerIds.Count(), i =>
   {
      results.AddRange( searchDelegate( containerIds[ i ], query) );
   }
   );
            
   return results;
}

Notice that the scheduling, join, state management, etc. it's all handled by Parallel.For. Hard to beat in simplicity, isn't it? If you still want some deeper control on task scheduling then Task & TaskCoordinator types are your friends.

I will continue some research on this and publish my findings. I'm quite intrigued with more performance information so I'll be working on more formal tests. All I need now is a machine with 16 cores ;-).

The Parallel Programming blog is here.

LitwareHR on SSDS - Part V - Searching across Containers

In SQL Server Data Services, the scope of a query is bound to a Container, but in LitwareHR we had a requirement of searching entities across multiple tenants, and because in our implementation each tenant gets its own Container, we had to create a way of performing queries (the same query to be more precise) that spanned across Containers.

Remember, LitwareHR actually "owns" all containers. LitwareHR's metadata (which is simply another Container as explained in a previous post), contains information about the tenant including a pointer to the tenant's Container.

The trivial way of implementing a cross-Container search would be to iterate on all tenant containers and query each of them. Something like this:

foreach (Container c in TenantContainerCollection)
{
    IEnumerable<Entity> result = proxy.Query(CreateScope(c.ContainerId), query);
    CombinedResults.Merge(result);
}

return CombinedResults;

Naturally, this doesn't scale very well and is not taking advantage of the fact that this is a highly parallel problem. In fact, most probably each Query sent to SSDS will end up in a different node of the SSDS fabric, minimizing chances of server side contention.

So we created a helper class: CrossTenantSearch, that essentially takes an array of TenantId's, a query statement (SLINQ) and returns the combined result sets.

Internally, it will asynchronously launch several concurrent searches, each one on a different thread of a ThreadPool and then will wait for all of them to complete, merge the partial results into a single collection and return.

The following diagram illustrates an example of running 3 concurrent queries (with different colors to identify each thread). Notice how the red thread returns even before the 3rd thread is launched, the green thread is longer (potentially returning a lot of results) and the yellow is very short.

After all of them return, results are combined into a single result set and sent back to the requestor. With this approach, the max time for a query is approximately the same as the longest query you have (e.g. the green thread in the diagram above). Caveat: this might not be valid for very large sets of containers, that is when there's a large number of containers you are searching in, because you might get contention issues on the client side and also the merging of all the results might take a lot of time.

There is of course room for improvements: paging, returning to the client partial results and allow the client to drive further fetches, etc. None of these have been implemented in LitwareHR and are left as an exercise to the reader.

Side note: when developing this feature we were puzzled by an exception we got related to thread apartments. It turns out by default, Visual Studio test runner runs in STA. There's a somewhat obscure configuration parameter that you need to add to the testrunconfig file manually, as it is not exposed in the configuration window:

<ExecutionThread apartmentState="MTA"/>

See here for more info.

LitwareHR on SSDS - Part IV - Data access enhancements 2: developing offline

SQL Server Data Service is well...an online service. That means that you have to be connected to the network 100% of the time if you are using it. What if you are not connected? well...you know the whole story.

Our goal while developing LitwareHR was to actually make the dev team as independent and autonomous as possible. Notice I say the dev team, not the application itself. We were comfortable in taking a dependency with SSDS for runtime, that is when LitwareHR would be deployed in a production data center. But we wanted developers to be able to work even if they are flying on a plane with no connectivity (like I was!). In one sentence we wanted a "mock SSDS".

With this in mind, we developed a new proxy implementing the same interface the real proxy implements, but against a local SQL Server Express database.

Again: our goal was not to create full-fledged offline SSDS, but rather an implementation complete enough to support our TDD development practices, and "complete enough" is key here. We gradually made it more complex as needed by LitwareHR, so no guarantees that it would actually work in another application, and no optimizations whatsoever.

Each dev in our team would configure he's own solution to use the offline version of the proxy, make sure all tests pass, etc. then we would test against the real one. Features on the offline client were strictly added as necessary as we made progress with the project and tests passed with the real proxy but failed with the mocked one.

The ProxyFactory class allows us to switch between one implementation and the other. Repository doesn't really know or care about what version is using.

The implementation is quite straight forward actually. There are 2 tables: CONTAINERS and DATA. The table CONTAINERS is only used to record the existence of a new container (because Containers can be empty). The table DATA contains the entity instances themselves. Structure of these tables is as simple as you could imagine:

CONTAINER:

[containerId] [nvarchar](255) NOT NULL

DATA:

[containerId] [nchar](255) NOT NULL,
[id] [nvarchar](255) NOT NULL,
[version] [int] NOT NULL,
[kind] [nvarchar](255) NOT NULL,
[entity] [xml] NOT NULL

containerId, Id, version and kind translate into SSDS entities' intrinsic properties. Queries against these being frequent. entity is the field that contains the serialized flexible entity.

The schema of the serialized entity is also quite simple and direct. A "book" entity with 3 fields: Title, ISBN and Author would be serialized like this:

<entity>
  <field name="Title" type="System.String">The Aleph</field>
  <field name="ISBN" type="System.String">78755</field>
  <field name="Author" type="System.String">J.L.Borges</field>
</entity>

Operations like Insert, GetById, Delete and Update are trivial. Query is trickier because we had to (re) implement a parser for SLINQ. Good for us that SLINQ is fairly simple today. Once again, we just developed the basics so LitwareHR would work.

In a nutshell, it's simply a translator from SLINQ syntax into a mix of T-SQL and XQuery. Searches involving intrinsic properties (Id, Kind, etc) are straight forward T-SQL statements. XQuery is only involved when the search criteria includes properties within the entity. For example:

This query against a "C_1" container, looking for an entity with Id = E_1:

from e in entites where e.Id=='E_1" select e

is translated into:

SELECT id, entity, version, kind FROM DATA WHERE containerId='C_1' AND Id =  'E_1'

and getting all entities with a flexible property named "Title" equal to "The Aleph":

from e in entities where e["Title"]=="The Aleph" select e 

will become:

SELECT id, entity, version, kind FROM DATA WHERE containerId='C_1' AND entity.value('(/entity/field[@name="Title"])[1]','NVARCHAR(MAX)') =  'The Aleph'

We definitely recovered the investments we made in creating this facility. When we started development, SSDS was still changing and under development (this was far before MIX08), it was only available from our intranet (making it more difficult for our non-MSFT team members to access it: VPN, etc). With this approach, we definitely increased our productivity.

For serious, production offline access I'd look at Sync Services.

LitwareHR on SSDS - Part III - Data access enhancements 1: caching

In most applications, the distance (in terms of bandwidth and latency) between the store (the database) and the application logic (like the web servers and web services) is usually very small. The connectivity between these two components is usually very reliable and with very high throughput. But because SSDS is on the other side of the cloud, latency and unreliability of the network could hurt the application performance and availability. Not to mention that in a production application, an ISV will very likely pay for each operation submitted against SSDS in addition to the storage he's using.

For all these reasons it is important to consider technical options that would minimize the degree of "chattiness" of the application (for decreased latency) and the total amount of calls made (for latency & for cost).

With this in mind, it is highly likely your app will have a spectrum of information that it deals with, with different levels of longevity and volatility.

If the data is immutable (e.g. reference information, read only, historic records, etc.) then there is a great chance you can aggressively cache it on the client side to avoid further calls. (Notice that "client side" is a relative term: in LitwareHR, the client side for SSDS is the web services layer).

In LitwareHR, for example, Resumes cannot be modified. Someone can submit multiple resumes, but once it is submitted, it is immutable. A perfect candidate then for caching.

To demonstrate this scenario, we included caching capabilities into LitwareHR's data access that can be enabled either programmatically and/or by configuration:

The CacheManager is simply using underneath Enterprise Library's caching block that gives us nice backing store capabilities and some advanced expiration policies.

The Repository will transparently get and store items (entities & full query results). To give the developer extra flexibility and control, entities only marked as "cacheable" (which is done through an interface implementation) are processed with the cache (if one is available). That means of course, that if an entity is not marked as "cacheable", then it will always go against the store, regardless of the existence and availability of a cache.

As an example of how this works, here's a small test. The class Book is an (extensible) entity and is also cacheable. 

public class Book : ExtensibleEntity, ICacheableEntity
{
   public string Title { set; get; }
   public string ISBN { set; get; }
   public bool IsCached { set; get;} //This is ICacheableEntity implementation
}

We want to test a Repository for this class and caching behavior:

[TestMethod]
public void ShouldInsertBookGetByIdDelete()
{
    using (Repository<Book> rp = new Repository<Book>(tenantId, 
                                                      new RepositoryCacheManager( "CacheManager", 
                                                                                  new SlidingTime( TimeSpan.FromMilliseconds( 3000 )), 
                                                                                  true, 
                                                                                  false )) )
    {
        Book bk = new Book { Id = Guid.NewGuid(), Title = "The Aleph", ISBN = "4374834" };
        bk.Fields.Add("Author", "J.L.Borges" );

        rp.Insert(bk);

        Book rbk = rp.GetById(bk.Id);
        Assert.IsNotNull(rbk);
        Assert.IsTrue(rbk.IsCached);
        Assert.AreEqual(bk.Title, rbk.Title);
        Assert.AreEqual(bk.Fields["Author"].Value, rbk.Fields["Author"].Value);

        Thread.Sleep(3500);

        rbk = rp.GetById(bk.Id);
        Assert.IsNotNull(rbk);
        Assert.IsFalse(rbk.IsCached);

        rp.Delete(bk.Id);
        Assert.IsNull(rp.GetById(bk.Id));
    }
}

Let's see piece by piece how this works:

1- We create a new Repository for Book, using a constructor that takes the tenantid and a CacheManager implementation. This CacheManager takes this arguments:

1. The name of the EntLib's cache to use (this translates into the configuration section to use)
2. The expiration policy (In the sample, it would be a sliding time of 3 seconds: items in the cache will be invalid after 3 seconds and discarded)
3. A boolean for caching entities (true in the example)
4. A boolean for caching queries (false in the example)

2- We create a new Book and add some fields, both explicit fields (as defined in the Book class) and an extension field (Author).

3- We insert the Book instance and then retrieve it immediately

4- Because the retrieval (hopefully :-)) will happen before 3 seconds, it should come from the cache (flagged with the IsCached field)

5- We then wait for slightly more than the expiration period (3.5 seconds to be precise) and then retrieve it again. This time, it should come fresh from SSDS, therefore IsCached should be false.

6- We clean up by deleting it

EntLib's underlying implementation has some interesting features we didn't use. For example, you can subscribe to expiration notifications and handle them. You could, for instance, proactively request a a renewal when an item expires proactively keeping your cache up to date.

LitwareHR on SSDS - Part II - The data access layer

The heart of LitwareHR implementation on SSDS is in it's data access layer of course. In fact, we created two different, but functionally equivalent implementations: one runs against SQL (LitwareHR's original implementation) and a second stack that runs against SSDS. Of course they are mutually exclusive and all layers above the DataModel cannot tell the difference.

As illustrated in the diagram above, the key class is Repository<T>, which among other things encapsulates all access to SSDS, translates T's into SSDS's Flexible Entities and in general provides a higher level of abstraction on top of SSDS artifacts. For example, Repository deals with tenants, which are then mapped into Authorities & Containers by a provider. The default implementation of this simply does a 1:1 mapping between Tenants and Containers, being Authorities fixed (as specified in configuration), but because it is using a provider model, you can supply a more sophisticated implementation.

T allows us to use typed objects that represent LitwareHR entities: Resumes, Positions, etc. so a hypothetical LitwareHR developer can write things similar to this:

public class Position : ExtensibleEntity
{
    public string Code {set;get;}
    public string Location {set; get;}
}

Position p = new Position { Id = Guid.NewGuid(), Code = "Code_1", Location = "Location_1", };

p.Fields.Add( "YearsOfExperience", 15 );

In LitwareHR all entities derive from ExtensibleEntity which is just a simple helper base class that provides some useful fields such as: Id, a collection of fields, etc.

In the example above, Id comes from ExtensibleEntity, Code and Location are defined at design time by the developer and "YearsOfExperience" might be a field that a particular tenant is adding to this position. Because it is a tenant specific field, it is defined at runtime.

Behind the scenes the Position entity will be translated into a SSDS flexible entity. Id, and Kind are intrinsic properties in SSDS and are automatically mapped into the Position.Id and typeof(Position).ToString(). The rest is mapped to flexible properties.

All those details are of course handled by Repository<T>. Its interface is straight forward and looks like the one below, which is self-explanatory.

Insert( T );

T GetById( id );

IEnumerable<T> ListAll();

IEnumerable<T> Search( string query );

Delete( id );

Update( T );

So a complete snippet (taken from one of our tests) looks like this:

LitwareHR on SSDS - Part I - Multi-tenancy & Flexibility

SSDS's application model and features map quite nicely to our customization and multi-tenancy requirements in LitwareHR.

A significant amount of code in LitwareHR is in the generic, multi-tenant, extensible data access. Our multi-tenant database performance guide, compares different extensibility approaches (XML datatypes, extended tables, fixed columns), their advantages and disadvantages, etc.

All of that is greatly simplified in the version of LitwareHR that uses SSDS because of the built-in support for extensibility and tenant isolation in the service.

The ACE Model:

SSDS model is quite simple actually, it is referred to as the ACE model and it is basically a 3 level containment architecture:

Authority (the top level container) contains zero or more Containers which in turn contain sets of Entities. Entities have properties: intrinsic (like Id, Kind & Version are present in any entity) and custom properties which are user defined. Properties have a type (string, dates, numbers).

This model maps very nicely to LitwareHR requirements:

Authorities & Containers give us out-of-the-box tenant isolation for storage. Entities give us exactly what we need to provide each tenant a different data shape for positions and resumes.

In our current implementation of LitwareHR, there's a single Authority (owned by Litware for LitwareHR) with multiple containers: one for LitwareHR itself where all application metadata is stored (this is the equivalent of the old TenantMetadataStore), and one Container for each tenant that is provisioned.

Here's a subset of this model:

Because LitwareHR allows each tenant to change the shape of the Position entity, instances of this type are all different between tenants.

Notice that SDSS is actually more flexible than how it is used in LitwareHR. SSDS allows instances of any entity to have any set of properties and there's no schema forced into any particular instance. LitwareHR restricts flexibility at the tenant level, so all positions of a given tenant will have the same schema, but each tenant can have any schema they want.

LitwareHR Metadata Container

Entities stored in LitwareHR's metadata container will be used to drive the application: tenant information, UI, menus, views, entities schema for the tenant, etc.

Here's the result of a query against the demo LitwareHR container. You can see various entities stored there, including a serialized recruiting workflow definition:   

The URI for this query is http://litwarehr.data.sitka.microsoft.com/v1/LitwareHR?q= which essentially means: "bring me all the contents of container LitwareHR in litwarehr authority" (note the format of the URI: http://<AuthorityId>.data.sitka.microsoft.com/<ContainerId>?q=).

SSDS query language (SLINQ) is a subset of LINQ syntax. For example, the query for retrieving all instances of particular entity type (e.g. "Tenant") would be:

"from e in entities where e.Kind=='Tenant' select e"

SLINQ is a subset and there are many features not available quite yet: projections, grouping, etc.

Currently, SSDS will return the whole entity, even if you just need a part of it. Of course you can still use LINQ on the client side to group, filter, etc. but over the wire the whole thing will be transported. 

Quid Pro Quo:

Nothing valuable is free, isn't it? SSDS gives you unlimited storage, tremendous flexibility, geo-aware location. You don't have to worry about backups, operations, disk failures, power, air conditioning, machines, etc. You just use it.

But all this goodness comes at a cost: you don't "own" the database any more, you can't assume the database is "close" to you (too chatty interactions with it, will lead into latency issues) and the programming model you were used to is different: there are no tables, stored procedures, views, joins, etc.

You will have to decide whether these cons, out-weight the benefits in your particular scenario, and hopefully some of the lessons learnt in Litware, although impossible to extrapolate to every scenario, will help you make that decision.

As Nigel mentioned in MIX though, it is highly likely that many more features will be added to SSDS in the upcoming months.

In the next chapters, I will go deeper into LitwareHR on SSDS architecture, the challenges we faced and how we solved them.

SQL Server Data Services - SSDS - New version of LitwareHR

Today, in his keynote, Ray Ozzie announced a new "cloud service" available from Microsoft: SQL Server Data Services (code name: Stika). It's a good that he announced it, because now I'm out of quarantine and I can talk about it :-).

For the last 2 months I've been working very closely with the SSDS team, understanding and exploring how their technology can be applied in the context of the work we do: (business) software delivered as a service. I've been playing the role of an ISV designing and creating solutions using SSDS. My playground has been, of course, LitwareHR: our reference application for and S+S app.

LitwareHR today looks like this:

LitwareHR on SSDS, looks like this:

Same client experiences (Web Client, Rich Client, APIs), same business logic exposed through WCF, completely new storage.

My team's job was to redesign Litware's (multi-tenant) data access layer to use SSDS, adapting it to the new application model. It's been a great and fun exercise, with lots of learning. 

If you are attending MIX08, Nigel Ellis (one of SSDS architects) will be delivering a session on SSDS on Thursday 6th at 8:30am in the Delfino 4005 Room, and guess what: one of his demos is Litware!

(Session recording as well as slides will be available within 24 hours at http://sessions.visitmix.com)

Ryan Dunn has more details on the beta program for the service: http://dunnry.com/blog/IntroducingSQLServerDataServices.aspx 

I'll be in Belgium (TechDays) and Hungary in the following weeks, and I plan to share and show more details there.

I will also post more details of LitwareHR on SSDS as I find the time to do so. Lot's of flying ahead of me, and great thing than Windows Live Writer works offline ;-). The beauty of Software and Services.

Northwind Hosting Concept Demo

For the last months our team has invested a lot of time exploring and researching the relationship between ISVs and Hosters. I shared some of the early thoughts and findings in a series of posts in this blog:

Part I - The Scenario

Part II - On-Boarding

Part III - Billing, Metering

Part IV - SLAs

Part V – Strategies for Capacity Management

We also wrote the white paper "ISVs are from Mars, Hosters are from Venus" and now finally I'm happy to share with you an end to end concept demo. We've implemented this to help illustrate the experience described in the white paper, between a hypothetical hoster (Northwind Hosting) and an ISV (Litware).

Northwind Hosting Concept Demo

Feedback, always welcome.

Fun with GeoRSS, Virtual Earth and WCF Syndication

WCF Syndication APIs are great. In just a couple of hours I wrote a simple GeoRSS feed, a fairly high level library to encapsulate the details of common entities (lines, points, polygons), all unit tests and a mashup with VE to test it.

I only encountered a couple of incompatibilities and problems (mainly related to XML namespaces) but everything was straight forward and easy. VE doesn't allow the RSS feed to come from a different server the map is hosted, so you need a proxy to walk around this limitation (Very well explained in this blog entry. A must read for anyone doing VE stuff. Thanks for the code!). 

Being fairly new to this is space, I was amazed at the power off all these infrastructure components, and the apps that could be built with this stuff; and I know I have just barely scratched the surface.

My library allows me to easily author a GeoRss feed:

GeoRssData is a simple object model for geo information. GeoRSSFeedBuilder translates the domain model into a RSS feed with the proper extensions. This is all quite easy with WCF Syndication classes. Finally this the result on IE:

Enabling RSS in LitwareHR

LitwareHR codebase now targets Visual Studio 2008 (beta 2). I believe many few changes are required for RTM, but we are still working on those details.

Anyway, .NET APIs in .NET 3.5 are richer and allowed us to offer a few fancy features almost for free. One of such features is a “multi-tenant”, customizations aware RSS feed for open positions of any tenant.

We used the WCF syndication APIs in .NET 3.5 to enable this. The code is smart enough to get the extra fields a tenant might have added to the Position entity (using Linq just for fun).

The best of all is that all this can be done in just a handful of lines of code:

Sweet!