SQL Server and Microsoft StreamInsight

SQL Server and Microsoft StreamInsight

  • Comments 3

As you might recall from our blog post on April 22nd, StreamInsight is packaged and licensed through Microsoft SQL Server. However, customers have pointed out various different deployment scenarios since our first StreamInsight CTPs. We reflected those in our approach to licensing, packaging and deployment of the product as follows:

  • We added Microsoft StreamInsight to the SQL Server flight of products to provide our data platform customers and partners with all the necessary components in one convenient package.
  • We decided to make StreamInsight installation and deployment as lightweight and flexible as possible. For instance, StreamInsight does not strictly require you to install any other SQL Server component besides StreamInsight.
  • Our adapter framework allows you to connect to other SQL Server and Microsoft platform products as data sources and sinks based on your scenario and your needs for deeper integration.

The rationale behind this is that we see event-driven applications emerge as a new paradigm of data-driven applications. For those scenarios, we view event processing as data processing and Microsoft's data platform would be the natural home for Microsoft's platform for event processing. However, customers have also pointed us to various use cases and scenarios that go beyond what is typically covered by Microsoft’s data platform.

Let's look at a couple of examples. Think, for instance, of static reference data that is not part of the incoming stream that StreamInsight processes. Instead, it is kept and maintained in a separate metadata store. You might still find it useful to correlate some of the incoming events from the stream with the corresponding reference data, for instance to drive grouping over a field like ZIP code that is only available from the reference data.
 
In many operational scenarios, for example in manufacturing, you have to deal with high input data rates. What is interesting in those scenarios are aggregates of the data, how the data changes, and the patterns or curves that the event data follows over time. Here, approaches do not scale that require you to put the data at rest in the database first and then run the same report over it again and again to check for the interesting patterns. Instead, you want to analyze the data while it is in flight, identify the patterns or complex events relevant to your business and then act on those.
 
Technically, StreamInsight facilitates these various deployment models by doing all its query processing in memory. StreamInsight has its own in-memory engine or runtime that does not need to write event data to disk.  Therefore, StreamInsight does not require you to install other SQL Server products if your scenario is already covered by the StreamInsight engine capabilities. Running StreamInsight just requires the .NET framework. This leads us to a follow-up question about the relationship between StreamInsight's engine and the CLR runtime. Although StreamInsight looks and feels like .NET, the StreamInsight engine does not depend on the garbage collector from the .NET runtime for its internal memory management. It instead uses its own native memory manager for the events to provide high throughput and predictable low latency.
 
SQL Server with StreamInsight provides you with the flexible platform to address all the various deployment scenarios mentioned above: You can either use existing or write your own input and output adapters to SQL Server engine databases, to Analysis Services cubes, to Velocity caches, BizTalk, or SQL Server Parallel Data Warehouse if you need to keep state or access persisted data. Or you can simply inject StreamInsight into your event data pipeline in order to do all the processing, transformations, calculations and analytics while the data is in flight. The most exciting deployments are those where you mix-and-match and connect all the pieces to a rich end-to-end application.

Regards,
Torsten Grabs

Leave a Comment
  • Please add 4 and 6 and type the answer here:
  • Post
  • Hi,

    Why is StreamInsight not licensed on its own in addition to being licensed with SQL Server?  This is one way of discouraging people from continuing with the Microsoft platform especially with the recent change in pricing model.

    Microsoft will continue to lose ground to open source and with the recent acquisition of Sun by Oracle, and Apple's latest innovations, and Google's recent run in the apps and Office space ...........

    StreamInsight has to use its own memory manager cos a lot of work still needs to be done to make the .NET GC more efficient.  Why cant the GC allow the compaction of LOH when process memory reaches critical levels even if they ordinarily dont allow it during normal execution.  I think that's better than the process/app crashing.

    The move to WPF for Visual Studio was a good one and builds people's confidence in the platform but as long as critical apps continue to steer clear of .NET (Office and Sharepoint for instance - they only allow integration on that front but are not written in it), developers will remain weary of porting some legacy apps to the platform and fully utilizing all .NET derived technologies.

  • I'd agree with Eniola, I investigated some CEP solutions a year or so ago, but the project got mothballed. Coming back to it, I was quite excited to see StreamInsight, but then was almost immediately disappointed by the fact it's only available for the enterprise SQL installations. Looking at other options now. This may be a good product, but it's a bad rollout MS.

  • Hi Eniola Benjamin, hi Joe,

    Let me jump in with a quick clarification: It is true that StreamInsight is available under the umbrella of a Microsoft SQL Server license. That rationale for this was that we see event-driven processing as a key capability in Microsoft's data platform. However, you can decide to just use StreamInsight from your SQL license - you'll then get the other SQL Server components for free.

    With regard to the enterprise SQL: installations that Joe mentioned, I first want to make sure that there is no confusion about the different SQL Server editions that include StreamInsight. StreamInsight is available in most SQL Server editions including the Datacenter, Enterprise, Standard and Web editions plus our Evaluation and Developer editions.

    Hope this helps.

    Torsten

Page 1 of 1 (3 items)