Introducing Storage Client Library 2.1 RC for .NET and Windows Phone 8

Introducing Storage Client Library 2.1 RC for .NET and Windows Phone 8

Rate This
  • Comments 28

We are pleased to announce the public availability of 2.1 Release Candidate (RC) build for the storage client library for .NET and Windows Phone 8. The 2.1 release includes expanded feature support, which this blog will detail.

Why RC?

We have spent significant effort in releasing the storage clients on a more frequent cadence as well as becoming more responsive to client feedback. As we continue that effort, we wanted to provide an RC of our next release, so that you can provide us feedback that we might be able to address prior to the “official” release. Getting your feedback is the goal of this release candidate, so please let us know what you think.

What’s New?

This release includes a number of new features, many of which have come directly from client feedback (so please keep this coming), which are detailed below.

Async Task Methods

Each public API now exposes an Async method that returns a task for a given operation. Additionally, these methods support pre-emptive cancellation via an overload which accepts a CancellationToken. If you are running under .NET 4.5, or using the Async Targeting Pack for .NET 4.0, you can easily leverage the async / await pattern when writing your applications against storage.

Table IQueryable

In 2.1 we are adding IQueryable support for the Table Service layer on desktop and phone. This will allow users to construct and execute queries via LINQ similar to WCF Data Services, however this implementation has been specifically optimized for Windows Azure Tables and NoSQL concepts. The snippet below illustrates constructing a query via the new IQueryable implementation:

 

var query = from ent in currentTable.CreateQuery<CustomerEntity>()

where ent.PartitionKey == “users” && ent.RowKey = “joe”

select ent;

 

 

The IQueryable implementation transparently handles continuations, and has support to add RequestOptions, OperationContext, and client side EntityResolvers directly into the expression tree. To begin using this please add a using to the Microsoft.WindowsAzure.Storage.Table.Queryable namespace and construct a query via the CloudTable.CreateQuery<T>() method. Additionally, since this makes use of existing infrastructure optimization such as IBufferManager, Compiled Serializers, and Logging are fully supported.

Buffer Pooling

For high scale applications, Buffer Pooling is a great strategy to allow clients to re-use existing buffers across many operations. In a managed environment such as .NET, this can dramatically reduce the number of cycles spent allocating and subsequently garbage collecting semi-long lived buffers.

To address this scenario each Service Client now exposes a BufferManager property of type IBufferManager. This property will allow clients to leverage a given buffer pool with any associated objects to that service client instance. For example, all CloudTable objects created via CloudTableClient.GetTableReference() would make use of the associated service clients BufferManager. The IBufferManager is patterned after the BufferManager in System.ServiceModel.dll to allow desktop clients to easily leverage an existing implementation provided by the framework. (Clients running on other platforms such as WinRT or Windows Phone may implement a pool against the IBufferManager interface)

Multi-Buffer Memory Stream

During the course of our performance investigations we have uncovered a few performance issues with the MemoryStream class provided in the BCL (specifically regarding Async operations, dynamic length behavior, and single byte operations). To address these issues we have implemented a new Multi-Buffer memory stream which provides consistent performance even when length of data is unknown. This class leverages the IBufferManager if one is provided by the client to utilize the buffer pool when allocating additional buffers. As a result, any operation on any service that potentially buffers data (Blob Streams, Table Operations, etc.) now consumes less CPU, and optimally uses a shared memory pool.

.NET MD5 is now default

Our performance testing highlighted a slight performance degradation when utilizing the FISMA compliant native MD5 implementation compared to the built in .NET implementation. As such, for this release the .NET MD5 is now used by default, any clients requiring FISMA compliance can re-enable it as shown below:

 

CloudStorageAccount.UseV1MD5 = false;

 

Client Tracing

The 2.1 release implements .NET tracing, allowing users to enable log information regarding request execution and REST requests (See below for a table of what information is logged). Additionally, Windows Azure Diagnostics provides a trace listener that can redirect client trace messages to the WADLogsTable if users wish to persist these traces to the cloud.

To enable tracing in .NET you must add a trace source for the storage client to the app.config and set the verbosity

 

 

<system.diagnostics>
<sources>
<source name="Microsoft.WindowsAzure.Storage">
<listeners>
<add name="myListener"/>
</listeners>
</source>
</sources>
<switches>
<add name="Microsoft.WindowsAzure.Storage" value="Verbose" />
</switches>

 

 

The application is now set to log all trace messages created by the storage client up to the Verbose level. However, if a client wishes to enable logging only for specific clients or requests they can further configure the default logging level in their application by setting OperationContext.DefaultLogLevel and then opt-in any specific requests via the OperationContext object:

 

// Disbable Default Logging
OperationContext.DefaultLogLevel = LogLevel.Off;

// Configure a context to track my upload and set logging level to verbose
OperationContext myContext = new OperationContext() { LogLevel = LogLevel.Verbose };

blobRef.UploadFromStream(stream, myContext);

 

New Blob APIs

In 2.1 we have added Blob Text, File, and Byte Array APIs based on feedback from clients. Additionally, Blob Streams can now be opened, flushed, and committed asynchronously via new Blob Stream APIs.

New Range Based Overloads

In 2.1 Blob upload API’s include an overload which allows clients to only upload a given range of the byte array or stream to the blob. This feature allows clients to avoid potentially pre-buffering data prior to uploading it to the storage service. Additionally, there are new download range API’s for both streams and byte arrays that allow efficient fault tolerant range downloads without the need to buffer any data on the client side.

IgnorePropertyAttribute

When persisting POCO objects to Windows Azure Tables in some cases clients may wish to omit certain client only properties. In this release we are introducing the IgnorePropertyAttribute to allow clients an easy way to simply ignore a given property during serialization and de-serialization of an entity. The following snippet illustrates how to ignore my FirstName property of my entity via the IgnorePropertyAttribute:

 

public class Customer : TableEntity
{
[IgnoreProperty]
public string FirstName { get; set; }
}

 

Compiled Serializers

When working with POCO types previous releases of the SDK relied on reflection to discover all applicable properties for serialization / de-serialization at runtime. This process was both repetitive and expensive computationally. In 2.1 we are introducing support for Compiled Expressions which will allow the client to dynamically generate a LINQ expression at runtime for a given type. This allows the client to do the reflection process once and then compile a lambda at runtime which can now handle all future read and writes of a given entity type. In performance micro-benchmarks this approach is roughly 40x faster than the reflection based approach computationally.

All compiled expressions for read and write are held in a static concurrent dictionaries on TableEntity. If you wish to disable this feature simply set TableEntity.DisableCompiledSerializers = true;

Easily Serialize 3rd Party Objects

In some cases clients wish to serialize objects in which they do not control the source, for example framework objects or objects form 3rd party libraries. In previous releases clients were required to write custom serialization logic for each type they wished to serialize. In the 2.1 release we are exposing the core serialization and de-serialization logic for any CLR type via the static TableEntity.[Read|Write]UserObject methods. This allows clients to easily persist and read back entities objects for types that do not derive from TableEntity or implement the ITableEntity interface. This pattern can also be especially useful when exposing DTO types via a service as the client will longer be required to maintain two entity types and marshal between them.

Numerous Performance Improvements

As part of our ongoing focus on performance we have included numerous performance improvements across the APIs including parallel blob upload, table service layer, blob write streams, and more. We will provide more detailed analysis of the performance improvements in an upcoming blog post.

Windows Phone

The Windows Phone client is based on the same source code as the desktop client, however there are 2 key differences due to platform limitations. The first is that the Windows Phone library does not expose synchronous methods in order to keep applications fast and fluid. Additionally, the Windows Phone library does not provide MD5 support as the platform does not expose an implementation of MD5. As such, if your scenario requires it, you must validate the MD5 at the application layer. The Windows Phone library is currently in testing and will be published in the coming weeks. Please note that it will only be compatible with Windows Phone 8, not 7.x.

Summary

We have spent considerable effort in improving the storage client libraries in this release. We welcome any feedback you may have in the comments section below, the forums, or GitHub.

Joe Giardino

Resources

Getting the most out of Windows Azure Storage – TechEd NA ‘13

Nuget

Github

2.1 Complete Changelog

Leave a Comment
  • Please add 4 and 8 and type the answer here:
  • Post
  • Where is the cors support?

  • @Luiz,CORS will be released sometime this Fall.

  • I get an ArgumentNullException whenever I try to deserialize a 3rd party object with ReadUserObject if that object has something like a List<> in it. It serializes just fine, ignoring the property types that aren't supported. But you cannot deserialize.

  • @ Keith Murray

    Thanks for your comment. We have identified this issue and resolved it in the refresh release which will be available later this week.

  • Hi guys,

    I'm not able to install the NuGet package for a Windows Phone 8 project. The ConfigurationManager dependency seems incompatible with Windows Phone 8. Here's the error:

    Install-Package : Could not install package 'Microsoft.WindowsAzure.ConfigurationManager 2.0.1.0'. You are trying to install this package into a project that targets 'WindowsPhone,Version=v8.0', but the package does not contain any assembly references or

    content files that are compatible with that framework. For more information, contact the package author.

    At line:1 char:1

    + Install-Package Microsoft.WindowsAzure.ConfigurationManager

    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

       + CategoryInfo          : NotSpecified: (:) [Install-Package], InvalidOperationException

       + FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PowerShell.Commands.InstallPackageCommand

    Any ideas?

  • Can I use IQueryable with DynamicTableEntity and complex conditions such as .Where(x => x.PartitionKey.StartsWith("something" && x.SomeProperty.TrimEnd(';').EndsWith("1234")))

  • Since making my last comment, I was able to manually install the component packages from WindowsAzure.Storage, all except for the ConfigurationManager package.

    Running these three commands worked for the respective packages...

    Install-Package Microsoft.Data.Edm -Pre (this installed Microsoft.Data.Edm 5.6.0-rc1, and added reference to Microsoft.Data.Edm.Portable)

    Install-Package Microsoft.Data.OData -Pre (this installed Microsoft.Data.OData 5.6.0-rc1, and added reference to Microsoft.Data.OData.Portable)

    Install-Package System.Spacial -Pre (this installed System.Spatial 5.6.0-rc1, and added reference to System.Spatial.Portable)

    Finally, running this command continues to fail...

    Install-Package Microsoft.WindowsAzure.ConfigurationManager -Pre (this attempted to install Microsoft.WindowsAzure.ConfigurationManager 2.0.1.0, but failed with the following error)

    Install failed. Rolling back...

    Install-Package : Could not install package 'Microsoft.WindowsAzure.ConfigurationManager 2.0.1.0'. You are trying to install this package into a project that targets

    'WindowsPhone,Version=v8.0', but the package does not contain any assembly references or content files that are compatible with that framework. For more information, contact

    the package author.

    I have also attempted to run the WindowsAzure.Storage install again, which now seems to pass the System.Spatial dependency I was previously getting, but now fails on ConfigurationManager

    Install-Package WindowsAzure.Storage -Pre (attempted to install Microsoft.WindowsAzure.ConfigurationManager 1.8.0.0, but failed with the following error)

    Install failed. Rolling back...

    Install-Package : Could not install package 'Microsoft.WindowsAzure.ConfigurationManager 1.8.0.0'. You are trying to install this package into a project that targets

    'WindowsPhone,Version=v8.0', but the package does not contain any assembly references or content files that are compatible with that framework. For more information, contact

    the package author.

  • @Deyan / Ryan

    I am sorry for the delay in the Windows Phone 8 library. The blog noted it would be published in the coming weeks, but that has been slightly delayed. We are now targeting releasing it as part of the RC refresh for 2.1 in the same package. During this time frame OdataLib has refactored to include a portable library for windows phone so we have decided to move to that and restart testing. (See ODataLib 5.6.0-rc1 on nuget for more details).  We will update this blog when the library is available publicly.

  • @Amit

    Yes you absolutely can do complex queries with DynamicTableQuery, however your example uses a few methods that are not supported such as startswith, trimend, endswith etc. The way to correctly do starts with is to use a lexically bounded query that specifies both upper and lower boundaries. For example I would use PartitionKey gte something && PartitionKey lt something{ ( note '{' is 1 + 'z' in AsciiTable). The way to do these comparisions is via the String.CompareTo method;

    Additionally when using a query of DynamicTableEntity type and addressing properties which are not included in ITableEntity (PartitionKey, RowKey,Timestamp)you need to access the property via the dictionary and use the correct typed getter. The library will be able to correctly construct the filter string by analyzing the property name argument used for the dictionary accessor and the type of the getter.  For example:

    var query =

    from ent in table.CreateQuery<DynamicTableEntity>()

    where ent.Properties["customerid"].StringValue == "customer_1"                    

    select ent;

    We have a draft of a more in depth blog that will detail specific examples of the IQueryable optimizations in the library that will be forth coming.

  • Are there any plans to introduce batching for Retrieve operations? (that is a big performance win)

    Are there plans to introduce/support compression? (especially for moving large amounts of data)

    Increasing the batch count would be huge tooo...at least support up to 4meg.

  • Is there any sample code on how to handle continuations?

  • Hi. Please see the following Microsoft connect bug report: connect.microsoft.com/.../azure-sdk-compute-emulator-controller-actions-hit-multiple-times-in-asp-net-mvc-4-app

    Thanks.

  • Hello thanks for the great updates. Here is my feedback:

    When you build a query for a partition: PartitionKey=samplePK and a RowKey that starts with a prefix for example “foo-“, you need to combine at least 3 filters using the table query CombineFilters.

    It would be nice for LinQ to support this scenario and do something like:

    var query = currentTable.CreateQuery<CustomerEntity>().Where(ent => ent.PartitionKey == “samplePK” && ent.RowKey.StartsWith(“foo-“));

    That for example could yield a query like: RowKey GreaterThanOrEqual “foo-“ and RowKey LessThan “foo.” Note the “.” Which is the lexicographic follower of the “-“

    Other thing that I was wondering about is asynchronous operations while using LinQ, Without creating Task.Factory.StartNew

    Whats the correct way of doing it?

    var tableQuery = query.asTableQuery();

    and then await table.ExecuteQueryAsync(tableQuery)

    Thanks!

  • System.Spatial seems to install OK with version 5.6.0 on WP8. So why are you forcing version 5.2.0 in the RC nugetpackage?

  • Nothing has been announced. That being said, the JSON support coming by end of CY13 will dramatically reduce payload sizes to and from the server which will have a dramatic affect on latencies, especially on slower network connections. In some cases payloads are reduced 45%-65% depending on the scenario and if type metadata is returned by the server.

Page 1 of 2 (28 items) 12