PowerShell Scripting for Microsoft project code named "Velocity"

Windows PowerShell allows you to invoke C# code directly from the PowerShell scripts. This offers a powerful way to “script” cache operations such as puts/gets etc. This can be used to prime the cache as part of an automated setup or to perform other management tasks such as exploring the cache etc. This blog shows some examples on how you can invoke the "Velocity" APIs directly from PowerShell scripts. Note that  I will use product and "Velocity" interchangeably .

Let us see how we can do it step by step in this write up. Windows PowerShell let's you create .NET objects,  you can create object with new-object cmdlet like this

[System.Random]$rand =new-object System.Random

Before creating any object make sure that the assembly for that class is loaded by PowerShell.

You can check the assemblies loaded by PowerShell by running this on PowerShell.

[System.Threading.Thread]::GetDomain().GetAssemblies()

By default, PowerShell loads a bunch of assemblies for you so that you can start working with the base types, and some common functions. If you want to use the functionality in an assembly that is not already loaded, you will have to load that assembly yourself.

you can load by doing something like this on PowerShell prompt

[Reflection.Assembly]::LoadFrom("CacheBaseLibrary, Version=1.0.0.0, " `

          + "Culture=neutral, PublicKeyToken=b77a5c561934e089")

 It will be good idea to use PowerShell-based cache administration tool which will be installed as part of product installation. Let us now see how we can create DataCacheFactory object in a PowerShell  Script.

 [Microsoft.Data.Caching.DataCacheServerEndpoint ]$server =new-object

 Microsoft.Data.Caching.DataCacheServerEndpoint("machineName", 22233, "DistributedCacheService")

[ Microsoft.Data.Caching.DataCacheServerEndpoint[]]$serverarray = new-object Microsoft.Data.Caching.DataCacheServerEndpoint[] 1

$serverarray[0]=$server

[Microsoft.Data.Caching.DataCacheFactory ]$cacheFactory = new-object Microsoft.Data.Caching.DataCacheFactory($serverarray, $true, $false)

 Once we have created this DataCachefactory we can use it as we do it in any C# client, what I mean is that you can call GetCache, GetDefaultCache  etc. on this DataCacheFactory. In the following code

I am creating a cache with a random name and going the use it with the DataCacheFactory created  above by calling GetCache on it. Creating a region and  putting a key and value in it.  At last a get

 call is getting the value for key we inserted by calling get on cache we created, after that we are  testing the value for its correctness.

[System.random]$rand =new-object System.random

$cacheName = $rand.next(100,134)

#Creating a Named cache in product cluster

New-Cache -CacheName $cacheName -NotExpirable true -Eviction LRU -TTL 1400 -Secondaries 0 -NotificationsEnabled true -Verbose

start-sleep -seconds 60

[Microsoft.Data.Caching.DataCache]$cache= $cacheFactory.GetCache($cacheName)

$regionName = $rand.next(22,127)

$cache.CreateRegion($regionName, $true)

start-sleep -milliseconds 15000

$key=$rand.next(45,292)

$value=$rand.next(83,120)

$cache.put($key, $value, $regionName)

$item = $cache.get($key, $regionName)

# Testing the value we got is what we put

if ($value -eq $item)

{    

      write-output("Test Passed")

}

In the above code snippet  you can see that is helping you in many ways. Firstly you are able to mix cache administration cmdlets with the client code,  secondly it is a script and you get all powers of script here. We will see how a Cache Administrator can solve some problems that he he/she may face.

1) Strating a product Cluster, Creating a Cache, Creating some regions and filling some data before giving it for other use.

# Starting Cluster

Start-CacheCluster

start-sleep -seconds 15

$errRegionAlreadyExists = [Microsoft.Data.Caching.DataCacheErrorCode]::RegionAlreadyExists

# Random Class I will use it to create random values for cachename, region name, keys and values

[System.random]$rand =new-object System.random

$cacheName = $rand.next(1,999)

#Creating a new cache using a cache administration cmdlet

write-output ($cacheName )

New-Cache -CacheName $cacheName -NotExpirable true -Eviction LRU -TTL 1400 -Secondaries 0 -NotificationsEnabled true -Verbose

start-sleep -seconds 60

# Product Exception handling block, this will handle all product exception in this script

trap [Microsoft.Data.Caching.DataCacheException]

{

  write-error $("TRAPPED: " + $_.Exception.Message);

  write-error $("TRAPPED: " + $_.Exception.ErrorCode)

  if($_.Exception.ErrorCode -eq $errRegionAlreadyExists)

  {  

      $regionName = $rand.next(1,200)

      $cache.CreateRegion($regionName, $true)

  }

  if($_.Exception.ErrorCode -eq 12011)

  {  

      exit

  }  

  continue;

}

# Creating Cache Factory with one DataCacheServerEndpoint

[Microsoft.Data.Caching.DataCacheServerEndpoint]$server =

new-object Microsoft.Data.Caching.DataCacheServerEndpoint("machinename", 22233, "DistributedCacheService")

# This code Microsoft.Data.Caching.DataCacheServerEndpoint[] 1 creates a array of one

[Microsoft.Data.Caching.DataCacheServerEndpoint[]]$serverarray = new-object Microsoft.Data.Caching.DataCacheServerEndpoint[] 1

$serverarray[0]=$server

[Microsoft.Data.Caching.DataCacheFactory] $cacheFactory = new-object Microsoft.Data.Caching.DataCacheFactory($serverarray, $true, $false)

# This block will handle any kind of exception in this script if it not of product type

trap [Exception]

{

    write-error $("TRAPPED: " + $_.Exception.GetType().FullName);

    write-error $("TRAPPED: " + $_.Exception.Message);

    write-error $("TRAPPED: " + $_.Exception.StackTrace);

    write-error $("TRAPPED: " + $_.Exception.Source);

    write-error $("TRAPPED: " + $_.Exception.TargetSite);

    continue;

}

# Getting the cache object in here which we created earlier in this script.

[Microsoft.Data.Caching.DataCache]$cache= $cacheFactory.GetCache($cacheName)

$regionName = $rand.next(22,127)

# Creating region with a random name, which is in this case a integer

$cache.CreateRegion($regionName, $true)

start-sleep -milliseconds 15000

$key=$rand.next(4,292)

$value=$rand.next(8,120)

$cache.add($key, $value, $regionName)

# Putting data in the Region we created above

for ($i=1;$i -le 100;$i+=1)

{

      $key=$rand.next(4,292)

      $value=$rand.next(8,120)

      $cache.put($key, $value, $regionName)

}

In the code above you can see how product exceptions being handled, you can also choose try-catch style in PowerShell Script.

Function TryCatch() {

   &{#Try Block

              throw (new-object Microsoft.Data.Caching.DataCacheException)

   }

   #Catch Block

   trap [Microsoft.Data.Caching.DataCacheException] {

      Write-Host " $_.Exception"

      Write-Output "Failure!"

      continue

   }

}

References:

http://blogs.msdn.com/powershell/archive/2006/04/25/583273.aspx

http://www.microsoft.com/windowsserver2003/technologies/management/powershell/default.mspx

http://blogs.msdn.com/wriju/archive/2008/07/01/how-to-find-public-key-token-for-a-net-dll-or-assembly.aspx

Troubleshooting “Velocity” Series – III (Continued)

How to control ETW sessions and collect traces

In this section I will describe how to work with an ETW sink. ETW stands for “Event Tracing for Windows”. It is a high speed tracing facility provided by the OS. It uses buffering and logging mechanism implemented in the OS kernel. The buffers used are written to the disk. The user can later retrieve these events in a human-readable format. Please follow these steps to subscribe to Microsoft project code named “Velocity” ETW sink.

·         For the first time logging metadata has to be registered for the machine. ‘Logs’ folder (under the installation folder: Microsoft Distributed Cache\V1.0\Logs) contains the following files:

Ø  Provider.man - manifest file for events for Microsoft Windows Vista and above.

Ø  Provider.mof - manifest file for events for downlevel (Microsoft Windows XP, Microsoft Windows Server 2K3).

Ø  ProviderUninstall.mof - manifest file to uninstall “Velocity” provider from WMI repository.

Ø  ProviderGUID.txt - text file which contains the GUID for “Velocity” session provider and is used internally.

Moving/deleting these files can break the functionality and is not supported.

From command prompt:

Ø  Downlevel users would have to do “mofcomp Provider.mof”

Ø  Users having Microsoft Windows Vista and above would have to do “wevtutil im Provider.man” (this requires elevation).

·         For an ETW sink, logs would go to ETW session and can be retrieved by command line utilities. Sessions can also be controlled in the same way.

·         The utilities required are ‘tracelog.exe’ and ‘tracerpt.exe’. These are standard Windows utilities and are publicly available.

·         Start collecting logs: first, an ETW session has to be started with a definite log level. Logs generated would be pumped to it. The log level can be changed dynamically.

tracelog -start <sessionName> -f <logFile> -guid ProviderGUID.txt -level <level>

o   sessionName: name of the ETW session.

o   logFile: file in which the logs would go to.

o   level: the desired Log level.

§  2 - Error

§  3 - Warning

§  4 - Information

§  5 - Verbose

o   To disable logging, log level should be given as 1.

·         Change log level: it can be changed dynamically by

tracelog -enable <sessionName> -guid ProviderGUID.txt -level <level>

·         Stop collecting logs: now, the ETW session needs to be stopped.

tracelog -stop <sessionName>

·         Trace dump can be viewed by : tracerpt <logFile> -y

Ø  Downlevel users would get a .csv file.

Ø  Users having Microsoft Windows Vista would get an .xml file. To get a .csv file, the switch ‘-of CSV’ would have to be added. Command then would be “tracerpt <logFile> -of CSV -y”.

This was all about ETW sink on a cache host. However, if you want this functionality on the client side, you need to copy Provider.man, Provider.mof, ProviderUninstall.mof and ProviderGUID.txt from the ‘Logs’ folder (under the installation folder: Microsoft Distributed Cache\V1.0\Logs) onto the client machine before executing the steps mentioned above.

Thanks,

Amit Kumar Yadav

(Microsoft project code named “Velocity” team)

Troubleshooting “Velocity” Series – III

How to use “Velocity” logging framework

Microsoft project code named "Velocity" provides the ability to trace events on the cache client and cache host. These events are captured by enabling log sinks. A sink represents a valid destination to which logs or events can be emitted. “Velocity” supports three log sinks:

·         Console sink - emits events to the console.

·         File sink - emits events to a text file.

·         ETW sink - emits events to the OS buffers which are written to the disk.

See related sections in the documentation for more details:

Log Sink Settings: http://msdn.microsoft.com/en-us/library/dd187434.aspx

How to: Set Log Sink Levels (XML): http://msdn.microsoft.com/en-us/library/dd169209.aspx

How to: Set Log Sink Levels (Code): http://msdn.microsoft.com/en-us/library/dd187351.aspx

How to change log file generation format

By default file sink emitted log files are generated hourly and use the following file format ‘dd-hh’. Hence you would see log files named with current date and hour and a new file would be generated every hour. For example, DCacheTrace[1371]13-2, DCacheTrace[1371]13-3, DCacheTrace[1371]13-4 and so on.

Now to increase or decrease the granularity of log file generation, you would need to change the above mentioned time format. Say for instance:

·         If you need log files per minute, you would want to change it to ‘dd-hh-mm’.

·         For a 24-hour format, change it to ‘dd-HH-mm’.

·         For AM/PM display, change it to ‘dd-hh-mm tt’.

Now I will list out the steps required to change the time format. Note that these settings are applicable to a single cache host since changes are done locally.

·         Open DistributedCache.exe.config (located in the installation folder).

·         Go to <fabric> <section name=”logging” path=”” > tag.

·         Add the following entry in “sinks” collection.

<customType className="System.Data.Fabric.Common.EventLogger,FabricCommon" sinkName="System.Data.Fabric.Common.FileEventSink, FabricCommon" sinkParam="C:\Program Files\Microsoft Distributed Cache\V1.0\Logs\DCacheTrace[$]/dd-hh" defaultLevel="0" />

·         The path in ‘sinkParam’ should be taken from ‘location’ attribute in the <log> tag.

·         The portion after ‘/’ in ‘sinkParam’ represents the time format. Change it as per your requirements.

·         Log level can be specified in ‘defaultLevel’. This represents the same property as ‘logLevel’ in the <log> tag.

·         Note that these settings override the ones defined in the <log> tag.

For more details, refer to http://msdn.microsoft.com/en-us/library/dd187434.aspx

In the next post, I will talk about the ETW sink.

Thanks,

Amit Kumar Yadav

(Microsoft project code named “Velocity” team)

Troubleshooting “Velocity” Series - II

How to ensure you have correct client libraries

This section is more important for people who are upgrading from CTP2 to CTP3. Since Microsoft project code named “Velocity” bits are not backward compatible, while upgrading the cache hosts, “Velocity” clients have to be upgraded as well. There are four client libraries:

1.       ClientLibrary.dll

2.       CacheBaseLibrary.dll

3.       CASBase.dll

4.       FabricCommon.dll

Even if these are updated, since the client needs references to only ClientLibrary.dll and CacheBaseLibrary.dll, in a scenario where by some chance only these two libraries are updated and the client still has older CASBase.dll and FabricCommon.dll, user would run into issues (has been reported earlier by multiple folks). The fact that file and product versions for the second two libraries are 1.0.0.0 for both CTP2 and CTP3 makes troubleshooting even more difficult. Hence you need to make sure that the correct set of assemblies has been copied by checking that their ‘Date modified’ timestamp is same in both locations (copied from, copied to).

Next time we’d be discussing “Velocity” logging framework usage.

Thanks,

Amit Kumar Yadav

(Microsoft project code named “Velocity” team)

Troubleshooting “Velocity” Series

Hi,

Hope all of you are having great time caching your data with Microsoft project code named “Velocity”. This post is first in the series of some practices and guidelines to follow in order to make best use of “Velocity” and for troubleshooting any common issues you may face. I also recommend that you see the "Velocity" readme file and the product help file for other known issues. You can find the code_name_velocity_readme.txt file on the "Velocity" download page.

How to work with common “Velocity” exceptions

The occurrence of any undesired behavior in “Velocity” cache requests results in DataCacheException being thrown to the client application. So the application using “Velocity” should be capable of catching them, recovering and taking appropriate actions, for instance, retrying a retry-able error or falling back to the persistent data store.

A “Velocity” client application should check for the exception’s ErrorCode field. These are explained in the documentation in detail. There are error codes like DataCacheErrorCode.KeyDoesNotExist, DataCacheErrorCode.RegionDoesNotExist, etc. which point to logical issues and represent definite failures. However, there also exist some error codes which may come due to cluster’s temporal state of re-alignment as a result of a cache host coming up, going down, named cache getting created/deleted etc. These are explained here and should be taken care of.

1.       DataCacheErrorCode.RetryLater : this error code represents transient incapability of the cluster to serve a request. Application can retry the request.

For instance, this would be thrown when the cluster is undergoing a configuration change or there are not enough cache hosts in the cluster to perform a Put/Add.

In the next release, this error code would be accompanied by an error sub-status which would tell the exact reason for this failure. For example - primary cache copy not found, secondary cache copy not found to update, etc to name a few.

2.       DataCacheErrorCode.Timeout : this error code is thrown when the client doesn’t receive a response from the server within a specified timeout. For instance, this would be thrown when there is a glitch in underlying communication between client and the server.

However, there is a possibility that the request may have been processed by the server, so application logic should take care of that while retrying.

To clarify a bit more, suppose client does

 cache.CreateRegion(“Foo”, false);

and gets a Timeout error code. Trying again may result in an exception with code DataCacheErrorCode.RegionAlreadyExists.

If the deployment mode is chosen as ‘Simple’, there is one more error code which the application should take care of - DataCacheErrorCode.CacheServerUnavailable. This is thrown when a “Velocity” client is not able to reach a particular cache host. This may be possible if that server is down or there is some network related failure.

Next in the series is ‘How to ensure you have correct client libraries’.

Thanks,

Amit Kumar Yadav

(Microsoft project code named “Velocity” team)

EF Query caching with Velocity

Jarek has written a blog and a sample of integrating caches into EF. One integration sample is using Velocity CTP3 bits. Take a look at http://blogs.msdn.com/jkowalski/archive/2009/06/11/tracing-and-caching-in-entity-framework-available-on-msdn-code-gallery.aspx

Sample code at http://code.msdn.microsoft.com/EFProviderWrappers

Proposed Performance Counters for CTP4

We are planning to add performance counters in the upcoming release, CTP4. The performance counters will be available for one or both of the following categories:

·         Host - A category for single-instance performance counters pertaining to a single cache host. Host counters track information such as total active connections and total client requests for a single cache host.

·         Cache - A category for multiple-instance performance counters pertaining to all cache hosts. Each instance of a cache counter corresponds to a separate named cache in the cache cluster.

Look forward to your feedback on identified counters or suggestions for new counters.

Sharique Muhammed

(Microsoft project code named "Velocity" Team)

Velocity CTP3 Samples are now available

Update on Velocity upcoming release - CTP4

As Velocity moves ahead with the next release milestone, there is an update that - Velocity is planning to release CTP 4 around mid-September 2009. Some of the features planned for CTP 4 would be - Velocity Setup/Configuration changes, Velocity Performance Monitor Counters, Enhancements to Security and more stability & reliability since CTP3

In coming days we will be sharing more details on these features to get your valuable feedback and comments.

Velocity Team

Velocity MSDN Magazine Article

Aaron has published a Velocity article on MSDN magazine. It also include sample code for the cool Northwind demos that I used in my recent talks.

 Check it out at - http://msdn.microsoft.com/en-us/magazine/dd861287.aspx

The dreaded error - Cache::Get: The request timed out

Sometimes you see this error in the Get calls and this can freak out people. There are a few reasons why this error may come. Internally we have a 15second timeout for calls (either Get or Put) .So if are not able to satisfy the request within that time, we timeout and throw this error to the user.

This can be fine tuned using the DataCacheFactory.Timeout  property to make it higher or lower. In typical scenarios, you should not his this error. However, there are cases where you will get this,

 * The specific machine that the client is routing the request to has just gone down. We try to establish a TCP connection twice before deciding to refresh our routing table. The TCP connection open timeout is 15seconds. (The minimum Send timeout is 10s). These limits are not exposed. Since we retry twice for a connection, it is possible that it takes as long as 40 seconds before we raise a complaint. So in that timeframe calls coming in would start timing out at 15 second intervals.

 The lease intervals are maintained between machiens and they are 3 minute long leases with a 1.5 minute update. So within 1 1/2 mins if the machine has not responded the neighbours will suspect something and try to establish a connection.  If the machine is running, but hte process is down, then the connection will be refused instantly. If the machine itself is down, then the TCP timeout applies and then an arbitration process is started to kick the machine out of the cluster. So all in all, it could take about 2 mins before the server side decides that a machine is down and reconfigure the cluster.

 * Machine is saturated - Here the connection would be slow and the Put/Get might start timing out. Nothing much can be done other than adding new machines or reducing the load. Post V1, we also have automated load balancing that will shift some of the load around in these scenarios.   But typically the distribution of load is good enough that if you have uniform sized objects and load, you wouldn't get in to this scenario unless you have not sized your servers properly.

* HA is needed and machine is down :  This is the most common problem that we see - people install the Velocity on either a single machine with HA on or on two machines and one machine is down. In either case, if we dont have a place that we can write a backup copy to, we throw this error. In V1, we will fix it so that we raise a different error (Secondary not available) or some such thing, so that you can do somethign different other than retrying.

 One of the diagnosability work that is happening for the V1 release is to make this error only occur when it is truly a transient problem.

More performance numbers

Anshul ran some more numbers to see the max throughput and latency that we have observed with CTP3. Here are some numbers to share.

Note that these are the max throughput numbers and latencies observed at those values. If you reduce the throughput, then latency goes down. For example, in the Read 2k, if the CPU are not maxed out , then you can get latencies of about 0.6 ms or so. 

** With all 17 dual core clients ( hardware I have) the 3 quad core server’s  CPU was not saturated L . The numbers are at around 65% CPU utilization.

*** With the 17 dual core clients the 3 node server  CPU was not saturated. The numbers are at around 75% CPU utilization.

****In all 20K numbers the network was saturated.

Server Configuration : Intel(R) Core(TM)  2 Quad CPU Q9300 \8 GB RAM \ 64 bit win 2k3\1GBPS network.

Thanks

Murali

Pushing Client Performance

You can increase the number of clients to increase the cache throughput. But sometimes if you want to have smaller set of clients and increase throughput, a trick is to use multiple DataCacheFactory instances. The DataCacheFactory instance creates a connection to the servers (e..g if there are 3 servers, it will create 3 connections) and multiplexes all requests from the datacaches on to these connections. So if the put/get volume is very high, these TCP connections might be bottlenecked. So one way is to create multiple DataCacheFactory instances and then use the operations on them.

 Warning - dont create too many of these factories as you will end up opening too many connections and waste memory and CPU.

-- MK

Regions and high Availability

When talking to some customers and partners, there was some confusion regarding regions and high availability. There is a assumption that if you use regions you will not get High Availability. That is not correct - High Availability is orthogonal to whether you put your keys within a region or not.

The way it works is that we map keys and keys within regions to partitions. The partition is nothing but a a range of hash values. (see my MIX talk) . All the high Availability and load balancing etc. is done at the partition level. So whether you put your keys within a region or not you will get HA.

-- MK

Announcing Velocity CTP3 !

Velocity Team is happy to announce release of Microsoft project code named "Velocity" CTP3

For community technology preview 3 (CTP3), Microsoft project code named "Velocity" offers several enhancements and revised APIs. The enhancements include cache notifications feature, performance improvements, security enhancements, and new cluster management options. There have also been updates to the installation program. Please checkout Velocity CTP3 Help/Documentation for more details [Microsoft Project Code Named “Velocity” CTP3 on the Microsoft Download Center]

Cache Notifications

When using local cache to store data that changes infrequently, you no longer have to wait for the configured timeout before the client pulls the latest object from the cluster. Using cache notifications, the object in the local cache automatically invalidates as soon as it is notified that the object has changed. This greatly reduces the possibility that your application is working with stale data.

When caching data that changes frequently, you may want your applications to perform certain tasks based on various cache operations. For example, you may want your application to do something specific when a cached object is added, replaced, or removed. You might also want your application to do something different when a region is added, cleared, or removed from the cache. Specifying an event-triggered task is now possible with cache notifications.

Performance Improvements

Throughput, latency, and memory usage have been further optimized to improve performance for your applications.

Security Enhancements

New Cluster Management Options

Previously, only lead hosts could manage the cache cluster operations. Now, when using SQL Server to store the cluster configuration information, you have the option to let SQL Server perform the cluster management role instead of the lead hosts. This option improves supportability and availability of the cluster, eliminating the possibility of the cluster going down due to an insufficient number of (running) lead hosts.

Updates to the Installation Program

To streamline deployment, the installation program now supports automated installation for the cache host.

Cluster Configuration Editing Options Decoupled from Storage Options

Previously, how you chose to store your cluster configuration settings in the cluster configuration storage location dictated the options available to you for editing those configuration settings. Now, regardless of where you store your cluster settings, you can change those settings with the PowerShell-based cache administration tool or by directly editing an XML representation of the cluster configuration.

Along with this paradigm shift, the option to store the cluster configuration settings in XML (the cluster's "working copy") has been removed. Now you can export and edit (and import back) the configuration settings in XML when you choose to store your cluster configuration settings in a SQL Server database or a SQL Server Compact data file in a shared network folder.

With this change, your decision for the cluster configuration storage location can be based on application availability requirements and the resources available to your application (that is, how challenging it would be to procure an instance of SQL Server that your distributed cache cluster could use).

Revised APIs

To minimize changes in the following release, the "Velocity" APIs have been changed to enhance usability and more closely resemble the naming conventions of other Microsoft namespaces. The namespace of "Velocity" has been changed to Microsoft.Data.Caching from System.Data.Caching. Additionally, all class and delegate names have been renamed to include the prefix DataCache. Along with the name changes, many overloads have also been changed. For more details about the CTP3 API changes, checkout Velocity Online Help Section: CTP3 API Changes (Velocity)

Velocity MSDN Link : http://msdn.microsoft.com/en-us/data/cc655792.aspx

Velocity CTP3 Download Link: Microsoft Project Code Named “Velocity” CTP3 on the Microsoft Download Center

Velocity Team