Failover Clustering and Network Load Balancing Team Blog
Cluster Shared Volumes (CSV) Cache is a feature in Windows Server 2012 and beyond which allows you to allocate system memory (RAM) as a write-through cache. The CSV Cache provides caching of read-only unbuffered I/O. This can improve performance for applications such as Hyper-V, which conducts unbuffered I/O when accessing a VHD file. Unbuffered I/O’s are operations which are not cached by the Windows Cache Manager. What CSV Block Cache delivers is caching which can boost the performance of read requests, with write-through for no caching of write requests.
CSV Cache delivers caching at the block level, which enables it to perform caching of pieces of data being accessed within the VHD file. This is fundamentally very similar in the value it provides to other solutions on the market you may already be familiar with. The primary difference is that CSV Block Cache reserves its cache from system memory, where most solutions on the market today deliver cache in the form a PCI card with an SSD or flash which you add to the server.
CSV Cache is completely integrated into the Failover Clustering feature and handles orchestration across the sets of nodes in the cluster.
CSV Cache will deliver the most value in scenarios where VMs are heavy read requests, and are less write intensive. Scenarios such as Pooled VDI VMs or also for reducing VM boot storms. Because the applicability of CSV Cache depends on the workload and your specific deployment considerations, it is disabled by default. The customer feedback on CSV Cache has been overwhelmingly positive and we generally recommend turning it on for all scenarios, including both Hyper-V Clusters using CSV and Scale-out File Servers using CSV.
You can allocate up to 20% with Windows Server 2012 and 80% with Windows Server 2012 R2 of the total physical RAM for CSV write-through cache, which will be consumed from non-paged pool memory.
There are two configuration settings that allow you to control CSV Cache.
1. Open an elevated Windows PowerShell prompt
2. Define the size of the size of the cache to be reserved (example of setting to 1 GB)
(Get-Cluster). BlockCacheSize = 1024
(Get-Cluster). SharedVolumeBlockCacheSizeInMB = 1024
3. Enable CSV Cache on an individual disk (must be executed for every disk you wish to enable caching)
Get-ClusterSharedVolume “Cluster Disk 1” | Set-ClusterParameter CsvEnableBlockCache 1
The cache size can be modified with no downtime, however for the Hyper-V root memory reserve in the parent partition to be modified to accommodate the memory allocated to the CSV cache it does require a server reboot with Windows Server 2012. To ensure resource contention is avoided, it is recommended to reboot each node in the cluster after modifying the memory allocated to the CSV cache. On Windows Server 2012 R2 the Hyper-V room memory reserve will be dynamically updated and a reboot is not necessary.Enabling CSV Cache on an individual disk requires that the Physical Disk resource be recycled (taken Offline / Online) for it to take effect.
CSV Cache will be disabled on:
The CSV Cache also provides a set of counters you can use to monitor the performance of the cache. You can leverage the Performance Monitor tool (PerfMon.msc) to add the following counts to monitor different aspects of the CSV Cache.
Open Performance Monitor, and under Add Counters you will find “Cluster CSV Volume Cache” with the following counters.
I/O satisfied from cache:
I/O satisfied from disk:
Thanks!Elden ChristensenPrincipal Program Manager LeadClustering & High-AvailabilityMicrosoft
Many thanks Elden, great post, good explanation
when will you provide caching to flash (ssd or pci)? we are waiting for this in sql, file server,etc..
I've turned this on in my lab environment, and seen great improvement of read performance in our Server 2012 RC Test Cluster.
Now-- can you provide a little more details on the resource allocation? I.E. If I were to allocate 20% of system RAM to this, would that cache get trimmed back as additional VMs are started (or ramp up their dynamic memory)?
I have only one file server and doesn't want to use file server cluster. Can I force file server read/write-back cache enabled regardless hyper-v's unbuffered io? otherwise it is so slow.
Is there any reason why this isn't enabled by default?
Are there any drawbacks to CSV cache except the extra memory utilization?
In general we recommend turning on the CSV Cache for any/all Hyper-V or Scale-out File Server deployments.
The only reason it is not enabled by default, is the default amount of memory may vary from deployment to deployment... depending on the amount of physical memory in the nodes, the VM density you are trying to achieve, and for a Hyper-V deployment it may be less and for a Scale-out File Server it will be more. So we just didn't have a good memory allocation number to set by default... but it is something we are definitely thinking about
Great write up and post for Windows Server 2012. This worked great on our Cisco UCS Blade Chassis deployments.
Does anyone have any more info on tuning the size of the cache? I see the perfmon counters, but not really sure how to interpret them as far as tuning. Can't seem to find much in any documentation.
I suggest you do a utility, which analyzes perflog or something, and offers to enable a certain amount of csv cache to each disk.
SAN vendors recommend for tired storage following relation: 5% - 25% - 70% (gold data - silver data - bronze data). This ratio may vary depending on read/writes IO. So, if you have 1 TB data your best size of cache - 50GB. You can use RAM or SSD accelerator card (HP, EMC, NetApp and others vendors have solutions with SSD).