Too Much Cache?

re: Too Much Cache?

someone — Wed, 28 Nov 2007 15:38:24 GMT

I'm getting "is not a valid Win32 application" error.

[I’m guessing that you are running SetCache.exe on Windows XP (or earlier). The GetSystemFileCacheSize and SetSystemFileCacheSize API functions require Windows Server 2003 SP1 or later (this includes Windows XP x64 Edition, since it is built from the 2003 SP1 codebase). Since these functions don’t exist on earlier versions of Windows, the SetCache.exe binary was compiled with the subsystem version set to 5.02, which prevents it from running on versions of Windows where the API functions do not exist.]

re: Too Much Cache?

Preshan Naidoo — Mon, 14 Jan 2008 10:49:11 GMT

Good article. keep it up to keep us up to date.

re: Too Much Cache?

Mark Edwards — Mon, 14 Jan 2008 19:49:14 GMT

Whilst coming out of L1/L2 cache may get you 20ns, coming from physical RAM is going to be between 80 and a 120ns on most systems (probably the latter), with the larger ones needing NUMA node traversal over a Xbar or AMD hyperlinks will also add to this number.

I've not seen any systems going under 80 ns, but not done performance work on DDR3 yet so it may be possible outside of cache.

However, your thought about access being a million times quicker is nearly right. Disk access on the big Arrays from EMC, Hitachi or HP would average (roundtrip) to be between 4 - 8 ms, if we average in cache utilization on their arrays.

Taking 8ms and 80ns to make the sums easier it's around 100,000 times quicker.

This does bring up a point about the new flash drives coming out the spec is access will be less than 1 microsecond, and typically at the slower end of RAM, which means 150ns, so at worst case (1us) these drives are about 8000 times better than mechanical spindles with a big front side cache. Hence Hybrid technology and ReadyBoost are being used to improve latency. Flash drives are too expensive for most consumers so far, but in the Enterprise the tipping point for mass production is about a year away if assume price/capacity rates keep diminishing linearly for the respective technologies. A single fibre flash drive can saturate a 4Gb/s Fibre link, so capacity is there.

This will give a very needed boost for many customers on the performance envelope.

re: Too Much Cache?

boyflex — Wed, 26 Mar 2008 17:14:29 GMT

how do u find out how much cache is on your pc?

(nice article learnt a bit keep i up)

[If you want to see how much cache is currently being used, you can do so with Performance Monitor counter /Memory/Cache Bytes. If you want to see the limits, you can use the SetCache executable or !filecache in a postmortem or live debug. !filecache will also show you the current file cache usage along with how much cache each file is using.]

re: Too Much Cache?

David — Thu, 08 May 2008 22:22:43 GMT

Very helpful. However, setting max cache size to >= 2 GB does not work for me. Tried on w2k3 server r2 and xp.

[This is a limitation on 32 bit systems. With this API and tool you are setting the working set of the file cache. The file cache resides in the kernel address space. On 32 bit systems, you are limited to 2 GB of Kernel address space. Additionally the cache needs to share the kernel address space with other kernel resources, so your cache's working set won't even get up to 2 GB. Even though your cache's working set is limited on 32 bit systems, many pages still may be "cached" on standby pages. If you use the 32 bit version on an x64 box, you can set the cache up to 4GB. At that point you reach the 32 bit SIZE_T limit (32 bits can only address up to 4 GB).]

re: Too Much Cache?

Steven Gill — Thu, 15 May 2008 13:36:00 GMT

Fantastic!

I have upgraded our backup server to W2k8x64 from w2k3x86 and it kept using all physical RAM, I even put 16GB in it and it used it all! this explains it all!

I have managed to reduce the cache to 1GB, but can't set it at more if I try anything over 1GB it sets to 8TB, e.g.:

C:\>setcache 2048

Current Cache Settings:

Minimum File Cache Size: 100 MBytes

Maximum File Cache Size: 1024 MBytes

Flags: 1

New Cache Settings:

Minimum File Cache Size: 100 MBytes

Maximum File Cache Size: 8388607 MBytes

Flags: 1

Any ideas? otherwise I'll take some of this expensive memory out if it can't be used usefully!

[Thanks for the feedback. It turns out there was a bug in the sample code. The bug reveals that I didn't have a modern system with a lot of RAM to properly test this code. I've updated the code and the binaries. Try this new version.]

re: Too Much Cache?

fuzb — Thu, 19 Jun 2008 13:13:56 GMT

Hello again - thanks for the updated setcache that works perfectly!

I have another problem though, after a month or so of our backup server being on (no reboots or installs, just doing it's normal backups as far as we can tell) the machine will reset the cache back to 8TB, at this point the machine grinds to a halt as all physical RAM is used and today it was so bad we had to actually press the power button because it wouldn't respond to ctrl-alt-del or pslist etc.

Any ideas what might cause that? there shouldn't be a time limit on how long the cache is set should there?

I have set up a scheduled task to run setcache daily and have to see how it behaves from now on.

Many Thanks!

[I don't see a direct timer that would reset the max size. There could be some crazy set of events that could end up triggering a reset, but I cannot guess on a possible scenario. There is also the possibility that another application on the system is calling SetSystemFileCacheSize() and resetting the cache size. If scheduling a daily task to reset the max size is not enough for you, I recommend opening a support incident with us so that we can investigate this further.]

re: Too Much Cache?

Sam — Sun, 06 Jul 2008 00:49:36 GMT

Very informative, thanks. But does the SetSystemFileCacheSize() work on Vista SP1?

Here's my test: on a system with 1.7 GB RAM, copy 1.5 GB of pictures, file size varies 5 - 15 MB.

With default settings (no tweaks), Task Manager's Physical Memory Cached grows while Free goes to 0.

I ran your SetCache, setting max cache size to 512 MB - no change, Cached still grows to max.

I set SystemCacheDirtyPageThreshold per the KB article, still no change.

BTW, if I delete the copied files the Free memory immediately jumps up, indicating to me the system is still caching the files that were just written. Which isn't necessarily bad, I just want the system to cache LESS of them!

What really bugs me is my Vista box with 4GB RAM will use 3+ GB for cache while copying files and slowing everything else down.

[These APIs will work with Vista SP1. The client requirements are XP x64 and Vista. For Server systems, you will need Server 2003 SP1 or Server 2008.

Do not rely on Task Manager for these values. It is not telling you what you think it is. Please read the Memory Shell Game. Task Manager is telling you that you are using Standby Pages. They are like cached pages and very close to freed pages. In your example the 3+ GB is probably in standby pages. Since they are not being actively used by processes, the system is using them as pseudo cache pages. We don't want memory to go unused.]

re: Too Much Cache?

egbvista — Sat, 30 Aug 2008 00:56:21 GMT

Aug 29, 2008

Re: Cache tuning API's and Registry values

The problem seems to be the one size fits all mentality of the operating

system, not bad applications. There is no excellent algorithm for an

operating system as broadly used as Windows, despite Mark Russinovich's

statement "...the Windows file copy engine tries to handle all scenarios

well" [from:

http://blogs.technet.com/markrussinovich/archive/2008/02/04/2826167.aspx].

People should read Mark's well written description of the efforts

expended by Microsoft in improving the cache system. It seems they

improve it in one place only to have bad performance immediately

pop up somewhere else.

The API's mentioned in this article are dangerous because they are

global and effect all applications on the system. If the system is a 16

core Datacenter, the ability to tune the system with these API's is like

using sledge hammer on ear rings.

There will come a time when the Windows I/O subsystem will be

redesigned. This will happen because of the amount of parallelism

arriving with the many cores of systems in the near future will

bring cache peformance, scalability and partitioning into focuse

where these issues were not visible to the original Windows Operating

system designers.

Until that time, I think we all must endure the

one-size-fits-all design of the Windows cache.

I could be wrong, though.

Just my 1 cent.

[There are many design challenges for a general use operating system. Windows isn’t a one trick pony. Seldom do servers only run one type of application. You have to throw in backups, management, and a host of other add-on services. While the Cache Manager does a good job for most of the usage scenarios, there are unique cases where it doesn’t work well. For the read I/O cache consumption on 64 bit case, the fundamental problem is a runaway working set. On 64 bit systems, working sets can grow larger than physical RAM on most servers. The default settings for the Memory Manager can’t handle processes that take more than their fair share of physical RAM. It can’t reliably determine whether one process deserves more pages than another. This is why administrators need to tune the server’s configuration. They can use things like WSRM to limit working set growth for individual processes or use the provided APIs to limit the working set of the system file cache. We are working on improving this experience in the next version of Windows. The changes are extensive and the risk of regression is far too high to backport to the current operating systems.]

Excessive paging on Exchange 2007 servers when working sets are trimmed

ExchangeVigilantes.com — Sun, 07 Sep 2008 15:40:29 GMT

Excessive paging on Exchange 2007 servers when working sets are trimmed

SetCache setting does not seem to work

mike suding — Sat, 04 Oct 2008 05:26:44 GMT

I used setCache.exe it on win2008 64bit enterprise to 2048MB and the "cached" value that shows in task manager (directly below memory guage) just keeps going higher than 2048MB. Mine is now 12,224MB! Does anyone know what is going on?

[Task Manager’s value for “Cached” is not what you think it is. In addition to the Cache Manager’s working set, this number includes the number of standby pages in physical RAM. While standby pages are like cached pages, they can be quickly disassociated with the previous working set, scrubbed and handed to a new process. Task Manager is just showing you another way of looking at the data. To see the real working set size of the System File Cache you need to use Performance Monitor. You can read more about this in the Memory Shell Game post.]

re: Too Much Cache?

Nihility — Tue, 16 Dec 2008 20:40:37 GMT

Using the sample code, is it possible to set the minimum cache size? It defaults to 100MB instead of the original 1MB.

[By default the sample code is hard coded to set the minimum to 100 MB, but not enforce it. This was done to allow the memory manager to reduce the working set size as needed. You can modify the sample code to set a hard limit by changing the Flags parameter in the call to SetSystemFileCacheSize().]

Nice!

TanMan — Wed, 17 Dec 2008 05:36:59 GMT

This reads and acts like a Russinovich post and tool - easily understandable, educational, small, fast, and useful. Thank you!

I have Vista Home Premium x64 with 4GB RAM. 1GB max cache on a 4GB system seems much more reasonable than the default of 8.4 TB!

Do I need to reboot in order for the new cache setting to take effect?

[Thanks for the feedback. In response to your question a reboot is not required. The setting is dynamically applied to the system file cache’s working set size. It is important to note this setting is not persistent so rebooting the machine will revert the size back to the default. In order to maintain the settings you’ll need to run the tool at least once per boot. One approach is the use of a machine start up script to automate the process after a reboot.]

Microsoft Windows Dynamic Cache Service

Microsoft Advanced Windows Debugging and Troubleshooting — Fri, 06 Feb 2009 23:09:27 GMT

Excessive cached read I/O is a growing problem. For over one year we have been working on this problem

Rilasciato Microsoft Windows Dynamic Cache Service

DevAdmin Blog — Sat, 07 Feb 2009 18:28:57 GMT

Rilasciato Microsoft Windows Dynamic Cache Service

re: Too Much Cache?

GeirW — Thu, 12 Mar 2009 11:45:20 GMT

Tool works fine on 2008 x64 in that it commits the change immediately after running the command. However the setting doesn't stick after a server reboot but defaults back to 8386607MB. Is there a way to make the setting permanent or do we have to resolve to running the tool during the Windows startup sequence?

[Good question. SetCache has been replaced by the Microsoft Windows Dynamic Cache Service. You read more about it here. These settings are not persistent and will revert to default values when the system starts. You can either use SetCache in a local system startup script or use the Microsoft Windows Dynamic Cache Service. The new service will auto-start with the system and set the cache limit based on many configurable options.]

re: Too Much Cache?

CecoM — Wed, 22 Apr 2009 18:22:54 GMT

Thank you very much for your post. I am so sory not found it 2 years ago... I am trying to donload binaries with no success. Tried several ISP's. Please, help?

re: Too Much Cache?

Nomgle — Mon, 08 Jun 2009 21:40:40 GMT

CecoM, this has now been replaced with Microsoft Windows Dynamic Cache Service - grab it from http://www.microsoft.com/downloads/details.aspx?FamilyID=e24ade0a-5efe-43c8-b9c3-5d0ecb2f39af&displaylang=en

re: Too Much Cache?

vincentk — Wed, 14 Oct 2009 09:10:21 GMT

I just tried to install this tool on Windows 2008 R2 but it failed to start with the notification that the tool was written for an earlier version of Windows.

Will there be an update soon or is there a tool provided within R2 that manages the cache size?

[ Thanks for the great question! The SetCache tool is not an installable tool. Additionally it has been replaced by the Microsoft Windows Dynamic Cache Service. Either way, these tools help to mitigate the problem of excessive growth of the system file cache on versions of Windows prior to Windows 7 and Windows Server 2008 R2. We have updated the memory manager algorithms in Windows 7 and Windows Server 2008 R2 to address this issue natively in the Operating System. You should not use these tools on the latest version of Windows. ]