John Gallardo's Weblog

The SQL Customer Advisory Team just released a Technical Note comparing SQL Server Reporting Services 2008 vs. 2005 from a scale-up perspective.  Its good to see that a lot of the work that we did over this release focusing on performance and scalability (across the board, from the core server/processing infrastructure to specific rendering extensions) has really paid off.  You can read the entire article here:

http://sqlcat.com/technicalnotes/archive/2008/07/09/scaling-up-reporting-services-2008-vs-reporting-services-2005-lessons-learned.aspx

Quoting from the summary (emphasis is mine):

Reporting Services 2008 was able to respond to 3–4 times the total number of users and their requests on the same hardware without HTTP 503 Service Is Unavailable errors compared with Reporting Services 2005, regardless of the type of renderer. In stark contrast, Reporting Services 2005 generated excessive HTTP 503 Service Is Unavailable errors as the number of users and their requests increased, regardless of the report renderer.

Our tests clearly show that the new memory management architecture of the report server enables Reporting Services 2008 to scale very well, particularly on the new four-processor, quad-core processors. With our test workload, Reporting Services 2008 consistently outperformed SQL Server 2005 with the PDF and XLS renderers on the four-processor, quad-core hardware platform (16 cores) both in terms of response time and in terms of total throughput. Furthermore, with these renderers on this hardware platform, Reporting Services dramatically outperformed other hardware platforms regardless of Reporting Services version, responding to 3–5 times the number of requests than when running on either of the other hardware platforms. As a result, we recommend that you scale up to four-processor, quad-core servers for performance and scale out to a two-node deployment for high availability. Thereafter, as demand for more capacity occurs, add more four-processor, quad-core servers.

When you install Reporting Services, we create a few directories:

1. LogFiles
2. ReportManager
3. ReportServer
4. RSTempFiles

Most of these are fairly self explanatory.  LogFiles... well we put our log files in there.  ReportManager contains the Report Manager application (what you get when you browse to http://<server>/reports) and ReportServer contains the Report Server application which is the proper 'Report Server' which implements our Web Service and HTTP handler endpoints... it is what you get when you go to http://<server>/reportserver.

So what about that RSTempFiles thing?  Well it contains files that are temporary of course!

These temporary files can be broken down into a few categories:

• Temporary Report Snapshot files.
• These files are only created if you have opted into using temporary file storage (see the WebServiceUseFileShareStorage).
• For RS2000/2005, snapshots are more or less completely independent and each is contained in its own directory (identified by a GUID).  Each snapshot contains a set of files (internally, they are referred to as "chunks"). 
• For RS2008, snapshots oftentimes contain some shared data.  The folder-per-snapshot hierarchy used in RS2000/2005 is replaced by a single directory called "Chunks."  Each chunk is a discrete file in this directory.
• These files are automatically cleaned up by the Report Server on a time-based interval (same as snapshot data stored in ReportServer and ReportServerTempDB catalog databases).
• Output/Intermediate Streams
• These files are all created directly within the the RSTempFiles directory.  There is no way to differentiate between these two sub-categories of files currently.
• Output streams - these streams are generated as output from the renderers.  The server spools them to disk if they grow large enough.  RS has an output cache that may cause these streams to survive beyond the lifetime of the request so that subsequent accesses can be served directly from the cache. 
• Intermediate files - these files contain results of intermediate calculations during report processing and rendering.  Generally, these contain data which is never returned to the client, but rather holds temporary results in order to relieve memory pressure.  These files are cleaned up when the request completes (no caching across requests).
• Conversion files
• These are stored under a folder named _Conversion. 
• Our compression format for snapshots changed between RS2000 and RS2005.  This folder would contain temporary files supporting the on-demand one-time upgrade of these persisted snapshots.  Unless you are upgrading from an RS2000 instance, you should never see this directory or files created. 
• Shadow Copy Files
• This is only for RS2008.
• After the CTP6 release of RS2008, we enabled shadow copy for our ASP.Net worker domains under the new hosting model.

I'm generally not a big fan of the "link dump" style blog.  However, the latest post on Joel On Software regarding web standards was too good not to share.

Some choice quotes...

Regarding "working around" problems in an implementation (bolded by me):

And eventually some tedious bore writes a lengthy article on her blog explaining a trick you can use to make Qxyzrhjjjjukltk 5.0 behave just like FireQx 3.0 through taking advantage of a bug in Qxyzrhjjjjukltk 5.0 in which you trick Qxyzrhjjjjukltk into deciding that it’s raining when it’s snowing by melting a little bit of the snow, and it’s ridiculous, but everyone does it, because they have to solve the hasLayout incompatibility. Then the Qxyzrhjjjjukltk team fixes that bug in 6.0, and you’re screwed again, and you have to go find some new bug to exploit to make your windshield-wiper-equipped headphone work with either device.

Regarding language used in a lot of standards documents:

Those documents are super confusing. The specs are full of statements like “If a sibling block box (that does not float and is not absolutely positioned) follows the run-in box, the run-in box becomes the first inline box of the block box. A run-in cannot run in to a block that already starts with a run-in or that itself is a run-in.” Whenever I read things like that, I wonder how anyone correctly conforms to the spec.

That one I have a lot of sympathy for.  In a previous life, I worked as a tester on the XML datatype implementation in SQL Server 2005.  There was a quite a bit of time spent going through various W3C specs on XQuery, XPath, and XSD.  IMO, the specs are insane.  In some cases they go into excruciating detail about the internal semantics of a particular operation -- as an example the XQuery 1.0 Formal Semantics spec says that path expressions should have the same semantics as (ie, they should normalize to) the XQuery for statement.  In other words, an expression like /foo/bar/baz should look like a set of nested loops.  Unfortunately, it also defines that the normalization process of each sub-expression (a relative path expression) should return the results in document order. There are a couple of problems here (directly related to SQL Server's implementation).  First of all, SQL Server uses static typing, and for breaks static typing because it implies zero-or-many cardinality.  This breaks functions which require a singleton unless you really want to put (/foo/baz)[1] around everything (note if you are using untyped xml, you need to do this anyways -- sorry).  AFAIK, SQL Server is not conformant in this regard (it has been a while though, so maybe I am missing something).  The other big hangup I have is that the implication of document ordering at each step in the path seems to preclude the potential for a more global optimization where you defer sorting until the very last step.

One of the major takeaways that I had from Joel's post was that it is incredibly difficult to implement against a standard when that standard actually has no implementation.  I don't know if it is possible to get around this.  Some standards are amazingly complex -- I can't imagine there being a reference implementation which is so robust that it becomes the standard.  Maybe if you go into the whole standards argument saying that the implementation _is_ the standard then you are better off, but I suppose that has its own problems as well.

This is a continuation of a previous posting that I made which introduced some of the memory management functionality in SSRS 2008.  In this post, I wanted to dig a bit more into the details of the system in order to illuminate some of the decisions that are happening under the covers. 

In order to get the system into a memory constrained state, I have configured the server to only use ~256MB of memory.  You can read about how to do this in this books online article entitled Configuring Available Memory for Report Server Applications.  I also set the "appdomainmanager" trace level to 4, which is the "Verbose" setting.  I wouldn't recommend having a production system run with this tracelevel since the memory management output can get pretty spammy when the system is under pressure and notifications are firing rapidly. 

I then ran a pretty simple report that has 100K rows.

So lets dig into the log file a bit.  The first couple of lines look like this:

appdomainmanager!DefaultDomain!aec!03/11/2008-16:56:17:: v VERBOSE: Received NotifyMemoryPressure(pressureLevel=MediumPressure, kBytesToFree=6056)
appdomainmanager!WindowsService_1!aec!03/11/2008-16:56:18:: v VERBOSE: Memory Statistics: 0 items, 0KB Audited, 0KB Freeable, 295544KB Private Bytes
appdomainmanager!DefaultDomain!aec!03/11/2008-16:56:18:: v VERBOSE: Appdomain (WindowsService) attempted to free 0 KB.
appdomainmanager!ReportServer_0-2!aec!03/11/2008-16:56:18:: v VERBOSE: Memory Statistics: 3 items, 83654KB Audited, 78534KB Freeable, 296900KB Private Bytes

This is basically telling the state of the current system.  We have two worker appdomains (WindowsService and ReportServer).  WindowsService is used for background processing (subscriptions) and the ReportServer appdomain handles all of the interactive requests from our HTTP endpoint (including all management operations such as publishing reports, creating folders, etc...)  In my test, I wasn't running any subscriptions, and you can tell that there were "0 items" reported.  However, I was running a single report interactively in the ReportServer appdomain.  You can see there that we indicated that there were "3 items."  What are these items?  As you might remember from the previous post, there is not a direct 1:1 mapping between "memory consumers" and requests.  Typically there are 3 "memory consumers" per request... this is a bit of an implementation detail about how out report processing and rendering engine divides up the large memory consuming components.  Additionally, you can see that a total of ~84MB was discovered as being used by these consumers, of which ~79MB is considered freeable.  We also report that we are currently using ~289MB private bytes.

appdomainmanager!DefaultDomain!aec!03/11/2008-16:56:18:: v VERBOSE: Appdomain (ReportServer) attempted to free 6056 KB.

This line is telling us that some consumer in the ReportServer domain agreed to try to release ~6MB of memory (which is equal to the amount we were trying to free in the previous request).  Some additional time passes, and we recieve another memory notification.

appdomainmanager!DefaultDomain!aec!03/11/2008-16:56:19:: v VERBOSE: Received NotifyMemoryPressure(pressureLevel=HighPressure, kBytesToFree=25120)
appdomainmanager!WindowsService_1!aec!03/11/2008-16:56:19:: v VERBOSE: Skipping shrink request for appdomain (WindowsService_1) because no memory consuming requests are registered.
appdomainmanager!ReportServer_0-2!aec!03/11/2008-16:56:19:: v VERBOSE: 1057 ms has elapsed since last memory shrink attempt for this appdomain (ReportServer_0-2-128497526482384832)
appdomainmanager!ReportServer_0-2!aec!03/11/2008-16:56:19:: v VERBOSE: Memory Statistics: 3 items, 77765KB Audited, 72645KB Freeable, 313648KB Private Bytes
processing!ReportServer_0-2!aec!03/11/2008-16:56:19:: w WARN: Processing Scalability -- Memory Shrink Request Received
appdomainmanager!DefaultDomain!aec!03/11/2008-16:56:19:: v VERBOSE: Appdomain (ReportServer) attempted to free 25120 KB.

From this, we can determine that the original memory shrink was not aggressive enough, and there are other consumers which are still allocating memory.  Note that the memory pressure is now "High."  However, note that the audited memory has actually been reduced from the previous iteration.  In general, the reason why this happens is that there are memory allocations that happen outside of the areas which are strictly controlled and audited (remember the CLR shares a single global heap for all code running in all appdomains).  Since we are in a high pressure state, in addition to requesting the large consumers to reduce memory usage, we will also instruct all memory consumers to freeze their current allocation so they do not grow larger.

As time passes, the processing and rendering engines continue to work on the report, the log shows additional memory shrink notifications which I am not going to duplicate here.  The majority of them look like this:

appdomainmanager!ReportServer_0-2!aec!03/11/2008-16:56:50:: v VERBOSE: Skipping shrink request for appdomain (ReportServer_0-2-128497526482384832) because no memory consuming requests have been added.

Indicating that we are not taking additional action because we have detected that no additional requests have been added to the system.  Internally, we use a time based heuristic here to basically bail out of the shrink notification early if we believe that we are not going to do any productive activity.

Eventually, about 30 seconds after starting execution of the report, we finish rendering it, and the following line is output to the log file:

webserver!ReportServer_0-2!9d0!03/11/2008-16:56:52:: i INFO: Processed report. Report='/homeadvisor_rowparameters', Stream=''

Ok, well hopefully that has given a little bit of insight into the system that we are using, and shows how you can examine the log file to get some details into the decisions that are happening.

Ok, it has been a little while.  But I have been pretty busy lately.  His name is Christopher.

We have also been pretty busy locking down Katmai to get CTP6 (aka known as the "February CTP") out the door and working bugs.  I recommend that people head over towards the "Katmai" forum on MSDN for Reporting Services if you have any questions.  There are lots of people there willing to answer questions and otherwise help out.

So what now?  I owe a part two post on some of the memory management stuff we are introducing in Reporting Services 2008.  I'm getting all of the info put together now, and the post should be up shortly.

One of the big pushes for SSRS 2008 has been to reduce the occurrence of OutOfMemoryExceptions caused during report execution.  A lot of work has gone into making this happen throughout the report rendering stack, including significant changes in the report processing engine to move a definition based object model as opposed to an instance based one.  I can't take the credit for this massive undertaking, but I did work on some of the supporting infrastructure to ensure that components of the report processing and rendering engines can make more efficient usage of the memory on the box while reducing the likelihood of OOM.  This is probably going to be a multi-part discussion, with the first going over some of the internal infrastructure, and the next drilling more into specifics with examples of the memory monitoring component in action.

One of the upfront design decisions that we made was the processing and rendering engines would continue to be managed code.  This decision essentially means that they continue to allocate memory from a shared pool -- the CLR managed heap.  Ultimately, it is the managed heap which makes policy decisions on whether or not specific allocations are going to succeed or fail.  From the perspective of our native hosting layer, SSRS 2008 only sees the managed heap as a single large pool of memory and there is no visibility into specific components, requests, or even appdomains.  Given the current hosting interfaces for the CLR, this just is not a tractable problem at this point. 

Given the design decision to continue to use managed code and thus a shared pool of memory, we opted to go for a two-pronged approach:

• Implement/leverage a process-wide memory notification infrastructure to detect when we are approaching/experiencing memory pressure.
• In response to this global notification, determine the specific memory consumers which should be trimmed.

The first was actually already done for us.  SSRS 2008 builds its native hosting layer on top of shared infrastructure from the SQL Server engine which already has memory monitoring components in the form of Memory Broker and other resource monitoring technologies.  These provided us with a configurable and predictive memory management infrastructure.  This is the key first piece.

The second piece of the puzzle is attributing memory usage to specific consumers.  You will note that I didn't say "requests" here.  Internally, there is a managed infrastructure which specific components can leverage to report their memory usage and receive notifications as to how much memory they should try to free.  This infrastructure is heavily utilized by our report processing and rendering engine to manage the size of data structures which grow as the amount of data in the report grows.  This allows RS to strike a reasonable balance between memory utilization and stability. 

Here is a rundown of the operations that occur when our native hosting layer detects memory pressure:

1. Native layer informs managed code that there is pressure and specifies the total amount of memory that must be shrunk.
2. Managed layer enumerates all memory consumers (generally 2-3 per report rendering request plus some global objects such as caches). 
3. We determine the minimal subset of memory consumers that have to be trimmed in order to satisfy the shrink request.
1. Actually -- there is a a bit of special sauce here.  Each component also gives the centralized memory manager information about "how difficult" it is to shrink its memory usage and this is factored into determining whom to shrink.

Step (3) is pretty actually interesting and is a key design decision for SSRS.  An alternative approach would be to perform more of a "fair share" approach to trimming memory usage.  The decision to minimize the number of components which are shrunk is really driven by the fact that a memory shrink operation is somewhat heavyweight.  The vast majority of components which subscribe to these events store everything in memory up until the point they receive the notification.  In general, these consumers are things like lists and dictionaries which are required to keep track of things that are put into them -- they can't just evict entries like a cache may be able to.  So in order to begin to use less memory, these data structures have to fall back to a mixed disk/memory mode which requires serializing a portion of their state to disk.  In order to keep things running as fast as possible when we are not under memory pressure, all of this serialization happens when they receive the shrink notification.  So you can imagine what would happen if we went with a fair share approach where every request is asked to trim some memory.  Each request currently being processed would grind to a halt as they race to serialize their state to disk.  Instead, with our approach what happens is a small fraction of requests (usually the big memory hogs) are temporarily suspended while they serialize some state to disk, and other requests are to a large degree unaffected.

A pretty common pattern that I see exporting of reports through URL Access using a simple hyperlink.  The hyperlink usually looks something like this:

<a href="http://localhost/reportserver/?/SalesData&rs:Command=Render&rs:Format=EXCEL">Sales Data in Excel Format</a>

At first glance, it doesn't appear that there is anything wrong with this.  However, if you are working in an environment where the data is highly volatile and the user needs to click on the link multiple times and see up to date information then you could experience some problems.  To understand why you encountering problems, you need to first understand that implicitly when you access a report through the Report Server, you are estabilishing a "session" with the server.  Fundamentally a session is a binding between a user, the report, and the data in that report.  We use session to ensure that when you navigate between pages and otherwise interact with the report that the data within the report remains consistent. 

Ok, so what does this have to do with the URL above?  Well, when a user clicks on that URL, we create a session for the user, and pass back a cookie their web browser which will identifier that session later on to the server so we can find the correct instance of the report with the data that was in it.  If the user, within that same IE session, clicks on the report again, and assuming the session hasn't timed out (10 minutes by default), we actually render the report out of the cached data that has been bound to that user session.  I know -- disaster, right?  The user was expecting to see that billion dollar deal that they just closed and instead they are just reminded that they haven't meant the numbers for October yet. 

So what do we do about it?  Well, there is actually an rs:* parameter which tells the server "Hey, I know that you already have a session created for this user, but forget about it for right now, and create a new one", that parameter is rs:ClearSession=true.  By providing the rs:ClearSession=true parameter along with the request to the report server, you are ensuring that the server constructs a new session for the user and runs the report from a clean state, ensuring (assuming you have the report configured to use live execution) that the user will see the most up to date information possible.  The nice thing about the rs:ClearSession=true parameter is that if there is not already a session estabilished, it is a no-op.

I guess it was an unintended vacation.  We've been busy here in the Reporting Services team.  CTP5 of SQL Server 2008 was a large milestone for us as it really started to put together all of the work we have been doing to make the processing engine more scalable as well as nail down some of the hard issues with our new native hosting layer.  I know you can't get your hands on it quite yet, but for the most part the major development work has been completed except for the setup and upgrade bits. 

Speaking of our native hosting layer, James Wu has just started a blog.  James did a lot (all?) of the initial prototyping work for our new native hosting infrastructure including integration with SQLOS and other technologies from the SQL Engine team.  Hopefully over the coming months James will provide a good bit of information about the native hosting layer and its impact on users, and maybe even a good story or two about the development process.

Today the first public release of SQL Server 2008 is shipping.  Of course, Reporting Services is still there in the box.  You can find general information about the release here:

https://connect.microsoft.com/SQLServer/content/content.aspx?ContentID=5395&wa=wsignin1.0

There are also some Katmai specific MSDN forums, and a Katmai Reporting Services MSDN forum. 

EMF rendering in Reporting Services has an interesting history.  When SSRS first shipped, we did not support "direct printing" from within the web interface.  Customers revolted.  They wanted a "Print" button on the toolbar which would provide a full featured print experience.  Many other web applications don't actually do this, choosing instead to present the user with a "printable" HTML page or some other format such as PDF which they then print within their PDF reader.  Rather than this workaround though, it was decided that we would implement an EMF renderer and provide a client-side ActiveX control which would send the EMF to the print spooler.

Turns out EMF has an interesting characteristic though.  Each page is a discrete stream.  Normally, this wouldn't cause a problem, but due to how renderers were architected, the cost restarting the renderer on a specific page is a function of which page it is within the report.  Specifically, when you request page N of the report from the EMF renderer, it needs to find page N, which means essentially internally walking across pages 1...N-1 in order to position itself on page N.  A quick back of the envelope calculation will tell you then that the cost of printing a report by requesting each page independently has an algorithmic complexity of

Where R(i) is the time to independently render a page, and T(i) is the time it takes to "paginate through" a given page without actually writing anything to the response stream.  This is not good.  Adjusting the way that the EMF renderer operated would require too much of a redesign to our processing and rendering architecture, so it was decided to instead address this issue by adding a new feature to the server.  This feature was "Persisted Streams."  When this mode is enabled, all of the requested streams are rendered in one pass from the EMF renderer so we don't incur the cost of restarting the renderer on a particular page.  The server returns the initial request when the first page is completed, and continues in the background to collect pages from the renderer and store them until they are requested by the client.  From the clients perspective, you merely keep requesting pages until the server returns an empty stream, which indicates that the report has finished rendering.  There are a couple of limitations to using the Persisted Streams feature though:

1. It is only accessible via the URL Access endpoint, and not via our SOAP APIs.  Unfortunately, this is what happens when features are implemented in a service pack.  It was only needed for a very narrow scenario, so from an implementation perspective we did not choose to incur the cost of making it available to our SOAP API.
2. It is kind of a "stateful" API, meaning that once you have read a stream, you cannot read it again.  So make sure you get it right the first time because you can't go back.

Here is some sample code that demonstrates how use our URL Access endpoint to retrieve streams in this fashion: 

            string requestUri =
                string.Format("https://{0}/reportserver?{1}&rs:Command=Render&rs:Format=IMAGE&rc:OutputFormat=EMF",
                m_serverName, m_reportPath);

            const string persistStreams = "&rs:PersistStreams=true";
            const string getNextStream = "&rs:GetNextStream=true";
            CookieContainer cookies = new CookieContainer();
            WebRequest request = WebRequest.Create(requestUri + persistStreams);
            (request as HttpWebRequest).CookieContainer = cookies;
            try
            {
                while (true)
                {
                    request.UseDefaultCredentials = true;
                    
                    using (WebResponse response = request.GetResponse())
                    {
                        Stream s = response.GetResponseStream();
                        BufferedStream buffered = new BufferedStream(s);
                        if (OnReceivedStream(buffered))
                        {
                            break;
                        }

                        request = WebRequest.Create(requestUri + getNextStream);
                        (request as HttpWebRequest).CookieContainer = cookies;
                    }
                }
            }
            catch (WebException ex)
            {                
                OnReceivedException(ex);
            }

Obviously, this code is not 100% complete.  It is part of a larger piece of a WinForms application that I have written which excercises this functionality.  It should be enough to get you started though.

I was poking around some other SSRS blogs from folks here on the product team, and I found this post from Lukasz about troubleshooting Reporting Services.  It is a great overview of some of the guidance we have given in the form of whitepapers, KB artciles, and a bunch of other sources.  I highly recommend you give it a read and bookmark it so you can use it is your "playbook" when running into problems.

Reporting Services uses a temporary database for storage of objects which are, well, temporary.  For example, report snapshots which are associated with a particular user session as the result of a live execution.  Cached report snapshots are also stored here because they begin life as a result of a live report execution.  This means that in systems where there are a lot of live report executions happening, we are churning through a lot of data in the ReportServerTempDB.  As your system scales out or up, you will undoubtedly experience a problem where you will begin to saturate the IO subsystem hosting the ReportServerTempDB.

Do I have an IO throughput issue? 

There are lots of really great articles written about this topic, but in general there are a couple of things which I look at:

• Perfmon counters for Physical Disk Usage:
  • Avg. Disk Sec/Read
  • Avg. Disk Sec/Write
• Check SQL Server Wait types
  • sys.dm_os_wait_stats
  • Look for high counts of wait stats like IO_COMPLETION, ASYNC_IO_COMPLETION, and the PAGELATCH_* class of waits

Ideally, you want to see Avg Disk Sec/Read or Write to be < 10ms. If you start seeing these counters creep over 50ms, then you are probably getting into a situation where the disk is not keeping up with the incoming rate of IO requests and this is causing waits.

So how do I deal with it?

The answer is pretty straightforward.  You start adding more disks to handle the IO load on the ReportServerTempDB.  Doing this is actually pretty easy.  From within Management Studio, find the ReportServerTempDB database in the Object Explorer.  Right click and go to properties.  From the tabs on the left side of the Database Properties dialog, you will see a "Files" section, go to that guy and start adding files.  Of course, you will want to spread the files across multiple disks if you are trying to increase IO performance.  Additionally, I would recommend sizing the files to an appropriate value rather than relying on autogrowth since autogrowth can cause performance issues when SQL Server actually has to grow the files.  The exact value will be a function of how much data is incoming.  An important fact to keep in mind though is that the ReportServerTempDB database should automatically remove "stale" data as user sessions and cached reports expire, so the data shouldn't in general continuously grow.

Once you have created the additional data files, SQL Server will automatically begin to load balance IO operations across the various files, so there shouldn't be anything else you have to do.  Nice!

The posts about things dealing with rsExecutionNotFound seem to be what people find most interesting these days.  In order to aggregate the posts into one place, I will be adding links that deal with this error here.

Maybe I will add a tag as well...

• Careful when using rc-Toolbar=false
• Session Timeout during execution

I'm in the middle of reinstalling Visual Studio, and so I thought I would share an interesting story from a couple of weeks ago.  An internal customer (Microsoft-speak for some other team at Microsoft) of Reporting Services was encountering seemingly random rsExecutionNotFound errors when navigating around their application.  Of course, things like this are never random and there was a perfectly good explanation.  After digging through their application for a little while, and with the help of Fiddler, I stumbled upon the problem.  They were rendering the same report in two different contexts, one of them was specifying rc:Toolbar=false, and the other wasn't.  Basically, their call to the report server looked something like this in the two places:

http://reports/reportserver/?/myreport&rs:Command=Render&rc:Toolbar=false

http://reports/reportserver/?/myreport&rs:Command=Render

Now, there is actually a subtle little problem with rendering using rc:Toolbar=false against the ReportServer virtual directory.  When this happens in a default installation, the Report Server sends a cookie back to IE to identify the session.  This cookie is then passed back to the report server to load up the appropriate snapshot, parameter, data source information, etc to view subsequent pages and images from the report.  What would happen is the user would navigate to the page where the report being rendered w/rc:Toolbar=false was sitting, which would give IE the cookie.  The user would navigate away from the page, and eventually head to the page where rc:Toolbar=false was not specified.  However, even in this case if the cookie is there, we try to use it to load the user session, which would fail because the session had timed out in the time between the two requests.  The user would be greeted with an "Execution Not Found" error message and would have to close and restart IE to get things working again.

The moral?  Don't render against the ReportServer virtual directory using rc:Toolbar=false unless you really know what you are doing, because it is tricky to get right.  If you want to render using the ReportServer virutal directory (so you don't have to roll your own Report Viewer Control ASP.Net application) then you should build a stylesheet which removes the toolbar, and use that. 

From Miguel de Icaza's blog:

http://tirania.org/blog/archive/2007/Jan-17.html

I had no idea that the Sansa was running Mono.  Personally, I think it is really great that there are multiple implementations of the CLR.  It is even cooler that a Mono-based device is being so highly praised.

Good job guys.