Musings on Reporting Services and Notification Services

High Availability – Frequently Asked Questions about Failover Clustering and Reporting Services

lukaszp — Wed, 28 Oct 2009 23:39:07 GMT

In SSRS high availability is achieved through a Scale-out deployment that is placed behind a Network Load Balancer (NLB). The NLB is then responsible for routing requests to Report Server nodes that are responding to requests.

High Availability Documentation Reference: http://msdn.microsoft.com/en-us/library/bb522745.aspx

However, many customers are interested in using Windows Failover Clustering with SSRS, since this is how they deploy their SQL Servers. This article discusses Windows Failover Clustering in regards to SSRS.

Windows offers built-in Failover Clustering capability. SSRS does not natively support Windows Failover Clustering. The use of Failover Clustering with SSRS service is possible using standard capabilities of Windows Failover Clustering; however it may not be desirable as explained below.

Standard features of Windows Failover Clustering include a Generic Service and a Generic Script cluster resource. Each of these resources can be used in conjunction with SSRS to achieve failover during a server down situation. The Generic Service cluster resources can be used to determine whether the service is running or not and to failover to another cluster node when the service stops running for whatever reason. This option has the limitation that it will not detect service hangs. The Generic Script cluster resource allows the IT administrator to monitor any desired attribute of the service (performance counters, response time, service started/stopped, etc.) to determine when to failover to another cluster node. Using a Generic Script resource allows you to overcome the limitations of the Generic Service resource, at additional implementation cost.

Generic Script Resource: http://msdn.microsoft.com/en-us/library/aa373089(VS.85).aspx

Generic Script Resource Example: http://msdn.microsoft.com/en-us/library/aa372844(VS.85).aspx

When configuring a failover group, a Network resource (IP Address, NetBIOS name) is usually added to ensure that client machine configurations to not need to change in case of failover. During an actual failover, the network resource is moved first to the new active cluster node, and then any other resources are moved based on the failover group dependency tree. This means that during the failover there is some time during which users will receive errors when accessing SSRS since the network names will resolve to a computer where the SSRS service is in the process of starting. This is expected given the design of failover clustering.

Though a failover clustered SSRS deployment is achievable, it may not be desirable. There are a number of factors to consider:

a. Impact on other applications that share the SQL Server: One common idea is to put SSRS in the same cluster group as SQL Server. If SQL Server is hosting multiple application databases, other than just the SSRS databases, a failure in SSRS may cause a significant failover impact to the entire environment. This consideration should be carefully evaluated before deploying SSRS in the same cluster group as SQL Server. We generally recommend that SSRS should be isolated from a high availability viewpoint from the SQL Server to minimize impact.

b. SSRS failing over independently of SQL Server: Another idea is to put SSRS in a different cluster group as SQL Server. When using this configuration after a failover different cluster nodes could be running SSRS and SQL Server. If you use this configuration, ensure you have sufficient licenses to cover each cluster node with a running component of SQL Server (i.e. each RS server & each SQL Server).

c. Scalability of SSRS: in a failover cluster, only one SSRS server is handling requests. In the NLB scenario, multiple nodes are handling requests, which will result in increased capacity.

Some additional considerations:

Passive SSRS server: Customers sometimes want to know what a ‘Passive’ SSRS server looks like, or how to tell that SSRS is ‘Passive’. This has to do with the failover clustering licensing terms that come with the SQL Server box. For SSRS, since we don’t implement a cluster resource ourselves, the way to make SSRS ‘Passive’ is to stop the SSRS service. Reason for this that if SSRS is running, it is going to do work on behalf of the overall deployment so it will be ‘Active’. So if you’re looking to get the licensing benefit of failover clustering for HA with SSRS, then you need to ensure that the SSRS service is stopped on all your passive cluster nodes.

Scale-out availability in Enterprise Edition: Customers sometimes discover that they need to purchase Enterprise Edition to have a scale-out deployment of SSRS. This is important to know if you’re trying to build a failover cluster with SSRS. SQL Server offers failover clustering support with Standard Edition. This will mean that if you’re using Standard edition SSRS, you will need to handle the backup & restore of the symmetric key in the report server database in order to achieve a failover. This can be difficult since the symmetric key needs to be protected (it encrypts users names & passwords stored in the report server database). So you don’t want just anyone being able to get a copy of it and knowing the password used to encrypt the symmetric key itself. Due to this, I would say that if you want to try to do a failover cluster deployment with SSRS, use Enterprise Edition.

To summarize, failover clustering with SSRS is possible, though there are a number of considerations and manual steps involved on your part. SSRS is designed to achieve High Availability through a Scale-Out architecture. As our documentation and best practices point out, that is your best option for achieving High Availability with Reporting Services.

Take care and good luck,

-Lukasz

Building a robust setup experience with Reporting Services

lukaszp — Thu, 02 Jul 2009 18:39:01 GMT

This post is intended for ISVs or Product Development organizations who are planning to integrate Reporting Services into their product. Over the years I have seen a number of products do this and some get it right and some encounter issues. The part of the problem I’d like to focus on is recommendations for building your product’s setup/installation program when you know you have a dependency on Reporting Services.

When building your setup program, you most likely want your installation to be easy for your users (painless) and result in a successful, working installation for the customer. This is the right focus to have for the customer, and is achievable. However, there is risk involved in putting too many eggs in your setup basket. What follows is a broad strokes discussion of some often overlooked risks, followed by some specific recommendations at the end.

The key problem when relying on another product’s installation, especially in complex deployment environments, is that the other product may require some additional configuration settings/customization in those environments. As an ISV/Product Development org, you have limited resources. You mitigate risk by testing in various common deployment scenarios, but there will always be a few machine configurations which cause your installation to fail, or result in a non-working product.

Reporting Services has a reasonably complex deployment story. Our installation tries to make it easy, and it has gotten easier in the latest release. Since we are a web service, we regularly encounter problems with authentication due to SPN configuration, SSL certificates mis-configuration, network/firewall policies, etc. We do a lot of work to try to avoid these problems and handle them, but sometimes it just doesn’t work out how we wanted it to. A key problem for us is that patching our setup program with the current infrastructure is really hard (we essentially have to re-release the product). Let’s just say that’s an extremely costly activity :). However, patching our code after it is installed is much easier – we have a regular Cumulative Update model and an On Demand Hotfix model which allows customers facing serious issues to get patches very quickly. This cost ecosystem applies generally to ISVs who ship packaged product. Even for those who offer downloaded products, the cost of re-verifying all the deployment scenarios in setup is very costly and will be undesirable. The key lesson is find ways not to have to change setup.

So if you are an ISV building a setup program which will configure or install Reporting Services, you carry additional risk that SSRS configuration will not work which could result in your application setup rolling back. If there is one sin in building your setup experience it is rolling back an installation when it is not absolutely necessary. Remember the customer can always use the uninstall feature to remove the product if it doesn’t work. The reason you want setup to complete, even if there are failures along the way is that once the bits are installed on disk, they can easily be investigated by support personnel and it is easy to patch the installation. If you’ve ever seen the pain of a developer trying to debug a setup program or repro bugs which cause setup roll back you will be keenly aware of how much time and effort goes into setup debugging. Frankly, much of the developer’s/tester’s time is wasted waiting for setup to complete or clicking “next” in the setup UI. So there are operational benefits to your org/development cycle as well.

So what should you do? The pattern that works best for products that use Reporting Services is as follows:

1) Build a setup program that completes regardless of whether the Reporting Services configuration succeeds.

2) Build a configuration utility which can be run manually after your setup completes to configure Reporting Services.

By doing #1, you build an easy setup experience for the majority of your customers. They will just install the product, the configuration of Reporting Services will succeed and everyone will be happy.

By doing #2, you ensure that in the case of failure to configure Reporting Services, your customers have a back up plan. This lets you debug the scenario and provide fixes to workaround any issues you may encounter. It is also very useful in more complex scenarios like backup/restore, server migration, and multi-server provisioning where your customers may not actually want to run setup each time they need to provision Reporting Services to work with your product. This side benefit is worth its weight in gold from an IT cost/time perspective and is often overlooked. Ideally, you will make this utility a command line utility which will enable IT to automate their installation procedures and save themselves no operational costs.

Take care and good luck,

-Lukasz

InternalCatalogException, ASSERT, or NullReferenceException when using a Custom Security Extension

lukaszp — Fri, 08 May 2009 23:50:01 GMT

We have seen multiple customers with issues when customizing the Security Extension Sample (2005, 2008) that is available on http://www.codeplex.com/. This is the sample that demonstrates how to create a Forms Authentication solution with SQL Server Reporting Services.

There are two scenarios which are the primary sources of the problems:

Scenario 1: “Anonymous” access

Many custom security extension authors want to allow ‘anonymous’ access – where users do not need to login, or to allow all users some basic level of permission without needing to login explicitly. In order to do this, they change the sample code’s implementation of AuthenticationExtension.GetUserInfo to return a NULL for out IIdentity userIdentity.

However, this approach breaks the contract SSRS has with the Security Extension. SSRS in all versions (2000, 2005, and 2008) require that Custom Security Extensions specify a user name for all requests and the IIdentity returned by GetUserInfo must not be NULL.

Scenario 2: Unexpected error in Authentication Extension

The implementation of AuthenticationExtension.GetUserInfo could throw an unexpected error.

Typically this happens when the cookie returned to the extension is not found or incorrectly processed. This can be a tricky area to get right and requires some iteration.

Solutions:

Key thing for Authentication extension authors is that the error messages are hard to diagnose. We’d recommend that in your extension you fail authentication on errors and use a try-catch block to log the error message to your own log file specific to your authentication extension so that you can diagnose the state of the system when the failure occurred.

In the implementation of AuthenticationExtension.GetUserInfo, ensure you are returning a non-null IIdentity object and that UserName has a value.

Diagnosis:

In RS 2008:

We changed the error message we return to:

rsAuthenticationExtensionError - The Authentication Extension threw an unexpected exception or returned a value that is not valid: <value>.

When the authentication extension returns a null identity <value> will be “identity==null”.

If the authentication extension returns an error message from the AuthenticationExtension.GetUserInfo method then <value> will be the name of the exception.

In RS 2005 and before:

The challenge to diagnose this issue is that the call stacks can vary depending on the call being made to the SSRS server. However, these typically will include an InternalCatalogException and in the call stack you’ll see ASSERT or NullReferenceException.

If AuthenticationExtension.GetUserInfo method throws an error, an InternalCatalogException or a NullReferenceException will be in the SSRS trace log file.

If UserName is NULL, you’re going to get an ASSERT on the server side and an error will be returned to the user.

Here are a couple of examples:

Example 1: ASSERT

w3wp!library!5!2/4/2009-14:31:47:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details., un-named assertion fired for component library;
Info: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details.
w3wp!library!1!2/4/2009-14:32:10:: i INFO: Call to RenderFirst( '/VP Reports/Client/Generic/AdvFinChargesByAccount' )
w3wp!library!1!2/4/2009-14:32:11:: a ASSERT: Assertion failed! Call stack:

Example 2: NullReferenceException

w3wp!library!8!06/21/2007-14:52:09:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details., ;
Info: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details. ---> System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.ReportingServices.WebServer.WebServiceHelper.ConstructRSServiceObjectFromSecurityExtension()
at Microsoft.ReportingServices.WebServer.Global.ConstructRSServiceFromRequest(String item)
at Microsoft.ReportingServices.WebServer.Global.get_Service()
at Microsoft.ReportingServices.WebServer.Global.DispatchRequest(Boolean& transferedToViewerPage)
at Microsoft.ReportingServices.WebServer.Global.Application_AuthenticateRequest(Object sender, EventArgs e)

Take care and good luck,

-Lukasz

IIS Troubleshooting for SQL Server 2005 Reporting Services

lukaszp — Wed, 04 Feb 2009 20:42:43 GMT

Over the years, we have seen a number of IIS related issues that cause trouble for users who deploy SQL Server 2005 Reporting Services. We have tried to resolve these in the MSDN Forums on a case by case basis. Now we have compiled the common issues with IIS into a single KB Article along with explanation of why these occur and what can be done to resolve them. Kudos to James Wu on the SSRS team for slogging through the issues and compiling them into this document.

How to troubleshoot IIS configuration issues in SQL Server 2005 Reporting Services

Take care and good luck,

-Lukasz

Several blogs I would keep on my reading list...

lukaszp — Fri, 23 Jan 2009 20:04:05 GMT

Recently I have had little time to devote to blogging, but I did want to pass along a couple of references that I tend to provide to customers, partners, and Microsoft internal folks as they dig deep into SSRS.

Robert Bruckner has a great blog which offers many tips on SSRS, especially focusing on report processing. The topic I tell customers to reference most often is how to use the ExecutionLog2 View for analyzing and optimizing reports.

Dean Kalanquin has just started a new blog. He's a old time SQL guy who can craft any SQL Query you can imagine. He's started a blog and has created a post on how to diagnose and trouble shoot subscriptions which I think would be very helpful, especially to those who have read my previous posts on subscriptions management.

Take care and good luck,

-Lukasz

SQL PASS Community Summit 2008 Presentation Links

lukaszp — Fri, 21 Nov 2008 21:03:33 GMT

Denny Lee and I presented on Wednesday at the SQL PASS Community Summit 2008 on Building SSRS 2008 Large Scale Solutions.

The slide deck is available for conference attendees at from the conference site. However, the format is PDF and during the transformation from PowerPoint to PDF many of the links were lost.

I have compiled all the links from the presentation deck below for everyone's reference:

SQLCAT.com

http://blogs.msdn.com/sqlcat

http://blogs.msdn.com/mssqlisv

http://technet.microsoft.com/en-us/sqlserver/bb331794.aspx

Planning for Scalability and Performance with Reporting Services

Upgrading Reporting Services (SQL Books Online)

Configuring a Report Server Scale-Out Deployment

Building and Deploying Large Scale SQL Server Reporting Services Environments Series

Using Visual Studio 2005 to Perform Load Testing on a SQL Server 2005 Reporting Services Report Server

Scaling Up RS 2008 vs. RS 2005: Lessoned Learned

Predeployment I/O Best Practices

Backing Up and Restore Databases in SQL Server

Optimizing Backup and Restore Performance in SQL Server

Backing Up and Restore Encryption Keys

Deploying a Scalable Shared Database

SQL Server Replication: Providing High Availability using Database Mirroring

Database Mirroring and Log Shipping

SQL Server Replication Features

Scale-Out Querying with Analysis Services

Scale-Out Querying with Analysis Services Using SAN Snapshots

Scaling out an Analysis Services Solution

Tune IIS

Tune Http.Sys:Windows 2003

Tune Http.Sys: Windows 2008

Take care and good luck,

-Lukasz

SQL Server 2005 Notification Services Components Package RC1 Availability

lukaszp — Wed, 10 Sep 2008 02:23:30 GMT

As part of this blog, I comment about Notification Services. I wanted to provide you the following information about the availability of a pre-release SQL Server 2005 Notification Services components package.

If you have questions about SQL Server 2005 Notification Services, visit the SQL Server Notification Services Forum on MSDN. You can help improve the final version by submitting bugs to the Connect Feedback Center.

Take care and good luck,

-Lukasz

----------------------

SQL Server 2005 Notification Services Components Package Availability

The SQL Server 2005 Notification Services component package made available in the February 2007 release of the Microsoft Feature Pack for SQL Server 2005 is being updated to include Notification Services components, such as Notification Services server and client components, but will not include Management Studio integration. These updated components are designed to work with SQL Server 2005 and SQL Server 2008 databases.

The SQL Server 2005 Notification Services components package update will be released in two phases:

Phase 1: RC1

The SQL Server 2005 Notification Services component package RC1 is available from the Microsoft Download Center.

This release is a pre-release version and is available only for testing. Product support is not available for this pre-release.

Phase 2: RTW

Release to Web (RTW) will be released as part of the Feature Pack update currently scheduled for release as part of SQL Server 2005 Service Pack 3 (SP3). SQL Server 2005 SP3 is targeted for calendar year 2008. The SQL Server 2005 Notification Services component package update will be subject to the SQL Server 2005 Product Support Lifecycle policies.

SQL Server 2005 SP3 Notification Services

Starting with SQL Server 2005 SP3, the updated Notification Services will support using either SQL Server 2005 or SQL Server 2008 to host Notification Services Instance and Application databases.

Changes introduced in the Updated Notification Services Components Package

We have not changed core functionality of Notification Services in the components package. We made limited interoperability fixes to enable Notification Services command line tools and service components to work correctly when using a SQL Server 2008 Database Engine to host the Notification Services Instance and Application databases.

No feature additions were made.

Phase 1: RC1 - Deploying SQL Server 2005 Notification Services Components Package

The RC1 package can be installed on the same computer as an existing SQL Server 2005 Notification Services deployment. The existing deployment will automatically use the new service components installed by the RC1 package. Any Notification Services instances running on the computer should be stopped prior to installing the RC1package.

Before installing the RC1 Package, you will need to install a SQL Server 2005 Cumulative Update to enable Notification Services to connect to a SQL Server 2008 database server.

For the purposes of testing the RC1 package, the following prerequisites are needed:

1) SQL Server 2005 Notification Services

2) SQL Server 2005 SP2

3) Cumulative update package 9 for SQL Server 2005 Service Pack 2, or later

If you do not have SQL Server 2005 installed and wish to test the RC1 Package, an option is to install the Evaluation Edition of SQL Server 2005 for the purposes of testing RC1. Please note that the Evaluation Edition does have a limited license term (please see the Evaluation Edition license terms for details).

The Future of Notification Services

SQL Server 2005 is the last planned release of Notification Services. Microsoft believes “alerting” scenarios are valuable to our customers and we are looking at ways to include them in future product offerings.

SQL 2008 RTM - Where is Report Builder 2.0?

lukaszp — Fri, 08 Aug 2008 03:16:00 GMT

We've just released SQL Server 2008 (it feels great by the way!) and there have already been many e-mails asking us where to find Report Builder 2.0. You can find an update on the release plan for Report Builder 2.0 on the main Reporting Services Blog:

http://blogs.msdn.com/sqlrsteamblog/archive/2008/08/07/report-builder-2-0-release-plan-update.aspx

Take care and good luck,

-Lukasz

Reporting Services HTTP 401 (Unauthorized) - Host Headers require your attention

lukaszp — Fri, 18 Jul 2008 21:00:38 GMT

In a previous post, I talked about issues with repeated login prompts when trying to access either Report Server or Report Manager using your browser.

Recently, during an upgrade of our internal test server we identified a new type of HTTP 401 (Unauthorized) message. This post provides a description of the problem and the solution.

Symptoms:

When all of the following symptoms apply, it could be this issue:

A. When accessing Report Manager or Report Server you use a URL that looks like http(s)://<foo>/reports. The name <foo> is NOT the computer name of the computer on which Report Server and Report Manager are deployed. However, DNS or the machine's hosts or lmhosts files (WINDOWS\system32\drivers\etc) are configured to send requests to <foo> back to the report server computer.

B. When accessing Report Server directly (e.g. http(s)://<foo>/reportserver) you get no problems - you can navigate the report server namespace and you can view all reports.

C. However when you access Report Manager (e.g. http(s)://<foo>/reports), you get what looks like the Report Manager UI but where you expect to see a list of reports, you get an error saying HTTP 401 (Unauthorized). If using SharePoint integrated mode, the error can appear on the SharePoint UI pages.

D. The ReportServerUrl element is specified in the rsreportserver.config file and is set to a URL that looks like http(s)://<foo>/reportserver.

The symptoms are most likely to occur when Reporting Services is deployed in a scale-out deployment or when Reporting Services is accessed using a user friendly name that does not match the machine name of the computer.

Solution:

<foo> is considered a "Host Header". It is an alternate name for the computer on which SSRS is installed. You will need to add the NetBIOS and Fully Qualified Domain Name (FQDN) for <foo> to the list of BackConnectionHostNames stored in the Windows Registry. Casing does not matter. Use the steps in Method 2: Specify host names in KB 896861, with the following adjustment. Step 7 of the KB article says "Quit Registry Editor, and then restart the IISAdmin service." Instead, just reboot the computer. This is the correct solution for security reasons that I'll explain later in this article.

For example, if <foo> is a Windows machine name like “contoso”, then it likely also can be referenced in FQDN form as “contoso.domain.com”. You will need to add both representations to the list in BackConnectionHostNames.

Cause:

Report Server and Report Manager communicate using a network connection. When configured for a scale-out deployment of Reporting Services, typically a single virtual server name is given to the deployment and the underlying machine names are never used by end-users. The same can happen even for non-scale-out deployments but I would consider it less common. For example, you might call the server www.contoso.com or on the internal network simply "contoso". These are called Host Headers. We have only observed the issue in this post when using a Host Header; we have not seen it when using just the machine name.

By default, Report Manager is configured for Windows authentication. In this mode, it impersonates the user making the request and uses that user's credentials to connect to the Report Server. Because the network request is local, the authentication between RM & RS can succeed without running into a double hop authentication issue. In the case described above, it fails mysteriously. Let's see why...

The exact authentication mode used by both RS and RM is defined in rsreportserver.config in the <AuthenticationTypes> element. When <AuthenticationTypes> includes <RSWindowsNTLM/> then RM & RS can use NTLM authentication. NTLM is generally a destination agnostic authentication scheme. What this means is that the client does not have mutual authentication and cannot be sure which machine it authenticated to.

When you perform NTLM authentication, you can optionally specify an intended destination in the form of an SPN. This helps mitigate authentication reflection attacks. Not all clients (e.g. browsers) or APIs packages (e.g. .Net Framework) do this by default. The specific change we observed is that the .Net Framework 3.5 SP1 now defaults to specifying the Host Name used in the request URL in an SPN in the NTLM authentication package. So if you make a URL request to http://<foo>/reportserver then the SPN “HTTP/<foo>” is added to the authentication information. This is good news from a security perspective, but has an unfortunate consequence.

As an aside - slide 24 of Jesper Johansson’s PowerPoint slide deck (download.microsoft.com) on authentication shows how these reflection attacks work.

You might think that since the user accessing Report Manager accessed http://<foo>/reports and then the report manager accessed http://<foo>/reportserver, everything should just work. However, Windows has a special case when you do NTLM authentication on the local computer (i.e. loop back connection) that affords protection against authentication reflection attacks. Part of the NTLM authentication process centers on a challenge issued by the destination computer and sent back to the client computer. Windows keeps track of the challenges it issues. If Windows receives a challenge it itself generated Windows will fail the authentication unless the connection is a loop back connection.

To determine whether the connection is a loop back connection, Windows reads the destination contained in the authentication information supplied by the client. If the destination is not specified or maps to the local machine then Windows allows the authentication to proceed. In our failing case, <foo> is neither the machine name nor the loop back IP address nor the machine's IP address, so Windows fails the authentication requests. This causes the requests between Report Manager and Report Server to fail with 401 (Unauthorized).

To address the issue, we need to tell Windows that <foo> is actually an alternate name for the local computer. The correct way to do that is to use the BackConnectionHostNames registry entry as described in Method 2: Specify host names in KB 896861. Solution Method 1: Disable the loopback check will result in a less secure system, as it disables the protection against reflection attacks. It is better to constrain the set of alternate names to only those you expect the machine to actually use.

SQL Server 2008 Reporting Services Scalability

lukaszp — Mon, 14 Jul 2008 20:51:16 GMT

Much has been said up to now about the scalability work the Reporting Services team has done in the SQL 2008 version. However, until now, there were no numbers to back up the architecture discussions. The SQL Server Customer Advisory Team (SQLCAT) just released a Technical Note titled Scaling Up Reporting Services 2008 vs. Reporting Services 2005: Lessons Learned.

This is a must read for anyone looking to compare the two products. It also shows that an investment, even now, in SQL 2005 Reporting Services provides a roadmap to higher scalability in the future. Previously, I discussed an approach for Scale Testing Reporting Services that is very similar to the approach used in the Technical Note.

Some key notes from the Technical Note:

1) "Reporting Services 2008 was able to respond to 3–4 times the total number of users and their requests on the same hardware without HTTP 503 Service Is Unavailable errors compared with Reporting Services 2005, regardless of the type of renderer."

2)"Our tests clearly show that the new memory management architecture of the report server enables Reporting Services 2008 to scale very well, particularly on the new four-processor, quad-core processors."

The key take away from the Technical Note is that SQL 2008 Reporting Services can scale-up better than SQL 2005 Reporting Services. This opens the door to hardware consolidation with maintained service quality. In SQL 2005 Reporting Services, we provided guidance that scale-out should be pursued relatively quickly - after 4 CPU cores was reached. In contrast, SQL 2008 works quite well with 16 CPU cores. We still recommend moving to a scale-out deployment topology for both the scalability and availability benefits that configuration offers. However, each of the machines in the scale-out can have higher specifications that will translate to increased scalability.

At the back of the paper, in the Miscellaneous section, there is some advice that goes to show it pays to read all the way through, and I wanted to highlight:

3) "Based on our tests, we believe that every Reporting Services 2008 installation or upgrade should pay close attention to the disk configuration of [the Report Server] databases."

It is important to understand that SSRS 2008 uses the Report Server database heavily. Now that memory utilization on the Report Server computer has been alleviated as a core scalability bottleneck , the other predominant limiters of scalability - CPU and I/O - are coming to prominence. On the I/O side, there are specific things to note:

Database performance is key to making Reporting Services scale. Investing in a good I/O subsystem underneath your Report Server database will yield significant benefits.
When SSRS 2008 reacts to memory pressure, it uses the local disk on the Report Server computer to cache objects and consequently reduces memory footprint. This will result in increased disk I/O relative to SQL 2005 Reporting Services. Administrators should start monitoring the I/O performance of their Report Server boxes and investing in faster locally attached storage when bottlenecks are identified.
Since the Report Server databases typically are located off box, administrators should monitor the network link between the Report Server and the Report Server databases to ensure there is sufficient bandwidth to efficiently handle the network I/O.

Take care and good luck,

-Lukasz

Solving the Reporting Services Login issue in the February CTP of SQL Server 2008

lukaszp — Thu, 27 Mar 2008 07:44:33 GMT

I’ve seen customers run into authentication issues using the SQL Server 2008 February CTP of Reporting Services (you should all give it a try!). This post provides solutions and a detailed discussion of the underlying causes. The material in this post is applicable to all SSRS deployments, beyond the scope of solving the specific issue.

Symptom:

A) When you access report server or report manager, you get a login prompt multiple times and eventually you get a blank screen.

B) When you try to publish reports/models from Business Intelligence Development studio, you get access denied errors.

Solutions:

1) Remove RSWindowsNegotiate and ensure RSWindowsNTLM is specified in the file rsreportserver.config

2) OR, change the report server service account to Network Service

3) OR, configure a specific hostheader for SSRS, configure your DNS server to understand that hostheader, and configure your domain controller to have an SPN for the hostheader and the report server service account.

Clearly #1 is the easiest but not necessarily the right change. The rest of this post attempts to explain the cause and the merits of each solution.

Cause:

In the cases I have seen, the problem is due to Kerberos authentication. Kerberos is an authentication protocol that allows clients that create authentication tokens to associate a specific destination to that token. In the failure case there is a mismatch between the destination specified in the token and the report server process configuration. Due to this mismatch, the underlying Kerberos authentication scheme supported by Windows prevents report server from authenticating the user.

So how does this apply to SSRS 2008?

By default in the February CTP, setup includes RSWindowsNegotiate in the AuthenticationTypes listed in the file rsreportserver.config. Negotiate is an authentication type that specifies the server can either accept Kerberos or NTLM authentication tokens. The client application decides what kind of token to pass to the server.

Side note: in later builds we changed the setup behavior to add RSWindowsNegotiate only if network service is used in setup as the report server service account.

Internet Explorer (perhaps other browsers, too) uses Kerberos authentication if the server it connects to supports it. In order to use Kerberos, IE (the client application) must specify the destination in the authentication token it creates. It does this by examining the URL and asking the Domain Controller for an SPN for the destination. SPNs are Service Principal Names and are used to identify resources on the network.

There are a number of default SPNs created by the Windows ecosystem. One of these is for the Network Service account for the HTTP/Host SPN. This is the SPN used by IE to authenticate to SSRS. If SSRS is not running as the Network Service account the SPN may not resolve correctly and the Kerberos authentication will fail.

The default SPN HTTP/Host for Network Service account applies to SSRS because in SSRS 2008 we use HTTP.SYS to listen for network traffic. All services that use HTTP.SYS share this SPN by default, and they all can be subject to this problem if they support Kerberos or Negotiate authentication.

If SSRS is running under the Network Service account, things just work. However, if SSRS is running under a domain account, IE will include an SPN in the Kerberos authentication token that does not resolve to the SSRS service account. This will cause continued failure to authentication, and thus the login prompt.

NTLM is different than Kerberos - NTLM does not include a destination in the token and it will allow report server to successfully authenticate the user. Because Kerberos authentication specifies a destination in the tokens it uses, it is in some degrees more secure than NTLM.

So what should I do to fix it?

Firstly, you need to determine how much benefit using the more secure Kerberos tokens are in your environment. Generally, I would recommend that you use Kerberos authentication since having destination specific tokens is a good practice.

Secondly, you need to determine if you are OK changing the SSRS service account to Network Service. We recommend that you use a Domain Account for the SSRS service account - this helps to isolate SSRS from other processes running on the computer. Thereby, it helps make the computer that hosts SSRS more secure.

Clearly the best solution security wise is to configure a hostheader and have a specific SPN. However, it is also the hardest solution to deploy. Most computer administrators do not have the ability/permission level to make the required DNS and Active Directory SPN changes.

Is there a clear winner here? No - Unfortunately the first and second options achieve different objectives from a security perspective. Option 3, though advised, will not commonly be used due to complexity of configuration. For our setup experience, we decided that we'll use Negotiate (Kerberos) if in setup you select Network Service and that we'll use NTLM if you select a domain account for the SSRS service account. This is a good compromise since it allows report server to work by default and be as secure as possible after installation.

More sophisticated IT organizations should consider option 3 for their data center deployments.

In the interest of completeness - you may encounter this issue when changing the service account through the configuration tool. At the time of writing, you will have to ensure the Authentication Types are correct based on the service account you're using.

Is this problem new in SQL 2008? Could I have seen it in SQL 2005?

No - the problem is not new. Yes - you could have seen it in SQL 2005. The problem would manifest differently. In SSRS 2005, we set the Report Server application pool identity to Network Service. The reason for this was the very same - otherwise there are authentication failures due to the SPN. You can try this for yourself by creating 2 application pools in IIS one with Network Service, the other with a Domain Account and trying to authenticate to each.

Take care and good luck,

-Lukasz

Finding Reporting Services Content faster

lukaszp — Wed, 19 Mar 2008 04:56:00 GMT

Hi everyone - I've been away from my blog for a little while. However, now again I have some time to start blogging again.

I get tons of customer questions over the course of a week. I’d like to share with you how I make my life easier by using Internet Explorer’s built-in search box to help me find things faster. Other browsers provide similar capabilities, so you should be able to get it working there as well.

The first issue I'd like to address is an issue that has come up a number of times in a number of ways. I first addressed it here. But since then SQL Server 2008 has made a lot of progress and I find I need to find the latest documentation, quickly.

How to Add a SSRS Search Provider to Internet Explorer 7:

1. Find the Search box in the upper right corner of the IE window.

2. Use the down arrow to open the provider selector menu.

3. Click Find More Providers…

4. Find the Create Your Own box (highlighted in Yellow/Mustard color)

5. In the text box labeled Paste the URL of the Search results page URL, paste the following:

http://search.live.com/results.aspx?q=TEST+%22reporting+services%22+site%3Amsdn2.microsoft.com

6. In the text box labeled Specify a name for the search provider Name, paste the following:

RS on MSDN2

7. Click Install.

8. Now type a search term into the search box (choose your favorite SSRS feature name), use the down arrow and select RS on MSDN2 from the selector menu. You will see the results of your search.

You can use additional search terms to find documentation that is specific to a version:

Find only SQL Server 2008 documentation

http://search.live.com/results.aspx?q=TEST+100+"reporting services"+site:msdn2.microsoft.com

Find only SQL Server 2005 documentation

http://search.live.com/results.aspx?q=TEST+90+"reporting services"+site:msdn2.microsoft.com

Find only SQL Server 2000 documentation

http://search.live.com/results.aspx?q=TEST+80+"reporting services"+site:msdn2.microsoft.com

Alternately, you can just type your search term followed by 100, 90 or 80 in the Search Provider text box.

Clearly this doesn't apply only to SSRS, but that's where I find it most useful :-).

Take care and good luck,

-Lukasz

Update 1: I corrected the search terms to be actual URLs. It should work better for people. Note, I'm using www.live.com as my search provider. You can do something similar with other search engines.

Keeping your report servers awake (or No more waiting for report server to startup)

lukaszp — Wed, 19 Sep 2007 00:26:00 GMT

Reporting Services in SQL 2005 is hosted in IIS. IIS has a performance optimization built-in that you cannot disable (AFAIK), which shuts down an application once there is a period of 20 minutes of inactivity. This is governed by the Idle Timeout IIS Metabase property.

You can read about this Idle Timeout here:

http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/b39a8ab9-7331-4bad-a89a-0210aadbd4a7.mspx?mfr=true

You can read about how to set the Idle Timeout here:

http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/83b35271-c93c-49f4-b923-7fdca6fae1cf.mspx?mfr=true

The behavior of the Idle Timeout causes the user’s perception that Report Server is constantly falling asleep. In fact, the same behavior applies to any web application hosted in IIS. Once IIS has shut down the report server, it can take anywhere from 5 to 20 seconds to respond to the next request. This value ranges based on the performance of the server hosting IIS & RS. This time is taken up by IIS loading ASP.Net, ASP.Net in turn loading the report server DLLs, and finally report server starting up. All of this takes some time. Naturally, to a user it looks like report server is dozing off - again.

To address this issue you need to ping the report server periodically to ensure it is never inactive. To ping the server, make a Web Services (SOAP) request to the report server virtual directory. You could do the same with Report Manager; though RM load time is quite fast if the report server is already running. You can do this using your favorite scheduling technology like the Windows Task Scheduler, a SQL Agent job that runs an extended stored procedure, etc.

But since we’re talking about Reporting Services, I thought I’d create a sample to show you how to use the solve this problem using Subscriptions.

I posted a report that lists the items in the report server namespace.

To get the report:

First add yourself to the lukaszpblog group on http://groups.msn.com/lukaszpblog.

Then you can get the report here:

ListChildren Report:

http://www.msnusers.com/lukaszpblog/Documents/Files/ListChildren.rdl

Take the report and publish it to your report server. Viewing it will show you a very basic listing of content at the root of the report server namespace. Edit the properties for the report and set the data source credentials the report will run as. The credentials will either need to be stored or none. In order to choose none, you’ll need to set the unattended execution account. The account you choose should have the minimum level of privilege (I’d suggest Browser role) on the report server. Test that the report executes to make sure everything works.

Now you can create a subscription to this report. Use the parameters in the subscription to ensure the report executes against the report server you want it keep alive. If you’re deployed in a scale-out deployment you should create one subscription for each node in the scale-out and set the parameters explicitly to access the machine directly (not through the virtual server name or NLB). Set subscription to execute on a schedule and set that schedule to occur every 9 minutes. I like this frequency since if something were to go wrong, like a networking issue, you the subscription will run twice within the span of the 20 minute IIS application shutdown window.

With this solution, the report server will keep itself alive by continually processing the subscription which pings the web service and therefore IIS will never unload it. Problem solved.

Take care and good luck,

-Lukasz

Scale Testing Reporting Services

lukaszp — Wed, 05 Sep 2007 21:25:00 GMT

I wanted to provide a quick note on how to scale test Reporting Services. Partially I'm posting this so I don't lose the link :-).

Often we are asked how best to scale test Reporting Services and what scale numbers to expect coming from a particular configuration. We always hesitate to promise specific numbers because so much depends on your usage patterns, your reports, the size of your data, and your architecture. To say that N users will always be supported by X hardware is just too much of a simplification. Therefore the guidance we give is based on patterns we have seen in our testing lab. You can see the guidance we gave for SQL Server 2005 Reporting Services here:

http://www.microsoft.com/technet/prodtechnol/sql/2005/pspsqlrs.mspx

You should be aware that SQL Server 2008 CTPs (Community Technology Preview) are becoming available as 2007 continues. To date, I believe we have provided 3 public SQL Server 2008 CTPs. You can find the latest one here:

https://connect.microsoft.com/SQLServer/content/content.aspx?ContentID=5395

As always with a new version, users want to know how well it scales relative to the old version. Bear in mind that these CTPs are kind of like Alpha or Beta releases of the product - they're not finished yet and performance/scale characteristics will change as we get closer to release.

However, if you're thinking of adopting the next version, now is the time to start planning your testing strategy. Specifically, you can start by building your test infrastructure to do scale testing against the existing product and reuse as new CTPs come available. The leg work to understand which reports are key to test and which scenarios you should be including is independent of your actual testing timeframes and having it done and out of the way can greatly simplify your acceptance testing later. In this regard, you may find the following article very interesting (I did):

Using Visual Studio 2005 to Perform Load Testing on a SQL Server 2005 Reporting Services Report Server (http://technet.microsoft.com/en-us/library/aa964139.aspx)

Whether you go with this approach or another solution, you will be well served to start planning this testing well in advance of having the final product in place. Also, we have not released any guidance on perf/scale for SQL Server 2008 Reporting Services; this will come in due course as we get closer to release.

Take care and good luck,

-Lukasz

Report Viewer Control - Reset() method

lukaszp — Fri, 24 Aug 2007 20:07:00 GMT

This post is about a consequence of the calling ReportViewer.Reset() method related to revaluating parameter default values.

The Report Viewer controls are a great way to integrate reports into your applications. You can see some examples here:

http://www.gotreportviewer.com

One aspect of using the controls that may not be easily found/understood is the value of the Reset() method.

http://msdn2.microsoft.com/en-us/library/microsoft.reporting.winforms.reportviewer.reset(vs.80).aspx

http://msdn2.microsoft.com/en-us/library/microsoft.reporting.webforms.reportviewer.reset(VS.80).aspx

The description of the method obsurely references to resetting the control to the 'default state'. This has an implication in the case of parameters - when you reset the control, you create a new 'session' if connected to the report server. This causes the report to be reexecuted when you run it. It also means that the parameter values will be reset to their initial defaults. Once done, you can again customize the parameter values in your code.

Take care and good luck,

-Lukasz