August, 2008

  • Never doubt thy debugger

    How many application pools can you cope with?

    • 1 Comments

    This is an interesting question I have to admit I had not thought to until we got a case from a customer whom was in trouble with his production servers: how many application pools can you run on IIS? Event better, how many w3wp.exe instances can the OS cope with? As quite often happens in our job, the correct answer is: it depends…

    First, the problem. The customer was a hosting company so they had a large numbers of sites on each web server, around 450, each of them with its own application pool, each application pool run under its own specific account and aspExecuteInMTA was set to 1 on each server. Under some circumstances asp applications from some web sites are failing with the following message (roughly translated from Spanish):

    Server error 'ASP 0177 : 80070057'
    Error in Server.CreateObject
    /site/page.asp, line 338
    80070057

    When the problem occurred the customer recycled the two or three top consuming web sites in terms of memory, threads and handles; then the failing web site was usually back online. Note that the failing web site was always only affected by other problematic ones, it was almost never the culprit.

    At first we were distracted by the fact that the failing web sites where using Access as the database and we also got some errors like:

    Server error 'ASP 0177 : 80070583'
    Error in Server.CreateObject
    /database.asp, line 9 -- "Provider=Microsoft.Jet.OLEDB.4.0;Data source=" & server.MapPath("../data/bd.mdb")
    80070583

    This is a deprecated configuration, see Using Microsoft JET with IIS:

    The Access ODBC Driver and Microsoft OLE DB Provider for Microsoft Jet are not intended to be used with high-stress, high-concurrency, 24x7 server applications, such as web, commerce, transactional, messaging servers, and so on.

    Anyway we soon found that was not the right direction. 450 application pools are a lot, plus there was the thing about recycling the most handles and threads consuming pools… this suggests desktop heap exhaustion.

    A bit of theory

    Desktop Heap overview has a good explanation of what Desktop Heap is and how things might start to go wrong when the system runs low in Desktop Heap storage; it also has a few interesting bits about IIS:

    Regarding non-ASP.NET applications, IIS 6.0 has been tested on a well-configured mainframe server running up to 2,000 concurrent worker processes, each serving one application pool, but not using unique identities. In practice, a design of up to 500 simultaneous application pools is achievable, depending on the application requirements and assuming hardware resources are not a significant constraint. It is important to configure the UseSharedWPDesktop registry setting mentioned above when using more than 60 application pools with unique identities

    My colleague Nicolas also has a post about Desktop Heap and Windows Stations here. Having one shared desktop can theoretically bring some security issues, but David Wang explains well this scenario:

    "we should use a common desktop when the number of application pools with unique identities configured on a server is greater than 60". In other words, IIS does not have a problem running >60 application pools which uses <60 unique identities. The limitation in question is the number of user desktops associated with each unique application pool identity that Windows simultaneously supports.

    The following are the interesting combinations and consequences:

    1. If UseSharedWPDesktop=0 and all application pools launch as Network Service, then there is exactly one Desktop (of Network Service) shared by all the application pools. You have no isolation of desktop nor process identity between application pools.
    2. If UseSharedWPDesktop=0 and all application pools launch as unique user identity, then there is a unique Desktop for each user identity, which is obviously not shared. This is limited to a number around 60 due to Desktop heap limit in Windows Server 2003. This means that you can only have ~60 application pools perfectly isolated by desktop and process identity simultaneously running.
    3. If UseSharedWPDesktop=1 and all application pools launch as unique user identity, then there is one Desktop that is shared by all the user identities. This removes the limit of 60 at the cost of not perfectly isolated application pools. You have no desktop isolation but retain process identity isolation between application pools.
    4. UseSharedWPDesktop=1 and all application pools launch as Network Service is pretty much a degenerate case that is not interesting. :-)

    Back to the problem

    The customer was already aware of those blog posts so they were already using UseSharedWPDesktop=1 (otherwise they were not able to run more than 50 concurrent application pools), set sessionViewSize to 64 Mb and the default value for “HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows”.

    To know more about Desktop Heap status on the server we used Desktop Heap Monitor: this comes with a comprehensive usage guide, anyway here are some quick steps to use it:

    1. Log in with an account that has Administrator privilege. Note that you must either do this at the physical interactive console (i.e. not through Terminal Services) OR connect to the console session using mstsc /console (only possible on XP or Server 2003)
    2. Open a command prompt and change to "c:\kktools\dheapmon8.0"
    3. For uni-processor systems run
      • DHEAPINST -f
    4. For multi-processor systems (including single processor hyper-threaded systems) run
      • SET _NT_SYMBOL_PATH=symsrv*symsrv.dll*C:\Symbols*http://msdl.microsoft.com/download/symbols
      • DHEAPINST
      • You will be prompted to agree to the symbol server terms of use. You must accept these for the tool to work
        The server in question must be able to make an outgoing connection to the Internet for this to work
    5. For either 3. or 4. you should see
      • dheapinst - Desktop Heap Monitor installed successfully
    6. DHEAPMON uses a driver in order to read information from the kernel memory. This must be loaded by the following command:
      • DHEAPMON –l
      • You should see this confirmation message: Dheapmon - Driver loaded successfully
    7. You are now ready to run the tool and capture the desktop heap information. Run the following command:
      • DHEAPMON -v > desktopheap.txt
      • Important: if the output text file does not contain a line like this: “Windowstation: (Service-0x0-3e7$) SessionID: 0” (without quotation marks) then it is almost certain that you did not connect to the server using Terminal Services Client using the /console switch. You must connect using /console to get the full information required

    To automate the process of gathering Desktop Heap information, create a batch file called HeapMon.bat containing the following:

    @echo off
    rem vars C B A are day month year in the for line below, change them to change date
    for /F "tokens=2,3,4 delims=/- " %%A in ('date /T') do set DT=%%C_%%B_%%A
    for /F "tokens=1,2 delims=/: " %%A in ('time /T') do set TM=%%A-%%B
    echo %DT%
    echo %TM%
    Dheapmon -v > dheapmon_out_%DT%_%TM%.txt

    This will create an output file uniquely named with the date and time included; you can use task scheduler to run this batch file at regular intervals, e.g. once per hour.

    Once finished all diagnostic activity with DHeapMon, unload the device driver with:

    DHEAPMON -u

    To uninstall dheapmon, run:

    DHEAPMON -r

    From the report I got is was clear that the customer was really using only 1 desktop within IIS WinSTA (remember he was using a shared desktop across his application pools); as IIS WinSTA is created as non interactive, this single desktop is 512 Kb but it was exhausted at 99.2% when then problem appeared:

    Windowstation: (WP_WINSTA-054f77ac-6ece-4080-b3a2-b0bf08dacaab) SessionID: 0
    Desktop: (Default) Addr: bbc30000
    Desktop Heap          524288 (0x   80000) Bytes
    Committed             520192 (0x   7f000) Bytes
    Uncommitted             4096 (0x    1000) Bytes
    Allocated             520072 (0x   7ef88) Bytes
    Total Freed              120 (0x      78) Bytes
    Unused                  4216 (0x    1078) Bytes
    Used Rate                    (      99.2) %

    To calculate SessionViewState size we should do something like:

    312Kb (Winsta0/Default) +
    64Kb (Winsta0/Disconnect) +
    128Kb (Winsta0/Winlogon) + 
    7 * 512Kb (all other non interactive winsta) +
    512Kb (IIS non interactive winsta WP_WINSTA-xxx)
    == 4600 Kb  (< 5 Mb)

    So apparently the problem is not on SessionViewSize bur rather just on the non interactive Desktop Heap size; moreover setting this value to 64 Mb as the customer did is not necessary because they were using a bit less than 4 Mb for that value so also the default (20 Mb) were far enough.

    The problem of increasing this one too much is that not only IIS owns one, so we first tried using 1024 Kb:

                    312 + 64 + 128 + 7 * 1024 = 7672 Kb (< 7 Mb)

    So we tried with the following (not the customer was not using /3gb):

    • UseSharedWPDesktop enabled
    • Delete the SessionViewSize registry key to get the default 20 Mb
    • SharedSection=1024,3072,1024

    This first try allowed the server to run fine a bit longer than before but at the end the problem was there again, and another heapmon trace showed we were still exhausting Desktop Heap… we gave another try with SharedSection=1024,3072,2048 (312 + 64 + 128 + 7 * 1024 = 14840 Kb (< 15 Mb).

    Bingo, this time we did it! smile_regular 

    Of course be very careful when you manipulate the Desktop Heap configuration and if you’re not sure how to do (or if you’re doing it right, or if you need help to fine tune your settings etc…) give Microsoft Support a call! smile_regular

    Conclusions

    To close with some general recommendations:

    1. Any kind of platform that has to support more than concurrent 450 processes should be considered to migrate under x64, and to really care about context switching
    2. Windows 2008 Server with Dynamic Session Space removes the SessionViewSize limits, but desktop heap limits would still apply
    3. 450 processes identity is also a very big because logon session & token cache might duplicate kernel memory
    4. Always be very careful when changing those values since this might have effect on other kernel quota

    The point is we will always have inherent x32 bit limitations that we cannot workaround and 450 application pools is a pretty close “practically tested” limit…

     

    Carlo

     

    Quote of the Day:
    Women like silent men. They think they're listening. - Marcel Archard
  • Never doubt thy debugger

    Visual Studio 2008 SP1 and .NET 3.5 SP1 available for download

    • 2 Comments

    Microsoft Visual Studio 2008 Service Pack 1 (exe)

    This download installs Visual Studio 2008 Service Pack 1 (SP1) and the .NET Framework 3.5 SP1. SP1 addresses issues that were found through a combination of customer and partner feedback, as well as internal testing. These service packs offer Visual Studio and .NET Framework users improvements in responsiveness, stability and performance.
    Click here for more information regarding about these service packs.

    Microsoft .NET Framework 3.5 Service Pack 1

    .NET Framework version 3.5 Service Pack 1 provides the following new features and improvements:

    • ASP.NET Dynamic Data, which provides a rich scaffolding framework that enables rapid data driven development without writing code, and a new addition to ASP.NET AJAX that provides support for managing browser history (back button support). For more information, see What’s New in ASP.NET and Web Development.
    • Core improvements to the CLR (common language runtime) that include better layout of .NET Framework native images, opting out of strong-name verification for fully trusted assemblies, improved application startup performance, better generated code that improves end-to-end application execution time, and opting managed code to run in ASLR (Address Space Layout Randomization) mode if supported by the operating system. Additionally, managed applications that are opened from network shares have the same behavior as native applications by running with full trust.
    • Performance improvements to WPF (Windows Presentation Foundation), including a faster startup time and improved performance for Bitmap effects. Additional functionality for WPF includes better support for line of business applications, native splash screen support, DirectX pixel shader support, and the new WebBrowser control.
    • ClickOnce application publishers can decide to opt out of signing and hashing as appropriate for their scenarios, developers can programmatically install ClickOnce applications that display a customized branding, and ClickOnce error dialog boxes support links to application-specific support sites on the Web.
    • The Entity Framework is an evolution of the existing suite of ADO.NET data access technologies. The Entity Framework enables developers to program against relational databases in according to application-specific domain models instead of the underlying database models. For more information, see Getting Started with the Entity Framework. The Entity Framework introduces some additional features, including support for new SQL Server 2008 types, default graph serialization of Entities, and the Entity Data Source. This release of the Entity Framework supports the new date and file stream capabilities in SQL Server 2008. The graph serialization work helps developers who want to build Windows Communication Foundation (WCF) services that model full graphs as data contracts. The Entity Data Source provides a traditional data source experience for ASP.NET Web application builders who want to work with the Entity Framework.
    • LINQ to SQL includes new support for the new date and file stream capabilities in SQL Server 2008.
    • The ADO.NET Data Services Framework consists of a combination of patterns and libraries, which enable data to be exposed as a flexible REST (Representational State Transfer)-based data service that can be consumed by Web clients in a corporate network or across the Internet. The ADO.NET Data Services Framework makes data service creation over any data source. A conceptual view model of the underlying storage schema can easily be exposed through rich integration with the ADO.NET Entity Framework. Services created by using the ADO.NET Data Services Framework, and also compatible Windows Live (dev.live.com) services, can be easily accessed from any platform. For client applications that are running on Microsoft platforms, a set of client libraries are provided to make interaction with data services simple. For example, .NET Framework-based clients can use LINQ to query data services and a simple .NET Framework object layer to update data in the service.
    • Windows Communication Foundation now makes the DataContract Serializer easier to use by providing improved interoperability support, enhancing the debugging experience in partial trust scenarios, and extending syndication protocol support for wider usage in Web 2.0 applications.
    • The .NET Framework Data Provider for SQL Server (SqlClient) adds new support for file stream and sparse column capabilities in SQL Server 2008.

    Here is the link to download the full package: Microsoft .NET Framework 3.5 Service Pack 1


    Carlo

    Quote of the Day:
    Resentment is like taking poison and hoping the other person dies. - St. Augustine

  • Never doubt thy debugger

    Charting with LogParser

    • 1 Comments

    A picture worth 1000 words and this is true also when you have to analyze IIS logs to get some statistics on your site; as you already know, LogParser is extremely flexible and very useful in such situation and its chart output format is what it’s needed.

    The first thing you need is a copy of Office Web Components installed on the machine; if you don’t, LogParser will complain and return this error:

    Error creating output format "chart": This output format requires a licensed Microsoft Office Chart Web Component to be installed on the local machine

    From LogParser help:

    chartType

    Values:
    name of chart type

    Default:
    Line

    Description:
    Chart type

    Details:
    The set of available chart types depends on the version of the Microsoft Office Web Components installed on the local computer.
    For a list of the available chart types, type the following
    help command from the command-line shell: 

         LogParser -h -o:CHART

    Currently the latest Office Web Components version available (the newest I’ve been able to find on the Internet) is version 11 which has been released for Office 2003.

    Changes in the 2007 Office System

    OWC11
    Description: OWC11 is removed from the 2007 Office system and now ships only with Microsoft Office Project 2007. The DataFinder functionality in OWC11 is also removed. This affects solutions that use the spreadsheet, chart, pivot table, and data source control functionality of OWC11 in a Web page or client program that requires installation of this ActiveX control on the user's computer. The user experience varies depending upon the host container of the OWC11 ActiveX control.

    For Web pages, users might see a broken icon or a notification that OWC is required. For client programs, users might receive a runtime error or other notification. In these situations, users download OWC11 from the Microsoft Download Center. The notification message generated by Office programs provides a link to the download location for OWC11. Neither Microsoft Office Excel 2007 nor SharePoint Designer 2007 now generate Web pages that use OWC.

    Reason for change: New technologies such as Windows SharePoint Services 3.0 with Excel server side capabilities and the new charting component in the 2007 Office system replace some features of OWC11.

    And if you have Office 2007 like me? I installed OWC11 anyway and everything is still working fine so you should be fine; just in case you need the download links, here they are: Office 2003 Add-in: Office Web Components and Microsoft Office 2003 Web Components Service Pack 1 (SP1) for the 2007 Microsoft Office System.

    A few samples

    Bytes per data type as exploded pie

    LogParser 
        "SELECT TO_UPPERCASE(EXTRACT_EXTENSION(cs-uri-stem)) AS PageType, MUL(PROPSUM(sc-bytes),100.0) AS Bytes 
        INTO Pie.gif FROM *.log 
        GROUP BY PageType ORDER BY Bytes DESC" 
        -chartType:PieExploded -chartTitle:"Bytes per data type" -categories:off -o:chart -i:iisw3c

    Bytes per data type as exploded pie

     

    Calculating percentages per data type

    logparser 
        "select extract_extension(cs-uri-stem) as Resource, mul(propcount(*),100.0) as ResourceHits 
        into percentage.jpg from *.log 
        group by Resource order by ResourceHits desc" 
        -chartType:PieExploded3D -chartTitle:"Percentages per data type" -categories:off -o:chart -i:iisw3c
    Percentage per data type

     

    .aspx page hits per minute

    logparser 
        "select quantize(time, 60) as TimeGenerated, count(*) as Hits 
        into qnt.gif from *.log 
        where to_lowercase(extract_extension(cs-uri-stem))='aspx' group by TimeGenerated" 
        -i:iisw3c -o:chart -chartType:Line
    aspx hits per minute

     

    Getting the number of unique visitors (actually the number of unique client IP addresses) is a bit more complex. We could try the following:

    select date, count(distinct c-ip) into UniqueIPs.gif from *.log group by date" -i:iisw3c -o:chart -chartType:Line

    but we get this error:

    Error: Semantic Error: aggregate functions with DISTINCT arguments are not supported with GROUP BY clauses

    The solution in this case is to split the query into two separate commands; for example you could create a .bat file (or script, or whatever method you prefer) and run the following:

    @echo off 
    
    logparser 
        "select distinct date, c-ip into dates.txt from *.log" -i:iisw3c -o:w3c
    
    logparser 
        "select date, count(c-ip) as UniqueIPs into UniqueIPs.gif 
        from dates.txt group by date order by date" 
        -i:iisw3c -o:chart -chartType:SmoothLine -view:on

    unique IPs per day

    Advanced carting with configuration scripts

    Using an external script file (JScript or VBScript) it is possible to have a deeper control over the produced chart; this is based on two global objects which expose methods and properties that can be used to modify parameters such as the chart colors, the chart fonts, and many other attributes. Those two global objects are instances of the chartSpace and chart objects of the Microsoft Office Web Components ChartSpace object model, and they are named "chartSpace" and "chart"; for more information on those objects you can have a look at the MSDN ChartSpace Object Model documentation.

    Taken from the LogParser documentation, we can use the following sample script to add a caption to the previous UniqueIPs chant and make the background color transparent:

    // Add a caption
    chartSpace.HasChartSpaceTitle = true;
    chartSpace.ChartSpaceTitle.Caption = "Generated by Log Parser 2.2";
    chartSpace.ChartSpaceTitle.Font.Size = 8;
    chartSpace.ChartSpaceTitle.Position = chartSpace.Constants.chTitlePositionBottom;
    
    // Change the background color
    chart.PlotArea.Interior.Color = chartSpace.Constants.chColorNone;
    @echo off 
    
    logparser 
        "select distinct date, c-ip into dates.txt from *.log" -i:iisw3c -o:w3c
    
    logparser 
        "select date, count(c-ip) as UniqueIPs into UniqueIPs.gif 
        from dates.txt group by date order by date" 
        -i:iisw3c -o:chart -chartType:SmoothLine -config:MyScript.js -groupSize: 800x600 -view:on

    unique IPs per day with config script

    From here you can experiment using the ChartSpace object model to further customize the appearance of your charts: the limit is your imagination smile_regular

     

     

    Carlo

    Quote of the day:
    Ability will never catch up with the demand for it. - Malcolm Forbe
Page 1 of 1 (3 items)