Alik Levin's : Deployment Phase

Use FREE Tools From IIS Resource Kit To Warm Up Your ASP.NET 1.1 Application By Batch Compilation

alikl — Sun, 20 Jul 2008 18:24:42 GMT

Have you noticed that when ASP.NET web application is accessed for the first time the response is slow? The reason for such behavior is batch compilation that occurs on the first hit.

ASP.NET batch compilation is the process of compiling ASP.NET markup (content of aspx files) into temporary dll’s. Compilation requires invoking compiler (csc.exe for C#) – that is pretty heavy activity. Process Explorer shows it clearly:

ASP.NET batch compilation occurs on per folder basis. Said that, if your application divided into multiple sub-folders that contain ASP.NET pages each time any of the folders accessed for the first time the batch compilation is invoked.

Note that starting with ASP.NET 2.0 compilation model has changed. Also, there is a tool Aspnet_compiler.exe that allows pre-compile your ASP.NET web application to improve performance.

Customer’s case study

Customer’s web application is built with ASP.NET 1.1. It is divided into multiple subfolders reflecting logical modules that are hosted across about 20 application pools. The application connects to Oracle database.

QA team complains that the application responds slowly each time any of the modules (subfolders) accessed for the first time.

Using Process Explorer and profiler we identified three main latency points:

Creating the application pool – w3wp.exe.
Batch compiling the application for each subfolder.
Creating Oracle connection pool when Oracle is accessed for the first time.

The solution

We decided to create a Warmer – solution that will hit each subfolder’s page in unattended manner thus warming up the application before the first user hits it.

For the solution we used free tools from IIS resource kit:

LogPrser.exe to identify the URL’s of the pages to hit.
TinyGet.exe to actually hit the pages identified by LogParser.

To identify what pages to hit we took IIS log files from QA environment and than we ran the following query using LogParser:

LogParser.exe "SELECT DISTINCT STRCAT('XXX', cs-uri-stem) AS cs-uri-stem-strcat INTO 'C:\result.txt' FROM 'C:\yourIISlogFile.log' WHERE INDEX_OF(cs-uri-stem, 'aspx') > 0" -o:w3c

Notice XXX – it has nothing to do with XXX rated content rather it is a placeholder to replace it with tinyget command.

Open resulting yourIISlogFile.log file in Notepad, hit Ctrl+H for “Replace” and replace all occurrences of XXX with the following command:

tinyget -srv:www.YourServer.com -uri:

yourIISlogFile.log before the Replace:

yourIISlogFile.log after the Replace:

Remove the header and save the file with BAT extension - your Warmer is ready for action. Run it each time you deploy new version.

Do not forget to remove old temporary files in ASP.NET temporary folder:

C:\Windows\Microsoft.NET\Framework\<<NET FX VERSION>>\Temporary ASP.NET Files\

CAUTION. This action may potentially corrupt your application if you do not provide proper exception handling. On one hand it is good check to make. on other hand – be aware of it and do not do it on production sites unless you are completely sure it will not corrupt the application.

Related materials

Performance Sin - Chatty Database Access And Loops (Plus Another Free Performance Tool)

alikl — Mon, 28 Apr 2008 15:31:08 GMT

Chatty database access is the surefire way for slow performance caused by resources starvation that might even lead to denial of service. Following is a real world case.

Customer

Service Unavailable message is consistently observed when there are more than 150 users access the web site. We think IIS cannot handle more than 150 users. What would you suggest?

Support

Let's see what "Service Unavailable" means. "IIS cannot start any new worker processes because of limited system resources...". Let's see the code.

The code

The code was calling on DB Access function that was put inside for loop that was called inside event handler. The event handler was called on each GridVew's row creation. Meaning, DB access in nested loop. In case of small GridVew of 10 rows and 10 cells the database would be accessed 100 times for each request. 150 concurrent users would create significant load of 15,000 connections:

SQL Server Profiler

After running SQL Server profiler it became clear that database access should be significantly improved. Following is the number of SQL Commands performed as a result of single request:

Conclusion

For improved performance avoid chatty data base access. Apply caching techniques instead.

More free performance tools

If you use SQL Server 2005 Express which does not come with built-in profiler you might find useful the following free tool:

Profiler for Microsoft SQL Server 2005 Express Edition

My related posts

Securing IIS7 - Windows Server 2008 Security Guide

alikl — Thu, 28 Feb 2008 22:09:42 GMT

Windows Server 2008 Security Guide is out.

It covers many crucial aspects but my favorite of course is IIS7 chapter:

Chapter 6: Hardening Web Services

This chapter provides prescriptive guidance for hardening the Web Server role. The chapter discusses how the Web server role installs Microsoft® Internet Information Services (IIS) 7.0, which has been redesigned into forty modular components that you can choose to install as needed.

It points to the following resources:

ASP.NET 2.0 Internet Security Reference Implementation - Have It Handy

alikl — Wed, 21 Nov 2007 13:04:58 GMT

JD Meier writes in his blog:

The ASP.NET 2.0 Internet Security Reference Implementation is a sample application complete with code and guidance. Our purpose was to show patterns & practices security guidance in the context of an application scenario. We used Pet Shop 4 as the baseline application and tailored it for an internet facing scenario. The application uses forms authentication with users and roles stored in SQL.

It is master piece, nothing less. Simple to get a grip, practical, and focused.

Now that GotDotNet is down I thought it would be useful to have it handy somewhere else that is on.

Download ASP.NET 2.0 Internet Security Reference Implementation Document here

Download ASP.NET 2.0 Internet Security Reference Implementation Code here

Enjoy

Composite Application Block (CAB) Programming Essentials - Crucial For CAB Performance

alikl — Mon, 29 Oct 2007 23:30:16 GMT

Rich Newman posted awesome guides for Composite Application Block (CAB) programming:

Table of Contents: Introduction to CAB/SCSF

Part 1 Modules and Shells
Part 2 WorkItems
Part 3 Introduction to Dependency Injection
Part 4 An Aside on Inversion of Control, Dependency Inversion and Dependency Injection
Part 5 Dependency Injection and the Composite Application Block
Part 6 Constructor Injection in the Composite Application Block
Part 7 Introduction to Services in the Composite Application Block
Part 8 Creating and Using Services in the Composite Application Block
Part 9 The Command Design Pattern
Part 10 Commands in the Composite Application Block
Part 11 Introduction to Events in the Composite Application Block
Part 12 Events in the Composite Application Block
Part 13 Introduction to UIExtensionSites

It is important to understand these core principles. Recently I was involved with a project where CAB was used extensively. Too extensively... The application was actually over-CAB'ed causing performance hit. When we ran the profiler we saw that many functions calls were empty while adding up to execution time. The only solution was redesigning the application and CAB usage.

Lessons Learned:

Do not over CAB
Load modules on demand
Cache static data

Related resources:

Notes on (extreme) Performance requirements for CAB
Mobile Client Software Factory (includes includes CABgen and OBgen for NGEN'ing CAB modules)
How To: load CAB modules on demand

Examining WCF Diagnostic Traces Using Service Trace Viewer Tool (SvcTraceViewer.exe)

alikl — Tue, 23 Oct 2007 16:52:57 GMT

Service Trace Viewer Tool (SvcTraceViewer.exe) tool comes with Microsoft® Windows® Software Development Kit for Windows Vista™ and .NET Framework 3.0 Runtime Components. It allows to view WCF diagnostics traces in very convenient way. "Using Service Trace Viewer for Viewing Correlated Traces and Troubleshooting" article explains how.

Specifically I used it for two scenarios

Troubleshooting when implementing Message Security with a Windows Client without Credential Negotiation. Using SvcTraceViewer.exe I spotted the following error message: "The security timestamp is invalid because its creation time ('9/19/2007 5:43:39 PM') is in the future. Current time is '9/19/2007 8:43:42 AM' and allowed clock skew is '00:05:00'." I immediately understood that machines clocks are out of sync - which is super important for Kerberos to work properly.
Examining message sizes with different protection mechanisms and levels. I needed to understand the impact of data protection mechanisms on general performance. See How Message and Transport Security compare.

Related posts

WCF Security In Intranet Scenario : Thoughts On Cons and Pros

Authentication And Identity Flow When ASP Page Consumes ASP.NET Web Service

alikl — Wed, 05 Sep 2007 19:04:00 GMT

"Classic" ASP has application isolation that is different from ASP.NET. Here is one of the real world scenarios where it might matter.

There is a legacy web application written in ASP and hosted on Win2K3 box (IIS 6.0). It is of course in the process of migration to ASP.NET. As part of the migration process there were several ASP.NET web services factored out of the classic ASP app. These web services are hosted on another Win2K3 box and require windows authentication. Classic ASP must consume these web services while satisfying the requirement of windows authentication. ASP page consumes the web service via .Net COM interop invoking .Net component:

The question here is what is this account that ASP page authenticates to ASP.NET web service on another machine?

It is common mistake assuming that the account is the application pool's one. ASP does not run in the context of the application pool. In case of anonymous access It runs in the context of what defined for anonymous user:

Said that, in order to let ASP page authenticate to ASP.NET web service based on windows authentication one needs to define domain account in above property page for virtual directory where ASP resides. This is the account that will hit the ASP.NET web service.

Client Certificates Authentication - Dirty Trick To Disable CRL Check. For Demos Only!

alikl — Tue, 14 Aug 2007 18:12:55 GMT

My lab domain has MS CA installed in it so I am able to issue certificates to the left and to the right. Recently I spent some time to understand why client certificates authentication does not work. More precisely the certificates dialog box was offering no client certificate to chose, as depicted below:

I first thought it is something on the client machine but after some investigation it turned out that it is IIS' part. IIS was unable to verify CRL. I was not in the mood of deploying CRL's so I decided to look into how to disable this feature. Here it is:

http://forums.iis.net/t/1100044.aspx

   Set oWeb = GetObject("IIS://localhost/W3SVC")
   oWeb.CertCheckMode = 1
   oWeb.SetInfo
   Set oWeb = Nothing

Open notepad paste the code above and save with vbs extension. Run it by double clicking it. Your IIS now do not give a damn about CRL.

NOT THE BEST OPTION FOR PRODUCTION ENVIRONMENT.

Good enough for demos.

Enjoy.

Web Services Over SSL - Is It Really That Slow Like They Say?

alikl — Wed, 01 Aug 2007 17:35:18 GMT

My answer is "no".

I am working on solution where there is no Windows Active Directory Domain so we cannot utilize our beloved Kerberos and Windows Integrated Authentication saving big on configuration and management while taking advantage of increased security it offers.

Other technique that we thought that could give us a lots of benefits in terms of strong authentication, transport level protection, and interoperability was using Client Certificates.

Here is the scenario.

ASP.NET web page calls on ASP.NET Web Service on separate machine. Think of scenario where Internet facing ASP.NET web site calls on Web Service deployed in internal zone:

The other scenario would be so called B2B scenario where intranet facing ASP.NET web site calls on Web Service over the Internet:

Another scenario would be calling Java Web Service.

Not that friendly for Windows Integrated Authentication.

The question we asked ourselves was - will it be fast enough? The following post by my colleague Eddie - Fast and Secured: Performance Impact of SSL gave us a lots of hope. But it discussed SSL between Web Browser and the Web Server. Web Browser (IE) has nice feature of caching SSL state so what depicted below happens less thus improving performance (think of OLEDB Connection pooling and you got the idea):

Well, our beloved Internet Explorer does a great job, what about .Net?

After some research we happily discovered the following:

http://msdn2.microsoft.com/en-us/library/system.net.httpwebrequest.aspx

Note:

The Framework caches SSL sessions as they are created and attempts to reuse a cached session for a new request, if possible. When attempting to reuse an SSL session, the Framework uses the first element of ClientCertificates (if there is one), or tries to reuse an anonymous sessions if ClientCertificates is empty.

That was encouraging and I decided I need to see it my eyes, so I set sample code and deployed to my lab domain. I also have used diagnostics technique described in Use Sysinternals DebugView To Diagnose The Application. When I fired up DebugView this is what I saw:

Each pair of records reflects on single Web page access telling me how many milliseconds was spent on each action to complete during the page processing.

Notice first two records - one for Web Service proxy creating and adding certificate to it:

//START STOPWATCH

Stopwatch stopwatch = new Stopwatch();

stopwatch.Start();

//GET HOLD ON CERT

X509Store store = new X509Store(StoreName.My, StoreLocation.CurrentUser);

store.Open(OpenFlags.ReadOnly);

X509Certificate2Collection certs = store.Certificates.Find(X509FindType.FindBySubjectName, "w3w1", true);

X509Certificate2 cert = certs[0];

//CREATE WEB SERVICE PROXY

Service1 proxy = new Service1();

proxy.ClientCertificates.Add(cert);

Debug.WriteLine("Web Service Proxy Created: " + stopwatch.ElapsedMilliseconds.ToString());

and the second one is actual Web Service call:

// CALL ON WEB SERVICE

string result = proxy.HelloWorld(cert.Subject);

Debug.WriteLine("Web Service Call Completed: " + stopwatch.ElapsedMilliseconds.ToString());

Notice that all subsequent calls are pretty fast. It should prove that SSL session caching is in place also with .Net as promised. Cool.

Conclusion

While these numbers have been taken on lab environment for super simple scenario it can serve as talking point when considering applying SSL to protect your sensitive data to its way to downstream servers. Also client certificate authentication should be considered as a strongest authentication available today when Kerberos is not available.

Man-In-The-Middle-Attack: Protecting Http Traffic With SSL Might Be Not Enough - Consider Protecting SQL Traffic Too

alikl — Mon, 30 Jul 2007 17:06:37 GMT

Think configuring SSL for your web site is enough to protect against prying eyes?

Here is how the sensitive data can be exposed by sniffing your SQL traffic.

Consider common simple 3 tier web architecture for data driven web site. The Web and DB server (it really does not matter what vendor it is) are physical separate machines as depicted below:

It is common practice to apply SSL for web traffic so that bad guy won't be able to easily see what runs on the wire. It is less common practice to apply network protection to the traffic that goes between the Web and DB server. In some cases it is possible to pretty easily sniff that traffic, or as they call it to launch Man-In-The-Middle-Attack.

One of techniques to sniff the traffic is first launch ARP poisoning attack and then just fire up any network protocol analyzer. For ARP poisoning the tools can be found freely here - www.insecure.org - and I used Cain and Abel. Linux friends will probably use Ettercap (what a nice "how to" they have ) and I used new shiny and freely available Microsoft Network Monitor for protocol analyzing.

Here is how my web site looks (I just returned some sample data from AdventureWorks database over SSL):

And here is what I fished in my net using MS Netmon while sniffing the traffic between the Web and DB server:

And here is how it looks after pasting it into my favorite tool - Notepad:

Go back to the first image and compare.

This is just email address, now think about a bit more sensitive data like your bank account balance or credentials. Is it secure on its way from the Web server to DB server?

Conclusion and Countermeasures

To build more secure system it is not enough to apply SSL or IPSEC here and there - it is about the holistic approach through the whole development process. Specifically for the case at hand consider the following resources when designing, building, and deploying your next system:

This post was inspired after watching presentation by Marcus Murray he has given during Microsoft Tech·Ed 2007 in Orlando - Why I Can Hack Your Network in a Day! [A live demonstration of techniques and tools used by hackers to compromise your network].

I am curious (or should I say I am freaked out) who else was inspired to do what after watching it?.....

It is scary presentation - do not watch it before you go to sleep. Especially if you are responsible for your org's IT in some way...

WCF Security In Intranet Scenario : Thoughts On Cons and Pros

alikl — Thu, 26 Jul 2007 15:54:13 GMT

I am researching on best practices with WCF security in terms of "YOU SHOUD" vs "YOU CAN". While it is great to have "How to" stuff I am also interested in "Why" angle. I have common simple scenario of WinForms client consuming WCF service inside corp walls with Active Directory deployed. Here is what I came up with while looking at Hosting Services and Common Security Scenarios.

I have built simple Hello World WCF service that accepts Person object/message and echoes back "Hello, " + Person.FirstName. I was using DataContract serialization. I think it really does not matter what it does in terms of biz logic. What really matters for me is how to host it, what binding to apply, and what security settings to configure.

For my intranet scenario "Message Security with a Windows Client without Credential Negotiation" would fit most I think. It utilizes Active Directory for authentication and message protection in transit as well saving me from messing with certs, transport level protection SSL, IPSEC style and plain vanilla UserName and Passwords. I think it is great and apparently it is a pro part from security stand.

Since it uses wsHttpBinding binding I thought it would be natural choice to host it in IIS rather self hosted as it shows up in the example. Here I earn what IIS has to offer vs what I can write in C#...

I also implemented scenario where I used basicHttpBinding with security set to None...

I fired up Fiddler2 to see what runs on the wire in both cases. The difference was pretty notable in terms of response time and payload size. Of course all these goodies that come from using Kerberos from my Active Directory friend - authentication and message protection - have their cost in terms of performance. I guess it is cons part from performance stand. I presume the time is spent for negotiation with Domain Controller and for cryptographic operations - encryption and signing - for message protection. The other cons part would be message size that naturally inflates when encrypted.

All my experiments are done with my demo lab domain that is totally based on VPC 2007 so the numbers should be taken with caution but I presume that it can give some food for thoughts. Consider simple message as follows:

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">

<s:Body>

<a:FirstName>Alik</a:FirstName>

<a:LastName>Levin</a:LastName>

</person>

</SayHello>

</s:Body>

</s:Envelope>

Here is what I captured using Fiddler:

	basicHttpBinding	wsHttpBinding
Message Encryption		V
Message Signing		V
Bites sent	584	8,014
Bites received	420	5,286
Time to last byte	20 ms	231 ms

Conclusion

When designing my next WCF solution for intranet scenario I will sure try to utilize current IT investment like AD and IIS to provide first class security and hosting services while saving on maintenance costs. On other hand I must take into account performance part, this consideration should reflect on message size - for example, think twice when tempted to use DataSet as DTO. It should also reflect on hosting options - using IIS 6.0 allows me to utilize Http traffic only.

Here is my take on intranet scenario where Windows 2003 Active Directory and IIS 6.0 deployed:

Utilize AD for message security and authentication.
Host WCF in IIS 6.0. If I had Windows Server 2008 with IIS7 I'd go for WAS hosting for sure since it allows binding other than Http and I presume this should improve performance.
Do not use DataSets for DTO rather carefully design custom most lightweight DTO (yes, invest as much time as needed for that one).
Use DataContract serialization with opt-in approach (allows fine tune what gets to the wires and what is not).

You have better suggestion for me around intranet scenario? If so, do not hesitate and drop me a line.

Here is even more food for thoughts A Performance Comparison of Windows Communication Foundation (WCF) with Existing Distributed Communication Technologies

Bon appetite.

Ubuntu And Apache Web Server Join My Lab Network

alikl — Mon, 09 Jul 2007 22:44:00 GMT

I have my lab network, my playground Active Directory Domain( more on it here - How I Setup Lab Domain Using VPC 2007 ).

I have customers who explore on interoperability between .Net applications and Java application that run on Windows/Linux. They seek for help.

To get started I decided that I need to have Linux machine on my lab network with Http server. From quick research on the Internet I understood that Ubuntu 6.06 would be the easiest for me to install on my VPC 2007. I followed the instructions from Installing Ubuntu on VirtualPC Step by Step.

After the Linux machine was up and running I needed to install apache web server. To do so I ran the following command line:

sudo apt-get install apache2

but it failed since some packages were not there.

I consulted with people who are in the know and what was done is the following:

Add "universe" to resource.list file. It lets Ubuntu get all repositories of updates.
Run command sudo apt-get update. Get all latest updates lists.
Run sudo apt-get install apache2 command again to install apache

The apache is now installed. Here is the look at it from Ubuntu machine on my lab network (notice "localhost" in Firefox's address bar):

Here is the look at it from Windows machine on my lab network:

Next would be starting to build .Net and Java applications and make them interop.

T-Shooting Kerberos

alikl — Thu, 05 Jul 2007 00:16:00 GMT

I was delivering "Authentication Explained" session for Security User Group.

First off - thanks for attending the session!

The session was based on "Authentication Explained" workshop. During the session I was demoing the following topics:

During the preparation for the session I struggled a bit with our friend Kerberos and I stumbled on very nice resource on technet:

The articles are very comprehensive and detailed - very handy.

Happy Kerberos t-shooting (troubleshooting)

Enjoy

IIS 7 Configuration File - applicationHost.config - Password Management

alikl — Tue, 24 Apr 2007 22:52:57 GMT

From my learning of IIS7 I understand that IIS7's metabase is actually XML configuration file very familiar to me and similar to ASP.NET's web.config. It is called applicationHost.config and sits in C:\Windows\System32\inetsrv\config

My first interest was to see how it manages passwords when specifying specific accounts for application pool.

I created demo application pool called xxxx, then I created demo account and specified my application pool to run under it. Then I navigated to C:\Windows\System32\inetsrv\config\applicationHost.config and opened it Notepad. I needed to run Notepad as administrator since UAC prevents from opening it directly into Notepad (I cannot say it about Visual Studio, which opens it gladly without running as Administrator). The following picture depicts what I found there:

Seems like the password encrypted using RSA and the cipher was stored in the config file, not the clear text password.

Very cool.

Where is the key? Digging deeper...

I learned it from:

http://www.iis.net/default.aspx?tabid=2&subtabid=25&i=992&p=3

http://www.iis.net/default.aspx?tabid=2&subtabid=25&i=992&p=4

Enjoy

IIS 6.0 Was True Love, New Romance Is About To Begin - IIS 7

alikl — Sun, 08 Apr 2007 00:01:41 GMT

I just could not hold it back - it is midnight and I am watching Richard Turner's screencast - New Screencast: How to configure IIS7 for Windows CardSpace sites

It was humiliatingly :) easy to set up test server cert, so I've done it, here is the prove:

Next he talks about how to configure it for CardSpace - I stopped watching since I know if I continue I may end up watching the sun set together with my laptop.

Richard, good job!!

Good night.