Adam Fazio

Hyper-V VLANs part II

adamfazio — Wed, 24 Jun 2009 04:03:00 GMT

In my first post on Hyper-V VLANs, I talked about the most common scenario for VLANs with Hyper-V, which is using VLAD IDs on the Virtual Machines or the Hyper-V Virtual Switch. Now that I’ve been around the block a few times working with enterprise customers on some more sophisticated scenarios, I think it’s time to touch on other uses for VLANs on Hyper-V Hosts.

To review, there are 3 places to set a VLAN ID: on the physical NIC, on the Virtual Switch (Host), on the Virtual Machine (Guest). Each Virtual Switch and Virtual Machine may only have one VLAN ID assigned, and physical NICs may have multiple.

Use cases

So, first of all, why do we care about this? Why exactly would we want to make life even more complicated by adding more VLANs at the Host level?

1) Storage Network Segregation. A good example of this is using iSCSI via on-board or add-in NICs. Windows Server iSCSI requirements state: “A physically separate, dedicated storage network” (yes, that means switches dedicated only to iSCSI traffic). However, many datacenters have very expensive, large, beefy core switches with plenty of available capacity they want to take advantage of. VLANs provide a way to isolate that storage traffic without a physically separate storage network.

2) Security. Another use may be that you have traffic on one or more NICs that you want to isolate from the rest of the network for security reasons. In other words, you want to ensure that certain traffic never ends up on an unintended end-point.

3) Performance. In a tradition sense, a VLAN is a logical broadcast domain. Therefore, you can create isolated network broadcast domains without the need for additional routers or devices with VLANs.

4) Other. Just FYI, the original main benefits of VLANs are things like spanning a LAN segment across multiple physical locations, etc.

Anyway, back to our Hyper-V Host… a typical Hyper-V Host using iSCSI storage in a Failover Cluster will need 5 or more gigE NICs:

1 NIC for Host management
1 NIC for Cluster Communications on a private network
1 NIC or more for Virtual Networks (Virtual Machine traffic)
2 NICs or more for iSCSI w/ MPIO

However, in our example we’re using Blades which are constrained to 4 NICs only. In this case we can combine the Host Management & Cluster Communications networks onto one NIC, use the Virtual Machine NIC as a backup for Cluster Communications, and dedicate 2 NICs to iSCSI. In the below pic, NIC#2 needs to be connected to a switch port configured for Trunk Mode in order to allow Virtual Machines to also use VLAN IDs.

We’re also using HP’s Network Configuration Utility to assign VLANs and other advanced functions.

A very good plan is to include the VLAN ID in your NIC naming scheme so you can correctly identify it in different utilities, such as the Hyper-V Virtual Network Manager:

Q & A

Q: Woah there buddy, you’re allowing Cluster Communications on the Virtual Machine Network (NIC#2). Isn’t that against best-practice?

A: Not really, it’s making the best out of the situation. In Failover Cluster manager we are configuring NIC#1 for “Allow Cluster Communications and Client Connections”. In NIC#2 we’re allowing only Cluster Communications. This provides redundant paths for Cluster Communications while keeping NIC#2’s bandwidth nearly solely dedicated to VM traffic.

Q: What if I want to Team the iSCSI NICs?

A: Don’t do that! It’s not supported. Instead use MPIO which will have the same net-affect.

Q: What if I want to have multiple Virtual Networks and Team them for greater performance and availability of my Virtual Machines?

A: Go for it using a vendor-supported Network Teaming Solution. HP, Intel & Broadcom all have Teaming solutions which work with Hyper-V. Make sure to obtain the very latest NIC drivers and Teaming software and make sure to check with those vendors for specific configuration requirements, such as Using HP ProLiant Network Teaming Software with Microsoft® Windows® Server 2008 Hyper-V which is the first paper I’ve seen which doesn’t tell you how to install and use the software, but rather how to uninstall it! This is because Hyper-V & HP Teaming must be installed in a specific order.

Q: What if I want multiple VLAN IDs on a NIC?

A: Please do! Again, use the vendor-provided software do to so and make sure your switch port is configured for Trunk Mode.

Special thanks to Anthony Ramirez at Fresno Unified School District.

Microsoft Server Virtualization Resources

adamfazio — Fri, 27 Mar 2009 00:29:00 GMT

Here's my obscenely vast list of resources that I actually do reference and send to customers on a very regular basis. Hopefully it's at least catagorized in a reasonable enough manner to make it useful to you!

Assessment / Downloads / Tools
Microsoft Virtualization ROI Tool (calculate the total return on investment)
Microsoft Hyper-Green Tool (estimate power, cooling, and CO2 savings)
Windows Server 2008 Hyper-V Trials
System Center Virtual Machine Manager 2008 Eval
Microsoft Assessment and Planning Solution Accelerator
Linux Integration Components for Hyper-V
VMC to Hyper-V Import Tool Available
Necessary Updates / Hotfixes
Comprehensive List of Hyper-V Updates
Recommended hotfixes for Windows Server 2008-based server clusters
Recommended hotfixes for System Center Virtual Machine Manager 2008
Updates / Hotfixes for Specific Issues
Why can't I start my VM when there is plenty of free memory?
Availability of a Volume Shadow Copy Service (VSS) update rollup package for Windows Server 2003 to resolve some VSS snapshot issues
Prerequisites and Known Issues with Hyper-V Protection (DPM SP1)
Troubleshooting Hyper-V

Learn	Do
Web Sites	Installation
Virtualization TechCenter on Microsoft TechNet	Step-by-Step Guide to Getting Started with Hyper-V
Microsoft.com/Virtualization - Resources	Install the Hyper-V Role on a Server Core Installation of Windows Server 2008
Microsoft.com/WindowsServer2008 - Virtualization and Server Consolidation	Server Core Installation Option of Windows Server 2008 Step-By-Step Guide
TechNet Library - Hyper-V	Command-Line Syntax Key
TechNet Library - Windows Server 2008	Netsh Commands for Wired Local Area Network (LAN)
System Center Virtual Machine Manager 2008	Server Core Configurator
TechNet - System Center Virtual Machine Manager 2008	Server Core Blog
Failover Clustering Resource Center	Microsoft iSCSI Software Initiator User's Guide 2.x
Posts / Articles / Whitepapers	Clustering & HA
Hyper-V Planning and Deployment Guide	Step-by-Step Guide for Testing Hyper-V and Failover Clustering
Infrastructure Planning and Design Guide	Checklist: Failover Cluster in Which All Nodes Run Hyper-V
Performance Tuning Guidelines for Windows Server 2008	Configuring a Two-Node File Server Failover Cluster
Microsoft System Center Virtual Machine Manager 2008 Reviewer’s Guide	Failover Cluster Requirements
MSDN and TechNet Powered by Hyper-V	How are virtual machines (VMs) failed over between nodes in a Windows Server 2008 failover cluster?
MS IT Showcase: Identifying Server Candidates for Virtualization	Failover behavior on clusters or three or more nodes
Virtualization Strategy Provides Tools, Processes, and Compliance Capabilities to Enhance Business Support and Drive Adoption	iSCSI Cluster Support: Frequently Asked Questions
TechNet Magazine October 2008: Virtualization	HYPER-V QUICK MIGRATION & VMWARE LIVE MIGRATION PART 1...
	SNW Demo: Windows Server 2008 Core, Hyper-V and Failover Clustering - with screenshots
Blogs	Storage & Virtual Disks
Virtualization Team Blog	Storage options for Windows Server 2008 Hyper-V
Windows Server Team Blog	More on Storage Options for Windows Server 2008 Hyper-V
Virtualization Feed	Hyper-V Storage Analysis
HyperVoria	Boot from SCSI in Virtual Server vs. Boot from IDE in Windows Server virtualization
Cheng - SCVMM Program Manager	NetApp and Microsoft Virtualization Storage Best Practices
Ben Armstrong - virtual_pc_guy	MPIO - NetApp DSM vs. Microsoft's generic
Jose Barreto - Storage Architect	Storage devices that are certified for Windows Server 2008
Rakeshm - SCVMM program manager	Backups
Robert Larson - MCS Architect	Backup and Disaster Recovery for Server Virtualization
Mike Sterling - Hyper-V PM	How to back up Hyper-V virtual machines from the parent partition on a Windows Server 2008-based computer by using Windows Server Backup
Communities	Invoking diskshadow to back up a Virtual Machine from a Hyper-V Host
TechNet Forums - Virtualization / Hyper-V	DPM SP1 prerequisites and Known Issues with Hyper-V Protection
TechNet Forums - Virtual Machine Manager	SCDPM Product Documentation
Events	DPM 2007 Planning Guide
getVIRTUALnow - Register for a Microsoft Virtualization Launch event	DPM 2007 Operations Guide
Virtualization Congress 2008	VM Snapshots
Microsoft TechEd	Virtual Machine Snapshotting under Hyper-V
Microsoft Management Summit	Virtual Machine Snapshots with Hyper-V
Webcasts/Podcasts	Getting Undo Functionality with Hyper-V Snapshots
TechNet Edge - Virtualization	Hyper-V and Snapshots Part 1 and Part 2
Transitioning to Windows Virtualization	Dev
TechNet Webcast: High Availability with Hyper-V (Level 300)	PowerShell Management Library for Hyper-V
TechNet Webcast: A 360 View Inside the Virtual World (Level 200)	Creating an awesome developer sandbox using Hyper-V (Part 1)
TechNet Webcast: Windows Server Virtualization Under the Hood (Level 200)	Workloads & Applications
TechNet Webcast: Assess Your Server IT Infrastructure for Windows Server 2008 Migration and Virtualization (Level 300)	Microsoft server software and supported virtualization environments
TechNet Webcast: Virtualization of Production Workloads Including Active Directory, SQL Server, and Others (Level 200)	Support policy for Microsoft SQL Server products that are running in a hardware virtualization environment
TechNet Webcast: Using Virtual Machine Manager and Windows PowerShell to Deploy HP Windows Server 2008 Academy Labs (Level 300)	Microsoft Support Policies and Recommendations for Exchange Servers in Hardware Virtualization Environments
Windows Server 2008 Hyper-V R2	Running Domain Controllers in Hyper-V
Windows Server 2008 R2 Overview Feature Overview	Active Directory in Hyper-V environments Part 1 , Part 2, Part 3
Hyper-V: Step-by-Step Guide to Using Live Migration in Windows Server 2008 R2	Microsoft Support Policies and Recommendations for Exchange Servers in Hardware Virtualization Environments
Windows Server 2008 R2 & Microsoft Hyper-V Server 2008 R2 - Hyper-V Live Migration Overview & Architecture	Exchange 2007: To Hyper-V or not to Hyper-V?
	Running SQL Server 2008 in Hyper-V Environment
	Oracle Database 10g Performance on Windows Server 2008 Hyper-V Virtual Machine
	Virtualizing PeopleSoft with Windows Server 2008 Hyper-V
	System Center Integration
	Configuring Operations Manager Integration with VMM 2008
	Enabling PRO in VMM 2008
	Configuring SCVMM 2008's PRO Feature with Ops Manager
	Operations Manager 2007 Accounts

Understanding Hyper-V VLANs

adamfazio — Sat, 15 Nov 2008 01:02:00 GMT

(please note part II to this post which covers other advanced scenarios)

If you are interested in this feature, it’s assumed you are familiar with the basic concept of a VLAN. Therefore I’ll focus just on using VLAN IDs with Hyper-V here. A VLAN ID is the integer which uniquely identifies a node as belonging to a particular VLAN. As per the 802.1Q specification, the VLAN ID itself is encapsulated within the Ethernet frame, which is how multiple VMs using the same physical NIC can communication on different VLANs simultaneously.

Firstly, you need physical NICs which support VLAN tagging and you need to enable the feature. However, you should generally not set the VLAN ID at the physical NIC, it should be set on either the Virtual Switch or the individual Virtual Machine’s configuration. The VLAN ID on the Virtual Switch is what the Host or Parent Partition uses. The VLAN ID setting on the individual Virtual Machine’s settings is what each VM will use.

VLAN ID setting at the Host’s Virtual Switch

VLAN ID setting at the Virtual Machine

When creating an External network in Hyper-V, a virtual network switch is created and bound to the selected physical adapter. A new virtual network adapter is created in the parent partition and connected to the virtual network switch. Child partitions can be bound to the virtual network switch by using virtual network adapters. The diagram below illustrates the architecture.

In addition to the above scenarios, Hyper-V also supports the use of VLANs and VLAN IDs with the virtual network switch and virtual network adapters. Hyper-V leverages 802.1q VLAN trunking to achieve this objective. To utilize this functionality, a virtual network switch must be created on the host and bound to a physical network adapter that supports 802.1q VLAN tagging. VLAN IDs are configured in two places:

· The virtual network switch itself which sets the VLAN ID the parent partition’s virtual network adapter will use

· The virtual network adapter of each guest which will sets the VLAN ID the guest will use

The diagram below illustrates an example of using a single physical NIC in the host which is connected to an 802.1q trunk on the physical network carrying three VLANs (5, 10, 20). The design objective in this example are:

· An 802.1q trunk carrying 3 VLANs (5, 10, 20) is connected to a physical adapter in the host

· A single virtual switch is created and bound to the physical adapter

· The VLAN ID of the virtual switch is configured to 5 which would allow the virtual NIC in the parent to communicate on VLAN 5

· The VLAN ID of the virtual NIC in Child Partition #1 is set to 10 allowing it to communicate on VLAN 10

· The VLAN ID of the virtual NIC in Child Partition #2 is set to 20 allowing it to communicate on VLAN 20

The expected behavior is that there is a single virtual switch, the parent and two children can only talk on their respective VLANs, and they can’t talk to eachother.

December 2008 issue of TechNet Magazine dedicated to ... Mac & Linux?

adamfazio — Tue, 11 Nov 2008 21:46:00 GMT

Ok, at this point I'm honestly impressed with the dedication Microsoft has shown for interop in the past couple years. Ever wonder how to get Macs working on your Windows network? Ever want to have your *nix boxes authenticating with Active Directory? This is for you.

The December 2008 issue of TechNet
Magazine is now available online. And
it's filled with information about interop.

Find out how to integrate Linux clients
with Active Directory, Manage Macs in
a Windows environment, and more.

TechNet Magazine, October issue is 100% Virtualization

adamfazio — Wed, 01 Oct 2008 20:12:00 GMT

Since it's October 1st, I thought this may be a timely post. Not-so-coincidentally, I'm one of the authors in this issue. :-) (please forgive the shameless self-promotion)

Virtualization: An Introduction to Hyper-V in Windows Server 2008

The introduction of Hyper-V makes virtualization an even more compelling solution for IT environments. Get an overview of today’s virtualization market and see how Hyper-V improves the manageability, reliability, and security of virtualization Rajiv Arunkundram

Virtualization: Manage Your Virtual Environments with VMM 2008

System Center Virtual Machine Manager provides a consolidated interface for managing your virtual infrastructure. The latest version adds support for Windows Server 2008 Hyper-V, as well as for VMware virtual machines. Explore the new features and get an overview of using VMM to centralize your management tasks. Edwin Yuen

Virtualization: Getting Started with Microsoft Application Virtualization

Microsoft Application Virtualization (App-V) allows you to deliver virtualized desktops to client systems throughout your organization. This simplifies system management and liberates employees from their desktops. Take a close look at how App-V works and discover how you can deploy it in your organization. Anthony Kinney

Virtualization: Achieving High Availability for Hyper-V

Consolidating servers onto fewer physical machines has many advantages, but it is extremely important that you plan for your systems to be highly available. Here’s a guide to using Windows Server 2008 Failover Clustering to bring high availability to your Hyper-V virtual machines. Steven Ekren

Virtualization: Backup and Disaster Recovery for Server Virtualization

Virtualization brings significant changes to disaster recovery. Here’s an introduction to how the Microsoft virtualization platform factors into your disaster recovery plan, as well as a deeper look into backup and restore options and considerations for Windows Server 2008 Hyper-V. Adam Fazio

Virtualization: Essential Tools for Planning Your Virtual Infrastructure

Is your infrastructure ready for virtualization? The Microsoft Assessment and Planning Toolkit, a network-wide infrastructure assessment tool, can help you better understand your IT infrastructure and determine whether your systems are ready for upgrade or migration to a variety of technologies, including virtualization. Jay Sauls and Baldwin Ng

Columns

From the Editor: It’s a Virtual(ized) World Joshua Hoffman
Letters: Readers Speak Out	Toolbox: New Products for IT Pros Greg Steen	SQL Q&A: Large Transaction Logs, When to Use Repair, and More In this installment, Paul Randal answers questions about backing up and restoring, looks at the differences between log shipping and database mirroring, and explains why the Repair function should only be used as a last resort. Paul S. Randal
Utility Spotlight: Offline Virtual Machine Servicing Tool Virtual machines that are stored offline don’t automatically receive the necessary updates to keep them safe and compliant. This, in turn, can pose a risk to your entire IT environment. Find out how the free Offline Virtual Machine Servicing Tool lets you automate the process of updating virtual machines. Peter Skjøtt Larsen and Suveen Kumar Reddy Vuppala	Windows PowerShell: The Power of Profiles Ever wish Windows PowerShell would launch with a work environment tailored to your needs? Don Jones demonstrates how you can use profiles to customize the Windows PowerShell shell. Don Jones	Hey, Scripting Guy!: Famous Last Words The Scripting Guys discuss Socrates and revisit the topic of querying an XML file . This time, however, the XML file is structured so that rather than using child nodes, additional property values are configured as attributes. The Microsoft Scripting Guys
The Desktop Files: Customizing Windows Deployment Services Wes Miller delves into Windows Deployment Services, showing you how you can customize and configure WDS to meet the needs of your organization. Wes Miller	Security Watch: Revisiting the 10 Immutable Laws of Security, Part 1 It's been about 8 years since "The 10 Immutable Laws of Security" were first published, and a lot has changed since then. This month, Jesper Johansson kicks off a three-part series in which he analyzes the laws from today's perspective to see if they still hold true. Jesper M. Johansson	Field Notes: High-Capacity Color Bar Code Gavin Jancke has developed a new bar code, using colors and triangles, that has much higher data capacity than traditional black and white bar codes. Take a closer look at these high-capacity color barcodes. Gavin Jancke
Windows Confidential: Work Harder, Not Smarter Raymond Chen looks at the skewed relationship bugs have to errors, and explains why it's important that programmers suffer as well as give results. Raymond Chen

Hyper-V’s readiness for primetime

adamfazio — Tue, 05 Aug 2008 00:04:00 GMT

I was browsing the blogoshpere today and ran across some posts that erked me. So I emailed the below to Bink over at HyperVoria and he posted it, and I figured I may as well post it too.

The Virtualization RDP (Microsoft Rapid Deployment Program) has over 100 customers running Hyper-V & SCVMM 2008 in production. Some of these are very large enterprises. The evidence produced from these deployments will be available at the Virtualization Launch event. https://www.getvirtualnow.com/main.aspx

A couple early public case studies

One question that bubbles up around any product’s v.1 RTM is “is it (Hyper-V) ready for the enterprise”? In addition to the case studies produced by RDP program, here are just a couple examples if you’re ever faced with this question, or if you’re just curious about how far Hyper-V can be pushed.

· Collectively, the TechNet and MSDN websites handle over 4 million hits per month and have been running 100% on Hyper-V since April 2008. Read about the implementation at Microsoft.com Operations Virtualizes MSDN and TechNet on Hyper-V.

· Microsoft.com is one of the heaviest traffic websites on the internet, at 15,000 requests per second, 1.2 billion page views per month, and 280M worldwide unique users per month as well as supporting ~5000 content contributors from within the company. On June 5th the Operations Team turned up a full sixteen VM cluster hosting www.microsoft.com on Hyper-V. Read more at Microsoft.com Powered by Hyper-V

Hey Exchange 2007, move the mailboxes, email me the log when you're done!

adamfazio — Sat, 27 Oct 2007 21:23:00 GMT

Here's a simple script for moving mailboxes, I talked about how to schedule an Exchange 2007 script in the last post. Here's what it does:

Reads user aliases from a text file in the local directory (MoveMbx1.txt)
Moves the mailboxes to a specified Server & Mailbox Store using 10 threads and suppressing the annoying "are you sure?" prompt
Puts today date in a variable in format YYYYMMDD
Get Exchange Migration Logs that have the above date in their name
Emails the logs in plain text

Yes, we could certainly do more sophisticated things, such as find the smallest Mailbox Store and move users there, or format the XML logs better, but hey, I ran out of time. :-) Just copy the below to a .ps1 file.

#############################
#                           #
# Exchange 2007 Migration #
# Move-mailbox script      #
#  Date & Author            #
#                         #
#############################

# NOTE: to run PowerShell scripts, you need to set the PS execution poloicy "set-executionpolicy unrestricted"

# load the list of users - this must be 1 user alias per line
# the user list file must be in the same directory as this script

$users = get-content MoveMbx1.txt

# move em to a specific target database using 10 threads

$users | move-mailbox -TargetDatabase "MailboxServer\StorageGroup01\MailboxStore01" -MaxThreads 10 -confirm:$false

# put today's date in a variable

$date=get-date -uformat "%Y%m%d"

# read today's migration log and store it as $EMailBody

$EmailBody = get-content "C:\Program Files\Microsoft\Exchange Server\Logging\MigrationLogs\*$date*.xml"

# when done, send us an email with the log text

$SmtpClient = new-object system.net.mail.smtpClient
$MailMessage = New-Object system.net.mail.mailmessage
$SmtpClient.Host = "smtpserver.domain.com"
$mailmessage.from = ("<emailaddress>@domain.com")
$mailmessage.To.add("<emailaddress>@domain.com")
$mailmessage.Subject = “MoveMbx1.ps1 script has completed”
$mailmessage.Body = $EmailBody
$smtpclient.Send($mailmessage)

Exchange 2007 Script: how to run as a scheduled task?

adamfazio — Fri, 26 Oct 2007 04:29:00 GMT

Because this information was so unbelievably hard to track down I thought I'd post this. This is so strange to me, I mean, isn't the idea of most scripting to run jobs unattended at specific times? Whew!

Basically, the process is you have to call powershell.exe, then load the Exchange extensions, then pass it the command that calls the script. You're supposed to be able to use -noexit so you can watch it run, and call exshell.psc1 directly via the run line -psconsolefile "C:\Program Files\Microsoft\Exchange Server\Bin\exshell.psc1", but I could not get it work using either of these. SO, we run the task starting in the Exchange bin directory as below.

Run: powershell -psconsolefile exshell.psc1 -command "& {c:\yourscript.ps1}"

Start in: "C:\Program Files\Microsoft\Exchange Server\Bin"

Run as: <account with rights to do whatever is in your script>

Adam’s unofficial Exchange 2003 troubleshooting tips

adamfazio — Sat, 23 Jun 2007 00:45:00 GMT

Adam’s unofficial Exchange 2003 troubleshooting tips

All items are listed in the order in which I usually reference/perform them. This is general guidance based on my 8 years working with Exchange, not prescriptive Microsoft Guidance.

As always, for the fastest, most methodical approach to fixing a problem and finding the cause, call Premier Support! Those guys do nothing but troubleshoot all day & night, we’ll never be as good as them!

Exchange troubleshooting

· Always consider the most basic things first - networking, name-resolution, AD, services running, databases mounted, etc.

· Turn on Diagnostic logging for the suspect component

· Look in the Event Log (Exchange uses the Application Log) (lookup events in the KB or eventide.net, search the Exchange Newsgroups or your favorite forum)

· Setup a Perfmon to capture historical & real-time performance data (see Troubleshooting Microsoft Exchange Server Performance doc)

· Cmd-line: netstat -na to view all active connections to/from the server

· Look at mail-flow using Message Tracking Center in ESM - for example look at the last hour of message flow, is it extremely higher than usual?

· Look at active POP/IMAP/MAPI sessions via Virtual Servers in ESM & EXMON

· Most client performance problems are Exchange being Disk I/O bound or Memory bound. CPU & Network are very very rarely the cause.

· Use ProcessExplorer by sysinternals to see a particular process and/or thread’s CPU, disk I/O, network sessions, etc.

Documentation (there are dozens more, but these I use most often)

Microsoft Exchange Server 2003 Technical Reference Guide

http://www.microsoft.com/downloads/details.aspx?FamilyID=3768246d-c9ed-45d8-bece-a666143cba4e&DisplayLang=en

Troubleshooting Microsoft Exchange Server Performance

http://www.microsoft.com/downloads/details.aspx?FamilyId=8679F6BD-7FF0-41F5-BDD0-C09019409FC0&displaylang=en

Microsoft Exchange Server 2003 Transport and Routing Guide

http://www.microsoft.com/downloads/details.aspx?familyid=C092B7A7-9034-4401-949C-B29D47131622&displaylang=en

Microsoft Exchange Server 2003 High Availability Guide

http://www.microsoft.com/downloads/details.aspx?familyid=FE6A573C-11FF-4CB6-BE2E-9B6F2164C54A&displaylang=en

Microsoft Exchange Server 2003 Performance and Scalability Guide

http://www.microsoft.com/downloads/details.aspx?FamilyID=62fb1297-4c6b-4d84-84cc-060989f2f305&DisplayLang=en

Tools

Tools for Exchange Server 2003

http://technet.microsoft.com/en-us/exchange/2003/bb288491.aspx

(The ones in the above URL I use most often)

NetDiag & DCDiag (already on all your servers) - run from cmd prompt - helps identify the most common problems - DNS, WINS, networking, & AD

User Monitor (EXMON) when users are experiencing RPC disconnects

SMTPDiag -Determine whether SMTP and DNS are configured to reliably deliver mail to an external e-mail address

Exchange Troubleshooting Assistant (EXTRA)

Exchange Best Practices Analyzer (EXBPA)

ExchDump - export configuration information

Process Explorer - very deep process & memory analyzer

http://www.microsoft.com/technet/sysinternals/ProcessesAndThreads/ProcessExplorer.mspx

IIS 6 Resource Kit

http://www.microsoft.com/downloads/details.aspx?familyid=56FC92EE-A71A-4C73-B628-ADE629C89499&displaylang=en

Debugging Tools for Windows 32-bit Version - analyze crash-dumps, etc,

http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx

Monitoring - From the MOM Management Pack for Exchange doc

Table 4.2 Minimum messaging functions to monitor

Test	Details
Server availability	· Server heartbeat. · Required services are running. · Databases are mounted. · MAPI logon check verification is running without errors. · Mail flow verification is running without errors. · No unexpected service termination. · Front End Server Monitoring test is running without errors.
Services running	· Verify that all required services are running on each server. Note that you can configure the list of monitored services for each server. · Generate an alert when a service is not running.
Databases mounted	· Verify that all databases are mounted. · Generate an alert if any database becomes dismounted.
MAPI Logon check	· Verify that the Server Availability Report shows no errors. This test verifies that each store can be accessed by a MAPI client, and implicitly verifies both Exchange and Active Directory functionality.
Log on to the mailbox of a test account	· Verify client to server connectivity, including verification that Exchange is running, the database is mounted, and Active Directory is functioning correctly. · Use this data to compile server availability statistics.
Front-end Server Monitoring	After you edit your registry to enable Front-end server monitoring, the following tests are performed: · Verify that services are running on the front-end server. · Verify that Internet clients can connect, including Outlook Web Access, Outlook Mobile Access, and Exchange ActiveSync (for computers running Exchange Server 2003). · Verify localhost monitoring occurs by default. · Verify that the public URL is resolvable and successfully connects to your front-end servers. · Verify that connectivity through your firewall and/or proxy server is functioning. · Verify that load balancing is occurring.
Mail flow verification	· Verify mail flow between selected servers by sending periodic e-mails to test mailboxes on each server. · Generate an Alert for successive failures. · Record mail delivery latency.
Server Health Monitoring	Scripts and rules are configured by default to monitor key health indicators. These indicators include: · Free Disk Space · Mail Queue Thresholds · Configuration and Security · Performance Thresholds · SMTP Queues
Free disk space	Running out of disk space is a common, preventable source of Exchange failures. This test monitors counter thresholds that you specify for the following performance objects: · All disks · Log disks · SMTP queue disks The Free disk space test is cluster and IFS aware, and uses WMI to collect information. It does not use performance data.
Mail Queues	· Verify that all mail queues (SMTP, MTA, internal mail delivery queues) are processing messages according to your thresholds · Verify that mail is flowing properly · Identify queue length problems that may lead to slow e-mail delivery and identify issues in your infrastructure that require attention · This data is based on performance data and Exchange WMI classes.
Server Configuration and Security Monitoring	· Verify that the IIS Lockdown Tool started. · Verify that Message Tracking Log shares are locked down. · Verify that the URLScan ISAPI filter is installed and running. · Verify that SMTP Virtual Server cannot anonymously relay (spam prevention). · Check for the existence of mailboxes on Front-End Servers. · Determine if SSL should be required. · Verify that the Log Files are being successfully purged after backup. · Verify that the SMTP directories are on a NTFS formatted drive. · Verify that circular logging is disabled for each Storage Group. · Verify that the value of the HeapDeCommitFreeBlock Threshold Registry Key is correct. · Verify that Message Tracking is enabled.
Server performance	· Generate an alert if thresholds for disk response are exceeded, indicating a slow disk. · Generate an alert if the RPC requests queue length exceeds expected thresholds. A consistent high value can indicate that you have a resource bottleneck. · Monitors the average RPC latency of all RPC requests submitted to the server. · Monitors the Outlook Mobile Access Latency response time. Server performance issues quickly become user response time issues. You can quickly solve these problems if you monitor the correct objects and act upon the issues that MOM brings to your attention.
Database checkpoint depth and memory usage	An alert is generated by default if any of the following counters exceed the identified threshold: · Disk Read Latencies: 50 msec · Disk Write Latencies: 50 msec · ESE Log Checkpoint Depth: 800 · Information Store Private Bytes: 1 GB · Information Store Virtual Bytes: 2.9 GB · MSExchangeIS: RPC Requests: 25 · MSExchangeIS: RPC latency: 200 ms · Outlook Mobile Access: Last response time: 60 sec

Uh ... you need to see this

adamfazio — Wed, 06 Jun 2007 20:04:00 GMT

Microsoft Surface - watch the videos, imagine the possibilities!

http://seattlepi.nwsource.com/business/317737_msftdevice30.html

http://www.microsoft.com/surface/

Windows Server 2008 / Longhorn – My Favorites

adamfazio — Wed, 30 May 2007 22:02:00 GMT

I had some training recently that got me excited about the upcoming release of the next-gen Windows Server platform, 2008 aka Longhorn. I thought you might like a no-marketing take on what I think are the most exciting new features and reasons to check it out. No, I don’t sell software (or anything else), I’m just a geek. If you want the full feature list and business value proposition and blah blah blah, go here. http://www.microsoft.com/windowsserver2008/default.mspx

If I wrongly describe or characterize the value of a feature, please feel free to let me know. Also, feel free to ask if there’s anything you want more information about.

Disclaimer: these are my favorite things and my opinions, not Microsoft’s. Also, it’s 5/29 today, beta 3 is the current release. These features are subject to change.

IIS 7

This is as big an upgrade as was IIS 4 to 5. It’s huge. Here are some highlights:

· Bye-bye metabase! At long last, we have a totally portable XML configuration file, take it with you!

o Replication: Put the same configuration on multiple machines

o Synchronization: Keep the configuration in sync

o Portability: Quickly move sites or applications

o Staged Deployment: Stage and rollback configuration changes

o Optionally can install metabase for backward compatibility

· New IIS Manager

· Delegated Administration

o Yes, you can now granularly delegate administration of IIS

· Nice command-line interface & new WMI provider

· PowerShell support

· Powerful, never before seen built-in diagnostics

o Trace a request! See each HTTP request performed, the duration, the authentication method used, etc. This takes hours of debugging down to minutes.

· Totally modular. Don’t need a feature, don’t load the module.

The one and only heartbreaking thing missing for IIS 7 is it’s not an available Server Core role. Bummer. This is because it requires the .Net framework which requires other components, etc.

Server Core

Finally, a no-GUI Windows Server. Nice. It’s basically a minimal server version with only subsystem components needed to support the “roles”. This means less patching, less security vulnerability, lower resource utilization, & better uptime. You manage it via remote admin tools, cmd-line, etc.

· Server Core reduces # of patches by ~60%

· Low surface area server for targeted roles

· Current server roles: DHCP, File, Print, AD, AD LDS (ADAM), Media Services, and DNS

· Optional features: WINS, Failover Clustering, Subsystem for UNIX-based applications, Backup, Multipath IO, Removable Storage Management, Bitlocker Drive Encryption, SNMP, Telnet Client, and QoS

· Command Line interface

Some lacking things to be aware of: no PowerShell support, no .NET framework, no upgrade to full Windows, you must reinstall.

Terminal Services

Admittedly, TS is not my deepest area because typically I’ve always used Citrix. However, this is a whole new animal and can in some cases be used where Citrix was cost-prohibitive. The big win is the ability to use TS with SoftGrid to virtualize applications and get around traditional compatibility conflicts allowing you to fully load each TS. Some features include:

· TS Web Access - web portal for accessing apps

· TS RemoteApp – new TS Client

· TS Gateway - for securely extending TS to the net.

· TS Easy Print – self-explanatory

· TS Session Broker – aka session load balancer

Terminal Services team blog: http://blogs.msdn.com/ts/

Active Directory

· Can run on Server Core

o With Read-Only DC running on Server Core + BitLocker you now have the most secure DC ever

· Read-Only DC

o Replicate only the passwords needed at that site to the branch DC

o Delegate “local admin” on a DC?

§ What? What’s that you say? How can you have a local admin on a DC? Well now you can. Use this with caution as a “local RODC admin” does of course have direct access to the AD database (NTDS.DIT) and therefore should be trusted. However, this prevents the most common reason for AD directory data loss: accidental deletion.

· Multiple Password Policies in same Domain

· Auditing improvements

· Without a reboot, you can now

o Apply DS patches

o Perform offline defragmentation

· Improved Backup/Recovery features via “Windows Server Backup” – replaces NTBackup

Clustering

The goal of cluster improvements is to bring clusters to the masses. There is a lot of FUD around clustering today, and rightly so. Although this is not a complete overhaul, I think there are enough improvement to give it a second look for clustering services you passed up previously.

· Greatly simplified setup (from a 50 page whitepaper down to a 3 step wizard)

· Validate – now if your cluster passes Validate, it’s “on the HCL” (not yet official)

o Configuration Issues

o Networking mistakes

o SP and Hotfix binaries

o Driver mismatches

o Inconsistent Settings

· Geo-clustering, can now span subnets

Windows Server Virtualization

Yes yes, we all know it’s not coming as soon as we would like, but SOON. Probably mid-2008. This is going to revolutionize the datacenter much in the same way VMWare ESX has. However, and unfortunately for VMWare, this will be included with every copy of 2008 at no extra cost and is on-par performance and otherwise. This is big-time.

· Hypervisor virtualization

· Run 32 & 64-bit VMs on same host

· 2, 4, 8 processor VMs

· Large memory support (theoretical 2TB limit in 64-bit Windows)

· Drag n’ drop VM conversion from Microsoft & VMWare VM images

· The features & benefits are countless … get ready

Ok, at any rate, this is a major release server platform. When I say major, I don’t mean “Microsoft is saying it’s major so go buy it” or “look, it’s got a new year tacked onto the name”. I mean in my opinion it’s the best server platform ever offered and really elevates the value and position of Windows in the Enterprise.

Links

Very nice post for resources
http://blogs.technet.com/jeffa36/archive/2007/05/26/windows-server-2008-resources.aspx

Windows Server 2008 Technical Library
http://technet2.microsoft.com/windowsserver/longhorn/en/library/bab0f1a1-54aa-4cef-9164-139e8bcc44751033.mspx?mfr=true

Webcasts on Longhorn
http://www.microsoft.com/events/webcasts/calendar/MonthView.aspx?stdate=5%2f1%2f2007&audience=0&series=0&product=Longhorn&presenter=0&tz=0

Windows Server Longhorn – TechNet Virtual Lab
http://www.microsoft.com/technet/traincert/virtuallab/longhorn.mspx

Know Thyself

adamfazio — Wed, 09 May 2007 09:03:00 GMT

Recently it has become apparent that perhaps the most overlooked aspect of a shop is its own staff’s capabilities. These are the aspects of an organization that are seemingly uncomfortable to discuss, but are arguably the most vital to our own success or failure. The overall capability of an IT shop becomes most apparent when it's attempting an extremely ambitious new endeavor, something maybe the staff is not familiar with or is very large and complex in nature.

The problem becomes that too often we prefer to focus on the technology itself. Even when we do admit that maturity or relative health needs to be looked at, it's still usually the technology or processes. But the fact is only the people have the power to improve these things and therefore the maturity and capability of the people must be constantly evaluated as well. This is generally accepted where I work because in consulting the people are the commodity. They are the item being bought and sold and so it's accepted that their evaluation must be out in the open and encouraged. But it seems in a more conventional environment the open evaluation of people is not a popular thing to do, or is even an insulting or threatening. But I ask you, how is it fair to those who strive for excellence when those who are lacksidazical with their work are openly valued to be of equal worth? How do you encourage all to strive for continual improvement when there is no obvious or visible recognition for performance and dedication? Would it not even encourage mediocrity?

I guess the message I’m trying to send here is to not only evaluate your people behind closed doors, but also to foster positive growth amongst your teams organically. If it’s not possible monetarily due to some outside regulation, I think the statistics have shown that open recognition is actually more important and effective at the end of the day than monetary or career opportunity. I can’t tell you how often I’m asked to change or improve something with technology when it’s so obvious that the people are the thing that need changing.

The reason this should matter to us all is that without inter-team growth and positive recognition we are not evolving our capabilities as well as we could be. Like I said, if you need proof of this just try to take on a project that utilizes your entire team as well as members outside your team. You’ll soon learn who really cares and is committed to personal and professional improvement and who’s just hanging out. My advice to those who are hanging out is to go find something you love and do that; you’ll not only inevitably be better at it, you’ll also be happier and consequently so will those around you.

The Importance of IT Operations

adamfazio — Wed, 29 Nov 2006 23:02:00 GMT

Lest we forget the impact our operations have on our users, and in some cases, the world.

The Business Vs. IT

adamfazio — Tue, 22 Aug 2006 18:41:00 GMT

This article was written based on experiences at a previous company. I think it's such a common scenario, I wanted to get this published. Maybe I'm not the writer I thought I was because no one picked it up, so I'll just publish it here myself! If you think you're too far down the totem pole, it doesn't apply to you, there's nothing you can do, etc., you're wrong. Success is the art of changing.

The Business vs. IT 5/26/2006

I work for a typical high-tech company on a typical IT team. We’re called “Infrastructure Systems Services”. We provision and manage critical datacenter systems that are largely responsible for keeping the business running. I’m a lead on my team, I provide general direction and senior level systems engineering. Sounds pretty cool huh?

My company has evolved over the years, during which time it’s been very successful, has acquired dozens of companies, and now spans numerous offices all over the world. I work at the largest site, where about 30% of the employees are located, along with the major business units (HR, Legal, Finance, etc.).

The jobs on my team are the kinds of jobs most IT people strive to get. Here’s the problem, my team is often highly dissatisfied with their jobs and the business is often dissatisfied with my team. Why is this? Let’s look closer …

As I said, we provide “critical business systems”. So what does that mean? It’s supposed to mean that at some point the business and IT looked at a using technology to meet a business need, it was deemed that the benefit of the need outweighed the cost of the technology, IT agreed it had the resources and know-how to implement and manage the technology, and we moved forward. So, the business is now more efficient and/or offering new services or products via this technology, and IT is happily providing this important service. The company’s customers are happy with the improved service or product, the return on the investment outweighs the cost, and the shareholders and market analysts are also happy. The company’s value increases, and everybody’s happy!

So how did we get from there to where both IT resents their jobs and the business resents IT???

Let me describe another scenario. This is just as hypothetical as the first, but based on actual events.

It’s Tax season and Tom in Finance spends a whole work-week every year doing the tax filings. He also has to update the filing quarterly because it’s a publicly traded company, that takes him about a day. This is Tom’s least favorite part of his job. So, he starts to shop around for software to help him automate this manual process. He finds one, talks to the vendor, buys it, and calls up Joe in IT. Tom knows to call Joe, they’ve worked together before, and Joe is always helpful. Even when Tom has called the Help Desk, they always send him straight to Joe for “server support”, and the vendor said this should run on a server so Tom’s whole team can use it.

Tom: hey Joe, how are you? Listen, I have this really important software that needs to run on a server, can you help me? It’s tax season and I need it running really soon…

Joe: Ok, sure Tom. Does it need it’s own server, or can it run on one you already have here in the datacenter?

Tom: Oh no, they said it needs it’s own server.

Joe: Ok, well, a new server costs about $4,500

Tom: ok Joe, send me the quote and I’ll buy it and have shipped to you

So the server arrives, Joe spends a day installing it, racking it, etc. Because Tom said it’s really important, Joe also sets up regular backups, monitoring, automated patching, antivirus, etc. So what’s the problem here? Joe & Tom are both happy right? Tom saves about 1.5 weeks per year doing taxes, and it only cost him $4,500 But let’s look at the real cost of this server:

Item	On time	Yearly
Server Hardware & OS	4,500.00
Server support		250.00
Tax software	1,000.00
SQL software	1,500.00
SQL maintenance		300.00
Backup agent	300.00
Backup maintenance		60.00
Backup tapes	50.00
Monitoring agent	350.00
Monitoring maintenance		70.00
Antivirus	100.00
Power & Cooling		500.00
Network & misc		100.00
Totals	$7,800.00	$1,280.00

Wait a minute, those numbers can’t be right, can they? What are all those maintenance costs? All enterprise class software includes yearly maintenance at about 20% of the software price, which includes 24/7 phone support, free patches and upgrades, and usually a vendor rep to make sure you’re happy.

Ok, well what about that SQL line? Well, Tom didn’t tell Joe that the Tax software uses a database. Luckily, Joe already has the software and knows how to install and configure it. However, the company must pay for using it.

So, let’s say Tom makes $70k/year. That means 1.5 weeks of his time costs the company $2,019. So over the course of 5 years, it costs $10,095 for the company to pay Tom to manually do the taxes. But with this new technology, the initial purchase plus 5 years of operating costs is $14,200. Fair enough, but over the next few years it will even out, not a big deal right? Wrong. Five years later, the server is outdated and must be upgraded. That’s at least another $4,500 every 5 years, plus the costs of maintaining the Power, Cooling, and Network equipment. Also, the tax software is outdated and must be repurchased because Tom didn’t know to buy support. The company will never break even on this deal. Multiply this situation by dozens and dozens of servers and applications and you’ve got a problem.

But it’s just a numbers problem you say, if the company is very profitable, it can take a loss here and there, right?

Now here’s the real problem: Joe and his team are spending more than half their time managing all these semi-important systems. After meetings, paperwork, phone calls, emails, etc. he’s now got very little time left for the systems the company and IT both decided were absolutely critical to running the business. The quality of those systems is diminishing. Joe can’t keep up with planning for increased capacity, keeping the systems healthy and secure, regular upgrades and maintenance, etc.

Over time the business starts to view Joe and his whole team as ineffective. They just can’t seem to keep up. Their budget has grown out of control and they’re always having to take the systems offline for some reason or another. Not to mention in a few years the current datacenter won’t be adequate, and the company will have to re-invest millions to either build a new one, or expand. Worst of all they’re either late or unable to deliver projects to implement more business-critical systems to increase profitability and customer satisfaction.

So in short, IT believes it’s delivering exactly what the business is asking for, but the business is never happy - and the business now sees IT as a very expensive roadblock to achieving it’s goals.

The real problem here is a disconnect between the business, itself, and IT. The business no longer knows what it’s employees are requesting of IT, and IT can’t differentiate between a critical business system and a nice-to-have, and is overwhelmed with the demands.

How do we fix this problem? Should we replace the IT manager? No, we’ve done that several times. Maybe the team needs more people? No, their budget is already astronomical. Maybe some people like Joe are not very efficient and should be replaced? Maybe they should be outsourced?

People

Processes

Technology

Oh! You knew I was going to through out some industry cliché, didn’t you?

People: The right people must not only have awareness of the situation, they must also care. Who are the right people? That will vary, but typically IT Directors on up to CIO, and on the business side CFOs and the like. Awareness below that level is important, but little change can be influenced.

Process: TCO. If you don’t know what the real costs are , you aren’t doing your job. Yes I mean you. Any person responsible for IT, from the sys admin up to the CIO should know this. Then a decision process should exist for determining the value of a service vs. it’s cost. No changes to the environment should be made without going through these processes. This again involves the business and IT working together.

Technology: Notice this is last on the list? My colleagues may argue “I would virtualize that server and centralize the databases, etc, etc, but you still have to pay for and manage the software, and the impact on the team’s resources is the same or worse. You simply cannot solve a people & process problem with technology. You may lessen the financial impact, but the real problems will be ever-present.

A direct line of communication between IT and the business is critical; someone on both sides who has deep understanding and visibility in to each other’s world. The IT folk should sit in on business discussions and have clear line-of-sight into the direction of the business and it’s customers. The business folk should regularly review the state of the environment, the road ahead, and the TCO. Together, they can then assess the real business value and costs of these needs, wants, and nice-to-haves.

Of course, I’m greatly over-simplifying this problem, and it’s only one among many inter-related complex problems businesses and IT shops face, the point is they must face them together.

Secure your internet-facing IIS 6.0 servers!

adamfazio — Mon, 14 Aug 2006 23:56:00 GMT

Well, of course! Who wouldn’t secure their most exposed servers, right? Well, here’s the situation which I’m very sympathetic to since it was my situation for many years. You’ve heard the slogan “do more with less”? Well that applies to people, too. In a typical mid-sized company an IT Admin may be faced with handling environments ranging from Directory, Email, Web, Database, Monitoring, Network equipment, etc, etc. These folks don’t need dozens of 500 page-long books detailing the inner workings of each technology. They need specific quick & dirty real-world guidance on how to get things done.

As promised, here’s my first attempt at helping the overworked IT Pro.

This article will not be all-inclusive of everything you need to know to be secure and highly available. It targets what I feel are the most beneficial things you can do to your IIS servers to achieve better security and availability. It assumes you’re familiar with basic OS and network security and builds from there. If you think I missed something really important, add a comment, this a community blog after all!

Summary:

· Use Security Configuration Wizard and normal process to create a secure web server baseline

· Locate website content on a non-system partition and audit NTFS permissions

· Regular Internal Security Scans & Audits

· Regular Log Analysis

· IIS 6.0

o Use Host Headers on all sites

o Use URL Scan

o If you have logins, enforce SSL site-wide

o Install only needed OS & IIS components

o Enable only needed Web Service Extensions

o Delete Default Web Site and C:\Inetpub

o Restrict use of Anonymous IIS Account

o Enable all IIS & SMTP extended logging properties

o Use custom error messages

· Web Farm

o Automate Website Content Replication

o Automate IIS Metabase replication

o Use a standard name for all IUSR_servername accounts

· General Web Infrastructure Design Considerations

· References

· Tools

Security Configuration Wizard (SCW)

Used in combination with a standard server build and configuration process, the SCW can produce dramatic results in reducing the attack surface and saving administrative time when compared with manual hardening. Additionally, the templates can be saved and used on other systems, rolled-back & deployed via Active Directory GPOs, (note: roll-back feature not available if deployed via GPO). Developing standard security settings is also needed to complete the security picture, such as Password Policies, NTFS security, etc.

Explanation: Security Configuration Wizard (SCW) is a tool for reducing the attack surface of computers running Microsoft® Windows Server™ 2003 with Service Pack 1 (SP1). It determines the minimum functionality required for a server's role or roles, and disables functionality that is not required. Specifically, SCW helps you author and deploy a security policy that:

· Disables unneeded services.

· Blocks unused ports.

· Allows additional address or security restrictions for ports that are left open.

· Prohibits unnecessary Internet Information Services (IIS) Web extensions, if applicable.

· Reduces protocol exposure to server message block (SMB), LAN Manager, and Lightweight Directory Access Protocol (LDAP).

· Defines a high signal-to-noise audit policy.

Website Content

Housing the content files on a non-system partition implies you’ll be pointing your website virtual directories also to this same location. This prevents such things as directory traversals, where an attacker can request system files.

Even more important is auditing the NTFS permissions of the content. This is what ultimately grants and restricts access to files and should be dumped and sent to the website content owners for review regularly.

Security Scanning & Auditing

Regularly scanning and auditing your own systems is a proactive measure to ensure you find any weaknesses before someone else does. It also can help maintain system configuration, user accounts, etc. Generally, you will need to audit the server and the website separately. Auditing the server can be done via MBSA and manual checks or scripts. The website or web application may have unique requirements, consider using a 3^rd party vulnerability scanning tool designed for this.

Log Analysis

The idea here is that one is only guessing they’ve not been breached if they don’t regularly audit logs. The most secure way to do this is in real-time so that you don’t miss anything if someone covers their tracks by modifying logs. This can also be accomplished by pointing IIS Logging to a UNC share or other remote repository.

Parsing the logs can be challenging. Here are some examples of what to look for:

· Repeated Failed Logon attempts in a short period of time.

· Many 404 (page not found), 401 or 403 (access denied) errors in IIS Logs.

· Funky requests that are obviously not your website

Here’s a sample of an unsuccessful login to a SharePoint site as Administrator. First the IIS Log entry, then the Windows Event Log.

---------------------------------------------------------------------------------------------------------------------------------------

IIS Log

· 2006-08-09 21:00:46 W3SVC1371773545 IIS01 192.168.1.120 POST /_vti_bin/sitedata.asmx - 47451 - 192.168.1.120 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.42) - - iis01:47451 401 2 2148074254 1918 331 0

Security Event Log

Event Type: Failure Audit

Event Source: Security

Event Category: Logon/Logoff

Event ID: 529

Date: 8/9/2006

Time: 1:59:28 PM

User: NT AUTHORITY\SYSTEM

Computer: IIS01

Description:

Logon Failure:

Reason: Unknown user name or bad password

User Name: administrator

Domain: IIS01

Logon Type: 8

Logon Process: Advapi

Authentication Package: Negotiate

Workstation Name: IIS01

Caller User Name: NETWORK SERVICE

Caller Domain: NT AUTHORITY

Caller Logon ID: (0x0,0x3E4)

Caller Process ID: 3372

Transited Services: -

Source Network Address: 192.168.1.120

Source Port: 1048

---------------------------------------------------------------------------------------------------------------------------------------

Host Headers

Host Headers basically tell the website to only respond when the header name is requested. This prevents the website from responding to any name other than the intended URL (i.e., IP scans, localhost spoofs, etc). Create a new Host Header for any and all URLs you want the site to be available from and certainly all registered DNS records that resolve to the web server (i.e. example.com, www.example.com, example.intranet.com, etc.)

This seems to be very commonly overlooked, maybe because the default website has a blank host-header upon install. Remember CodeRed? That nasty worm that propagated through the internet attacking IIS 5’s indexing service? Even you were still unpatched today, 5 years later, I’d bet if you used host headers you would never have been affected. Almost all worms propagate via IP scans and a server properly using host headers will not respond to them. Of course, I’m not at all advocated playing with fire like that, but consider the benefits of this simple measure.

URL Scan

UrlScan version 2.5 is a security tool that restricts the types of HTTP requests that Internet Information Services (IIS) will process. By blocking specific HTTP requests, the URL Scan security tool helps prevent potentially harmful requests from reaching the server. A good approach to this tool is if you know the website’s needed HTTP verbs (GET, POST, etc), just allow those and deny everything else. Same goes for file extensions.

Note: IISLockdown is no longer needed in IIS 6. All of the default security-related configuration settings in IIS 6.0 meet or exceed the security configuration settings made by the IIS Lockdown tool. Therefore, you do not need to run this tool on Web servers running IIS 6.0. However, if you are upgrading from a previous version of IIS, you should run the IIS Lockdown Tool before the upgrade to enhance the security of your Web server.

Enforcing site-wide SSL

Here's a small point I left out but see every day: IIS is using SSL, but it's optional!. Yes, there's a little checkbox that you don't want to miss if you want to ensure usernames, passwords, creditcards, etc are encrypted, check it! (Web Site Properties, Directory Security, Secure Communications, Edit, check Require secure channel (SSL), check Require 128-bit)

Web Content Replication

Automating the replication of the web content itself is that much more important in the case of a load-balanced, multi-tiered web farm. Publishing changes to the web application content across Dev, Test, Staging, and Production offers many opportunities for files and permissions to become out of synch. Automating this process eliminates any doubt that all data is the same, and intended changes are indeed being applied.

This also can allow for self-service publishing to the Production servers by way of allowing developer access to a Dev replication share, and automatically replicating that share to Staging, then Production on scheduled intervals.

IIS Metabase Replication

Replicating the IIS Metabase allows web site administration to be performed on only one server and automatically replicated to all the others. This ensures standard settings are applied uniformly and increases configuration compliance across all systems reducing the opportunity for human error.

Use a standard name for all IUSR_servername accounts

If you’ve enabled Anonymous access, you’re most likely using this account. In a web farm scenario, you can rename all local IIS_machinename accounts to IIS_whatever, change their passwords to be the same, and ACL the website content with the local account. Don’t worry, since the SID on these accounts is the same, if one account gets hosed you can just remove it from the farm and fix it while your site continues running happily.

General Web Infrastructure Design Considerations

Do not put your internet-facing web servers in your corporate Active Directory. Don’t laugh, this is very common. Removing an AD dependency may pose challenges from an infrastructure perspective, but generally speaking web servers should not be members of the corporate AD. If they simply must be in an AD, a separate AD DMZ Forest should be built, or if only authentication is required you may consider ADAM (Active Directory Authentication Mode). The number of ports required to be open from the DMZ to the corporate LAN is a key consideration here, as well as the fact that if a web server is compromised, so is your corporate AD. Because of their internet exposure, web servers should always be contained as much as possible, and not be joined to the corporate domain or use any domain accounts.

Do not use AD accounts to run IIS Anonymous Authentication. This has two problems, one being security and other being availability. If you have 3 load-balanced web servers, but they all use the same domain account you still have a single point of failure, the domain account. If the account is locked out, disabled, deleted, or AD is unavailable, so is the entire web farm. Also, any member of AD has default rights to do basic LDAP queries which potentially exposes your directory to the internet.

Consider real-time monitoring. There are several tools which can be used to monitor in real time account logons, event & IIS logs, server health & performance, etc. This is necessary for critical servers to proactively address issues before they become outages.

Application Pool Settings. The default Worker Process Recycling interval is every 1740 minutes. Is this appropriate? Consider using one Application Pool for each web application and a shared pool for websites with static content. Also, consider using multiple Worker Processes to service the same pool (Web Garden). This way, if one process is busy or hung, others are available to service requests.

Enable Logging Everywhere. It’s better to have logs and not need them than to need logs and not have them. Enable logging for SMTP, full extended properties. Enable all Success & Failure for logon events - Security Settings, Local Policies, Audit Policy.

Disable Internet Explorer. This way admins cannot use IE locally on servers. This also means you no longer have to patch IE monthly, you can patch during planned maintenance cycles. This can be done via either Security Settings, Software Restriction Policies or via a bogus Proxy Server defined in IE Internet Options.

Fingerprinting

Fingerprinting is whereby someone can identify the web server vendor, version, OS, etc. I have a hard time this one. If you’re paranoid you can reduce the ways in which someone can do this, however, even if you’re successful (which is difficult and in some cases impossible), if an attacker cannot identify the web server he’s likely just going to throw all popular exploits for all popular platforms at it, probably starting with IIS or Apache. In my opinion you don’t gain much from this, security though obscurity is not a good plan.

References

Internet Information Services (IIS) 6.0 Resource Guide
http://www.microsoft.com/downloads/details.aspx?FamilyID=80a1b6e6-829e-49b7-8c02-333d9c148e69&DisplayLang=en

Windows Server 2003 Security Guide - Chapter 9: The Web Server Role
http://www.microsoft.com/technet/security/prodtech/windowsserver2003/w2003hg/s3sgch09.mspx

Security Configuration Wizard for Windows Server 2003
http://www.microsoft.com/windowsserver2003/technologies/security/configwiz/default.mspx

Windows Server 2003 Deployment Kit
http://www.microsoft.com/windowsserver2003/techinfo/reskit/deploykit.mspx

IIS on TechNet
http://technet2.microsoft.com/windowsserver/en/technologies/featured/iis/default.mspx

IIS 6.0 Security Best Practices (IIS 6.0)
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/596cdf5a-c852-4b79-b55a-708e5283ced5.mspx?mfr=true

IIS Log Codes
http://support.microsoft.com/?id=318380

Tools

Internet Information Services (IIS) 6.0 Resource Kit Tools
http://www.microsoft.com/downloads/details.aspx?familyid=56fc92ee-a71a-4c73-b628-ade629c89499&displaylang=en

URL Scan
http://www.microsoft.com/technet/security/tools/urlscan.mspx

Copying IIS Configurations Using iiscnfg.vbs (IIS 6.0)
https://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/05001ec3-be42-431a-bfe8-08c865564037.mspx?mfr=true

Log Parser 2.2
http://www.microsoft.com/downloads/details.aspx?FamilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en

AccessEnum (for auditing NTFS permissions)
http://sysinternals.com/Utilities/AccessEnum.html