SQL Server Policy-Based Management

Generate the List of All Available Facets and Their Properties

sqlpbm — Mon, 20 Apr 2009 20:40:40 GMT

Our colleague Jens Suessmeyer in Germany has figured out how to generate the list of all facets and their properties. He has posted the list on his blog. Below is the code snippet from Jens:

foreach (FacetInfo info in PolicyStore.Facets)

{

     Console.WriteLine(info.Name);

     foreach (System.Reflection.PropertyInfo p in info.FacetProperties)

     {

          Console.WriteLine("\t" + p.Name);

     }

}

About the Author:

Hongfei Guo is a PM on the SQL Server Manageability team at Microsoft

Policy Evaluation Modes

sqlpbm — Tue, 14 Apr 2009 03:39:53 GMT

I hosted a PBM break out session in Microsoft MVP Summit 2009 in March. One feedback I got is that the rules for what evaluation modes a policy support are obtuse. I thought that would be a good topic to cover in the blog.

In PBM, users can specify the “Evaluation Mode” for a policy, as shown in the Policy Dialogue below:

The Evaluation Mode defines the combination of 1) when the policy will be evaluated and 2) what the system will do in case of policy violation. The following evaluation modes are supported:

Evaluation Mode	When to Check	Action upon Violation
On Demand	On demand only	Users can choose to configure the system to comply to the policy
On Schedule	Periodically according to the specified schedule	log
On Change – Log only	When there is a relevant (DDL) change to the DBMS	log
On Change – Prevent	When there is a relevant (DDL) change to the DBMS	Roll back the transaction

You may have noticed from the GUI that all policies support On Demand evaluation mode. Most policies support On Schedule evaluation mode. Among those, some policies support On Change – Log only. Again among those, some polices also support On Change – Prevent. As shown in the Venn diagram.

On Schedule

For 2008, we support On Schedule mode for all policies referring to facets on the Database Engine. There are two facets that are not in this category: ISurfaceAreaConfigurationForAnalysisServer is defined for the Analysis Services and ISurfaceAreaConfigurationForReportingServices is defined for the Reporting Services. We only support On Demand mode for policies defined on these two facets.

On Change

A policy supports On Change modes if and only if both of the following rules hold:

1) The facet referred to by the policy supports On Change modes

2) If there are any target-set-filter conditions (see the Policy Dialogue Figure) specified in this policy, they are all Naming conditions (conditions in the form of <@name, [Operator], [value]>) and there is no server restriction condition.

let’s look at the first rule first. The On Change evaluation modes are implemented using the DDL eventing mechanism of SQL Server. And the “roll back” action relies on transactional support of the corresponding DDL statement. The change of some facet properties does not trigger any event, for example, database size in the Database facet. And accordingly we don’t support On Change modes for such facets. For some facets, although there are always events for every facet property, there is no transactional support for the DDL statement. For such facets, On Change – Log Only mode is supported, but not On Change – Prevent. IDatabaseOption facet is such an example. You can use the following query to check out what evaluation modes a facet support:

select * from msdb..syspolicy_management_facets

The execution_mode column is a bit mask defined as follows:

4 (100): On Schedule

2 (10): On Change - Log Only

1(1): On change - Prevent

The following query checks what events are monitored for a facet supporting On Change modes:

select * from msdb..syspolicy_facet_events

Next let’s look at the second rule. In a policy you can use conditions to filter the set of objects the policy applies to. Theoretically, you can use any condition to filter out the targets. However, there is a tradeoff between flexibility and performance. The performance penalty in allowing arbitrary filter conditions becomes significant with On Change evaluation modes since the policy evaluation is done alongside the transaction. It is less an issue for On Schedule mode since users can choose to run the policy at off-peak time. For that reason, we made a tradeoff:

1) For the common use cases where the filter is a “name condition” on Database, that is: <@name, [Operator], [value]>, we allow the full set of applicable evaluation modes because there is an efficient way in calculating the target set.

2) For the use cases where arbitrary filter conditions are needed, we allow the flexibility but for performance reason we don’t allow On Change evaluation modes.

The syspolicy_conditions_internal.is_name_condition indicates if this condition is a “name condition” (when the value is not 0).

Is_name_condition has the following enum values (representing the operator in the filter condition):

0 : None

1: Equals

2: Like

3: NotEqual

4: NotLike

Summary

Not all policies support On Change evaluation modes. We only support On Change evaluation modes for a policy when the referred facet has DDL event coverage and the target filter conditions for the policy are simply Naming conditions if there are any.

About the Author:

Hongfei Guo is a PM on the SQL Server Manageability team at Microsoft

Defining Complex Server "Health" Policies in SQL2K8

sqlpbm — Fri, 12 Sep 2008 06:07:25 GMT

Bart Duncan (a dev on the manageability team and one of the masterminds behind the Management Data Warehouse and Data Collector feature in SQL Server 2008) has a blog posting where he walks through using PBM to author a complex server "health" policy.

Out-Of-Box Policies

sqlpbm — Tue, 15 Jul 2008 23:10:01 GMT

SQL Server 2008 includes several predefined policies. These are generally associated with best practices and overtime the rules from the Best Practices Analyzer will be implemented as policies for PBM. A number of these have already been implemented and are available to you in one of two forms:

First, they're installed as part of every Management Studio installation. There is no specific feature option to select in the setup GUI, just install Management Studio (Management Tools - Basic) and the policies will be installed. The policies are placed on your local machine in the form of XML files. These files can be imported into an instance of SQL Server 2008 or can be run directly from the XML files. The location of the files depends on where you chose to install the client tools. On a default install, you'll find them here: C:\Program Files\microsoft sql server\100\Tools\Policies

The second way to get the policies is to download them from the feature pack. Here's the link to the RC0 feature pack. If you scroll down the page you'll find an entry for Microsoft SQL Server 2008 Policies. You can download and run the MSI. You'll have an option to install the policies to any location you like.

It's a good idea to periodically check for updates to the policies. Overtime we'll add new policies, change existing ones as needed and remove out-dated policies. If you decide to modify one of these policies we recommend you move it to a different location to avoid it from being overwritten by an update.

_____________

About the Author:
Dan Jones is a PM on the SQL Server Manageability team at Microsoft.

Policy Categories

sqlpbm — Mon, 07 Jul 2008 22:06:18 GMT

Policy categories have two purposes in SQL2K8: 1) organization of policies and 2) policy scope. In this posting we'll explore creating categories for each of these purposes.

Creating Categories

There are two ways to create a category: 1) the Manage Categories dialog and 2) The Policy Properties dialog.

To create a category from the Manage Categories dialog, connect Object Explorer to an instance of SQL2K8. Expand the Management node, right-click on the Policy Management and select Manage Categories.

This will launch the Manage Categories dialog. On this dialog you can create, update, and delete categories and change the subscription state for each category. I'll cover the subscription state later on.

To create a new category click in the Name field on the last row and type in the name for the category. This will create a new category which is mandatory for every database. Click OK to dismiss the dialog and you're done.

You can also create a new category from the Policy Properties dialog.

On the Description tab click New next to the Category field. This launches the new Create new category which accepts a category name. The category will be created as mandatory for all databases. You also use this dialog to assign a policy to a particular category. Use the category combo box to select a category.

Organizing Policies

There are a few key places where the use of categories simplifies certain actions.

Object Explorer Details

When the Policies folder is selected in Object Explorer, Object Explorer Details lists all of the policies on the system with a select set of properties, including Category.

The columns in Object Explorer Details can be sorted, just click on the column heading you wish to sort by.

Sorting on category can make it easier to perform certain operations.

Evaluate, Export, and Script are all valid operations on a set of policies. You could use ctrl-click to select multiple disjointed policies but I've found sorting to be quicker and less error prone.

View Policies Dialog

Right-click any object in Object Explorer and select Policies -> View to see the policies which apply (or can apply) to the selected object. The View Policies dialog is displayed.

Category is a property in the table which can be used to sort the policies. This makes it easy to see which policies in a particular category apply to the object or are in violation.

Evaluate Policies

The Evaluate Policies dialog allows you to manually evaluate one or more policies against a single object or against the computed target set (all objects the policies applies to).

This dialog includes the Category property which you can sort on. If you wish to run all of the policies in a particular category sorting them first prevents you from having to hunt and peck for each policy in a particular category.

In this section we looked at several places where using categories can expedite certain policy tasks. Now let's look at category subscriptions which controls the applicability of policies.

Delegation of Policy Applicability

Using categories databases can subscribe to certain sets of policies. Subscribe simply means that all of the policies in the category may apply or do not apply to the particular database. This is only applicable to policies which target databases or objects contained in a database. Policies which target the instance and instance objects (excluding database) are excluded from the subscription.

Categories can be configured for mandatory subscription, meaning all databases must subscribe to the category and cannot opt out, and non-mandatory subscriptions, meaning individual databases can opt-in/opt-out of policies in the category.

When a database subscribes to a category it's not implicit that all policies in the category apply to the database and all objects in the database. The policy may contain a target filter which eliminates certain objects from the scope of the policy. However, if a database does not subscribe to a category the database and all database objects in that databases are explicitly excluded from the scope of the policy.

There are certain situations where optional categories make sense. A set of policies may only apply to certain databases but the Policy Administrator doesn't know which databases the policies apply to. However the owner of the database does know and can choose which policy categories apply and which don't.

For example you may have a set of policies that only apply to reporting (read-only) databases, but only the database owner knows if their database is a reporting database (there is no database attribute to designate this type of database and therefore no way to construct a target filter). The Policy Administrator would create the policies for reporting databases in a category called "Reporting Database Policies". The category is configured to be optional. Each database owner can then choose to subscribe to the category or not.

Security Notice: You must be SA or DBO of the database to control policy category subscriptions.

This launches the Categories dialog.

Mandatory categories appear subscribed to and disabled - meaning they cannot be unsubscribed from. The "<Default>" category is always mandatory. Non-mandatory categories have the subscription checkbox enabled to allow the database owner to subscribe/unsubscribe as appropriate. When the category is expanded the list of policies in the category is displayed. To subscribe to the category select the checkbox under the Subscribed column and click OK.

Summary

Categories serve two purposes: 1) organization (group like policies together) and 2) policy scope. It's a best practice to categorize your policies by their purpose (such as industry compliance or security). Choosing to mark the categories as optional or mandatory depends on your particular situation.

Suggestion: If you have SQL2K and/or SQL2K5 servers in your topology and you want to create policies for these servers you can create policies on a SQL2K8 server. Create two non-mandatory categories, "SQL Server 2000" and "SQL Server 2005" and place each policy in the appropriate category. This will make it easier to track what the policies are for. If you automate the policies using PowerShell you can easily select the correct set of policies for each server version.

_____________

About the Author:
Dan Jones is a PM on the SQL Server Manageability team at Microsoft.

Wide Format

sqlpbm — Sun, 06 Jul 2008 20:24:50 GMT

You may notice I've changed the blog format from narrow to wide. I think this makes it more readable as it uses more screen real estate - less scrolling! Let me know what you think. Or let me know if you think I'm obsessing too much about something insignificant and I should be directing that energy to more postings.

Cheers,
Dan

Using PBM Against SQL2K and SQL2K5

sqlpbm — Fri, 04 Jul 2008 22:32:56 GMT

We get this question a lot: can I use PBM against a SQL2K & SQL2K5 instances. The quick answer is yes but in a limited fashion. PBM is ultimately based on SMO (SQL Server Management Objects) and SMO supports SQL2K, SQL2K5, and SQL2K8. PBM relies on some changes to the DB engine which are not available in versions below SQL2K8, therefore, not all PBM functionality is available in SQL2K and SQL2K5.

Let's divide the PBM functionality into four key areas:

Authoring policies
Storing policies on a server
Automated policy evaluation
Manual policy evaluation

Authoring Policies

There are two ways to author a policy: connected and disconnect. Connected policy authoring is only supported in SQL2K8. When authoring in connected mode (Object Explorer is connected to an instance) the policy is stored on the server. There is no way to alternatively save the policy to the file system (other than exporting it after it's created on the server). Disconnected authoring saves the policy to the file system (as an XML file). This functionality is only available in SQL2K8 Management Studio. Since it doesn't require a connection to a back-end server you don't need an instance of SQL2K8 available. Therefore, authoring policies is somewhat dependent upon SQL2K8 - you need the toolset but not a server.

Storing Policies On A Server

There are two ways to store a policy on a server: create it on the server and import a policy XML file to the server. Both of these are only available on SQL2K8 servers.

Automated Policy Evaluation

Policy automation is dependent upon enhancements to dependent features only available in SQL2K8 (SQCLR, SMO, Agent and DDL Eventing.) Therefore, automated policy evaluation (Check on Change - Prevent, Check on Change - Log, and Check on Schedule) is only available on SQL2K8 servers. In addition, the automation requires the policy to stored locally on the server which is only supported on SQL2K8.

Manual Policy Evaluation

Manually evaluating a policy is a client-side operation (simply meaning it doesn't run in the context of the DB Engine service) and therefore it requires SQL2K8 Management Studio (or at least the SQL2K8 management stack - PBM & SMO APIs). There are four ways to manually evaluate a policy:

From Object Explorer in Management Studio
From Registered Servers in Management Studio
The PowerShell cmdlet Invoke-PolicyEvaluation
From the PBM API

Object Explorer (OE): Policy evaluation is supported from OE when connected to SQL2K5 and SQL2K8 servers (note: this may also be available when connected to a SQL2K server but I didn't have one available to me at the time of this writing to verify). You get the same Evaluation dialog in both cases. However, when connected to a SQL2K5 server there are no locally stored policies which means you need to select policies from file or from a SQL2K8 server.

Registered Servers (RegSrvrs): Policy evaluation is supported from RegSrvrs for SQL2K, SQL2K5, and SQL2K8 servers. You can evaluate policies that are saved to file or stored on a SQL2K8 server. You can also evaluate policies against a group of servers. The server group can contain mixed versions.

Invoke-PolicyEvaluation: This PowerShell cmdlet can be run against SQL2K, SQL2K5, and SQL2K8 servers. You just pass in the server name to the TargetServerName parameter. You can evaluate policies stored on a SQL2K8 server or on the file system. This blog post shows how to use the Agent PowerShell subsystem in SQL2K8 to create a job that runs a PowerShell script to evaluate policies against a group of servers. These servers can be SQL2K, SQL2K5 and SQL2K8 servers.

PBM API: The PBM API is a public API and contains methods for evaluating policies. This is exactly what we use under the covers for our policy evaluation. You can create your own .Net application which calls this API. Similar to the PowerShell cmdlet you need to pass in a connection to the server to evaluate. This connection can be a SQL2K, SQL2K5, or SQL2K8 connection. The details for how to do this (the code you need to write) is beyond the scope of this posting but is a good topic for a future posting.

Summary

When we designed PBM we accounted for the fact that not all customers immediately upgrade their environment - you have to deal with mixed environments. We wanted to be sure PBM would add value to these mixed environments but we also need to give you a reason to upgrade :-). Therefore, we enable you to run policies against SQL2K and SQL2K5 just not with the same fidelity as SQL2K8.

ExecuteSql()

sqlpbm — Thu, 03 Jul 2008 19:06:45 GMT

One of the key development tenets of Policy-Based Management was to raise the level of abstraction of management. The analogy I've used time and time again is Security Zones in Internet Explorer. Prior to Security Zones we had to scroll through pages and pages of configuration options. Security Zones greatly simplifies this task by providing ready made templates for the settings. However, every so often it's necessary to by pass the security zones and configure the properties directly or to configure a property that's not covered by the Security Zones. We recognized that in a v1 release we wouldn't be able to cover the entire system with facets (necessary to raise the level of management abstraction) so we created a so called backdoor. The backdoor allows the policy author to create a condition expressed in Transact-SQL. Effectively this means if you can write a query to check for the desired state you can wrap that query in a policy. For security purposes the policy can only be automated to check for compliance on a schedule. I'll double back to security in a moment. First let's look at the basics of the ExecuteSql() function.

Here's the function signature:

Variant ExecuteSql(string returnType, string sqlQuery)

Here are the arguments:

returnType - Specifies the return type of data that is returned by SQL Server. The valid literals for returnType are Numeric, String, Bool, DateTime, Array, and GUID.
sqlQuery - String that represents the query to be run.

Note: Be sure to wrap both the returnType and sqlQuery in single quotes. If the sqlQuery contains quotes you'll need to escape them by doubling up the single quotes.

The return value:

Return value - The first column in the first row of the result set returned by the Transact-SQL query.

The ExecuteSql() function can be used against any facet and it'll be executed once for each target in the target set. For example, if it's used on the server facet it'll only execute once. If it's used against the database facet it'll run once for each database. When run against databases or objects within a database, the context for the execution is the current database.

For added flexibility and power the context of schema bound objects is passed into the SQL statement using ADO.NET parameter substitution.

@@ObjectName - corresponds to the name field in sys.objects. The variable will be replaced with the name of the current object.

@@SchemaName - corresponds to the name field in sys.schemas. The variable will be replaced with the schema for the current object if the current object belongs to a schema.

Security

The sqlQuery can contain any valid DDL or DML Transact-SQL statement. This includes SELECT, DELETE, UPDATE, TRUNCATE, CREATE, GRANT, etc. Because of this the initial implementation prevented policies using ExecuteSql() from being automated. They could only be manually run and they would run in the context of the current user. It is the responsibility of the user to ensure the policy isn't doing something malicious. In other words the user should treat the policy just like they'd treat any file containing Transact-SQL - don't blindly execute it!

We subsequently relaxed this restriction. We recognized how powerful this capability is and the early user feedback indicated a strong desire to automate policies containing SQL. To allow this we had to address the security concerns. A user only needs to be a Policy Administrator in MSDB to author policies, a very low privileged role. Due to the security concerns we considered eliminating this role and forcing all policy administration to be performed by an SA. There are times I think we should have done this but instead we opted to keep the Policy Administrator role and instead run the Transact-SQL under a proxy account. The proxy account only comes into play when the policy is run on a schedule. When it's run on demand it executes under the current user's context.

The proxy account is called ##MS_PolicyTsqlExecutionLogin##. You'll find this login under the "Logins" folder in Management Studio and master.sys.syslogins. By default the login has very limited permissions. You will likely need to extend the privileges of this user to accommodate the SQL you need to run under policy.

Note: Every permission you grant to ##MS_PolicyTsqlExecutionLogin## you are effectively granting that permission to all users in the Policy Administrator role.

Let's walk through an example. Someone recently came to me asking how to create a policy that checks that all tables have data compression enabled. They didn't care if it's row or page level compression, they simply want to check that compression is on.

Compression is a property of a partition. Tables can have one or more partitions. This means for each table we need to make sure that all partitions are compressed. Unfortunately there is no facet over partitions. This was simply a scoping decision for SQL2K8. There is no technical reason we couldn't do it. But this means the only way to author this policy is to write Transact-SQL. It's always best to state the policy in plain English (or in your native language) "All tables must have all partitions compressed." Since we're dealing with table I'll author the policy/condition on the Table facet. We could do this at the database level but I want to flag individual tables that have uncompressed partitions rather than flagging databases with tables with uncompressed partitions.

I'll start by constructing the raw Transact-SQL that returns schema name, table name and partition count where the partition isn't compressed. In an ideal situation this query would not return any rows. This query can be run in your favorite query tool.

SELECT sch.name, tab.name, COUNT(*)
FROM sys.partitions part
JOIN sys.tables tab
ON (part.object_id = tab.object_id)
JOIN sys.schemas sch
ON (tab.schema_id = sch.schema_id)
WHERE part.data_compression = 0
GROUP BY sch.name, tab.name

There are a few modifications we'll need to make to the SQL before we wrap it in a condition. First we need to set it up to accept the context of a specific table. Remember, we'll run the SQL for each table. To do this we need to add to the WHERE clause and use our special parameters.

SELECT sch.name, tab.name, COUNT(*)
FROM sys.partitions part
JOIN sys.tables tab
ON (part.object_id = tab.object_id)
JOIN sys.schemas sch
ON (tab.schema_id = sch.schema_id)
WHERE sch.name = @@SchemaName
AND tab.name = @@ObjectName
AND part.data_compression = 0
GROUP BY sch.name, tab.name

This gets us almost all the way there but remember, the return value is only the first value in the first row of the result set. This means we need to only return COUNT(*) and while we're at it we need to remove the GROUP BY.

SELECT COUNT(*)
FROM sys.partitions part
JOIN sys.tables tab
ON (part.object_id = tab.object_id)
JOIN sys.schemas sch
ON (tab.schema_id = sch.schema_id)
WHERE sch.name = @@SchemaName
AND tab.name = @@ObjectName
AND part.data_compression = 0

Excellent, the final step is to construct the full function. The return value is numeric and we don't have any strings to escape so this is a pretty straight forward step.

ExecuteSql('Numeric', 'select COUNT(*)
FROM sys.partitions part
JOIN sys.tables tab
ON (part.object_id = tab.object_id)
JOIN sys.schemas sch
ON (tab.schema_id = sch.schema_id)
WHERE tab.name = @@ObjectName
AND sch.name = @@SchemaName
AND part.data_compression = 0')

The SQL is constructed so that it returns 0 if all of the current table's partitions are compressed and some value greater than 0 if at least one partition is not compressed. Since conditions are constructed in the positive we'll compare the result to 0. In other words if it equals 0 the table is considered to pass the policy else it's considered to violate the policy.

Here are screen shots of the condition I created and the advanced condition editor.

Here's a screen shot of the evaluation results for the policy when run against one of my test databases which has two tables, one has all partitions compressed and one has none.

ExecuteSql() is a powerful function that allows you to create some very powerful policies. Given the security considerations you'll want to use it wisely and choose carefully who you allow to create policies.

Have you created a cool policy using ExecuteSql()? If so you should post it up on http://www.codeplex.com and share it with others. Or you can reply to this posting with the details.

_____________

About the Author:
Dan Jones is a PM on the SQL Server Manageability team at Microsoft.

APBMB

sqlpbm — Thu, 19 Jun 2008 03:39:28 GMT

Another PBM Blog (APBMB)! Lara Rubbelke has been blogging a bunch about PBM. Check out her blog here.

PowerShell Script for Creating a Policy to a File

sqlpbm — Thu, 19 Jun 2008 03:39:02 GMT

Sethu Srinivasan, a developer on the PBM team, wrote a blog post for a sample PowerShell script that will create a new policy to check XPCMDSHELL and save it to an XML file.

Running Against SQL Server 2005 and SQL Server 2000

sqlpbm — Sun, 15 Jun 2008 05:35:59 GMT

@ TechEd SQL Server MVP Peter DeBetta and I presented a session where we created a PowerShell script for running a group of policies against a group of servers. The script was then automated using the PowerShell subsystem in SQL Server Agent. The result of each policy evaluation was loaded into a table for reporting purposes.

Because PBM is built on top of SQL Server Management Objects (SMO) and SMO supports SQL Server 2000, SQL Server 2005, and SQL Server 2008, PBM will work against all of these versions.

The script reads in a text file containing a list of the target servers (the servers each policy will be evaluated against). It connects to a primary server to grab the policies and then evaluates each policy in a particular category against each server.

The script is intended to be a proof of concept and is not ready for a production environment. Some alterations that you would want to make include adding error handling and reading the servers from a Central Manageability Server or at the least a table in the primary server rather than from a file.

Here's the script for creating the database and table to store the results of the policy evaluation:

CREATE DATABASE [PolicyResults]
GO

USE [PolicyResults]
GO

CREATE TABLE [dbo].[PolicyHistory](
     [EvalServer] [nvarchar](50) NULL,
     [EvalDateTime] [datetime] NULL,
     [EvalPolicy] [nvarchar](max) NULL,
     [EvalResults] [xml] NULL
) ON [PRIMARY]

GO

ALTER TABLE [dbo].[PolicyHistory] ADD CONSTRAINT [DF_PolicyHistory_EvalDateTime] DEFAULT (getdate()) FOR [EvalDateTime]
GO

You might want to add a column to store the policy result or you can use an XQuery to grab it out of the EvalResults column. I'll show you the query below.

Here's the PowerShell script. I cleaned up the version of the script that was presented - reorganizing it a little to make it easier to follow what's going on.

#Evaluate Policies in a Particular Category against a Server list
#Uses the Invoke-PolicyEvaluation & Invoke-SqlCmd Cmdlets

function InsertPolicyHistory($ServerVariable, $DBVariable, $EvalServer, $EvalPolicy, $EvalResults)
{
    #Escape single quotes so we can insert
    $EvalResultsEscaped = $EvalResults -replace "'", "''"
    $EvalPolicyEscaped = $EvalPolicy -replace "'", "''"

    #Setup the insert statement
    $QueryText = "INSERT INTO PolicyHistory (EvalServer, EvalPolicy, EvalResults) VALUES(N'$EvalServer', '$EvalPolicyEscaped', N'$EvalResultsEscaped')"

    #Run the insert statement using the Invoke-SqlCmd Cmdlet
    Invoke-Sqlcmd -ServerInstance $ServerVariable -Database $DBVariable -Query $QueryText
}

#CONSTANTS
#Declare the Server\Instance & database to post the policy results
$HistoryServer = "myServer\myInstance"
$HistoryDatabase = "PolicyResults"
#Declare the server\instance for the Policy store
$PolicySourceServer = $HistoryServer
#Setup the file containing the list of servers
$ServersFile = "c:\Servers.txt"
#Setup the location to dump the policy evaluation result output
$PolicyDir = "c:\PolicyEvaluation\"
#Setup the policy filter - only policies in this category will be processed
$PolicyCategoryFilter = "Custom Automation"

#Setup a connection to the policy store
$Conn = new-object Microsoft.SQlServer.Management.Sdk.Sfc.SqlStoreConnection("server=$PolicySourceServer;Trusted_Connection=true");
$PolStore = new-object Microsoft.SqlServer.Management.DMF.PolicyStore($Conn);

#Read the servers file into a variable
$Servers = Get-Content $ServersFile

#Clear out the directory where the evaluation results will go
del c:\PolicyEvaluation\*

foreach ($TargetServer in $Servers)
{
    foreach ($Policy in $PolStore.Policies)
    {
        if ($Policy.PolicyCategory -eq $PolicyCategoryFilter)
        {
            #Clean-up any invalid file system characters
            $PolicyNameFriendly = (Encode-SqlName $Policy.Name)
            $TargetServerFriendly = (Encode-SqlName $TargetServer)

            #Setup the output file as Server_Policy.xml
            $OutputFile = $PolicyDir + ("{0}_{1}.xml" -f $TargetServerFriendly, $PolicyNameFriendly);

            #Evaluate the policy
            Invoke-PolicyEvaluation -Policy $policy -TargetServerName $TargetServer -OutputXML > $OutputFile;

            #Read in the policy evaluation results to load it into the result table
            $PolicyResult = Get-Content $OutputFile;
            #Insert the results to our result table
            InsertPolicyHistory $HistoryServer $HistoryDatabase $TargetServer $Policy.Name $PolicyResult;
        }
    }
}
#Clean-up the evaluation result files
del c:\PolicyEvaluation\*

After you run the script you can use SQLCMD or SSMS to query the result table. The query below will return all rows where the policy was violated. To get back all of the rows where the policy passed change "false" to "true".

USE [PolicyResults]
GO
WITH XMLNAMESPACES ('http://schemas.microsoft.com/sqlserver/DMF/2007/08' AS Pol)
SELECT *
FROM dbo.policyhistory
WHERE EvaluationResults.exist('//Pol:EvalDetail/Pol:Result/text()[. = "false"]') = 1

In closing, if you automate the script using SQL Server Agent it will run under the context of the SQL Server Agent job. This can either be the Agent service account or you can use a proxy account. Which ever path you choose you'll need to make sure the account has access to the servers.txt file, the location where the results will be written to, the server containing the policies, the server where you want to write the results and each of the servers listed in servers.txt.

_____________

About the Author:
Dan Jones is a PM on the SQL Server Manageability team at Microsoft.

Facets

sqlpbm — Sat, 24 May 2008 21:23:49 GMT

The American Heritage Dictionary of the English Language (Fourth Edition) defines facet as: One of numerous aspects, as of a subject. The definition of Aspect (from the same dictionary) is: Appearance to the eye, especially from a specific vantage point. The definition of Facet in SQL Server Books Online is: A set of logical properties that model the behavior or characteristics for certain types of managed targets. The number and characteristics of the properties are built into the facet and can be added or removed by only the maker of the facet. A target type can implement one or more management facets, and a management facet can be implemented by one or more target types. Some properties of a facet can only apply to a specific version. For example, the Database Mail property of the Surface Area Configuration facet only applies to SQL Server 2005 and later versions.

Though Policy-Based Management relies heavily on facets, they're really concept that goes beyond PBM. When I was a kid I'd get these pictures that had hidden items. To find the items you had to use different pieces of colored translucent paper (like red or green). Just looking at the picture you couldn't see the "hidden" elements. But placing the red sheet over the picture revealed the hidden treasures. Facets are sort of the same way.

Another analogy I've used to describe facets is Internet Explorer. Think back to the early days of IE and you wanted to change the way IE handled ActiveX controls. You'd navigate to Tools -> Options, go to the Advance tab and scroll through a long list of settings. No one every really knew if their browser settings were secure or not. Think of the handling of ActiveX controls as a property of the browser with the values Do not Allow, Prompt, Always Allow. Requiring a user to reason about this, and many other properties, is insane. My mom, for example, has no idea what this means or what the proper setting should be. In IE 5.5 the IE team introduced a security facet. They didn't call it that but that's what it is. They introduced the concept of a Security Level with the values: High, Medium-High, and Medium. Effectively this is a new logical property in IE. I call it a logic property as it has no intrinsic meaning - it doesn't really control anything. Ah, but what it does do is incredibly powerful (yes, I'm contradicting myself). This logical property is really an aggregation of several physical properties (e.g. ActiveX Controls) a proxy for a group of settings. By creating the logical property the administration of IE is greatly simplified. My mom can understand Medium-high security - more importantly I can walk her through the steps to properly configure IE. Facets in SQL Server are very similar concept.

Take Database, for example. There are lots of physical properties for a database: Name, Collation, Compatibility Level, Auto Close, Encryption Enabled, Log File location, Data File Location, etc. In fact, a database has something like 60 properties. Wow, how does one reason about 60 properties? The facet allows us to do three powerful things: 1) create a single view of all of the physical properties for a database, 2) create specific views of database properties; for example, security, compliance, performance, etc and 3) create logical properties which are derived from one or more physical properties. There was a key point I just introduced: facets are over objects (target types) in the system and an object (target type) can have more than one facet. Databases, tables, views, logins, can have facets. in SQL Server 2008 we didn't cover every object (target type). For example, we didn't cover Replication, Agent, or DB Mail. But more on that later. So just think of a facet as a collection of related properties for a given object or similar set of objects.

The implementation of a facet is .Net code - for us it's C#. In SQL Server 2008 users cannot create their own facets, although we designed the system with extensibility in mind.

Let's take a closer look at how facets are exposed in the Management Studio. There are two ways to look at a facet: 1) the facet definition and 2) an object (target type) with respect to a particular facet.

Facet Definition

The facet definitions, meaning the description and properties that make up the facet, can be found in OE. See the picture below.

Right-Clicking any facet and selecting Properties displays the Facet Properties dialog. In this dialog you'll get a description of the facet, the target types it applies to and a list of all of the properties (with description) that make up the facet. The sample below is for the Database facet.

This covers the first view of facets: the facet definition. Now let's look at viewing a particular target in the context of a facet. The screen shot below shows the context menu for a database.

Look about halfway down the context menu and you'll see menu item called Facets. Selecting this menu item launches the Facet dialog for database. The screen shot below shows the dialog for the AdventureWorks sample database.

The Facet combo box is expanded. This allows you to choose which facet you want to view for the selected object. Only the facets that apply to the selected object type are displayed. In this screen shot the Database facet is selected. This shows you all of the properties on a database and the current value for each property for the selected database.

Two other key things to point out: 1) You'll notice that some properties are bold while others are grayed out. The bolded properties are read/write and can be updated; the grayed out properties are read-only. This means, instead of going to the standard Database properties dialog you can come here and change a database property. 2) You'll notice the button in the bottom right labeled Export Current State as Policy...". Let's suppose you've configured the database exactly as you like it but you want a create a policy that monitors any changes from the original state. This option creates a policy and condition for the selected facet using for the current state. It's a very nice shortcut for creating policies, saving you a bunch of time typing in properties and values.

The final thing a facet does is define the supported evaluation modes for policies. All facets for the Database Engine can be checked on schedule. Check on Change: Prevent and Check on Change: Log depend upon the eventing model of the Database Engine. Because the supported evaluation modes are on the facet all of the properties in the facet must support support the same set of events. This means for a facet to support Check on Change: Prevent all of the properties in the facet must raise a DDL event.

The three tables below are a great reference to print out and keep handy when using PBM.

This table maps each facet to the supported evaluation mode(s):

Facet Name	CoC: Prevent	CoC: Log	CoS
Application Role	X	X	X
Asymmetric Key	X	X	X
Audit			X
Backup Device			X
Broker Priority			X
Broker Service			X
Certificate			X
Credential			X
Cryptographic Provider			X
Data File			X
Database			X
Database Audit Specification			X
Database Ddl Trigger			X
Database Maintenance			X
Database Option		X	X
Database Performance			X
Database Role	X	X	X
Database Security			X
Default			X
Endpoint	X	X	X
File Group			X
Full Text Catalog			X
Full Text Index			X
Full Text Stop List			X
Index			X
Linked Server			X
Log File			X
Login			X
Login Options	X	X	X
Message Type			X
Multipart Name	X	X	X
Name			X
Partition Function			X
Partition Scheme			X
Plan Guide			X
Remote Service Binding			X
Resource Governor			X
Resource Pool	X	X	X
Rule			X
Schema	X	X	X
Server			X
Server Audit			X
Server Audit Specification			X
Server Configuration		X	X
Server Ddl Trigger			X
Server Information			X
Server Performance			X
Server Security			X
Server Settings			X
Server Setup			X
Service Contract			X
Service Queue			X
Service Route			X
Statistic			X
Stored Procedure	X	X	X
Surface Area		X	X
Surface Area for AS
Surface Area for RS
Symmetric Key			X
Synonym			X
Table			X
Table Options	X	X	X
Trigger			X
User			X
User Defined Aggregate			X
User Defined Data Type			X
User Defined Function	X	X	X
User Defined Table Type			X
User Defined Type			X
User Options	X	X	X
View			X
View Options	X	X	X
Workload Group	X	X	X
Xml Schema Collection			X

The tables below map the relationship between Facets and Target Types. The table on the left shows the facets for each target type. The table on the right shows the target types supported by each facets.

Facets by Target Type		Target Types by Facet
ANALYSIS SERVICES		Application Role
Surface Area for AS		APPLICATION ROLE
APPLICATION ROLE		Asymmetric Key
Application Role		ASYMMETRIC KEY
Name		Audit
ASYMMETRIC KEY		AUDIT
Asymmetric Key		Backup Device
Name		BACKUP DEVICE
ASYMMETRIC KEY USER		Broker Priority
User Options		BROKER PRIORITY
AUDIT		Broker Service
Audit		BROKER SERVICE
BACKUP DEVICE		Certificate
Backup Device		CERTIFICATE
BROKER PRIORITY		Credential
Broker Priority		CREDENTIAL
BROKER SERVICE		Cryptographic Provider
Broker Service		CRYPTOGRAPHIC PROVIDER
CERTIFICATE		Data File
Certificate		DATA FILE
Name		Database
CERTIFICATE USER		DATABASE
User Options		Database Audit Specification
CREDENTIAL		DATABASE AUDIT SPECIFICATION
Credential		Database Ddl Trigger
CRYPTOGRAPHIC PROVIDER		DATABASE DDL TRIGGER
Cryptographic Provider		Database Maintenance
DATA FILE		DATABASE
Data File		Database Options
DATABASE		DATABASE
Database		Database Performance
Database Maintenance		DATABASE
Database Options		Database Role
Database Performance		ROLE
Database Security		Database Security
DATABASE AUDIT SPECIFICATION		DATABASE
Database Audit Specification		Default
DATABASE DDL TRIGGER		DEFAULT
Database Ddl Trigger		Endpoint
DATABASE ROLE		ENDPOINT
Name		File Group
DEFAULT		FILE GROUP
Default		Full Text Catalog
Name		FULL TEXT CATALOG
ENDPOINT		Full Text Index
Endpoint		FULL TEXT INDEX
FILE GROUP		Full Text Stop List
File Group		FULL TEXT STOP LIST
FULL TEXT CATALOG		Index
Full Text Catalog		INDEX
FULL TEXT INDEX		Linked Server
Full Text Index		LINKED SERVER
FULL TEXT STOP LIST		Log File
Full Text Stop List		LOG FILE
FUNCTION		Login
Multipart Name		LOGIN
User Defined Function		Login Options
GROUP USER		LOGIN
User Options		Message Type
INDEX		MESSAGE TYPE
Index		Multipart Name
Name		FUNCTION
LINKED SERVER		PROCEDURE
Linked Server		SYNONYM
LOG FILE		TABLE
Log File		TYPE
LOGIN		VIEW
Login		XML SCHEMA COLLECTION
Login Options		Name
MESSAGE TYPE		APPLICATION ROLE
Message Type		ASYMMETRIC KEY
PARTITION FUNCTION		CERTIFICATE
Partition Function		DATABASE ROLE
PARTITION SCHEME		DEFAULT
Partition Scheme		INDEX
PLAN GUIDE		RULE
Plan Guide		SCHEMA
PROCEDURE		SQL ASSEMBLY
Multipart Name		STORED PROCEDURE
Stored Procedure		SYMMETRIC KEY
REMOTE SERVICE BINDING		SYNONYM
Remote Service Binding		TABLE
REPORTING SERVICES		TRIGGER
Surface Area for RS		USER
RESOURCE GOVENOR		USER DEFINED FUNCTION
Resource Governor		USER DEFINED TYPE
RESOURCE POOL		VIEW
Resource Pool		XML SCHEMA COLLECTION
ROLE		Partition Function
Database Role		PARTITION FUNCTION
RULE		Partition Scheme
Name		PARTITION SCHEME
Rule		Plan Guide
SCHEMA		PLAN GUIDE
Name		Remote Service Binding
Schema		REMOTE SERVICE BINDING
SERVER		Resource Governor
Server		RESOURCE GOVENOR
Server Configuration		Resource Pool
Server Information		RESOURCE POOL
Server Performance		Rule
Server Security		RULE
Server Settings		Schema
Server Setup		SCHEMA
Surface Area		Server
SERVER AUDIT		SERVER
Server Audit		Server Audit
SERVER AUDIT SPECIFICATION		SERVER AUDIT
Server Audit Specification		Server Audit Specification
SERVER DLL TRIGGER		SERVER AUDIT SPECIFICATION
Server Ddl Trigger		Server Configuration
SERVICE CONTRACT		SERVER
Service Contract		Server Ddl Trigger
SERVICE QUEUE		SERVER DLL TRIGGER
Service Queue		Server Information
SERVICE ROUTE		SERVER
Service Route		Server Performance
SQL ASSEMBLY		SERVER
Name		Server Security
SQL USER		SERVER
User Options		Server Settings
STATISTIC		SERVER
Statistic		Server Setup
STORED PROCEDURE		SERVER
Name		Service Contract
SYMMETRIC KEY		SERVICE CONTRACT
Name		Service Queue
Symmetric Key		SERVICE QUEUE
SYNONYM		Service Route
Multipart Name		SERVICE ROUTE
Name		Statistic
Synonym		STATISTIC
TABLE		Stored Procedure
Multipart Name		PROCEDURE
Name		Surface Area
Table		SERVER
Table Options		Surface Area for AS
TRIGGER		ANALYSIS SERVICES
Name		Surface Area for RS
Trigger		REPORTING SERVICES
TYPE		Symmetric Key
Multipart Name		SYMMETRIC KEY
USER		Synonym
Name		SYNONYM
User		Table
USER DEFINED AGGREGATE		TABLE
User Defined Aggregate		Table Options
USER DEFINED DATA TYPE		TABLE
User Defined Data Type		Trigger
USER DEFINED FUNCTION		TRIGGER
Name		User
USER DEFINED TABLE TYPE		USER
User Defined Table Type		User Defined Aggregate
USER DEFINED TYPE		USER DEFINED AGGREGATE
Name		User Defined Data Type
User Defined Type		USER DEFINED DATA TYPE
VIEW		User Defined Function
Multipart Name		FUNCTION
Name		User Defined Table Type
View		USER DEFINED TABLE TYPE
View Options		User Defined Type
WINDOWS USER		USER DEFINED TYPE
User Options		User Options
WORKLOAD GROUP		ASYMMETRIC KEY USER
Workload Group		CERTIFICATE USER
XML SCHEMA COLLECTION		GROUP USER
Multipart Name		SQL USER
Name		WINDOWS USER
Xml Schema Collection		View
		VIEW
		View Options
		VIEW
		Workload Group
		WORKLOAD GROUP
		Xml Schema Collection
		XML SCHEMA COLLECTION

_____________

About the Author:
Dan Jones is a PM on the SQL Server Manageability team at Microsoft.

Creating A Policy With the GUI

sqlpbm — Sun, 27 Apr 2008 04:44:25 GMT

In the previous posting we introduced the main PBM concepts through the use of a sample policy: All stored procedures in the database must start with "usp_". In this posting we will walk through the creation of that policy using the PBM GUI accessible from SQL Server Management Studio (SSMS).

The first step in creating the policy is to create a condition. To do this we use Object Explorer to navigate to the Policy Management node.

Right-clicking on the Conditions folder we select New Condition... which launches the New Condition dialog.

First we give our Condition a name. Conditions can be reused by multiple policies so we want to give it a meaningful name. Since conditions define the desired state of the system we'll give it a name in the positive. Something like SP Name Prefix usp_ should do. Next we have to pick which facet we want to build our expression over. Remember, facets define two things for us: target types and properties. We have a choice of three facets: Stored Procedure, Multi-Part Name, and Name. For this example we'll pick the Multi-Part Name facet.

Now that we have our condition created we right-click on the Policies node in OE and select New Policy... We give our policy a name and select the condition we created above. We also could have started from the New Policy dialog and launched into the New Condition dialog.

We want to ensure no stored procedures are created on the system which violate the policy, therefore, we choose the On change: prevent evaluation mode.

Next we need to choose which targets the policies applies to. Because we built our condition on the multi-part name facet we see all of the targets which support that facet. We select Stored Procedure since that's the only one we care about for this example.

On the Description tab we can enter a description for the policy and a link (http, https, and mailto) for where to go for additional help.

After we enter this information click OK to save the policy. The policy and condition show up in Object Explorer.

Now we can attempt to create a stored procedure which violates our policy. In this example we create a nonsensical SP but the important point is the name, MyProcedure. Since our policy states names must start with 'usp_' our procedure should not be created. And that's exactly what happens. Let's take a closer look at the information we get back. First, the message tells us which policy we violated along with which object violated the policy. We also see the condition, policy description, and the link for additional help. All of this is useful information if the person is not familiar with the policy. The statement which violated the policy is included in the output. If this was a long SQL script this can help us trouble shoot the offending code.

This simple example showed you everything you need to get started using Policy-Based Management. Though our condition was straightforward they can get quite complex. In future postings we'll explore building more complex conditions in addition to some of the other advanced capabilities of Policy-Based Management.

_____________

About the Author:
Dan Jones is a PM on the SQL Server Manageability team at Microsoft.

Policy-Based Management Overview

sqlpbm — Sun, 20 Apr 2008 19:56:50 GMT

This posting provides an overview of Policy-Based Management (PBM). You can read more about PBM in SQL Server 2008 Books On-line. Many of the ad hoc problems DBAs experience are caused by unwanted configuration changes or configuration changes that don't comply with the "standard". Some shops have implemented custom consistency checks. On a routine schedule these consistency checks run against the production environment comparing specific configuration properties against the desired state. This information, however, is locked away in custom scripts which makes it more difficult to maintain and reuse.

PBM allows the DBA to declare the desired state of the system and check the system for compliance with that state. Think how group policy controls Windows system settings like password strength and expiration. PBM provides similar capabilities to the DBA.

Let's walk through an example: suppose the DBA needs to enforce a naming standard on stored procedures. In simple English the DBA states their intent as follows: All stored procedures in the database must start with "usp_" - ("usp" stands for user stored procedure). The picture below maps this policy to the main PBM concepts.

Starting from the bottom and working our way up the stack:

Target Type: These are entities to be managed by PBM. The list of entities should be familiar to a DBA. They include: database, table, view, login, user, stored procedure, etc. In our example the target type we're interested in is Stored Procedure.
Management Facet: A set of logical properties that model the behavior or characteristics for certain types of target types. Our policy will be written over the Multi-Part Name facet which contains two properties: Name and Schema. This facet models all schema-bound target types (tables, views, etc). In our example we only care about stored procedures.
Condition: A Boolean expression that specifies a set of allowed states of a target type with regard to a management facet. Said another way, the condition is stated as the desired state of the target type. We want the name of our stored procedures to start with usp_ - since underscore is a wildcard we need to escape it with brackets. The percent at the end wildcards the rest of the name which we don't care about.
Policy: A policy includes the condition and the expected behavior, for example, evaluation mode. We already built our condition, our expected behavior is only stored procedures that meet our condition can be created in the database. This translates to an evaluation mode of "Check on Change - Prevent"; meaning whenever a stored procedure is created or altered the transaction will be checked for compliance with the policy. If the transaction does not comply with the policy it will be blocked from completing. There are other supported evaluation modes which we'll explore in a future posting.

In this example we've introduced the key concepts in PBM: Target Types, Facets, Conditions, and Policies. In a future posting we'll look at the actual GUI in Management Studio for creating policies. We'll also explore advanced topics such as target filters and categories.

_____________

About the Author:
Dan Jones is a PM on the SQL Server Manageability team at Microsoft.

The DBA Morning Checklist

sqlpbm — Tue, 15 Apr 2008 03:20:00 GMT

Bill Richards recently wrote a nice article for SQL Server Central. You can read the original article here. The article covered a checklist DBAs should follow each morning. Buck Woody in his weekly podcast, Real World DBA Episode 5 - The DBA Checklist, covered the same topic.

Routinely checking that your systems are in compliance with your guidelines/policies is a critical component of every DBA's job. Often times these simple checks are enough to prevent a bad situation from turning even worse.

Manual checklists, though, are error prone. What happens when you're late getting into the office, or have an early morning meeting, or your on vacation? Any situation that alters your morning ritual increases the likelihood the checklist is missed for that day.

This is where Policy-Based Management can help. Most of the checks listed in these articles can be turned into an automated policy check. If the system is out of compliance with one or more policies it will write to the event log. If your shop is using a system management tools (MS System Center, IBM Tivoli, HP Openview, etc) you can monitor the logs for specific messages.

If your shop doesn't use a monitoring tool PBM can be configured to send email (using DB Mail and Agent Alerts) when it detects an out of compliance situation. I recommend setting up each policy in the daily checklist to run nightly. The out of compliance emails should go to a DBA distribution list rather than an individual. This way there are multiple eyes looking at violations - redundancy should be a golden rule.

Depending on the size of your environment the manual check list could take you 15-30 minutes each morning. At 15 minutes a day, you're spending 65 hours a year, assuming you check only on week days. That's a week and a half a year spent verifying everything is what it should be. I don't mean to imply this isn't time well spent. If it avoids a disastrous situation it's worth more than that. But I know I'd love to get back a week and a half a year.

With PBM you'll get this time back which you can spend on other valuable DBA activities. This sounds like the start of a compelling reason to upgrade to SQL Server 2008. Okay, that last bit sounded too much like marketing speak. Sorry!

In a future posting we'll turn a few of these checklist items into policies.

Cheers,

Dan