Windows Security Logging and Other Esoterica

I get a lot of questions about how ACS event retention works.  So here you go, I'm blogging it so I can just answer with a link :-)

There are two DWORD registry values which affect backlog transmission.  Both are on the collector machine under HKLM\System\CurrentControlSet\Services\AdtServer\Parameters.

EventRetentionPeriod, if present, is expressed in hours (I forget the default).  It takes precedence over MaximumEventAge, which is in days (default=1).  Both of these values control the backlog of events that will be sent from agents to the collector on agent connect, but as mentioned, EventRetentionPeriod wins any conflict.  MaximumEventAge used to control database retention in early beta builds but does not anymore, since the database moved to a partitioning mechanism.  You might encounter MaximumEventAge if you are migrating from ACS beta to Operations Manager 2007 ACS.

Grooming is now governed entirely by the grooming algorithm.  The grooming algorithm is simple: partitions will be deleted by the next grooming job as soon as they are eligible for deletion.

Eligible for deletion means:

• dtPartition.Status == 2 AND
• dtPartition.LastCreationTime < (now() - (partitionDuration * numPartitions))

Think of (partitionDuration * numPartitions) as the retention period before data is groomed from the database. 

• partitionDuration = dtConfig[5]
• numPartitions = dtConfig[6]

Note that dtPartition[<partitionId>].LastCreationTime defaults to 12:00am 1/1/2000 (collector local time).  After successful execution of the close partition script, this field’s value is set to max(dtEvent_<partitionId>.CreationTime) for the partition in question.  There is an implication here that if you update status to 2 without updating LastCreationTime, then the partition is immediately eligible for grooming assuming your clock is accurate.

The partition switch offset (time of day to switch partitions) value in dtConfig has no effect on grooming, other than that grooming will not occur during a partition switch.

Grooming runs at startup and immediately after checkpointing.  The default checkpoint interval is 198 seconds but this interval can be configured  by the DWORD registry value CheckPointInterval on the collector, in the same location as the other registry values.  A successful checkpoint logs an event in the database, event ID 0 with a source of “_acs” (you might have seen these on an “idle” ACS and wondered how they got there…)

We got several reports recently of a bug in ACS that certain DS Access events, primarily for dnsNode and dnsZone objects, don't properly get looked up.

Some background: the event log in Windows prefers to log invariants such as message IDs, parameter message IDs, SIDs (security IDs which represent users and groups, etc.), and GUIDs (globally unique IDs which represent objects in Active Directory), rather than the actual names of the objects.  At view time the viewing application is expected to look up the name associated with the invariant and display it to the user.

The reasons that Windows does this are (1) that it enables localization, so that English speakers can see "Administrator" and French speakers can see "Administrateur", and (2) that it provides rename safefy- many objects are rename-able, such as domain accounts and other AD objects.

Anyway in ACS we had to solve the problem of how to store mountains of log data in a database, make it queryable in meaningful ways, preserve original format, present to users in a recognizable/understandable format, etc.

The way we chose to solve our several problems was to take strings that contained an invariant and append the translated name or string.

For example:
%{e0fa1e8c-9b45-11d0-afdd-00c04fd930c9}
would be translated to:
%{e0fa1e8c-9b45-11d0-afdd-00c04fd930c9}=”dnsNode”

and
%%7685
becomes:
%%7685=”Write Property”

As I mentioned, though, we ran into a problem recently.  Some of our customers were monitoring AD objects with ACS and noticed that ACS was not translating the GUIDs for certain objects.  When they manually looked up the GUIDs they noticed that they were for AD-integrated DNS objects.

After investigation, we found that AD was logging certain audit events for the objects, before all the attributes of the objects had been populated- DNS was populating the objects in multiple operations per object and each operation causes a separate event.  So ACS, which operates as close to real-time as we could get, was actually noticing the first event and asking AD "what's this?" before DNS had finished updating AD with things like the object's name.  The difference in time was literally only milliseconds.

Anyway I didn't really feel it was an ACS bug and wanted to file a bug against Windows DNS Server.  However the Operations Manager team has prototyped a configurable behavior for the ACS agent that lets it wait a very short time (configurable number of ms) and retry, when it fails to look up an AD object because the object doesn't exist.  This might be released as a public patch and/or in a future Service Pack.

I thought you might appreciate stories of the kinds of weirdness we run into.

A judge in New Zealand declined to convict the admitted (guilty plea) botherder of a million-bot botnet, citing the negative consequences a conviction would have on the young man's future prospects.  See the story here.

Well duh.  The whole theory of crime and punishment is that if you do something bad, you get punished, and punishment is something that is unpleasant, so you try to avoid it, hopefully by not doing the crime.  See?  One would hope that a judge would understand this concept.

I could understand if the judge said "this is just a stupid kid, he doesn't deserve to do 20 years", and gave the kid probation, community service and a big fine.  I don't know if New Zealand has such options, or if the judge has latitude in sentencing.  There is probably more to the story than is being told.  But you don't take over a million computers that don't belong to you, personally making tens of thousands of dollars, and not realize that you're doing something wrong.  Unless you're a sociopath.  And in either case, you either need punishment (for doing something you know is wrong) or separation from society for the protection of society while you get treatment (if you are a sociopath).  So whatever the case, the judge got it wrong, and as a result is practically encouraging future behavior of the same sort.

If you haven't used wevtutil.exe to script event log tasks in Windows Vista or Windows Server 2008, you're missing out.  The new tool makes getting events out of the log pretty easy, but the main thing is that it doesn't suffer from any of the drawbacks around getting field delimiting correct.

The tool's command to query events from a log is "qe", and takes a log name as a parameter.

If you want to specify a query expression, then you can use XPath with the /q switch.  The easiest way to do this is to use Event Viewer to build a filter for just the events that you want, and then copy just the XPath expression out of the XML tab of the filter dialog in Event Viewer.  Be careful to copy only the filter expression and not the XML that surrounds it. 

Finally, the default output format of wevtutil is XML.  However it dumps each event as XML, but does not include a root element- in other words it's not well-formed XML by default.  To include a root element you need to include the /e switch and a root element name.

I put this all together in a batch file, with an example XPath filter that just gathers interactive logon events (event ID=4624, logon type=2).  You can save this as a .cmd file and run it as an administrator on Vista or WS08 and it will pull up a list of your interactive logons in Internet Explorer (or your default XML handler application if you've changed the registration).  It has to run as admin because it accesses the security event log.

If you're really good (better than me, which is not hard) you could write an XSL style sheet and put this into a report format.

Good luck!

I often talk about Ned, who is the current subject matter expert in Microsoft product support for the auditing feature in the US (Fadi is your guy in the Middle East and we have a couple of guys in Europe).  Well, Ned has a blog and I thought I'd point you guys there.  His recent posts on auditing include a description of how to deploy the special groups logon auditing feature with group policy.

Fadi, Ned and Brian of the auditing team have documented all the auditing events by audit policy category and subcategory for your reference.

Check it out in the Knowledge Base.

Even better, they documented all the events in spreadsheet format, and that's propagating to the Microsoft Download Center.  I'll publish the link when it's online.

2008-04-17 UPDATE:  Brian just sent me the link: here is the spreadsheet.

There's one topic that I know is on everyone's mind- no, not American Idol- it's "What's new in Auditing in Windows Server 2008?"

Well, funny that you brought that up.  My friend Jesper Johanssen just wrote a new book, the Windows Server 2008 Security Resource Kit, and he invited me to write a chapter about auditing for it, which I did.  So you, dear reader, are getting information straight from the horse's mouth, so to speak.

Anyway I think the book hits store shelves on March the 10th.  A number of distinguished individuals contributed to the book: Susan Bradley, Darren Canavor, Kurt Dillard, Roger Grimes, Brian Komar, Alun Jones and others.

I'd also like to send out special props to my auditing posse: Raghu (who was the primary developer for auditing for Vista & WS08) and Ned (who is the resident guru for auditing in Microsoft Customer Support Services), both of whom made significant contributions.  Raghu introduces the new "special group logon tracking" feature, and Ned contributed a spreadsheet mapping all the events (360-ish) to the policy category and subcategory and giving other key information about each event; this is included on the CD bundled with the book, along with an XML file defining the schema for all the events and event messages.  Ned's also working on getting a version of the spreadsheet available for download from the Microsoft download site.

In other news, the Windows Server 2008 Security Guide is also out, and yes, yours truly contributed in small part to the auditing guidance in there too, although I seem to have been overlooked in the credits (in all fairness my work delta from the Vista Security Guide was really small so maybe it did not meet their "credits bar").

Anyway, download the security guide and buy a copy of the book.  Buy more than one copy of the book, and give copies to your friends and loved ones.  Nothing says "Happy Anniversary, Honey" quite like a book or white paper about computer security.  OK, so maybe I should stick to computer security and stay away from relationship advice.  Flowers work well in my experience.

I've decided to start dumping my knowledge of ACS for posterity's sake.  My first installment is here, and it's an excerpt from an external email I put together which describes how event transformation works on ACS.

Transformation is performed on the agent (using instructions provided at connect time by the collector) and on the collector.  Transformation instructions are all stored on the collector in a file called EventSchema.xml which is in the AdtServer directory (%windir%\system32\security\adtserver).  This file is pointed to in the collector’s registry and is read during startup of the collector service; failure to successfully read and parse this file at startup is a fatal error for the collector (the debug log will complain about parsing).

The collector reads EventSchema.xml and builds in-memory binary tables of event transformation instructions and event string types by OS version/event log/event source.

The collector (as explained elsewhere) also reads AcsConfig.xml to get its persistent state and configuration for all known agents, to know what logs/sources to collect for each agent/agent group, etc.  This is all read into in-memory state for each agent.

At connect time, the agent sends version information- what the OS and agent version and service pack are, etc.  The collector first looks in its in-memory agent state to see what configuration applies to the agent.  Then it looks in its transformation tables and extracts the appropriate version-specific transformation instructions for the events that the collector is configured to collect from that agent.  Then it packages these instructions and sends them to the agent.

The agent starts reading events, transforming them according to its instructions from the collector, and sending the transformed events to the collector.  The collector finishes the transformation, services real-time subscriptions and loads the events into the database as appropriate.

If the agent encounters an event that is it configured to send (by log/source) but does not have transformation instructions for, then it simply builds a copy the event string for string and sends the copy of the event to the collector as an “unschematized” event.  The collector will handle this event without problems but will not extract non-header user fields (no primary/client/target user fields) and will not add string type information.

I’ll take Windows Server 2003 (build 3790), Event Log: Security, Event Source: Security, Event ID: 644 as an example.