Windows Security Logging and Other Esoterica

Farewell for now...

2012-06-11T05:44:09Z

I have resigned from Microsoft and am moving to another company. I hope my blog has been helpful to you all!

Feel free to contact me at my hotmail address (eric_fitzgerald). I'll be up and blogging somewhere else soon.

Best regards,

Eric

Off Topic: Unicode Right-to-Left Override character used by malware

2011-08-22T20:00:00Z

Here's an interesting thing for you security types to be aware of. Many of you probably are careful to screen attachment types to make sure that you don't unintentionally execute code that might be malicious.

Malware authors have discovered that by embedding a unicode control character in file names, they can cause the file name to read right-to-left (instead of the normal English left-to-right) and therefore obfuscate file extensions.

For example, "innocuous_cod.exe" could have the RLO character inserted after the underscore, and then it would read as "innocuous_exe.doc" (everything after the "_" is read right-to-left).

Here's a write-up with links to detected variants: http://blog.commtouch.com/cafe/malware/exe-read-backwards-spells-malware/

An interesting logging regulation that doesn't apply to Windows event logs...

2011-05-27T17:08:00Z

I was browsing around looking for logging regulations and stumbled across this. It's the United State's federal regulation on EDRs - Event Data Recorders - installed in automobiles.

EDRs are little log engines, like the "black box" flight data recorders on commercial airliners. They are typically part of the airbag system on an automobile. They record specific data about the operation of the vehicle including speed, etc., and they record incident data like airbag sensor activation or airbag deployment.

Anyway, if you want to see an example of a really different kind of logging, the PDF document of NHTSA rule 2006-25666 describes what data must be logged, and how often (sampling rate), and retention policy.

Sample logs are available here.

Decoding UAC Flags Values in events 4720, 4738, 4741, and 4742

2011-04-28T18:19:12Z

In Windows Vista, Windows Server 2008, Windows 7, and Windows Server 2008 R2, there are four events that contain a user account control (UAC) flags value:

4720 - user account creation
4738 - user account change
4741 - computer account creation
4742 - computer account change

This value is a bitmask value, and it's represented in textual format as a hexadecimal value, e.g. 0x1234.

The "decoder key" for this value is in Knowledge Base article 305144. If you're a developer type, the actual declaration is in IADS.H in the Windows SDK.

Ned points out that the article is missing an entry:

0x04000000 - PARTIAL_SECRETS_ACCOUNT (i.e. "Read-Only Domain Controller")

I also want to point out that Windows will set the undeclared value 0x4. I don't know what this value does, if anything.

To decode this value, you can go through the property value definitions in the KB article from largest to smallest. Compare each property value to the flags value in the event. If the flags value in the event is greater than or equal to the property value, then the property is "set" and applies to that event. Subtract the property value from the flags value in the event and note that the flag applies and then go on to the next flag. Here's an example:

Flags value from event: 0x15

Decoding:

PASSWD_NOTREQD 0x0020
LOCKOUT 0x0010
HOMEDIR_REQUIRED 0x0008
(undeclared) 0x0004
ACCOUNTDISABLE 0x0002
SCRIPT 0x0001

0x0020 > 0x15, so PASSWD_NOTREQD does not apply to this event

0x10 < 0x15, so LOCKOUT applies to this event. 0x15 - 0x10 = 0x5

0x4 < 0x5, so the undeclared value is set. We'll pretend it doesn't mean anything. 0x5 - 0x4 = 0x1

0x2 > 0x1, so ACCOUNTDISABLE does not apply to this event

0x1 = 0x1, so SCRIPT applies to this event. 0x1 - 0x1 = 0x0, we're done.

So this UAC flags value decodes to: LOCKOUT and SCRIPT.

Auditing Changes to Audit Policy

2010-07-16T20:21:46Z

Mitsuru, one of our support engineers in Japan, actually did some excellent research recently into exactly what our behavior is for auditing audit policy and I wanted to share that with you.

In Windows, we've always had auditing for changes to security policy. Audit policy has always been one aspect of that policy.

However, it's not so clear how to audit changes to audit policy. The reason is, because the change itself might affect whether or not the audit is generated. Usually in Windows, we generate audit after the operation that we are auditing, is performed. When we generate audit, we always check audit policy to see if we need to generate an event.

So what would happen if you turned off the setting "audit changes to audit policy"? Well, if we implemented it in the way we generally implement audit policy, nothing would happen- no event. As described above, if we checked audit policy after we disabled audit policy, then the effective policy would say "don't generate audit".

But consider the case where a malicious audit or system administrator wants to cover their tracks. One thing such a person might do, to not leave as much of a trace, is to disable audit policy before they do the bad thing, and re-enable it afterwards. If we implemented audit normally, then there would be no trace of this.

To avoid this undesirable case, we changed around the instrumentation a little so that we always generate audit for certain audit policy change events. This means that you might not get EXACTLY what you intended, but it also ensures that you can always find the significant events when someone disables audit policy.

Anyway, to sum up, the following events are always audited when audit policy is disabled regardless of the "Audit Policy Change" subcategory setting in Windows Vista+:

4715 The audit policy (SACL) on an object was changed.
4719 System audit policy was changed.
4906 The CrashOnAuditFail value has changed.
4908 Special Groups Logon table modified.
4912 Per User Audit Policy was changed.

The following events are only audited when success auditing is enabled for the "Audit Policy Change" subcategory:
4902 The Per-user audit policy table was created.
4904 An attempt was made to register a security event source.
4905 An attempt was made to unregister a security event source.
4907 Auditing settings on object were changed.

Special thanks to Mitsuru for documenting this.

XPath to generate a list of NTLM authentications on Windows Vista or Later

2010-05-13T22:51:00Z

Hi Everyone,

Sas sent me an email complaining that I am not posting as often as I should- sorry about that. I am working on a different project now but I am still in close touch with the auditing team and I'll try to do better.

Anyway a question that I hear regularly is, "how do I find all the NTLM authentications on my network"?

Other than running a network trace, the best way I have found (ok invented :-) to do this is to look at the logon events in the audit log.

One of the changes we made to the logon events in Windows Vista (and therefore subsequent releases of Windows) was to include the NTLM protocol level in the logon events, if the NTLM auth package was used.

Now, with the new EventLog ecosystem, it's easy to generate some XPath to find just these events.

Here's the query:

*[System

[Provider

[@Name='Microsoft-Windows-Security-Auditing']

and Task = 12544

and (band(Keywords,9007199254740992))

and (EventID=4624)

]

and

EventData

[Data

[@Name='LmPackageName'] != '-'

]

To use this in Event Viewer:

Find the Security log under Windows Logs in the tree pane.
Right-click the Security log, and choose "Filter Current Log..."
Select the "XML" tab.
Check the "Edit query manually" box.
Replace the default query ("*", or everything in the "<Select>" element), with the text in the box above. I've formatted it for readability.
Click OK

The event view will now be filtered and you'll only see NTLM logon events. Additionally, each filtered event will contain a "Detailed Authentication Information" section containing the protocol level (e.g. LM, NTLM, NTLM V2) in the "Package Name" field, and the session key length, if one was negotiated.

Detailed Authentication Information:
            Logon Process: NtLmSsp
            Authentication Package: NTLM
            Transited Services: -
            Package Name (NTLM only): NTLM V2
            Key Length: 128

Auditing system impact on performance

2009-08-10T17:26:00Z

UPDATE 2010-06-06 (EricF) - Fixed Vista+ architecture image; link was broken on migration to new blog platform

I get questions from time to time, such as my recent offline question from Steve, about what performance impact auditing has on the system as a whole.

To answer this you need to understand a couple of things:

Auditable activity is implemented as instrumentation (e.g. a function call to the auditing system) inside the code that does something auditable.
The auditing system in Windows has two sets of programmatic interfaces for introducing an event, one in kernel mode and one in user mode - so the component generating audit does not need to switch between kernel and user modes.
Although audit policy is stored in user mode, we cache a copy of the relevant policy for kernel-mode components, in kernel mode. This means that no mode switch is necessary to check audit policy to decide whether to generate an event.
There are user-mode and kernel-mode queues for audit events. The call to generate an audit event actually just queues the event, assuming the queue is not full. So from the perspective of the component generating audit, audit has an "asynchronous" flavor under light-medium loads. Under heavy loads when the queues fill, audit blocks the component raising the audit until the event can be queued, showing its true synchronous behavior.
Dequeuing audit events always occurs on a separate thread than enqueueing so that raising audit events and writing them to the log don't affect each other's perf under light to moderate load.
The pre-Vista auditing system in the kernel delivers events to LSA. The Vista+ auditing system in the kernel delivers most events directly to ETW, the kernel mode event trace engine, which means that most of the kernel audit (including the potentially perf impacting object access events) doesn't require a mode switch at all.
The LSA formats events and then delivers them to the event log. In WS03, events are batched in the RPC call to eventlog. In Vista+, delivery is done by means of ETW in almost all cases.
ETW queues events and spools them to the Windows eventlog service as fast as the service will accept them.
The eventlog service writes the events to the log file as they arrive.

I have uploaded graphics of the Windows XP/Windows Server 2003 auditing architecture, and the Windows Vista/WS08/Windows 7 architecture, to make this process more clear:

Windows Vista+ Auditing Architecture

So now back to the original question- what is the impact of auditing on performance?

At low auditing loads, auditing generally has no discernable impact on perf. If you were hardcode with a profiler and iterated an auditable activity a million times I am sure you'd be able to measure it, but for reasonable audit policies you won't notice a significant difference.

At high auditing loads, auditing has a significant performance impact. This is more true of pre-Vista multiprocessor systems than of systems with the new eventlog system.

For example, a multi-processor domain controller (say a 32-processor box) running Windows Server 2003, might run into problems under extreme load. Why is that? Because ultimately the limiting factor on event rate is how fast you can write the events to disk. Pre-Vista eventlog has a single thread writing events to disk. So even though you might have 32 threads servicing authentication requests (an auditable activity), each of them is queueing to a single audit queue which is ultimately despooling to eventlog via RPC on a single thread, and eventlog is only writing to the security log with a single thread. What we observe in practice in this case is that a single processor on the system goes to 85-100% utilization, and the other processors drop to a very low utilization as the authentication threads are blocked waiting for the audit function call to return. This call won't return until the queue is not full, and the queue is waiting on RPC which is waiting on eventlog... so eventlog governs the rate.

In Windows Server 2003, we added a particular optimization only for the security event log, which batches events in the RPC call to eventlog. This means that you can get more event throughput in the security log than in other logs on the system. It didn't eliminate the bottleneck, but it pushed back the limit, so WS03 on typical hardware should be able to log several thousand events per second to the security event log. Previous versions were only able to log about 1000 events per second.

Note that the change in performance characteristics occurs all at once. So the impact tends to be trivial until the queues fill, at which point the impact is severe. It does not scale linearly, there's a discrete behavior change. What this means realistically is that if you ever encounter a performance problem with auditing, then you probably just need to turn it down a little and you won't have a problem any more.

In Vista and subsequent releases, audit queues events via ETW. ETW was designed for high-performance kernel tracing, and in the auditing team we tested it to over 10,000 (10.000 for you folks in Germany :-) events per second before we decided that we had hit our scale targets. We never tested exactly how high it would go, but we were satisfied that the eventlog service was no longer a bottleneck in realistic scenarios.

There are some edge cases where you might run into performance problems by trying to audit too much in a critical path. For instance, it is a really really bad idea to put SACLs on your entire registry. If you monitor registry activity with a tool like Process Monitor, you will notice that when a system is not idle, there are often hundreds or thousands of registry accesses per second. If you impose an auditing tax on each of those activities you will notice a degradation in performance. Not to mention that the resultant mountain of events is probably not very valuable. Of course you can tune SACLs as I have mentioned before, but I doubt that it's useful to take the time to tune SACLs for the entire registry.

One last point is that the eventlog is writing the events somewhere. Wherever it is writing events, it is consuming disk I/Os and competing with anything else writing to the same volume. If you have a disk performance problem on that disk, it can result in an auditing performance problem, as everything else will back up if the eventlog can't write events to disk fast enough. So one thing you can do is ensure that the disk where your log is placed has enough I/Os.

In summary audit has very minimal impact unless you do a whole lot of it, in which case it can have severe impact on your system. The change happens suddenly, not gradually, so you can do a lot of auditing with no problem. If you run into a problem, turn it down just a little (or little by little) and at some point the behavior will change such that you won't have any significant perf impact anymore.

Mapping pre-Vista Security Event IDs to Security Event IDs in Vista+

2009-06-11T02:09:00Z

I've written twice (here and here) about the relationship between the "old" event IDs (5xx-6xx) in WS03 and earlier versions of Windows, and between the "new" security event IDs (4xxx-5xxx) in Vista and beyond.

In short, EventID(WS03) + 4096 = EventID(WS08) for almost all security events in WS03.

The exceptions are the logon events. The logon success events (540, 528) were collapsed into a single event 4624 (=528 + 4096). The logon failure events (529-537, 539) were collapsed into a single event 4625 (=529+4096).

Other than that, there are cases where old events were deprecated (IPsec IIRC), and there are cases where new events were added (DS Change). These are all new instrumentation and there is no “mapping” possible- e.g. the new DS Change audit events are complementary to the old DS Access events; they record something different than the old events so you can’t say that the old event xxx = the new event yyy because they aren’t equivalent. The old event means one thing and the new event means another thing; they represent different points of instrumentation in the OS, not just formatting changes in the event representation in the log.

Of course I explained earlier why we renumbered the events, and (in the same place) why the difference is "+4096" instead of something more human-friendly like "+1000". The bottom line is that the event schema is different, so by changing the event IDs (and not re-using any), we force existing automation to be updated rather than just misinterpreting events when the automation doesn't know the version of Windows that produced the event. We realized it would be painful but it is nowhere near as painful as if every event consumer had to be aware of, and have special casing for, pre-Vista events and post-Vista events with the same IDs but different schema.

So if you happen to know the pre-Vista security events, then you can quickly translate your existing knowledge to Vista by adding 4000, adding 100, and subtracting 4. You can do this in your head.

However if you're trying to implement some automation, you should avoid trying to make a chart with "<Vista" and ">=Vista" columns of event ID numbers, because this will likely result in mis-parsing one set of events, and because you'll find it frustrating that there is not a 1:1 mapping (and in some cases no mapping at all).

Eric

Minimizing Directory Service Audit Event Noise

2008-09-05T00:51:00Z

I've written before on noise reduction in the Windows security event log. I've also written to describe how object access auditing works. But, I still get questions on how to reduce noise from object access events. The other day I got that question, specific to Directory Service objects, on an internal discussion list so I thought I'd clean up the answer a bit and share it with the world. In general the same is true for any type of object, although there are a few more knobs to control for DS objects.

Object access audit is generated when the system access control list (SACL) on the object matches the access that was performed on ALL of the following conditions:

Object - the object that was accessed must have either an explicit or inherited SACL. The access performed is compared against the ACEs in that SACL.
Success or failure of activity - every audit access control entry (ACE) in a SACL will be either of type AUDIT_SUCCESS or AUDIT_FAILURE. The access performed must match the access type of the ACE for the rest of the ACE to be considered.
User account - the accessing user's token is compared against each ACE matching the access type. If the user, or a group the user belongs to, matches the SID in the ACE, then an audit might be generated.
Access - the access being performed must match the audited accesses in the access mask in an otherwise matching ACE.

The specific auditing algorithm is discussed here.

So the way to reduce the number of audit events (566 on Windows Server 2003, 4662 on Windows Server 2008, or one of the new DS Change events on Windows Server 2008) is to cause one or more of those conditions to fail, except in the specific cases that you care about.

The SACL which will generate the most audit events is "Everyone:Success & Failure:All accesses" on the domain head with OI,CI (object inherit & container inherit flags) for all object types. This SACL matches all of the above conditions in all cases. (Incidentally I think that this is pretty close to the default SACL- with the exception of failures- for Windows 2000 Active Directory installations, and SACLs are not updated when DCs are upgraded from version to version. Windows Server 2003 has much more conservative SACLs for new installations of AD.)

To reduce noise, I offer the following suggestions, addressing each of the above conditions:

Audit only the objects that you care about. User accounts and groups already are well-audited with "Account Management" auditing, so don't audit them with DS access. Perhaps audit OUs, or other DS objects. Use the Object Type and attribute type restrictions that you have in DS Access auditing. Also, in Windows Server 2008, you can affect auditing on a per-object basis by adjusting the SearchFlags attribute in the AD schema for the object. SACLs are more easily reversed so are probably a more acceptable method of controlling audit for most organizations.
Audit successful accesses only. Failed accesses are common and are NOT indicative of any security problem; in fact many failures are not even explicit requests by the user but are just normal requests made by the OS, and the OS will re-try with less access if the operation fails. In my experience failure auditing is primarily useful for troubleshooting, not for security.
Audit the "Everyone" group. Although this matches any user, you will not accidentally miss any accesses that you care about due to failing to audit a user account who has access to the objects in question. The only time that you would NOT audit "Everyone" is if you had an application or service account which was very noisy; in that case you'd need to create a group with all accounts EXCEPT the noisy accounts, and audit that group.
Audit only the accesses that you care about. Specifically, read accesses occur much more often (in my experience, a conservative estimate is about a 100:1 ratio) than write accesses. If you restrict your auditing to "write" type accesses (including change, delete, change permissions, create, etc.) then you will end up generating far fewer events. Auditing for read access is very noisy. If you must audit for reads, consider auditing fewer objects, perhaps only auditing reads on the container object instead of the objects in the container, or on one "interesting" object in any given container as a "canary".

Tracking User Logon Activity Using Logon Events

2008-08-21T02:32:00Z

I get the question fairly often, how to use the logon events in the audit log to track how long a user was using their computer and when they logged off.

As I have written about previously, this method of user activity tracking is unreliable. It works in trivial cases (e.g. single machine where the user doesn't have physical access to the power switch or power cord), and it works most of the time in simple cases where there is good network connectivy and the user is not trying to evade detection. If the user has physical access to the machine-- for example, can pull out the network or power cables or push the reset button-- and if the user is actively trying to evade time tracking, then the only reliable solution is to surreptitiously put a video camera (subject to local laws) in a place that can monitor the user's presence in front of the keyboard (yes I am aware of research done to track sound of keyboard clicks, etc.).

There is no way to instrument the OS to account for someone who just backs away from the keyboard and walks away. The screen saver, if configured, will come on after a configurable delay since the last keypress or mouse movement. Yes, if you know the SS delay then you could just work that into your calculations. However the workstation does not lock until the screen saver is dismissed (some of you might have noticed that when you bump the mouse to dismiss the screensaver, sometimes you see your desktop for a fraction of a second- that’s because your machine isn’t locked while the screen saver is being displayed). And the events don't tell you whether the workstation was locked or auto-locked so you don't really know whether to add in the screen saver delay factor. Plus, prior to Windows Vista, there is no workstation lock event at all, only an unlock event, which is constructed in a way which makes it difficult to correlate with the original logon event.

So the bottom line is, I don't advocate or recommend this method for tracking the time a user spends at the keyboard. If I were hypothetically called as an expert witness, I would testify that such a method is unreliable and trivially circumvented. You have been warned, I've beaten that dead horse enough I guess.

Given that you are disregarding all my contrary advice, how are you going to accomplish this?

First, we need a general algorithm.

Use time (for a given logon session) = Logoff time - logon time

Now, what about the cases where the user powers off the machine, or it bluescreens, or a token leak prevents the logoff event from being generated, etc.? We can use the BEGIN_LOGOFF event to handle token leak cases. We can use the shutdown event in cases where the user does not log off. And in case of crashes, the only event we can use is the startup event. Note that each of these introduces increasing levels of uncertainty.

Logoff time = (logoff time | begin_logoff time | shutdown time | startup time)

This is good, but what about the time the workstation was locked?

Workstation lock time = unlock time - lock time
Total workstation lock time (for a given logon session) = SUM(workstation lock time)

How about remote desktop & terminal server sessions, and fast user switching? You can connect and disconnect from logon sessions, during which time the user technically isn't using the computer.

Session idle time = session connect time - session disconnect time
Total session idle time (for a given logon session) = SUM(session idle time)

How about times when the machine was idle? We can estimate that by looking at the time the screen saver was in place and adding the screen saver timeout.

Console idle time = (screen saver dismiss time - screen saver invoke time + screen saver delay)
Total console idle time = SUM(console idle time)

Putting all of this together and modifying our original formula, we get:

Use time (for a given logon session) =
   Logoff time - logon time
      - SUM(workstation lock time)
      - SUM(session idle time)
      - SUM(console idle time)

When we expand it, it is not quite so pretty:

Use time (for a given logon session) =
   ( (logoff time | begin_logoff time | shutdown time | startup time) - logon time )
      - SUM(unlock time - lock time)
      - SUM(session connect time - session disconnect time)
      - SUM(screen saver dismiss time - screen saver invoke time + screen saver delay)

You have to be very careful that you only look at events that are properly contained chronologically between two other appropriate events, to avoid accidentally pairing the wrong logon and logoff events, or pairing a lock workstation event from one logon session with a different logon session. The best correlation field is the Logon ID field, the next best are timestamp and user name. At various times you need to examine all of these fields.

Now, which event IDs correspond to all of these real-world events?

They are all found in the Security event log. The pre-Vista events (ID=5xx) all have event source=Security. The Vista/WS08 events (ID=4xxx) all have event source=Microsoft-Windows-Security-Auditing.

512 / 4608 STARTUP
513 / 4609 SHUTDOWN
528 / 4624 LOGON
538 / 4634 LOGOFF
551 / 4647 BEGIN_LOGOFF
N/A / 4778 SESSION_RECONNECTED
N/A / 4779 SESSION_DISCONNECTED
N/A / 4800 WORKSTATION_LOCKED
* / 4801 WORKSTATION_UNLOCKED
N/A / 4802 SCREENSAVER_INVOKED
N/A / 4803 SCREENSAVER_DISMISSED

* prior to Windows Vista, there was no event for locking the workstation. Unlocking the workstation generated a pair of events, a logon event and a logoff event (528/538) with logon type 7. These events had the same user name as the "original" logon session and were completely enclosed chronologically by the logon/logoff events for the "real" logon session, but did not contain the Logon ID of the original logon session or other unambiguous correlator. This makes correlation of these events difficult.

All of these events are generated in the Logon/Logoff audit policy category, although on Windows Vista and Windows Server 2008 they are scattered among the various subcategories in this audit policy category. The audit event spreadsheet that Ned wrote has all the policy subcategory mappings as well as the event descriptions.

Sorry that this is more of a do-it-yourself than a solution-in-a-box, but this is pretty difficult to script and so far I haven't worked on a project that required this.

Eric

ACS Event Retention Mechanism

2008-07-18T01:56:00Z

I get a lot of questions about how ACS event retention works. So here you go, I'm blogging it so I can just answer with a link :-)

There are two DWORD registry values which affect backlog transmission. Both are on the collector machine under HKLM\System\CurrentControlSet\Services\AdtServer\Parameters.

EventRetentionPeriod, if present, is expressed in hours (I forget the default). It takes precedence over MaximumEventAge, which is in days (default=1). Both of these values control the backlog of events that will be sent from agents to the collector on agent connect, but as mentioned, EventRetentionPeriod wins any conflict. MaximumEventAge used to control database retention in early beta builds but does not anymore, since the database moved to a partitioning mechanism. You might encounter MaximumEventAge if you are migrating from ACS beta to Operations Manager 2007 ACS.

Grooming is now governed entirely by the grooming algorithm. The grooming algorithm is simple: partitions will be deleted by the next grooming job as soon as they are eligible for deletion.

Eligible for deletion means:

dtPartition.Status == 2 AND
dtPartition.LastCreationTime < (now() - (partitionDuration * numPartitions))

Think of (partitionDuration * numPartitions) as the retention period before data is groomed from the database.

partitionDuration = dtConfig[5]
numPartitions = dtConfig[6]

Note that dtPartition[<partitionId>].LastCreationTime defaults to 12:00am 1/1/2000 (collector local time). After successful execution of the close partition script, this field’s value is set to max(dtEvent_<partitionId>.CreationTime) for the partition in question. There is an implication here that if you update status to 2 without updating LastCreationTime, then the partition is immediately eligible for grooming assuming your clock is accurate.

The partition switch offset (time of day to switch partitions) value in dtConfig has no effect on grooming, other than that grooming will not occur during a partition switch.

Grooming runs at startup and immediately after checkpointing. The default checkpoint interval is 198 seconds but this interval can be configured by the DWORD registry value CheckPointInterval on the collector, in the same location as the other registry values. A successful checkpoint logs an event in the database, event ID 0 with a source of “_acs” (you might have seen these on an “idle” ACS and wondered how they got there…)

ACS' first bug from being too performant

2008-07-16T20:48:00Z

We got several reports recently of a bug in ACS that certain DS Access events, primarily for dnsNode and dnsZone objects, don't properly get looked up.

Some background: the event log in Windows prefers to log invariants such as message IDs, parameter message IDs, SIDs (security IDs which represent users and groups, etc.), and GUIDs (globally unique IDs which represent objects in Active Directory), rather than the actual names of the objects. At view time the viewing application is expected to look up the name associated with the invariant and display it to the user.

The reasons that Windows does this are (1) that it enables localization, so that English speakers can see "Administrator" and French speakers can see "Administrateur", and (2) that it provides rename safefy- many objects are rename-able, such as domain accounts and other AD objects.

Anyway in ACS we had to solve the problem of how to store mountains of log data in a database, make it queryable in meaningful ways, preserve original format, present to users in a recognizable/understandable format, etc.

The way we chose to solve our several problems was to take strings that contained an invariant and append the translated name or string.

For example:
%{e0fa1e8c-9b45-11d0-afdd-00c04fd930c9}
would be translated to:
%{e0fa1e8c-9b45-11d0-afdd-00c04fd930c9}=”dnsNode”

and
%%7685
becomes:
%%7685=”Write Property”

As I mentioned, though, we ran into a problem recently. Some of our customers were monitoring AD objects with ACS and noticed that ACS was not translating the GUIDs for certain objects. When they manually looked up the GUIDs they noticed that they were for AD-integrated DNS objects.

After investigation, we found that AD was logging certain audit events for the objects, before all the attributes of the objects had been populated- DNS was populating the objects in multiple operations per object and each operation causes a separate event. So ACS, which operates as close to real-time as we could get, was actually noticing the first event and asking AD "what's this?" before DNS had finished updating AD with things like the object's name. The difference in time was literally only milliseconds.

Anyway I didn't really feel it was an ACS bug and wanted to file a bug against Windows DNS Server. However the Operations Manager team has prototyped a configurable behavior for the ACS agent that lets it wait a very short time (configurable number of ms) and retry, when it fails to look up an AD object because the object doesn't exist. This might be released as a public patch and/or in a future Service Pack.

I thought you might appreciate stories of the kinds of weirdness we run into.

If you're gonna herd bots, do it from New Zealand!

2008-07-16T20:37:00Z

A judge in New Zealand declined to convict the admitted (guilty plea) botherder of a million-bot botnet, citing the negative consequences a conviction would have on the young man's future prospects. See the story here.

Well duh. The whole theory of crime and punishment is that if you do something bad, you get punished, and punishment is something that is unpleasant, so you try to avoid it, hopefully by not doing the crime. See? One would hope that a judge would understand this concept.

I could understand if the judge said "this is just a stupid kid, he doesn't deserve to do 20 years", and gave the kid probation, community service and a big fine. I don't know if New Zealand has such options, or if the judge has latitude in sentencing. There is probably more to the story than is being told. But you don't take over a million computers that don't belong to you, personally making tens of thousands of dollars, and not realize that you're doing something wrong. Unless you're a sociopath. And in either case, you either need punishment (for doing something you know is wrong) or separation from society for the protection of society while you get treatment (if you are a sociopath). So whatever the case, the judge got it wrong, and as a result is practically encouraging future behavior of the same sort.

WEvtUtil Scripting

2008-07-16T19:13:00Z

If you haven't used wevtutil.exe to script event log tasks in Windows Vista or Windows Server 2008, you're missing out. The new tool makes getting events out of the log pretty easy, but the main thing is that it doesn't suffer from any of the drawbacks around getting field delimiting correct.

The tool's command to query events from a log is "qe", and takes a log name as a parameter.

If you want to specify a query expression, then you can use XPath with the /q switch. The easiest way to do this is to use Event Viewer to build a filter for just the events that you want, and then copy just the XPath expression out of the XML tab of the filter dialog in Event Viewer. Be careful to copy only the filter expression and not the XML that surrounds it.

Finally, the default output format of wevtutil is XML. However it dumps each event as XML, but does not include a root element- in other words it's not well-formed XML by default. To include a root element you need to include the /e switch and a root element name.

I put this all together in a batch file, with an example XPath filter that just gathers interactive logon events (event ID=4624, logon type=2). You can save this as a .cmd file and run it as an administrator on Vista or WS08 and it will pull up a list of your interactive logons in Internet Explorer (or your default XML handler application if you've changed the registration). It has to run as admin because it accesses the security event log.

If you're really good (better than me, which is not hard) you could write an XSL style sheet and put this into a report format.

Good luck!

@echo off

set outputfile=%temp%\interactive-logon-events.xml

if "%1" NEQ "" set outputfile=%1

REM The next command is all one line and has no carriage returns

REM The only spaces in the XPath are around the AND keywords

wevtutil qe Security /q:"*[System[Provider[@Name='Microsoft-Windows-Security-Auditing'] and Task=12544 and (EventID=4624)] and EventData[Data[@Name='LogonType']='2']]" /e:Events > %outputfile%

start %outputfile%

set outputfile=

Ned on Auditing

2008-04-20T06:14:00Z

I often talk about Ned, who is the current subject matter expert in Microsoft product support for the auditing feature in the US (Fadi is your guy in the Middle East and we have a couple of guys in Europe). Well, Ned has a blog and I thought I'd point you guys there. His recent posts on auditing include a description of how to deploy the special groups logon auditing feature with group policy.

Windows Server 2008 Security Events Posted

2008-04-17T06:09:00Z

Fadi, Ned and Brian of the auditing team have documented all the auditing events by audit policy category and subcategory for your reference.

Check it out in the Knowledge Base.

Even better, they documented all the events in spreadsheet format, and that's propagating to the Microsoft Download Center. I'll publish the link when it's online.

2008-04-17 UPDATE: Brian just sent me the link: here is the spreadsheet.

2010-04-01 UPDATE: Here is the link to the updated spreadsheet for Windows 7 and Windows Server 2008 R2.

Shameless Self-Promotion

2008-03-05T23:47:00Z

There's one topic that I know is on everyone's mind- no, not American Idol- it's "What's new in Auditing in Windows Server 2008?"

Well, funny that you brought that up. My friend Jesper Johanssen just wrote a new book, the Windows Server 2008 Security Resource Kit, and he invited me to write a chapter about auditing for it, which I did. So you, dear reader, are getting information straight from the horse's mouth, so to speak.

Anyway I think the book hits store shelves on March the 10th. A number of distinguished individuals contributed to the book: Susan Bradley, Darren Canavor, Kurt Dillard, Roger Grimes, Brian Komar, Alun Jones and others.

I'd also like to send out special props to my auditing posse: Raghu (who was the primary developer for auditing for Vista & WS08) and Ned (who is the resident guru for auditing in Microsoft Customer Support Services), both of whom made significant contributions. Raghu introduces the new "special group logon tracking" feature, and Ned contributed a spreadsheet mapping all the events (360-ish) to the policy category and subcategory and giving other key information about each event; this is included on the CD bundled with the book, along with an XML file defining the schema for all the events and event messages. Ned's also working on getting a version of the spreadsheet available for download from the Microsoft download site.

In other news, the Windows Server 2008 Security Guide is also out, and yes, yours truly contributed in small part to the auditing guidance in there too, although I seem to have been overlooked in the credits (in all fairness my work delta from the Vista Security Guide was really small so maybe it did not meet their "credits bar").

Anyway, download the security guide and buy a copy of the book. Buy more than one copy of the book, and give copies to your friends and loved ones. Nothing says "Happy Anniversary, Honey" quite like a book or white paper about computer security. OK, so maybe I should stick to computer security and stay away from relationship advice. Flowers work well in my experience.

ACS Event Transformation Demystified

2008-02-28T04:43:00Z

I've decided to start dumping my knowledge of ACS for posterity's sake. My first installment is here, and it's an excerpt from an external email I put together which describes how event transformation works on ACS.

Transformation is performed on the agent (using instructions provided at connect time by the collector) and on the collector. Transformation instructions are all stored on the collector in a file called EventSchema.xml which is in the AdtServer directory (%windir%\system32\security\adtserver). This file is pointed to in the collector’s registry and is read during startup of the collector service; failure to successfully read and parse this file at startup is a fatal error for the collector (the debug log will complain about parsing).

The collector reads EventSchema.xml and builds in-memory binary tables of event transformation instructions and event string types by OS version/event log/event source.

The collector (as explained elsewhere) also reads AcsConfig.xml to get its persistent state and configuration for all known agents, to know what logs/sources to collect for each agent/agent group, etc. This is all read into in-memory state for each agent.

At connect time, the agent sends version information- what the OS and agent version and service pack are, etc. The collector first looks in its in-memory agent state to see what configuration applies to the agent. Then it looks in its transformation tables and extracts the appropriate version-specific transformation instructions for the events that the collector is configured to collect from that agent. Then it packages these instructions and sends them to the agent.

The agent starts reading events, transforming them according to its instructions from the collector, and sending the transformed events to the collector. The collector finishes the transformation, services real-time subscriptions and loads the events into the database as appropriate.

If the agent encounters an event that is it configured to send (by log/source) but does not have transformation instructions for, then it simply builds a copy the event string for string and sends the copy of the event to the collector as an “unschematized” event. The collector will handle this event without problems but will not extract non-header user fields (no primary/client/target user fields) and will not add string type information.

I’ll take Windows Server 2003 (build 3790), Event Log: Security, Event Source: Security, Event ID: 644 as an example.

Here’s the WS03 schema for 644 (excerpt from %systemroot%\system32\security\adtserver\EventSchema.xml in the path “Schema\Log[@Name=’Security’\Source[@Name=’Security’]\Version[@MinBuild=’3790’]\Event[@SourceId=’644’]”).

</Event>

The instructions are all applied in order. “Call” instructions are executed agent-side; “Param” instructions are executed server-side.

These instructions can be translated as:

· Take string 1 from the original event and make it string 1 in the new event. It is of type “typeUserDn”.

· Take string 3 from the original event and make it string 2 in the new event. It is of type “typeComputerName”. Note that we are doing reordering here by appending original string #3 before original string #2. Nifty, eh?

· Take string 2 from the original event and make it string 3 in the new event. It is of type “typeTargetSid”.

· Take string 4 from the original event and make it string 4 in the new event. It is of type “typeClientUser”.

· Take string 5 from the original event and make it string 5 in the new event. It is of type “typeClientDomain”.

· Take string 6 from the original event and make it string 6 in the new event. It is of type “typeClientLogonId”.

· Take string 4 from the original event and treat is as a user name, and take string 5 from the original event and treat it as a domain name, look up the associated SID and make it string 7 in the new event. The new string is of type “typeClientSid”.

· Take string 3 from the new event, treat it as a SID, look up the user/domain name associated with it and append the user name as string 8 to the new event and the domain name as string 9 to the new event. String 8 is of type “typeTargetUser” and String 9 is of type “typeTargetDomain”.

See the reordering? Now here is an instance of the event with the original event data. If you’re not familiar with the XML, it’s the XML output of Crimson, the new eventlog service introduced in Vista/WS08, but this is a WS03 [pre-Crimson] machine; we're looking at a saved event log (evt) file.

<Channel>C:\Users\ericf\AppData\Local\Temp\SERVER34_SecEvts.evt</Channel>

<Computer>SERVER34</Computer>

</System>

<Data>user09</Data> // String 1 – user name

<Data>SERVER34</Data> // String 2 – looks like a machine name, confirmed by string 4

<Data>%{S-1-5-21-5998314728-109421381-169156293-611111}</Data> // String 3 – definitely a SID

<Data>SERVER34$</Data> // String 4 – definitely an account name (machine account)

<Data>CONTOSO</Data> // String 5 – looks like a domain name

<Data>(0x0,0x3E7)</Data> // String 6 – definitely a logon ID

<Data>-</Data> // String 7 – empty null string at the end of the event (ignored by ACS)

</EventData>

</Event>

When the event arrives at the collector, type information is applied, and then the user fields (typePrimary*, typeClient*, typeTarget*) are extracted from the string data section and the strings that are left are re-numbered starting at 1 (no reordering occurs).

Here’s a chart of what the event looks like at the various points in the system. The changes at each step are shown in red.

Original Event in Event Log		Client-Side Transformation at Agent		Server-Side Normalization (WMI/SQL output)
Field	Content Description (implicit)	Field	Content Description (implicit)	Field	Content Description (explicit)
		Client User		Client User	typeClientUser
		Client Domain		Client Domain	typeClientDomain
		Client Sid		Client Sid	typeClientSid
		Client Login Id		Client Login Id	typeClientLogonId
		Target User		Target User	typeTargetUser
		Target Domain		Target Domain	typeTargetDomain
		Target Sid		Target Sid	typeTargetSid
String01	typeUserDn	String01	typeUserDn	String01	typeUserDn
String02	typeTargetSid	String02	typeComputerName	String02	typeComputerName
String03	typeComputerName	String03	typeTargetSid	String03
String04	typeClientUser	String04	typeClientUser	String04
String05	typeClientDomain	String05	typeClientDomain	String05
String06	typeClientLogonId	String06	typeClientLogonId	String06
String07		String07	typeClientSid	String07
String08		String08	typeTargetUser	String08
String09		String09	typeTargetDomain	String09

To finish off a description of transformation, there are 7 transformation functions, each of which can optionally take 2 integers as parameters. Note that there is no “destination event” field specifier; all references are only to the original event. That’s because when constructing the destination event, any data added to the event is always appended- it is constructed from beginning to end- so the implicit destination field is “at the end of the event as it is now”.

Function	Parameter 1	Parameter 2	Description
AppendString	Reference to a string parameter in the source event in the event log	Unused	Appends the referenced string to the event which will be sent to the collector
AppendStringFromTable	Reference to a constant string in the statically defined <Strings> table (1-based) in the relevant Source\Version element in EventSchema.xml	Unused	Appends the referenced constant string to the event which will be sent to the collector
AppendProcessNameFromPid	Reference to a string parameter in the source event in the event log (source string is expected to be a numeric process ID)	Unused	Looks up the process image path name for the referenced PID and appends it to the event which will be sent to the collector
AppendTimeFromDatetime	Unused	Unused	Not Implemented/No Action
AppendSidFromNames	Reference to a string parameter in the source event in the event log (source string is expected to be a user name)	Reference to a string parameter in the source event in the event log (source string is expected to be a domain name)	Looks up the SID for the account represented by the specified user and domain names, and appends the SID to the event which will be sent to the collector
AppendNamesFromSid	Reference to a string parameter in the source event in the event log (source string is expected to be a security ID)	Unused	Looks up the user name and domain name for the account represented by the specified SID, and appends the user name and the domain name as separate strings to the event which will be sent to the collector
AppendNumber	Unused	Unused	Not Implemented/No Action

Out of range params cause the transformation instruction to be ignored and skipped. Non-integer params or other XML formatting/malformation problem (including non-UTF8 formatting) cause an EventSchema.xml parsing error at collector startup which in turn causes collector startup failure.

So that’s ACS transformation in a nutshell. I hope this helps you guys understand ACS functionality a little better.

Shortly I will finish my write-up on AcsConfig.xml but that is a simple file and not too hard to figure out if you are into experimentation.

Here are some cool things that you can try with the event schema file if you are adventurous:

1. Drop fields. We have modified eventschema.xml successfully to cause it not to collect certain fields (e.g. logon GUIDs) of certain events:

// try deleting a line here

// or, to preserve ordering of subsequent strings

// try replacing “AppendString” with “AppendStringFromTable (param1=1)”

2. Add an event source. Some caveats are:

· You must have a unique, well-formed GUID for the new source

· You have to get events of the new source into the log (try “AuthzReportSecurityEvent” from MSDN)

· You have to modify AcsConfig.xml to tell the agent(s) to collect the new source

NB I have used the C/C++ comment syntax throughout this post but note that ACS does not support either C/C++ nor XML style comments in the XML config files it uses

You learn something new every day- Logon Type 0

2008-02-26T23:49:00Z

Today I encountered something new in the logon event- I thought that was old hat and I knew all there was to know about that but I guess I was wrong.

The logon event (528/540 prior to Windows Vista, 4624 in Vista and Windows Server 2008) has a field called a Logon Type. This is a code that is passed into the logon API that tells the authentication system in Windows which policy to check the logon against. Windows has separate policy checks for network logons, interactive logons, etc., so that you can allow users to access a system in some ways but not in others.

The logon type code is, in C/C++ parlance, an enumerated value- it's an ordered list of numeric values, each with an associated name, and these are defined in a publicly available file in the source code (ntsecapi.h). In the source code, the values are always referenced by name.

Today on one of the internal aliases someone actually found a logon event with a logon type of 0- I have never personally seen one of these before and 0 is not defined in the SECURITY_LOGON_TYPE enumeration, so I would have assumed that it was a bug- but it turns out that we are aware of this case and use it occasionally for system logons.

So there you are.

ACS Tidbits

2008-02-02T04:04:00Z

Well there has been a lot happening on my old project, ACS (Audit Collection Services, a feature of SystemCenter Operations Manager 2007).

Two more of our partners, Enterprise Certified and NetPro, have released compliance solutions on top of ACS.

Another of our partners with ACS-based compliance solutions, SecureVantage, has started a new blog where ACS is a frequent topic.

Anyway I'm pleased to see that ACS is becoming a successful platform and I'm happy to answer ACS questions! To you ISV's out there, Joseph and I welcome your questions as well (if we aren't already talking to you). Let us know who you are so we can stay in touch with you!

I always wondered who Björn was...

2008-01-18T01:47:00Z

OK here's something I just remembered today. I may be the last person who remembers this so it's important that I record this somewhere.

In the RTM bits of Windows NT 4.0, for the German language release only, someone snuck in a string resource into the auditing message file. I'm guessing that it was one of our localization engineers, but I don't know- I was over in the support side of things at the time. I stumbled across the message one day while looking at source code.

Here's Björn's momentous message: "Björn grüßt den rest der welt". Basically Björn says hi to everyone. He's a friendly guy.

This is string resource zero in the message table resource- it's not a code resource, it's properly formed and it's not used by the code anywhere. You would not know it exists unless you slog through source code (like me) or use a hex editor or string dumper to analyze binaries AND happen to be so bored that you pull out an NT 4.0 RTM German CD and examine msaudite.dll. NT4 RTM CD's are pretty rare, btw, because we replaced them with slipstream SP1 CD's very shortly after release.

If I remember correctly somebody else came along in a later service pack and changed Björn's name to their own (maybe it was Ulli? I can't remember and I'm too lazy to find the source- it requires a lot of effort to dig that far back). I do remember that shortly thereafter there was a huge Easter Egg crackdown here at Microsoft probably brought to a head by the Excel 97 Flight Simulator. Björn's message of goodwill to mankind was erased forever.

I did a search using the Officially Santioned Search Engine and the other one too; evidently the internet has forgotten Björn's message. But I still remember, Björn.

Anyway I thought you might like this bit of arcana. If you are bored, have a hex editor and a German NT4 CD, knock yourself out...

Why does Windows XP generate so many logon failure events?

2007-11-10T02:23:00Z

I got the question last week, why there are so many logon failure events on Windows XP when it is not domain joined.

The short answer is, by design. (Yes, bad design.)

The longer answer is that the shell team is working around the fact that there is no "tell me if this user account has a blank password" API.

When in a workgroup (not domain joined), Windows XP displays a welcome screen that has little pictures (called "tiles") for each user who is permitted to log on to the computer.

The shell team wanted the experience that when you click on a tile, that you will immediately be logged on if your password is blank (we have good data that a large percentage of home users have blank passwords). They only want you to be prompted for a password if you actually have a password. Fair enough, and it also helps with accessibility for people for whom typing is challenging.

The XP Welcome Screen, when it is initialized each time it is to be displayed, attempts to log on each user for which a tile will be displayed, using a blank password. Users with non-blank passwords will cause failures in this case (other users will cause logon success events followed by logoff success events). [2007-11-21 correction]

The Welcome Screen uses the result of these logon attempts to decide whether to display a password box when you select a user's tile. If the user has a blank password, they will be logged on instead of being prompted for a password.

Why are they logging on the account? Well it turns out to be the easiest way to tell if your password is blank. We don't have a "is your password blank" API- that would be a security disaster- and we would prefer that the shell team not go mucking about in the SAM, retrieving hashes and computing the blank password hash for each account so that it could compare them.

I asked for this behavior to be changed prior to XP's release. Specifically I asked that the blank password check be moved from Welcome screen initialization to tile selection- this would still cause logon failures but many fewer of them. I was declined. I asked for fixes to it in SP1 and SP2 and was declined. At this point we will not be revisiting this "feature"; the Welcome Screen was redesigned to eliminate this problem.

The shell team who designed the Welcome Screen did not feel that auditing was a common scenario for workgroup machines, and I didn't (and still don't) have any business case to dispute that.

List of Windows Server 2003 Events

2007-10-12T21:45:00Z

So a long time ago, back in my days of providing technical support for Windows NT 4.0, I published "Security Event Descriptions". This article was the "schema" so to speak, for the Windows NT 4.0 security event log events.

Technically Windows events are not schematized until Windows Vista; or put another way the schema is implicit based on the instrumentation in the code- since the event is raised by some function in the code, the "schema" could be interpreted as the parameter order in the call to that function.

Anyway security monitoring types love that article, but I hate it. It's just better than nothing. It doesn't state which events map to which audit policy categories. It does tell you whether the event is a succss or failure event but it doesn't alert you to the cases where the same event is used for success and failure (e.g. event 560).

When Windows 2000 came around and we added two new audit policy categories (DS Access and Account Logon [which was a huge naming blunder]), I wrote an article for the Windows 2000 security events. However it was so large I broke it into two articles.

I didn't write an article for Windows Server 2003. At first I didn't think it was necessary because we propagated all the WS03 events to the Technet Events & Errors Message Center web site. I wrote custom content for the top 30 or so events by volume of searches

(On a side note, did you ever wonder what happens when you click the "More Information" link at the bottom of the Event Viewer event description? We send the event source, event ID, OS version and so forth to the Technet E&E site and display the content that is returned. We count the number of hits for each OS Version/Source/Event ID combination and then our writing teams pester the component owners to populate that content.)

Anyway, I was making excu^h^h er, explaining why I didn't write the KB articles for Windows Server 2003 security events. So I thought the E&E message center would be all that anyone needed. It didn't strike me as that important that you had to have seen the event (or at least know it exists) before you could use the site. However since then I have received a large number of requests for the event definitions, mainly from people who were creating security event management solutions.

So here's what I have for you, courtesy of Ned, one of the audit log posse here at Microsoft. If you want a complete list of WS03 security events, then I suggest you look at chapter 4 of the Windows Server 2003 Security Guide. This documents the event IDs of all the security events on Windows Server 2003. Plus, it groups them by policy category, in case you ever wanted to know what you are in for if you enable one of the categories for audit. If you want the layout of the event (what data is in the description field, and in what order) then just look for that specific event on the Technet E&E site or click the link in the bottom of the event description in Event Viewer.

I've already described how the Vista and Windows Server 2008 (and subsequent releases) event systems are self-documenting, so I won't go into that further here.

One last tip: If you own Microsoft System Center Operations Manager 2007, then you can search for a file called EventSchema.xml on the media. It is an XML document that describes one possible normalization all the security events from Windows 2000 forward, and the semantic content of the normalized events.

2007-10-31 UPDATE: There is also an event-id-to-audit-policy-category map here.

German court bans retention of logged IP addresses

2007-10-03T20:53:00Z

A German court has ruled that a government web site may not retain IP addresses and other personally identifiable information (PII) in their logs for any longer than the user is actually using the site.

The judges pointed out that in many cases it was simple to map an IP address to an identity with the help of 3rd parties, and declared that logging IP addresses was a "violation of the right to informational self-determination."

OK whatever.

Germany does not seem to be of one mind regarding logging. On the one hand their draconian privacy laws (how's that for an oxymoron?) are pretty much in opposition to any meaningful user activity logging. On the other hand, their law enforcement folks at least seem to know the value of logs, even if they are a little draconian in the other direction. Finally the article above notes that even the Bundestag, the lower house of the German Parliament, doesn't comply with with the privacy laws that body created- the web site logs and retains PII.

Attention Germany: the privacy horse has left the barn. Technology has far outpaced the capability of an individual to control where his or her information flows. Expecting to both receive service from an online provider, and to remain "private" (whatever that means) from the provider, is unreasonable- and in fact denying the provider the right to log prevents the provider from systematically improving service to you. Logging is a best practice for administrative activity, including maintenance-related activities, marketing & service planning, and security-related activities such as forensics. Everything generates logs nowadays. It would probably be better to write laws restricting what can be done with logs rather than to outlaw logging. In this manner you could mitigate abuses such as those by the ambulance chasers but still provide organizations of all sorts, including the government itself, the information they need to do their jobs.

Ensuring that there's no useful data in your logs...

2007-09-01T00:23:00Z

As I wrote about earlier, TorrentSpy, a file-sharing search engine, was ordered by a U.S. magistrate to enable logging on its servers and to subsequently make those logs available to the MPAA, the plaintiff in an illegal file-sharing lawsuit against TorrentSpy. They have lost their appeals and as a result have decided to block US IP addresses from their web servers (which will effectively ensure that no information interesting to the MPAA will reach their logs). This ruling also puts copyright law squarely at odds with privacy rights, as pointed out by the Electronic Frontier Foundation. The whole case seems to hinge on the fact that the judge interpreted the fact that information such as IP addresses temporarily reside in a computer's RAM as meaning that information is "stored" by the computer and therefore discoverable; many computer experts reject that argument. More analysis of the implications of the ruling are found here.