Pointless Blathering : DMA

GetScatterGatherList will coallesce contiguous SG list entries

PeterWie — Fri, 26 May 2006 03:24:00 GMT

GregH asked me an interesting question at WinHEC today - one for which I didn't immediately know the answer. He was wondering whether the new DMA DDIs (Get|PutScatterGatherList) would return a single SG entry for each page, or whether they would coallesce physically contiguous pages into a single entry.

The answer is that the coallescing is "kind of complex and probably won't happen in a way that will make complete sense to someone just looking at the generated [SG] lists". To anyone who is familiar with the DMA DDIs that sort of answer probably isn't surprising :)

On X86 & for non-scatter-gather devices, the bounce buffers are allocated in big contiguous blocks and the DMA DDI will coallesce entries where possible. Of course non-SG devices always get one big contiguous block no matter what, and so they'll only have a single entry.

On AMD64 the bounce buffers for an SG adapter aren't necessarily contiguous so there may be less coallescing.

Still, since you always have to plan for the worst case, this ends up being an optimization but shouldn't change your driver's behavior (unless your device uses single-page SG entries rather than address/length pairs).

What is DMA (Part 9): I/O MMUs and why you'll wish you'd used the DMA DDI in 3 or 4 years

PeterWie — Thu, 11 May 2006 18:19:00 GMT

Perhaps you've read through all of my posts about DMA and still think that using the DMA DDIs is optional. After all you've built a 64-bit card, and you're not using DAC so you don't have to worry about busses being downgraded on you (or maybe you don't believe in the boogyman either). Sure someone might build an X86 machine that has actual map registers (companies have done such things before), but what are the chances you'll ever have to run on one of them? The reason is coming in the next few years.

There are certain problems i hate debugging. Any problem that involves a mis-programmed DMA controller is high on my list. These look like random memory corruption and usually don't reproduce well. If you're lucky the corruption is identifyable (like a text file, or WAV data). If not you sit back and collect repros until the pool of machines is large enough to determine that they all have the same network controller.

And that's just DMA that's accidentally gone wrong. Since DMA gets around all page protections it could be used to steal data as well. Of course that's a moot point for a kernel-mode driver since they can steal anything they want anyway. But if we ever want to allow direct hardware access from a guest OS (on a Virtual PC system) or from user-mode DMA is going to be the killer issue. It's hard to build a secure system when one of the restricted components can tell its device to read any page on the computer.

Enter the IOMMU*. Just like the regular MMU creates virtual address spaces from physical memory, the IOMMU creates logical address spaces for each device (function actually) on your PCI-X bus. Smart people tell me these should be in "every" new system by the end of the decade, and I for one am very excited.

In a nutshell - the IOMMU has page tables for each bus/device/function that describe a logical address space (similar to the CPU pages tables). When your devices attempts to read a logical address L, the IOMMU does a lookup, finds the appropriate physical page, and returns that page's contents. If there's no mapping, or if the protections only allow write, then the DMA is "logged" and isn't allowed to happen. It's much more complex than that, involving lots of electrons and transistors and such, and i've only seen a glimmer of it so i can't talk in detail about how it works.

In short, the IOMMU lets us control which pages each device can access through DMA. We could, for example, block access to core kernel information (interrupt & system call tables, non-paged kernel code, etc...) by leaving them out of the device page maps. We could allow a Virtual PC guest to directly access a device since we could now virtualize the DMA operations as well - ensuring the device can't access any pages that aren't in the guest's physical address pool. We could coallesce fragmented transfers or make 32-bit devices work on 64-bit systems without any copies (if there are still 32-bit devices in three years :).

Of course we'd want to enable this in a way which didn't break existing drivers, or at least "well-behaved" ones. To do this, we'd need to add code to some function calls the driver makes before and after every DMA operation. Maybe something like GetScatterGatherList and PutScatterGatherList?

We're only just starting to make decisions about what we'll do with this in the base OS. It's clearly interesting, but how much of it we can use is up in the air. However even now a couple things are clear:

This can help with some of the reliability problems around DMA
This helps most if we can tie it into every DMA operation a driver initiates
The most logical place to do that is building it into the DMA DDI (whatever that may look like in the future).

So to summarize: even if you don't see a reason you need to use the DMA DDI today, that reason is looming on the horizon. I think we'll see demand from system administrators, IT departments & OEMs with high support costs to start using the IOMMU as a protection mechanism. It's going to see usage in the virtualization space to allow direct hardware access (and on an enlightened OS i'm sure it will leak into the DMA functions).

If you're using the DMA DDIs already you're probably in good shape. If you're not, you should start thinking about how you'll integrate it in during one of your future design changes.

* IOMMU is AMD's term for it. Intel uses some different term that starts with V, which is less catchy and which i can't rememeber. So for now assume IOMMU applies to both.

What is DMA (Part 8) - BuildScatterGatherList

PeterWie — Fri, 07 Apr 2006 03:05:00 GMT

Since i'm on vacation next week, i thought i'd tackle something light this week.

Last time i talked about GetScatterGatherList and PutScatterGatherList and how much better they are than the older method of doing DMA. But as much as I like these two functions, they have one major problem that hit us while we were working on the storage stack - they allocate memory.

During the development of Windows XP one of our goals in the storage team was to ensure we could successfully issue disk I/O operations even if the pool was exhasted or the system address space was full. In Windows even kernel memory is pageable, and when you can't page in a kernel thread's stack the system can't really continue. The kernel can crash if a page-in fails due to a bad block on disk, or due to some driver returning STATUS_INSUFFICIENT_RESOURCES.

AllocateAdapterChannel & MapTransfer would work for us, but the performance isn't good on modern systems since you can't issue more than one request at a time through the channel. We needed something new.

The trick to making forward progress even when you can't freely allocate memory is to preallocate all the resources you need for at least one I/O operation for use in emergencies. When you get an I/O you try to allocate what you need, and if you can't get it you try to use the "emergency" resources. If those aren't available you queue the request for later processing when the emergency resources are free.

In order to do this around DMA, we needed the ability to pre-allocate a scatter-gather list, then hand that to the DMA engine to fill in. This is exactly what BuildScatterGatherList does - it constructs the SG list within the supplied buffer but otherwise acts just like GetScatterGatherList.

There's only one problem. GetScatterGatherList doesn't just allocate space for your scatter gather list. It also allocates private memory so that it can track the DMA mapping operation - list entries for enqueing it, the map register base, the number of map registers - all of those things you would normally have to keep track of yourself. Obviously BuildScatterGatherList can't allocate memory, and your driver shouldn't have to guess how much extra space it might need. So how do you know how big to make the buffer you hand in?

You find that out by calling CalculateScatterGatherList(). It takes a CurrentVa and a Length along with an optional MDL. This function determines the size of buffer that BuildScatterGatherList requires. If you provide an MDL the function will compute the required size for any chained MDLs as well. If you provide NULL for the MDL then it uses CurrentVa and Length to determine how many pages you're transferring & determines the rest from there.

With these two functions you can ensure that you'll always have enough memory to handle one DMA mapping for some reasonable sized I/O operation (where that reasonable size is whatever you passed in for Length).