Networking, Games, and Virtual Environments : Programming

'Managed Prototypes'

John L. Miller — Fri, 27 Jul 2007 19:02:00 GMT

MSCD has a front-page story on research.microsoft.com. A friend of mine asked me about a quote in the article which could perhaps be misunderstood:

“It is as much as eight times faster than our original managed prototype, and it’s great that customers will have a chance to experience the benefits for themselves.”

The fact that our prototype was managed is orthogonal to the performance gains we've seen in our MSCD CTP. The speed-up is instead the result of algorithmic and architectural improvements that came out of our lengthy design and optimization efforts.

For the record, managed code is awesome. It does run a little slower (5% - 20%) for some things - especially if you're new to it and write your code in a way that makes the system do unneccesary work - for example appending to a string 50 times rather than using StringBuilder. Managed code also runs some things a little faster than C++, which surprised me at first. One thing that I think is incontrovertable: developing in C# and managed code is much, MUCH quicker than doing the same job in C or C++. My experience has been a factor of two or factor of three speed-up in development for the same quality results.

So, please don't misinterpret the quote in the MSCD story: we're patting ourselves on the back for our algorithmic improvement ingenuity, not dissing managed code.

Developing Distributed Systems

John L. Miller — Mon, 01 Jan 2007 17:23:00 GMT

Over the last five years I've had the fortune to do research and development on several different distributed systems for Microsoft. I've learned a lot from these efforts, some of them things I embarrassingly knew a decade ago and promptly forgot. Here's the three most important:

Know what you're building. As a corollary, make sure you can tell when you're done.
Establish 'correctness' criteria to let know you when things work.
Have a way to figure out what's wrong when your system doesn't work.

The first two aren't too hard as long as you think of them, but the third can be a real bear, especially for distributed system developers.

Take developing PNRP (the Peer Name Resolution Protocol) for example. This is a complicated P2P service which is a special-case of a distributed hash table (DHT). To the untrained eye, PNRP appears to route messages randomly through the set of all people running the service, making it difficult to know if it's working correctly except by final transaction results. By mid 2002 the PeerNet team had PNRP implemented and up and running in the lab. We did a simple test out in the wild with a couple dozen participants. The moment of truth came, and we started our services. Each of us registered our email name, and then we tried to resolve each other.

To our horror, it didn't work. Even worse, we didn't have any obvious way to diagnose the problems.

In the absence of custom tools, everyone ran NetMon (Microsoft's network sniffer) and saved 5-minute snippets of trace as we would all bring the system up and try something. The traces were saved off, collected together, and then handed to a very talented network developer to examine. To his credit, after a few days of work he had at least some answers for the first five minute session. You'll never grok just how big an achievement this was, unless you happen to be in the handful of people who have developed P2P protocols.

The next time I helped develop a P2P protocol from scratch was as part of a research project at MSRC. The researchers working on it were already familiar with the woes of debugging distributed systems, so from the very beginning we ensured we could debug any issues that arose. There are two good ways of doing this.

Simulation. This will help you catch a healthy chunk of systemic problems, but ultimately it's only as clever at finding problems as you are at manufacturing them. There's no way to get away from the need for #2,
Centrally coordinated logging. Servers are anathema to P2P, but the truth is you should never shy away from a great solution on a purely idealistic basis.

Centralized logging is easily the most important tool we had for validation and troubleshooting. It let us determine which nodes had connectivity, whether they were working they way they should, throughput and consistency numbers, node connectivity, and so on.

If you're developing a distributed system, I highly recommend implementing a logger with the following properties

Have a centralized log-server which receives all log data. Use a 'cloud ID' and 'node ID' for each party logging data to separate their activities.
For each message you generate and transmit, log both local generation time and central server receive time. This will let you establish a global ordering of logged messages, critical for understanding the system's behavior.
Have a (removable) logger for the P2P client which logs activities to the central server.

Logger transport should be reliable, to prevent message loss. In other words, even when everything else breaks, your logger should still be able to get all of its messages to the central log server. This can be a serious amount of work, but is well worth it.
Logs should be machine-parsable, to enable automated analysis and processing
Logs should be human-readable, or translatable into human-readable form. Once you work with the logs for a while you'll get an intuition for reading them, and find problems that would have been impractical to find with automated processing.

Figuring out the right granularity of messages to log can be tricky. Here's what was useful for us:

Log messages for each of the following:

Client startup, client shut down
Connection data: connection initiated, succeeded / failed, remotely initiated connection received, connection torn down. You'll be amazed how many problems boil down to the right peers not communicating to each other.
Periodic snapshots of activity. For example, every 60 seconds log bytes sent / received, connections received / rejected / initiated in that 60-second interval.
Significant events, or summaries of significant events. If you're doing file sharing, a significant event might be transferring a file block to a remote party. If you're doing name resolution, it could be receiving and retransmitting a name resolution request.

There's much more you can log, but even these essentials will go a long ways towards helping you understand your system's behavior and validating it.

Have you developed a distributed system? If so, did you use logging? Or, if you're just getting ready to start developing, are you planning on logging? I'll be very interested to hear...

Threading models for network services

John L. Miller — Fri, 24 Nov 2006 02:08:00 GMT

One of the first steps in writing a multi-layered network service is determining a threading model. Common wisdom for a performant network service is that the socket layer, at the very least, should use some form of overlapped IO, such as async winsock calls or IO completion ports.

I do a lot of interviewing, and in the course of talking to people about code they've written, I'm always astonished at the variety of approaches to solving the same problem. For example, there are people whose idea of asynchronouse programming for a network service is spinning up a dedicated thread for each active connection. This is fine, to a point, but if you expect to have thousands of clients, probably not the best way to go.

One of my team's projects right now is a multi-layered network service. Our lowest layer sits on top of winsock. The next highest level uses logical transactions (send / receive a message to a neighbor) exposed by the lowest layer, and exposes additional multi-step transactions. Another layer builds on top of this, combining several sets of transactions into meta-transactions. And, of course, a user application sits on top of the whole shebang.

The question is, how do we thread this? It's a given that the lowest level needs a thread pool to manage socket IO, but what about the higher levels? Each of the higher levels need maintenance timers of one sort or another. They also issue transactions which are too lengthy to do in a single go on the IO thread when a packet is received, lest the socket buffers overflow.

We were evaluating three alternatives:

Per-level threads. Have a dedicated thread or thread pool at each level. Use events to notify each higher level that the level beneath it has data or transactions ready for consumption, then have that higher level use its own thread to retrieve the data from the lower level and process it. The cleanest and easiest design wise IMO, but it isn't elegant, and it has performance implications from context switches.
A single thread pool. Have a worker thread pool at the lowest level. Higher levels get work done by the lowest level thread pool, registering work items and time-based events for processing as appropriate. In some cases, process a received packet through every layer of the service on the thread which originally received the data. Attractive, but it requires the lowest level call into and do work in every other level. It's also possible to get into a situation where all worker threads are busy on long-lived tasks, to the detriment of high-priority tasks such as socket IO.
A combined approach. Do both. Use the receiving thread (or the application thread in the case of calling down) to do any easy work to minimize context switches, and queue any significant work items for later processing. The most efficient, but a potential bug farm as you try to keep straight how work is done in each layer, and what work is safe to do in a given context.

In the end we chose a variant of the single thread pool. We have two logical thread pools, one dedicated to servicing any winsock events, the other available for higher-level work. Timed events are serviced by another thread that will, when the timer fires, queue a work item to the front of the application queue to be serviced by the next worker thread. It's a fairly clean design, and though there'll be some tricky parts, it's the right way to go for us.