Networking, Games, and Virtual Environments : Networking

DVE Scalability - More to be done?

John L. Miller — Tue, 22 Jul 2008 13:53:00 GMT

Much of my last year has been spent reading about distributed virtual environment scalability. As it turns out, perhaps it shouldn't have been.

A lot of research papers I've read begins like this:

"DVE's consume loads of bandwidth, and there are lots of opportunities to improve their network usage."

Network games researchers have source access to - such as Quake III - have different behavior than the latest generation of multiplayer online games. Being an avid player of World of Warcraft, I finally spent some time analyzing traffic usage for a few basic scenarios:

Sitting in a capital city with a few hundred other players
Playing a battleground with 79 other players
Fighting in close proximity to 30+ players and several computer-controlled avatars.

I was shocked at how LOW the traffic usage was. Even when I was beating on the keyboard and mouse along with 30 other players next to each other in the battleground, my upload bandwidth stayed below 5 kbps, and download below 50 kbps. Better still, when I moved away from other players - out of interaction and viewing range - I stopped getting information about them.

There could still be issues with massively scaled games where thousands of players will be within interaction range of each other, but it's unclear if other computing resources (such as video cards) can keep up with the demand of displaying those avatars.

Does this make scalability research in DVE's obsolete? Absolutely not, but it does mean we need to be careful about what we assume is and is not already implemented.

I wonder if Blizzard talks to researchers about their Networking design...

Peer-to-Peer Content Distribution and download speeds

John L. Miller — Sat, 28 Jul 2007 18:30:00 GMT

When I talk to people about P2P content distribution, there's a common misperception. They assume that the more people there are downloading that file, the faster download goes. This isn't usually true, as I'll explain below. What is true is that a peer-to-peer system in which servers participate should always be faster than just using the servers alone.

The following formula - assuming everyone's download capacity exceeds the speed they can actually get the content at - is true for all three systems.

[per-user average download speed] = [total upload speed] / [number of downloaders]

Let's call the system with no peer contribution - such as traditional web downloads - 'client-server.' P2P systems where peers serve files even when they're not actively downloading them we'll call 'always on.' Finally, if peers only serve files for as long as they're actively downloading that specific file, we'll call them 'greedy.'

All three systems will usually have servers (called 'seeds' in the peer-to-peer world), so there's always someone with a full copy of the file that can make sure people can download.

We need a few other numbers to illustrate the speed for these systems. Let's say

An average user has 4 Mbps download and 250 Kbps upload.
The seed has 10 Mbps upload, 40x as fast as an average user's upload.
The always-on system has 2x as many users uploading as it has doing both upload & download, for a total of 3x as many uploading nodes as the greedy system.

Here's a table that shows average user download speed for 10, 100, 1000, and 10,000 users for each of the three systems

# Clients	Server	always on	greedy
10	1.000 Mbps	1.750 Mbps	1.250 Mbps
100	100 Kbps	850 Kbps	350 Kbps
1,000	10 Kbps	760 Kbps	260 Kbps
10,000	1 Kbps	751 Kbps	251 Kbps

From the table above, you can see that the P2P download *should* always be faster than the seed server on its own. However, the average download speed keeps dropping as the number of clients grows.

The always-on system has significantly faster download speed than the greedy system. This looks great on the surface, but it comes at a price. Users in always-on systems are donating their system's bandwidth even when they're not immediately benefiting. As long as they're OK with this, the system can usually offer improved download speed when the user does want content. But, it means a longer imposition to the user: the upload bandwidth being consumed can impact their other activites, such as web browsing, playing network games, etc. It's also more likely that the upload for a file retrieved earlier in an always-on system will interfer with the system's ability to provide a file the user wants to download NOW, at least in a system where users tend to want newer files rather than older ones.

The framework we use in MSCD leaves the choice of whether to behave as an always-on or a greedy system in the hands of the programmer. For the MSCD CTP which allows users to download Visual Studio 2008 Beta-2 images, we've configured the client to behave as a greedy client. In other words, you only share with other peers until you finish your download, and then you disconnect from the cloud.

Please post a comment if you have any questions in this area, I'm always on the look-out for new reasons to blog :)

'Managed Prototypes'

John L. Miller — Fri, 27 Jul 2007 19:02:00 GMT

MSCD has a front-page story on research.microsoft.com. A friend of mine asked me about a quote in the article which could perhaps be misunderstood:

“It is as much as eight times faster than our original managed prototype, and it’s great that customers will have a chance to experience the benefits for themselves.”

The fact that our prototype was managed is orthogonal to the performance gains we've seen in our MSCD CTP. The speed-up is instead the result of algorithmic and architectural improvements that came out of our lengthy design and optimization efforts.

For the record, managed code is awesome. It does run a little slower (5% - 20%) for some things - especially if you're new to it and write your code in a way that makes the system do unneccesary work - for example appending to a string 50 times rather than using StringBuilder. Managed code also runs some things a little faster than C++, which surprised me at first. One thing that I think is incontrovertable: developing in C# and managed code is much, MUCH quicker than doing the same job in C or C++. My experience has been a factor of two or factor of three speed-up in development for the same quality results.

So, please don't misinterpret the quote in the MSCD story: we're patting ourselves on the back for our algorithmic improvement ingenuity, not dissing managed code.

MSCD links to download Visual Studio 2008 Beta 2

John L. Miller — Fri, 27 Jul 2007 00:45:00 GMT

If you're interested in using Microsoft Secure Content Distribution to download Visual Studio 2008 Beta 2, just click here, install and run the downloader, and you'll be off and running! This version of MSCD will be available for four weeks, so you have until 22-August to give it a try!

Microsoft Secure Content Distribution

John L. Miller — Thu, 26 Jul 2007 19:06:00 GMT

A few years ago, Pablo Rodriguez and Christos Gkantsidis applied Network Coding to Peer-to-Peer file swarming, calling their system 'Avalanche'. I was lucky enough to be involved in their project. Over time, Cambridge Incubation at Microsoft Research Cambridge built a content distribution system around the Avalanche research results. Today that work culminates in a public customer technology preview (CTP) of the resulting system, Microsoft Secure Content Distribution (MSCD).

Before I go any further, let me stress: this is a four-week CTP. Microsoft has no announced plans to incorporate MSCD into any of its products, or to offer it as a separate product.

MSCD allows authorized content publishers to distribute their content to a large audience via file swarming. The publisher can choose to use MSCD to augment their existing server bandwidth, or use it to enable them to reach a much larger audience than they could have otherwise with a relatively small server investment. MSCD is NOT a file searching or file sharing technology: it's intended for a small number of publishers to distribute content to a large number of customers.

Security, preservation of publisher rights, and providing a good customer experience are core goals of MSCD.

The goal of the CTP is to gain real-world experience with MSCD. You can test and simulate to your heart's content, but with internet protocols, you never realy know how well it works until you deploy it.

As you may have heard, Visual Studio 2008 Beta-2 is expected out soon. In addition to the existing distribution mechanisms for VS Beta-2, the Visual Studio team is allowing Cambridge Incubation to make it available via MSCD for the next four weeks. Visual Studio 2008 has a lot of great new features which should make it extremely popular. My personal hope is that some of the more adventurous customers will choose to download it via MSCD.

We've been developing and testing MSCD for quite a while, but this is our first public CTP, and so our first opportunity to see how it works 'in the wild.' There may be hiccups in the experience of downloading the VS Beta via MSCD. If so, please remember these are issues with MSCD, NOT with Visual Studio. Anyone who chooses to try downloading via MSCD can also use the server-based download mechanism at any time.

If you're interested in more information about MSCD, you can read this article to start. If you'd like to read more about some of Visual Studio 2008's great features, you can check out Soma's blog or ScottGu's blog.

Writing a Packet-level Simulator

John L. Miller — Tue, 22 May 2007 20:56:00 GMT

Over the years I've gained a healthy respect for the value of simulation. Abstract algorithm simulation can help you fine-tune your core concepts and transaction structure. Low-level network simulation can give you a controlled environment in which to evaluate and debug your protocol and implementation. Each has a unique value proposition for development.

For high-level simulators, I'm used to seeing either mathematical simulations, or transaction-level simulations, where a transaction exchange is modeled in its impact on node state. Network packets aren't modeled, just the updates you hope network exchanges will make to node state. These I'm used to seeing as one-off implementations, because the algorithm being simulated varies from protocol to protocol.

Low-level simulaltors tend to have a more focussed mission, such as acting as a transport replacement. The first one that springs to mind is WiDS. These are a great way to validate protocols which ship a low-to-moderate amounts of data, and for which you already have a full implementation.

I have a need somewhere between these two. My protocol is already designed, and I want to model artifacts which won't show up in a mathematical/numerical simulation. The protocol's focus is shipping large amounts of data around, which makes it inappropriate for a full transport-level simulator: I wouldn't be able to simulate a cloud of any reasonable size.

So, I find myself making my own simulator I'm tentatively calling a packet-level simulator. It stops just short of constructing actual packets, but it tracks the size of packets exchanged for each transaction, and per-node limits on processing speed, bandwidth, and accessibility. The simulation's primary goal is modeling the impact on nodes of the actual protocol and node speed effects. As long as I can model the interaction of data transmissions and connect attempts, then I'm not so concerned about validating the transcoding of packets.

My simulator is working fairly well. I'm on my third iteration of optimizations.

The first iteration was a relatively naive approach, where I recalculate the speed of data flow across all connections each time a message started or finished transmission anywhere in the simulation. Although accurate, I could only run at about 20x realtime.

The second iteration involved adding a targeted optimization for the simplest case of a message transition where the impact on transmission speeds was restricted to the node sending that message. In the sorts of networks I'm modeling, this hits about 80%-90% of the cases, which results in an effective 5x-10x speedup.

Still, the simulator isn't quite where I want it to be. More than 96% of all CPU time is consumed in the graph speed updates for the cases that miss the optimization above. Simulating 200 nodes for 18 hours took several hours, and I want to simulate clouds of at LEAST 500 nodes for 24-hour stretches.

I'm working on what I hope will be the final optimization, optimizing changes to packet transfer which affect up to two hops away from the sender / receiver of the message. This is proving to be quite difficult. I'm not aware of anything I can grab from graph theory, because I'm not optimizing flow: each node can be thought of as a source of data, with the goal of sending that data to all of its neighbors as quickly as possible. There's no multi-hop data flow to speak of, though: A sends one set of data to B, B sends a different set of data to C. The graph is directed with degree up to 8. There can be multiple edges going the same direction between any pair of nodes, with different messages flowing over them.

I've been trying to wrap my head around this third stage for two weeks now. Each time I get close I find an exception to my logic that makes me question all of the progress I've made on it. Hopefully I'll get it sorted by the end of this week.In the meantime, it helps to blurt out the scope of the problem. :)

If you ever find yourself working on a similar set of problems, I'll be happy to detail a little deeper what I'm doing and how it's useful. I'm considering making the packet-level simulator publicly available once I get it working well enough, and release at least one paper based on its results. If this would be useful to you, please let me know.

May the best bot win...

John L. Miller — Tue, 27 Mar 2007 22:16:00 GMT

There are two broad areas in gaming I want to take a closer look at. The first is distributed games, and models for ensuring fairness and cheat-proofing. The second is issues which affect game play quality, such as latency and jitter in network connections. As part of this research, I need a way of reproducibly simulating the impact of game system changes - including resulting jitter and latency - on playability.

One way to do this is to create a simulated player which reacts to perceived inputs, monkey with those inputs, and then see how the simulated player does. In other words, editing bots.

But which game to use? I'm looking seriously at two choices right now, and hoping for a few others to consider.

Quake supports bots, though I'm not sure if they're scriptable. Since Quake III source is available under GPL, though, it hardly matters except as a question of difficulty.
Unreal's got a very detailed scripting language. I'm going to load up UT2004 tomorrow for the first time in years, and check out exactly what's available.

I'm investigating a few other games as well, such as Half-Life 2 and Battlefield 2142, but I'm not too optimistic.

Ease of scripting the bots and affecting their inputs / storing state is a requirement. Equally important is whether the game is recent, and if there's a large enough online community for results for that game to be interesting. This is a harder question. I've mostly been playing Xbox-360 games for the last year, and am somewhat out of touch with the most popular FPS / latency sensitive PC games out there.

So, my questions for you, gentle reader:

Do you have a recommendation for a latency-sensitive game with scriptable bots I should investigate?
What are the most popular online PC games which are latency sensitive? Note that games like Warcraft and World of Warcraft aren't sensitive enough to qualify for this work.

Do tell :)

And now for something completely different...

John L. Miller — Fri, 22 Dec 2006 11:44:00 GMT

So far I've mostly written about issues that come up in my day-to-day development life, and not very often at that. Starting with my next post, most of what I write about will be related to my research efforts, and I should be posting more frequently. So what are my research interests?

So glad you asked!

I got into computers almost 25 years ago out of a love for gaming, especially simulation games. Playing them, sure, but also wanting to program them. Later my interests expanded to include all of computer science, and especially two frontier areas, virtual environments (think cyberpunk) and AI. I've figured out a way to combine my interest in at least two of these, as virtual environments are becoming commonplace in massively multiplayer games.

Having a planet-scale detailed virtual environment that everyone can use (and play games on) has been pure fantasy for quite a while. Among other challenges, the hardware just hasn't been there to handle the rendering, communications, and storage required for such an endeavor. And even if you had such hardware, communications at the rates required to support such environments was prohibatively expensive

I don't think hardware and connectivity are issues anymore. Even the previous generation of games and CAD packages underscore the ability of a new home machine to render synthetic 3D scenes at several times the resolution of home televison, more than high-def enough to fuel immersion. As for communication, typical home broadband is more than two orders of magnitude faster than modems, and again more than fast enough to show the data for complex interactions with many people, as World of Warcraft proves.

Still, there are challenges. World of Warcraft has more than 7 million unique subscribers, with hundreds of thousands online at any given time. But, these subscribers don't all get to play in the same environment at the same time. The world servers are carefully created to keep too many people from interacting at once. Simply put, the software design can't cope with too many players in any one part of the world. What's more, most players are limited to only ever interacting with a small fraction of the total player base. Consistency requirements force Blizzard to implement World of Warcraft as a server-based model, limiting the number of players who can be in a given copy of the world to tens of thousands. It's an impressive achievement, but a long ways away from a planet-scale virtual environment.

How do we fix this? Software, of course. But nobody's quite sure what that software should look like.

For my part I think the answer lies in peer-to-peer infrastructure. But, applying P2P to this domain is fraught with complications. The two most significant are consistency and fairness. How do you make sure everyone's view of the world is consistent, or at least reconcileable? How do you make sure that people aren't gaining advantage by manipulating the virtual world's rules to benefit themselves?

It's a fine question. I'll tell you my best guess as reserach moves forward :)

Threading models for network services

John L. Miller — Fri, 24 Nov 2006 02:08:00 GMT

One of the first steps in writing a multi-layered network service is determining a threading model. Common wisdom for a performant network service is that the socket layer, at the very least, should use some form of overlapped IO, such as async winsock calls or IO completion ports.

I do a lot of interviewing, and in the course of talking to people about code they've written, I'm always astonished at the variety of approaches to solving the same problem. For example, there are people whose idea of asynchronouse programming for a network service is spinning up a dedicated thread for each active connection. This is fine, to a point, but if you expect to have thousands of clients, probably not the best way to go.

One of my team's projects right now is a multi-layered network service. Our lowest layer sits on top of winsock. The next highest level uses logical transactions (send / receive a message to a neighbor) exposed by the lowest layer, and exposes additional multi-step transactions. Another layer builds on top of this, combining several sets of transactions into meta-transactions. And, of course, a user application sits on top of the whole shebang.

The question is, how do we thread this? It's a given that the lowest level needs a thread pool to manage socket IO, but what about the higher levels? Each of the higher levels need maintenance timers of one sort or another. They also issue transactions which are too lengthy to do in a single go on the IO thread when a packet is received, lest the socket buffers overflow.

We were evaluating three alternatives:

Per-level threads. Have a dedicated thread or thread pool at each level. Use events to notify each higher level that the level beneath it has data or transactions ready for consumption, then have that higher level use its own thread to retrieve the data from the lower level and process it. The cleanest and easiest design wise IMO, but it isn't elegant, and it has performance implications from context switches.
A single thread pool. Have a worker thread pool at the lowest level. Higher levels get work done by the lowest level thread pool, registering work items and time-based events for processing as appropriate. In some cases, process a received packet through every layer of the service on the thread which originally received the data. Attractive, but it requires the lowest level call into and do work in every other level. It's also possible to get into a situation where all worker threads are busy on long-lived tasks, to the detriment of high-priority tasks such as socket IO.
A combined approach. Do both. Use the receiving thread (or the application thread in the case of calling down) to do any easy work to minimize context switches, and queue any significant work items for later processing. The most efficient, but a potential bug farm as you try to keep straight how work is done in each layer, and what work is safe to do in a given context.

In the end we chose a variant of the single thread pool. We have two logical thread pools, one dedicated to servicing any winsock events, the other available for higher-level work. Timed events are serviced by another thread that will, when the timer fires, queue a work item to the front of the application queue to be serviced by the next worker thread. It's a fairly clean design, and though there'll be some tricky parts, it's the right way to go for us.

NAT Traversal

John L. Miller — Sun, 29 Oct 2006 20:31:00 GMT

Over the years I've done a lot of work on P2P protocols. One challenge which consistently arises is devising a good P2P NAT traversal strategy, i.e. one which doesn't require all data between clients be relayed through a server.

Common wisdom divides NAT's into several categories, depending upon how they map internal to external endpoints, and what traffic is allowed into the private network. For example, 'cone', 'restricted', and 'symmetric.' Several strategies are specified to allow connections with one or more endpoints behind one of these NAT's to succeed and persist.

Based on these categories (and a few additional heuristics), there's are several well-researched and tested work to choose from. STUN (UDP NAT traversal) and STUNT (TCP NAT traversal) both provide good behavioral guidelines for NAT traversal, and there are several third party implementatoins of each. Not to be left out, Microsoft's own Teredo protocol is a NAT traversal strategy similar to IP tunneling over STUN, and is built into Windows XPSP2 and later.

Unfortunately, nothing's ever as easy as you'd hope. Within each of the NAT categories mentioned earlier there is a huge variety of behavior. Some of a given NAT's behavior is intentional, some a resource-based compromise, and some can be chalked up to bugs. I don't believe any NAT-traversal strategy can succeed against the full set of intentional behaviors within a bucket (such as 'port restricted'), let alone the other kinds of challents. And so, there will always be machines behind NAT's for which all known P2P NAT traversal algorithms fail.

Still, it works well enough to be incorporated into your application. If you have a good success detection mechanism and a back-up data relay, you can ensure all of your clients can talk to each other.

Here's my (unsolicited) advice.

Don't trust the reliability claims of any NAT traversal mechanism unless you've seen good data on it based on wide (thousands of seats, at the least) deployment. Many people advertise success rates based on limited testing, because doing extensive testing is Hard.
Always have a fallback. If you really, truly need any arbitrary pair of machines to be able to talk to each other, you're going to have to have some sort of a data relay, for example a server which can relay data as well as NAT traversal signalling.
Don't try to invent a new NAT traversal mechanism yourself without knowing what else is out there. You might be the one person in a million who can do a better job, but there's a lot of great work which has already been done and tested. Make sure you read up on STUN, STUNT, and/or Teredo before you decide to roll your own. Trust me, making your own NAT traversal and having it work well isn't easy.

Good luck!

Using SCHANNEL and TLS

John L. Miller — Tue, 16 May 2006 16:00:00 GMT

A few areas of computer science are especially intimidating. The two worst for my money are networking and security. Networking because it involves LAL of TLA's, and security because, well, it's security.

Just because these areas are intimidating doesn't mean they're difficult. Especially if you follow the golden rule: don't invent it yourself.

What does this mean? Security protocols and models have been around for ages. There's a few very solid implementations of authentication protocols built into every copy of Windows. Don't write your own when you can use one that has already been tested, and deployed by millions of people!

How hard is it to use these? Not hard at all, as long as you skip becoming intimidated. If you know what a public key is in the abstract (not necessarily how to use it), then in the space of three days I think you can learn to write an authentication engine that uses TLS for authentication. Neat!

I enjoyed learning more about security, and would be happy to share what I learned here, if anyone is interested. If so, then drop me a line or leave a comment, and I'll happily write a few 'how to learn it' blog entries, which will no doubt reference many of the other 'how to do it's out there.