John L. Miller's Blog: Networking and more

Microsoft Touch Mouse: Engadget Readers' Choice Award!

2012-02-24T17:30:00Z

Engadget recently announced that the Microsoft Touch Mouse was given the 2011 readers' choice award for best peripheral. Nice!

Here's a story that was posted on research.microsoft.com about the touch mouse a while back. Ah, the memories. Touch mouse was a fun project, it was great getting to watch it go from research project through prototype to the final product.

Microsoft Touch Mouse Sensor API

2011-10-21T14:32:00Z

The CID Team worked on the touch tracking and gesture recognizer for the Microsoft Touch Mouse. This was based upon the research done for the Mouse 2.0 project by Microsoft Research and the Microsoft Hardware Team, including the Applied Sciences group.

The Touch Mouse is a wireless mouse with an integrated multi-touch sensor. The mouse takes a snapshot of the sensor every 8 or so milliseconds, cleans it up, and sends it to the host computer. If installed, IntelliPoint software on Windows 7 processes these bitmaps, tracking fingertips, recognizing gestures, and sending appropriate commands to Windows based on the gestures made.

The picture below shows an example frame of data from the sensor, with annotations showing where the software believes fingertips are.

The sensor images usually show up around 100 times a second.

If you've got a touch mouse and want to take a crack at writing your own applications which process the sensor, now you can! Microsoft recently released the 'Touch Mouse Sensor API,' which includes libraries to read data from the Touch Mouse, and several example projects showing how to process and render the data. Note that the API is intended for research / experimentation purposes only, NOT to release production software. See the EULA for more details.

The sensor API is available from http://research.microsoft.com/downloads. You can either visit the website and search for 'Touch Sensor API', or click on the following links to get it:

32-bit Touch Mouse Sensor API

64-bit Touch Mouse Sensor API

If you have questions, feel free to post them here. No promises, but I'll answer them if I can. Enjoy!

Recent research results

2011-09-08T16:55:00Z

Hard to believe it's been two years since I posted! I expect to be writing more regularly moving forward. Not that the bar is very high at this point :)

Over the past few years I've helped write a few papers related to distributed virtual environments in various workshops. In case you're interested, here's pointers:

The Near-Term Feasibility of P2P MMOG's. There's been a lot of research on creating peer-to-peer distributed virtual environments, e.g. a P2P version of World of Warcraft. Using World of Warcraft traffic patterns and UK consumer broadband capabilities, this paper suggests that it's not possible to operate a pure P2P massively multiplayer game on today's internet.

Group Movement in World of Warcraft Battlegrounds and Avatar Movement in World of Warcraft Battlegrounds. These two papers examine movement of actual player avatars in World of Warcraft, specifically in battlegrounds. They suggest that previous models of avatar movement are inconsistent with observed behaviors. This is relevant for tasks such as workload evaluation and simulation of online games.

massively multiplayer online game site

2009-04-12T10:20:00Z

I tripped across the daedalus project this morning while I was looking for player statistics for World of Warcraft. What a great site! The author, Nick Yee, has apparently been doing user surveys to MMOG players for years, and compiling various articles and statistics based upon them. The ones I've looked at so far are quite interesting!

Unfortunately it looks like I discovered his site a little late. There was a post on March 8th indicating he's not going to be updating it anymore.

Nevertheless, a treasure trove of data to comb through. Thank you Nick Yee, for putting this together and leaving it up!

Interesting Cheating Talk

2009-03-19T11:48:00Z

Jon Crowcroft pointed me at an extremely interesting talk by Dan Ariely on Cheating. Take a peek if you're interested in cheating and cheater behavior, especially as influenced by group dynamics, it'll be 15 minutes well spent.

Second Life paper answers many questions

2008-12-17T13:07:00Z

A recent paper by Matteo Varvello et. al. answers a lot of the questions I've found myself asking about Second Life and its community. The paper Is There Life In Second Life? and if you're interested in Second Life's popularity, community, and performance, it is a must-read!

A quick inspection of recent web articles about Second Life shows a common sentiment that perhaps popularity its is waning - or was never as great as total user account statistics suggested. When user comments are enabled on these articles, you also see that whatever their numbers, Second Life aficionados are extremely vocal when defending the popularity and functionality of their world.

It'll be interested to see how the Second Life community reacts to this paper.

It presents a lot of salient statistics about how many users are online at a time, and what those users are up to - at a macroscopic level.

So check it out! Is There Life In Second Life? is extremely accessible, so don't worry about needing a PhD to make heads or tails of it :)

Microsoft Research AutoCollage - try it now!

2008-09-04T12:39:00Z

I work on an incubation team at Microsoft Research in Cambridge, UK. Today we released our first public offering, AutoCollage. In a nutshell, AutoCollage lets you select a folder containing images, and creates a collage synthesized from interesting bits of some of the pictures in that folder. You can see examples of collages here, they're pretty cool IMO.

You can download AutoCollage and try it for thirty days free. If you like it, you can buy it.

AutoCollage works best on Vista, but you can use it on Windows XP SP2 and later, as long as you've got the .NET framework 3.0 or later installed.

AutoCollage is based on a lot of cool research technology from Microsoft. Carsten Rother developed the primary AutoCollage algorithm, incorporating research from Microsoft Research Asia, Redmond, and Cambridge. It's truly a global effort :)

I hope you enjoy it!

Crashing WoW servers

2008-07-30T11:35:00Z

Yesterday I was forwarded a link about retirement of a long-time player from World of Warcraft. While that in itself might be interesting, the really juicy bit was the way he went out: with a mighty 'crash' from the server. Scalability issues, or something else?

Check out Boom's Goodbye Blog, where he describes his farewell, including the convergence of more than 1,000 players in a single capital city, and three different 40-party groups (in addition to lots of individuals) engaging in combat.

It's nice to see that WoW can scale past the 100-200 player limit (within mutual interaction range) I had privately hypothesized. It would be very interesting to find out exactly how many it could support sustained!

DVE Scalability - More to be done?

2008-07-22T13:53:00Z

Much of my last year has been spent reading about distributed virtual environment scalability. As it turns out, perhaps it shouldn't have been.

A lot of research papers I've read begins like this:

"DVE's consume loads of bandwidth, and there are lots of opportunities to improve their network usage."

Network games researchers have source access to - such as Quake III - have different behavior than the latest generation of multiplayer online games. Being an avid player of World of Warcraft, I finally spent some time analyzing traffic usage for a few basic scenarios:

Sitting in a capital city with a few hundred other players
Playing a battleground with 79 other players
Fighting in close proximity to 30+ players and several computer-controlled avatars.

I was shocked at how LOW the traffic usage was. Even when I was beating on the keyboard and mouse along with 30 other players next to each other in the battleground, my upload bandwidth stayed below 5 kbps, and download below 50 kbps. Better still, when I moved away from other players - out of interaction and viewing range - I stopped getting information about them.

There could still be issues with massively scaled games where thousands of players will be within interaction range of each other, but it's unclear if other computing resources (such as video cards) can keep up with the demand of displaying those avatars.

Does this make scalability research in DVE's obsolete? Absolutely not, but it does mean we need to be careful about what we assume is and is not already implemented.

I wonder if Blizzard talks to researchers about their Networking design...

Distributed Virtual Environment Scalability

2008-01-10T11:26:00Z

In the previous post I parrotted scalability figures for World of Warcraft. While investigating DVE's, I tripped across interesting figures for WoW and several other environments.

Halo-3

From this press release, we can see that in the first week of Halo-3's release,

2.7 million people played Halo-3 online
They logged 40 million hours of online play that week

That's absolutely astonishing for a single game! 40 million hours is 19,230 US person YEARS of work, in a week!

World of Warcraft

From Neilsen video game figures for September 2007, we see that

WoW was the most popular PC game by a factor of 3
The average player played 1051 minutes per week - 17.5 hours!

If the math holds up, that's 10 million subscribers * 17.5 hours/wk = 175 MILLION hours of WoW play per week, or 84,135 US person years of work equivalent for each week of play.

As a side note, "Traffic Analysis and Modeling for World of Warcraft" describes WoW traffic overall, and says the median download bandwidth for a player is 6.9 kbps, and the median uplink is 2.1 kbps. If we accept the peak number of simultaneous active users as 900,000, that's a total of 6.2 Gbps peak average upload from WoW datacenters for gameplay. Imagine all the processing that goes into calculating what's being communicated... Zowie!

Second Life

It's difficult to find reliable statistics for Second Life, and I haven't gone far into the research literature. From my own observations and inferences, and a quick search on the internet

Online user population tends to be between 25,000 and 50,000 at the times I connect.
A given region (a land parcel whose simulation is handled by a single server, and whose inhabitants can interact) looks able to support no more than one or two hundred users. I've seen limits of 63 attendees at invitation-only performance events, for 'technical reasons', which makes me wonder if perhaps the limit isn't 64 users per region?
Checking just now, it says 1,271,025 users have logged in in the last 60 days. Several months ago I saw a figure of 25,000 to 50,000 new accounts per day. If those numbers hold true on an average day today, and each new user logs in once, that would be 1.8M to 3M unique user logins. This leaves me uncertain of the active returning population in Second Life. Does anyone have better figures?

Anyways, enough random numbers for now. If you have anything to add, please leave a comment or send me mail, this sort of stuff is fascinating for me.

World of Warcraft hits 9M + active subscribers!

2008-01-02T13:06:00Z

The holidays gave me a chance to re-acquaint myself with World of Warcraft (WoW). I have to say, it's still the single most impressive online game I've ever seen. For my money, it does everything right. It literally *is* for my money, since I'm one of those paying 9 million subscribers.

I'm researching distributed virtual environment (DVE) scalability. As part of that, I was curious about the populations in WoW, both numbers supported per server, and total number of simultaneously active users. "A Measurement Study of Virtual Populations in Massively Multiplayer Online Games" provides a great peek into WoW, based on results from the CensusPlus UI add-in for WoW. Based on that paper, and on CensusPlus results published at http://www.warcraftrealms.com/, it appears:

There's a peak of almost 900,000 simultaneous users logged in and playing in the US and EU.
Average daily populations on a given server fluctuate by around a factor of 4-5 between minimum and maximum number of players online.
The peak number of simultaneous users on a given server appears to be around 4,000.
Data on maximum users in a zone (able to interact directly) isn't provided, but appears to be on the order of a couple hundred.

I've downloaded and added CensusPlus to my characters addins. It's out of date, but still works fine with 2.3. It's fun to see the numbers for your own server.

In summary, congratulations Blizzard on the incredibly successful World of Warcraft, and thank you researchers and add-in developers for giving us insight into player populations!

Content versus Form

2007-11-15T13:58:00Z

If I could go back 30 years, I would tell myself to focus on the content and intention of each message, rather than its form.

I'm struggling through a technical report describing application of Bayesian techniques to a particular problem. Like papers in any new (to me) field, nomenclature and conventions for equations are quite confusing.

In the past I focused on - obsessed over, really - these nomenclature differences between fields. Why couldn't they be consistent in mathematical formulations, terminology, and symbols? In college inconsistencies between my pure math, physics, and chemistry texts drove me nuts. They frustrated me to the point where I had problems memorizing formulas, but more importantly, understanding the reason for those formulas and their derivation. My grades reflected my frustration and ignorance.

Today I find myself accepting that I don't necessarily understand nomenclature in the paper I'm reading, and assuming it will differ from what I already know, even if it looks the same. Instead of getting frustrated over three different interpretations for superscripts, I think "what is the author trying to tell me here?" It's a much more rewarding experience, and I'm already reaping dividends.

If you find yourself faced with inconsistencies in presentation of technical material, or in the formulas that back it up, remember the price I paid for focussing on form rather than content. Hopefully you can have a more productive 30 years to come as a result. :)

Peer-to-Peer Content Distribution and download speeds

2007-07-28T18:30:00Z

When I talk to people about P2P content distribution, there's a common misperception. They assume that the more people there are downloading that file, the faster download goes. This isn't usually true, as I'll explain below. What is true is that a peer-to-peer system in which servers participate should always be faster than just using the servers alone.

The following formula - assuming everyone's download capacity exceeds the speed they can actually get the content at - is true for all three systems.

[per-user average download speed] = [total upload speed] / [number of downloaders]

Let's call the system with no peer contribution - such as traditional web downloads - 'client-server.' P2P systems where peers serve files even when they're not actively downloading them we'll call 'always on.' Finally, if peers only serve files for as long as they're actively downloading that specific file, we'll call them 'greedy.'

All three systems will usually have servers (called 'seeds' in the peer-to-peer world), so there's always someone with a full copy of the file that can make sure people can download.

We need a few other numbers to illustrate the speed for these systems. Let's say

An average user has 4 Mbps download and 250 Kbps upload.
The seed has 10 Mbps upload, 40x as fast as an average user's upload.
The always-on system has 2x as many users uploading as it has doing both upload & download, for a total of 3x as many uploading nodes as the greedy system.

Here's a table that shows average user download speed for 10, 100, 1000, and 10,000 users for each of the three systems

# Clients	Server	always on	greedy
10	1.000 Mbps	1.750 Mbps	1.250 Mbps
100	100 Kbps	850 Kbps	350 Kbps
1,000	10 Kbps	760 Kbps	260 Kbps
10,000	1 Kbps	751 Kbps	251 Kbps

From the table above, you can see that the P2P download *should* always be faster than the seed server on its own. However, the average download speed keeps dropping as the number of clients grows.

The always-on system has significantly faster download speed than the greedy system. This looks great on the surface, but it comes at a price. Users in always-on systems are donating their system's bandwidth even when they're not immediately benefiting. As long as they're OK with this, the system can usually offer improved download speed when the user does want content. But, it means a longer imposition to the user: the upload bandwidth being consumed can impact their other activites, such as web browsing, playing network games, etc. It's also more likely that the upload for a file retrieved earlier in an always-on system will interfer with the system's ability to provide a file the user wants to download NOW, at least in a system where users tend to want newer files rather than older ones.

The framework we use in MSCD leaves the choice of whether to behave as an always-on or a greedy system in the hands of the programmer. For the MSCD CTP which allows users to download Visual Studio 2008 Beta-2 images, we've configured the client to behave as a greedy client. In other words, you only share with other peers until you finish your download, and then you disconnect from the cloud.

Please post a comment if you have any questions in this area, I'm always on the look-out for new reasons to blog :)

'Managed Prototypes'

2007-07-27T19:02:00Z

MSCD has a front-page story on research.microsoft.com. A friend of mine asked me about a quote in the article which could perhaps be misunderstood:

“It is as much as eight times faster than our original managed prototype, and it’s great that customers will have a chance to experience the benefits for themselves.”

The fact that our prototype was managed is orthogonal to the performance gains we've seen in our MSCD CTP. The speed-up is instead the result of algorithmic and architectural improvements that came out of our lengthy design and optimization efforts.

For the record, managed code is awesome. It does run a little slower (5% - 20%) for some things - especially if you're new to it and write your code in a way that makes the system do unneccesary work - for example appending to a string 50 times rather than using StringBuilder. Managed code also runs some things a little faster than C++, which surprised me at first. One thing that I think is incontrovertable: developing in C# and managed code is much, MUCH quicker than doing the same job in C or C++. My experience has been a factor of two or factor of three speed-up in development for the same quality results.

So, please don't misinterpret the quote in the MSCD story: we're patting ourselves on the back for our algorithmic improvement ingenuity, not dissing managed code.

MSCD links to download Visual Studio 2008 Beta 2

2007-07-27T00:45:00Z

If you're interested in using Microsoft Secure Content Distribution to download Visual Studio 2008 Beta 2, just click here, install and run the downloader, and you'll be off and running! This version of MSCD will be available for four weeks, so you have until 22-August to give it a try!

Microsoft Secure Content Distribution

2007-07-26T19:06:00Z

A few years ago, Pablo Rodriguez and Christos Gkantsidis applied Network Coding to Peer-to-Peer file swarming, calling their system 'Avalanche'. I was lucky enough to be involved in their project. Over time, Cambridge Incubation at Microsoft Research Cambridge built a content distribution system around the Avalanche research results. Today that work culminates in a public customer technology preview (CTP) of the resulting system, Microsoft Secure Content Distribution (MSCD).

Before I go any further, let me stress: this is a four-week CTP. Microsoft has no announced plans to incorporate MSCD into any of its products, or to offer it as a separate product.

MSCD allows authorized content publishers to distribute their content to a large audience via file swarming. The publisher can choose to use MSCD to augment their existing server bandwidth, or use it to enable them to reach a much larger audience than they could have otherwise with a relatively small server investment. MSCD is NOT a file searching or file sharing technology: it's intended for a small number of publishers to distribute content to a large number of customers.

Security, preservation of publisher rights, and providing a good customer experience are core goals of MSCD.

The goal of the CTP is to gain real-world experience with MSCD. You can test and simulate to your heart's content, but with internet protocols, you never realy know how well it works until you deploy it.

As you may have heard, Visual Studio 2008 Beta-2 is expected out soon. In addition to the existing distribution mechanisms for VS Beta-2, the Visual Studio team is allowing Cambridge Incubation to make it available via MSCD for the next four weeks. Visual Studio 2008 has a lot of great new features which should make it extremely popular. My personal hope is that some of the more adventurous customers will choose to download it via MSCD.

We've been developing and testing MSCD for quite a while, but this is our first public CTP, and so our first opportunity to see how it works 'in the wild.' There may be hiccups in the experience of downloading the VS Beta via MSCD. If so, please remember these are issues with MSCD, NOT with Visual Studio. Anyone who chooses to try downloading via MSCD can also use the server-based download mechanism at any time.

If you're interested in more information about MSCD, you can read this article to start. If you'd like to read more about some of Visual Studio 2008's great features, you can check out Soma's blog or ScottGu's blog.

Second Life - Reality sets in?

2007-07-15T10:46:00Z

Earlier I commented about the disparity in numbers quoted for Second Life's population. It's not that any of the numbers are wrong - for what's being expressed, they're no doubt correct. Rather, it's a question of what's being measured. For my money, steady-state population, and those willing to pay for the experience are both fine metrics.

A recent LA Times article talks about a trend of some large businesses either reducing or eliminating their presence in Second Life. The article is short, insightful, and worth reading IMO. Some key quotes from that article:

"Even at peak times, only about 30,000 to 40,000 users are logged on, said Brian Haven, an analyst with Forrester Research."
"[Philosophy professor Peter Ludlow] said most firms were more interested in the publicity they received from their ties with Second Life than in the digital world itself."
"Between May and June, the population of active avatars declined 2.5%, and the volume of U.S. money exchanged within the world fell from a high of $7.3 million in March to $6.8 million in June."

My two specific longest-term motivations for being a computer scientist are making games and to enable some sort of global cyber space, not that I've done much for either of these in the last decade.

Where's the future of virtual communities and communications, and for that matter, is there one? I still think lack of a payoff for time invested in Second Life is its biggest conceptual challenge. World of Warcraft (WoW) has an incentive and reward structure built in, with social interaction as a side-benefit. WoW continues to do well, with more than 8.5 million subscribers, each paying more than $10 a month for the privilege.

World of Warcraft taps a large existing community - computer gamers - and offers an experience attractive to that community with significant value-adds. Perhaps the biggest challenge in Second Life is that the only existing community it really taps into are those who frequent online social mechanisms such as ICQ, MUD/MOO, and others, and press and the curious inspired by the press, who are no doubt a highly transient population.

I definitely want to see Second Life (or similar environments) succeed. It seems like they need to offer something else, though, in order to be anything more than a niche application.

Data persistence in a digital world

2007-06-29T09:27:00Z

A while back I read a news article pointing out an issue largely overlooked, namely the transience of digital data.

For thousands of years, institutional and personal memory were stored solely in physical written form, on paper, papyrus, wax, stone, you name it. The systems for writing these memories are well known and easily accessible: anyone who can read the language can read virtually anything. The down side is that accessing a piece of data in its native form requires a physical token, such as the book containing that data. How many of you remember taking a trip to the library when you needed to know something 30 years ago? And waiting for a book from inter-library loan if the local library didn't have it, sometimes being shipped from hundreds of miles away...

We've switched ease of access and permanence. Thanks to computers and networking, virtually any information you're interested in is no further away than your desktop, or perhaps your pocket if you're especially well connected. This comes at a price, though: most institutional and personal memory will be lost in just a few years, thanks to rapidly evolving digital storage formats and the lack of a long-lived, easily accessible physical store.

It's not uncommon to find written diaries from both common folks and historical figures going back hundreds of years. However, I challenge you to find an electronic diary of someone from the 80's that can still be read today. For example, I did a lot of writing on my Atari 520ST, and stored it on 3.5" floppies. Assuming the floppies are even working today, I don't have any machines with a floppy drive. I certainly don't have any that can read the data files created by whatever word processor I used, and it didn't save in plain text.

Is the web a temporally localized phenomenon, or will it last for centuries? It's too early to tell. But if it *doesn't* last, how much information will be lost when it fades into obscurity? Will digital archaeologists be able to make heads or tails of what came even a few decades before?

With luck, we'll never know. But our grandchildren will.

Writing a Packet-level Simulator

2007-05-22T20:56:00Z

Over the years I've gained a healthy respect for the value of simulation. Abstract algorithm simulation can help you fine-tune your core concepts and transaction structure. Low-level network simulation can give you a controlled environment in which to evaluate and debug your protocol and implementation. Each has a unique value proposition for development.

For high-level simulators, I'm used to seeing either mathematical simulations, or transaction-level simulations, where a transaction exchange is modeled in its impact on node state. Network packets aren't modeled, just the updates you hope network exchanges will make to node state. These I'm used to seeing as one-off implementations, because the algorithm being simulated varies from protocol to protocol.

Low-level simulaltors tend to have a more focussed mission, such as acting as a transport replacement. The first one that springs to mind is WiDS. These are a great way to validate protocols which ship a low-to-moderate amounts of data, and for which you already have a full implementation.

I have a need somewhere between these two. My protocol is already designed, and I want to model artifacts which won't show up in a mathematical/numerical simulation. The protocol's focus is shipping large amounts of data around, which makes it inappropriate for a full transport-level simulator: I wouldn't be able to simulate a cloud of any reasonable size.

So, I find myself making my own simulator I'm tentatively calling a packet-level simulator. It stops just short of constructing actual packets, but it tracks the size of packets exchanged for each transaction, and per-node limits on processing speed, bandwidth, and accessibility. The simulation's primary goal is modeling the impact on nodes of the actual protocol and node speed effects. As long as I can model the interaction of data transmissions and connect attempts, then I'm not so concerned about validating the transcoding of packets.

My simulator is working fairly well. I'm on my third iteration of optimizations.

The first iteration was a relatively naive approach, where I recalculate the speed of data flow across all connections each time a message started or finished transmission anywhere in the simulation. Although accurate, I could only run at about 20x realtime.

The second iteration involved adding a targeted optimization for the simplest case of a message transition where the impact on transmission speeds was restricted to the node sending that message. In the sorts of networks I'm modeling, this hits about 80%-90% of the cases, which results in an effective 5x-10x speedup.

Still, the simulator isn't quite where I want it to be. More than 96% of all CPU time is consumed in the graph speed updates for the cases that miss the optimization above. Simulating 200 nodes for 18 hours took several hours, and I want to simulate clouds of at LEAST 500 nodes for 24-hour stretches.

I'm working on what I hope will be the final optimization, optimizing changes to packet transfer which affect up to two hops away from the sender / receiver of the message. This is proving to be quite difficult. I'm not aware of anything I can grab from graph theory, because I'm not optimizing flow: each node can be thought of as a source of data, with the goal of sending that data to all of its neighbors as quickly as possible. There's no multi-hop data flow to speak of, though: A sends one set of data to B, B sends a different set of data to C. The graph is directed with degree up to 8. There can be multiple edges going the same direction between any pair of nodes, with different messages flowing over them.

I've been trying to wrap my head around this third stage for two weeks now. Each time I get close I find an exception to my logic that makes me question all of the progress I've made on it. Hopefully I'll get it sorted by the end of this week.In the meantime, it helps to blurt out the scope of the problem. :)

If you ever find yourself working on a similar set of problems, I'll be happy to detail a little deeper what I'm doing and how it's useful. I'm considering making the packet-level simulator publicly available once I get it working well enough, and release at least one paper based on its results. If this would be useful to you, please let me know.

SChannel - careful how you use it...

2007-05-21T20:30:00Z

I recently had a very interesting experience with SCHANNEL, in particular with TLS authentication.

If you're not familiar with it, SCHANNEL is an example of something called an SSP. It's a Windows implementation of standard authentication protocols, and is included as part of the Windows operating system. If you can figure your way around SSPI (cryptic but relatively small) then you can perform authentication for any network connection with confidence that the security implementation has withstood the test of time.

One of my projects has a network service component which deals with a large number of transactions per unit time. Each transaction involves a new mutually authenticated TLS connection, and the work in setting up those connections was my primary scale-limiting point.

TLS authentication uses asymmetric key cryptography for authentication and validation of credentials. These operations are computationally expensive. As expensive operations, they are the primary limiting factor in my service scalability. Up until recently, I thought a modern machine could only do 10-15 mutual authentications per second. As it turns out, it all depends upon how you set up your connections.

I was doing the following each time a new client connected:

Bind to the SSP and get function pointers
Create local credentials and prepare to authenticate
Perform authentication activities

I'd thought that the SSP would be cached in memory, so the binding would be cheap. As it turns out, there's a better way to do this: Do (1) and (2) once, and then do (3) as many times as needed for your operations. It's obvious now, but it wasn't with the code base I was working on. As soon as I made this switch, my authentication speeds went up by a factor of 15!

If you need to use SCHANNEL on a server or server-like component, make sure you're not doing extra work for each connection.

Second Life Slashdot stream

2007-04-20T12:00:00Z

I stumbled across a recent Slashdot posting about Second Life. There are some interesting comments amidst the dross, worth a read if you're interested in virtual worlds or Second Life in particular.

The gist is, Second Life has made its client side open-source, and the article claims it'll be doing the same for the server side. Very interesting for someone who wants to research distributed systems and distributed system security. Getting the servers correctly federated and working well together is an interesting enough problem.

If the client code and the server code are both really open sourced, it may be worth a peek to see what would be required to make a P2P implementation. No doubt there are a plethora of legal and logistical requirements for servers to support monetization and integrity of the world, but heck, that just makes the problem that much more interesting!

While looking up pointers, I stumbled across a tech special report from business week online which is centered around 'Virtual Life,' with lots of Second Life articles. Looks like time for more reading...

Windows Software RAID5, we hardly knew ye

2007-04-07T19:47:00Z

Ever since I installed Vista Ultimate on my home machine, I've had grand visions of setting up Windows Media Center with my lifetime collection of media on a terabyte RAID5 array. Windows NT used to include a software RAID5, so if you weren't too concerned about performance, you could run a RAID array without having to buy any extra equipment.

Of course, my hardware purchases of late have been predicated around this. I bought three Samsung HD501LJ 500 GB SATA drives. Cheap, well reviewed, I knew I'd be in for a treat. They arrived a few days ago, and I got my first ugly surprise: not enough SATA power leads on my power supply to power them all. A quick trip to the local computer shop got me a standard-to-SATA power connecter adaptor, and I was back in business. I installed and powered up the new drives, and went to the disk manager to set up the RAID array.

Hmmm, that's odd - no option for 'stripe with parity.' I did a web search and got some instructions for using the command line to set up the RAID array. My sole reward was a pithy message informing me RAID5 was no longer supported by the utility.

Very... Disappointed!

Ah well. So much for doing it on the cheap. I'll order an SATA-based hardware RAID5 controller and set up my dream system.

In the mean time, though, doff your caps and join me in a moment of silence for Windows Software RAID5 support. It will be missed.

May the best bot win...

2007-03-27T22:16:00Z

There are two broad areas in gaming I want to take a closer look at. The first is distributed games, and models for ensuring fairness and cheat-proofing. The second is issues which affect game play quality, such as latency and jitter in network connections. As part of this research, I need a way of reproducibly simulating the impact of game system changes - including resulting jitter and latency - on playability.

One way to do this is to create a simulated player which reacts to perceived inputs, monkey with those inputs, and then see how the simulated player does. In other words, editing bots.

But which game to use? I'm looking seriously at two choices right now, and hoping for a few others to consider.

Quake supports bots, though I'm not sure if they're scriptable. Since Quake III source is available under GPL, though, it hardly matters except as a question of difficulty.
Unreal's got a very detailed scripting language. I'm going to load up UT2004 tomorrow for the first time in years, and check out exactly what's available.

I'm investigating a few other games as well, such as Half-Life 2 and Battlefield 2142, but I'm not too optimistic.

Ease of scripting the bots and affecting their inputs / storing state is a requirement. Equally important is whether the game is recent, and if there's a large enough online community for results for that game to be interesting. This is a harder question. I've mostly been playing Xbox-360 games for the last year, and am somewhat out of touch with the most popular FPS / latency sensitive PC games out there.

So, my questions for you, gentle reader:

Do you have a recommendation for a latency-sensitive game with scriptable bots I should investigate?
What are the most popular online PC games which are latency sensitive? Note that games like Warcraft and World of Warcraft aren't sensitive enough to qualify for this work.

Do tell :)

Vista Installation? Easiest EVER

2007-03-25T11:11:00Z

We've been encouraged to install Vista at work for quite some time, but I measure how difficult it'll be for most people to use based on what happens at home.

I bought my personal copy of Vista Ultimate about a week ago, and just got around to installing it yesterday. I have to say, kudo's to the Windows team for the easiest installation I've EVER had, and my fastest installation since I was putting DOS on PC's 15 years ago.

I had been running a hardware RAID mirror, so I removed one of the drives and then booted from the Vista Ultimate DVD. Within five minutes I was able to delete my remaining drive's partitions and create a new single partition. Vista wouldn't let me install on this drive, citing 'potential incompatibilities.' I booted into the Fatal1ty A8N-SLI's BIOS and disabled the hardware RAID, then booted from CD again. This time setup was happy.

The setup experience from here on in was amazing. To go from a blank partition to booting into Vista took less than half an hour, and perhaps two or three minutes of my time, all at the beginning. After booting into Vista and starting to play, it found drivers enabling ALL of my hardware to work. This impressed me because I've got, among other things, a USB microphone, USB smartcard reader, third party PCI USB card, 7800 GT display with multimon, Hauppauge PVR-150 TV adapter, USB remote control, gigabit network, and lots of other goodies. It found drivers for *everything*, even a legacy driver for my built-in Realtek audio, and it did it without me having to lift a single finger.

Will I like Vista? That remains to be seen. But I can't say a single bad thing about installation. In a word, AWESOME.

Now it's time to buy three 500 GB SATA drives and set up a software RAID5 :)

Welcome to the terabyte!

2007-01-06T03:23:00Z

I saw a press release today for a new Hitachi 1 Terabyte drive, supposed to be available for $400. Oddly, this is something I've wanted for a long time, and was planning on buying an external RAID array to achieve. It's not that I need the space, or that I'll do anything productive with it. It just sounds nice to say.

"Ooo, I wonder how much disk space I have left on my terabyte drive?"

I'm sure eventually I'll come up with a way to fill it. Maybe I'll make a new copy of every picture I take every time I look at it? Write an editor that saves a new full copy of the file for each keystroke? Or perhaps use it in my mediacenter PC to give me 500 hours of video storage. Now THAT sounds worthwhile.

Whatever the case, I'll buy one when it becomes available on the street, and revel in the memory of my first 10 MB hard drive in 1985, 100,000 times less storage than this much smaller, much quieter drive I will be able to hold in the palm of my hand.