Threading models for network services
One of the first steps in writing a multi-layered network service is determining a threading model. Common wisdom for a performant network service is that the socket layer, at the very least, should use some form of overlapped IO, such as async winsock calls or IO completion ports.
I do a lot of interviewing, and in the course of talking to people about code they've written, I'm always astonished at the variety of approaches to solving the same problem. For example, there are people whose idea of asynchronouse programming for a network service is spinning up a dedicated thread for each active connection. This is fine, to a point, but if you expect to have thousands of clients, probably not the best way to go.
One of my team's projects right now is a multi-layered network service. Our lowest layer sits on top of winsock. The next highest level uses logical transactions (send / receive a message to a neighbor) exposed by the lowest layer, and exposes additional multi-step transactions. Another layer builds on top of this, combining several sets of transactions into meta-transactions. And, of course, a user application sits on top of the whole shebang.
The question is, how do we thread this? It's a given that the lowest level needs a thread pool to manage socket IO, but what about the higher levels? Each of the higher levels need maintenance timers of one sort or another. They also issue transactions which are too lengthy to do in a single go on the IO thread when a packet is received, lest the socket buffers overflow.
We were evaluating three alternatives:
- Per-level threads. Have a dedicated thread or thread pool at each level. Use events to notify each higher level that the level beneath it has data or transactions ready for consumption, then have that higher level use its own thread to retrieve the data from the lower level and process it. The cleanest and easiest design wise IMO, but it isn't elegant, and it has performance implications from context switches.
- A single thread pool. Have a worker thread pool at the lowest level. Higher levels get work done by the lowest level thread pool, registering work items and time-based events for processing as appropriate. In some cases, process a received packet through every layer of the service on the thread which originally received the data. Attractive, but it requires the lowest level call into and do work in every other level. It's also possible to get into a situation where all worker threads are busy on long-lived tasks, to the detriment of high-priority tasks such as socket IO.
- A combined approach. Do both. Use the receiving thread (or the application thread in the case of calling down) to do any easy work to minimize context switches, and queue any significant work items for later processing. The most efficient, but a potential bug farm as you try to keep straight how work is done in each layer, and what work is safe to do in a given context.
In the end we chose a variant of the single thread pool. We have two logical thread pools, one dedicated to servicing any winsock events, the other available for higher-level work. Timed events are serviced by another thread that will, when the timer fires, queue a work item to the front of the application queue to be serviced by the next worker thread. It's a fairly clean design, and though there'll be some tricky parts, it's the right way to go for us.
