Transmitting vs. Generating Data
One difficult part of protocol design is optimizing network data transfer. Conventional wisdom holds that local processing is a cheap resource, and network transmission is expensive and limited. Logic suggests that applications be designed to transmit as little information as possible, even if that means doing more local processing on each network endpoint.
Compression is a critical tool for data transfer. Old modems had data compression built-in to try to allow more data to be transmitted over the same speed connection. Today, large file transfers often involve compressed files and archives rather than the raw, expanded form. This is only common sense, and an accepted practice.
This strategy can be extended further than compressing entire files. For example, binary delta compression allows a subset of information to be transmitted when updating software. When a binary is updated (for example going from v8 to v9 of a DLL), the change usually affects a subset of bits in the binary. Rather than transmitting the entire binary, the diff between the previous and current binary of the file is calculated, and that diff is transmitted, along with version information. A user might get away with downloading a 10kB patch rather than a 2 MB binary, a huge savings for all involved. The same strategy can be applied to any versioned data, such as documents.
Another variation of this strategy is employed in online games, where you need to make sure the game world is consistent between players, even when each player controls their own motion and actions. One way to do this is to send a snapshot of the entire world between players for each move or frame. An improvement over this naive approach is to ensure players have the same world state when starting up, and then communicate changes as state deltas (such as moves or position changes), with an occasional state checksum to ensure the endpoints are in sync. There are even more efficient approaches, such as transmitting motion vectors rather than positions, and only updating the vectors upon new player input. All instances of the game can calculate identical player locations based upon these vectors and elapsed time without requiring a network update for each frame. Again, occasional checks should be made to ensure the games remain in sync.
The idea of generating rather than calculating data is present in other types of network transactions. Stream ciphers are a well-known way to encrypt data between network endpoints. After exchanging a small amount of setup data, a stream cipher typically works by reversibly combining a generated sequence of cipher bytes with the data to be transferred, for example by XOR'ing bytes. Both sides of the connection are able to generate the same sequence of cipher bytes, and so the encrypted data can be sent across the wire safely, looking like garbage to anyone who intercepts the stream and doesn't know the generated byte sequence. The sequence generation setup can be as simple as determining a seed to be used on either side to seed a pseudo-random number generator, then repeatedly calling that generator to obtain bytes for combination with the data stream. Compare this to having to have a large buffer of bytes available to perform the same operation on both sides which wasn't generated. [NOTE: this is an example only, don't employ this rather insecure stream cipher algorithm without doing appropriate background research.]
The common thread in all of these examples is an emphasis on data generation rather than full state replication. Whenever you design a distributed application, think carefully about what state has to be transmitted, and what can be done more creatively, by well-known initial state, compression, and delta transmission. Doing this right can reduce your application's bandwidth requirements from an untenable amount of bandwidth to something that could be easily transmitted over a 20-year old modem.
