Shawn Hargreaves Blog
MotoGP had the ability to store and replay race events, and could save replays to disk. Players used this to view races after they finished, and in timetrial mode to race against a "ghost bike" that was playing back their previous best lap.
There are several ways to implement replays:
MotoGP used a mixture of techniques #2 and #3. We stored the entire physics simulation state at periodic intervals, and also stored controller inputs for the gaps in between these keyframes. This hybrid approach allowed us to jump back and forth between keyframes (so we could include forward and rewind controls in the replay viewer) while keeping the overall data rate quite low.
Remember how predictable memory usage is important in console games? This applies to replay data, too. We allocated a fixed size (one megabyte) buffer for the replay system, which was enough to store a default 3 lap race with keyframes every 10 seconds. But the game also had an option that let hard-core race fanatics replicate the 20+ lap races from the real MotoGP sport. To find room for such long races, we varied the keyframe frequency:
In a 20 lap race with a full pack of bikes, you got no keyframes at all. You could play back the replay in order from the start, but couldn’t jump back and forth through it. In a 10 lap race, you could jump in units of roughly 1 minute. In a 1 lap race, you got a keyframe every couple of seconds. In each case we tried to fit as many keyframes as possible into memory. If you messed around and deliberately failed to finish the race in a sensible amount of time, the replay would just stop recording when the buffer filled up, aka "ran out of film for the camera".
It is important to understand that replay keyframes only stored the state of the core bike physics. They did not store any data to do with rider animations, sounds, cameras, particles, skidmarks, etc. These other features did not affect the core gameplay simulation, so they did not have to be 100% identical during replays. As long as the bikes did the same thing, these other systems would stay at least similar to the original. We didn’t care if individual particles ended up in slightly different locations, as long as there was still a cloud of smoke in the right place.
The challenge was what to do when you jumped to a different keyframe. This would leave things like particles and rider animation in the wrong state, since they were not stored as part of the keyframe. To avoid obvious artifacts during these jumps, we deleted all active particles and skidmarks, restored bike state from the keyframe, then ran a single update before rendering a frame to the screen. This update gave the camera and rider animation a chance to catch up with the modified bike position, but left us without any particles or skidmarks. If you let the replay continue forward, new particles would soon appear, but any skidmarks from earlier in the race were gone for good. Ah well, such is the price you pay for making things work with limited memory!
The most painful part of implementing replays was making sure we got the same results each time we played back the same controller inputs. This technique only works if the game simulation is 100% deterministic. Over time, even the tiniest deviation will be magnified until you end up with objects flying around in crazy directions, running into walls, etc.
We discovered (and spend much time fixing) several reasons why a game may not be entirely deterministic:
Thanks for article.MotoGP is my favorite.
"Solution: don’t have any bugs!"
Now I'm really curious of what that last sollution might be.
The keyframes were really useful for debugging the purely input-based part of the replay system. While you're playing back the replay without the user jumping back and forth, you can just ignore the keyframe data - the input recording should have already got the physics into the right state. But while you're there, in debug builds, you might as well check that the keyframe state matches the current physics state. If it doesn't, you can report numerically exactly which bits of state didn't
match, and these give valuable clues to which bit of the system is not being deterministic.
Due to the butterfly effect you often find that by the time you reach the next keyframe pretty much all of the state is wrong, but you can set the keyframe interval very low to catch the errors sooner (at the expense of only being able to store about ten seconds of replay data) and hopefully you then find that only one member is wrong (e.g. mWheelieAmount) and you then know exactly what's causing the inconsistency.
In fact I think we actually did restore keyframes when we reached them anyway, even though it shouldn't make a significant difference. I don't remember exactly, but I think this actually "fixed" some bugs. I guess there are a few possible explanations, but I never thought of one that actually made sense in the end.
If I was implementing this kind of system again, there's one particular thing I would have wanted to do - and that's segregate the physics state into constant data, primary inter-frame state, derived state, and possibly temporaries (which were only necessary to communicate between methods during the course of the frame). Then you can mostly ignore constant data (just make sure it gets set up consistently next time the game runs), store the inter-frame state in keyframes, and recompute derived state after restoring keyframes. Crucially you also get the obvious option to zero out the derived state before recomputing it, and to zero out the temporaries at the start of every frame.
All of a sudden you can be a lot more confident that the physics engine isn't storing data in places it shouldn't, that storing a keyframe is writing the right data, and that your bugs are going to be reproducible (rather than depending on the state you switched from and to when you restored the last keyframe). The hardest bugs to diagnose were the ones where you can to skip to a keyframe where the bike was flat on the ground from a state where the bike was doing a wheelie.