A question came up a couple of times recently about an issue with the timestamps on the files involved in a build always being the current time. The issue is that folks have customized their deployment process to deploy only files where the timestamps are newer. Folks then ask for an option to have get set the timestamp to the timestamp when the file was checked rather than when it was downloaded. I don't think that's the best answer in this case (and if you aren't doing a clean build, it's not an answer at all, since that will lead to botched builds).
The biggest culprit here is likely that the default behavior for Team Build is to create a new workspace and get all of the files for every build. We've left that default in place in for TFS 2008, but we may change it in the future.
You can change Team Build to get only the files that have changed since the last build by having it not create a new workspace each time and doing an incremental get. Aaron wrote a post about how to do incremental gets in TFS 2005. It's a simple matter of setting IncrementalGet to true in TFS 2008.
After making the appropriate change, you'll want to switch to incremental builds if you need to have the binaries updated only when the corresponding source files change (there are links for that in the posts referenced earlier).