Base types, Collections, Diagnostics, IO, RegEx…
For the .NET 3.5 release, the BCL team has been trying to solve some problems related to dates, times and time zones that customers having been asking us to address for some time. We’ve invested in two significant features:
DateTimeOffset is a completely new DateTime type that stores the UTC Offset in addition to the DateTime. In presenting the design of this type, there are a couple of questions that keep coming up that we would like to address. The full rationale behind these provisional decisions requires going back through some of the history of DateTime over the last 8 years.
In .NET 3.5 there will be a new DateTime type called DateTimeOffset. The name of this type is an interesting topic in itself, but current plans are that this new type will complement rather than fully replace the existing DateTime type, and thus it will not be called DateTime2. A couple of questions keep coming up about this which I will address in this post by covering some relevant history:
It is necessary to start with the original rationale and design issues surrounding the original DateTime type.
The original DateTime type was mostly designed in 1998-1999, as part of the the new .NET Platform type system. Up until that time, almost all platforms (e.g. Win32, C, VB, COM, databases) represented dates and times with a combined date and time type that had no time zone information. The time zone context was outside the type’s data and in the application’s context, in the same way that using Decimal to represent money involves the context of the currency unit living outside the Decimal instance. The option of a DateTime with a UTC offset or a time zone ID was discussed and there was a lot of design back and forth. There was a lot of desire to keep the type system simple, and while a time zone aware DateTime was better at representing absolute points in time such as a file time-stamp, it got awkward when doing certain things:
Another factor was that many prior platforms had lived with DateTime types like this without running into serious issues. In the end, the desire to keep the platform simple won out, and the DateTime was a date and a time and nothing else, although support for conversion between local and UTC was provided using the machine’s local time zone.
Once this was shipped, a number of serious problems became clear:
In V2.0, when the severity of these issues was fully understood, design began on a solution. There was actually a fully developed specification to address the above issues by adding a UTC offset to the existing DateTime type. This was going to be a challenging operation since this was a significant change to what the type was. The only way to keep a reasonable compatibility guarantee was to make the time zone offset optional. However, this proposed change simply did not stand up to design scrutiny and implementation trials, and we discovered that our options of changing the existing type were significantly more limited than we had hoped. Here were some of the problems that emerged:
This was a difficult situation. It was extremely undesirable to leave every existing API taking DateTime and accepting or returning local instances with unreliable behavior around DST transitions. But a comprehensive change to DateTime was not possible in a compatible way either. It was decided to do a very minimal change to DateTime by using some unused bits in the structure to indicate if an instance was UTC or Local, and using an extra state to disambiguate the hour repeated during DST transition to solve the reliability problem. The DateTime.Kind property was introduced to solve the reliability problem without doing a full deprecation of usage of the type. However, it was not a comprehensive solution because to maintain compatibility, the Kind could only be very minimally used in existing APIs. It was decided that we would try to make it possible for existing APIs to use the Kind, but make any behavior changes on DateTime itself opt-in as much as possible. Thus the Kind is used in the critical conversions to and from Local and UTC, but not when DateTime instances are compared or formatted by default.
Even this minimal change presented significant compatibility challenges. An example of this was DateTime.ToString. If the instance was marked with DateTimeKind.Utc, it would makes sense to output its time zone offset as “+00:00” instead of pulling in the machine’s local time which only makes sense for local instances. At one point the code actually did this more correct behavior. However, we found that a significant number of users of DateTime were taking a dependency on this incorrect behavior and were broken by this change. They were serializing UTC instances in XML, and because it defaulted to formatting as a local time and because Parse converted back to the local time zone by default, their code was working most of the time, even though the persisted string would have been incorrect if parsed by a 3rd party system. To maintain compatibility even this change had to be backed out, and correct formatting of the time zone offset had to be made opt-in via the new “o” and “K” format options.
The DateTimeKind solved the most pressing issues with DateTime, but still left the space with a lot of unresolved issues. It was still very difficult to use correctly, and it could not deal well with the server scenarios where instances from different machines in different time zones needed to be handled.
So, this should address the first question of why we are not currently pursuing further the option of adding an optional offset to the existing DateTime type.
After .NET 2.0 it became clear that there was still considerable demand for a type with DateTime and a UTC offset combined. This is the type we are delivering in .NET 3.5 (DateTimeOffset). The second question then comes up as to why we are planning to a new type that complements DateTime instead of creating a complete replacement and calling it DateTime2. It is definitely compelling to have a single new type so that there does not have to be complex guidance around which of the types to use for what scenarios.
Even with compatibility taken out of the picture a lot of the reasoning above still makes it compelling to have new type that is always absolute. First, consider the landscape of date and time related scenarios:
(Scenarios C-D might be even better handled by dedicated Date and Time types. These options are still on the table, although they will not be in the .NET 3.5 release).
The existing DateTime can do scenarios A-E. But it does a mediocre job of E, because you have to convert it to and from UTC and Local depending on whether you want human-readable display or accurate arithmetic. For scenario F, you need to store an additional field to make it work. The proposed new DateTime with UTC offset is significantly better for scenario E and enables scenarios F with a single type instance. However, it is not as good for scenarios A-D where it would have to default to some arbitrary time zone offset. As a point of reference, the majority of DateTime scenarios involve E, although A and C are fairly common also, and F becomes more important when richer Time Zone support is available.
It is possible to create a new type that addresses scenarios A-D as well by making the UTC Offset optional rather than required. But this creates a lot of the same problems encountered during the aborted attempt to create such a type in .NET 2.0:
An interesting question that is always useful to ask about a design is: Given what we know now, what would we do differently if we could start again?
It would be hard to get broad consensus on this point. My personal opinion is that the date and time space is too complicated to address with a single type. If I could start again, I would have something like the V1.0 DateTime, but have a separate DateTimeOffset type also. The main difference to where we will land with the current plan of record is that we would not need the DateTimeKind that was added to DateTime in V2.0. This was really a band-aid that would not have been needed if a DateTimeOffset had been available in V1. That being said, my own opinion is that the DateTimeKind solution was critical to salvage reliable usage of the existing APIs given that there was no better type to use in V1.0.
PingBack from http://msdnrss.thecoderblogs.com/2007/06/18/a-brief-history-of-datetime-anthony-moore/
You've been kicked (a good thing) - Trackback from DotNetKicks.com
Great post Anthony, thanks!
<< DateTime is working just fine in the majority of existing uses>>
I don't think anyone thinks this is really the case. I think we'd all like to see APIs amended as soon as possible to support the new type.
(2) In the summary description of DateTime for intellisense, please can we have words to the effect that DateTime is effectively deprecated for many scenarios in favour of the new type. This would go a long way to preventing any more buggy .NET date/time code.
(3) Was there any consideration given in the design process around making DateTimeOffset an (immutable) reference type?
1. What about System.TypeCode support? Are there any plans for TypeCode.DateTimeOffset ?
2. Are there any plans to support DateTimeOffset as a column type in SQL 2008?
3. Will DateTimeOffset be supported in XSD schemas and System.IO classes?
What can be possible issues with mass replacing DateTime to DateTimeOffset ?
While I appreciate your efforts, these solutions do not truly address the real problems.
I am also a customer and have been reporting my concerns and issues for many years.
I need a UTC-DateTime that *is a* DateTime. If I use DateTimeOffset, I will still need to convert it to DateTime when passing it as a parameter to existing methods. This introduces an opportunity for errors.
> "there is no practical way to do
> it without breaking compatibility."
Being a structure, extension is difficult. Could it be promoted to a class? Probably not a very good idea either. Ideally I would prefer an IDateTime interface, but then many existing signatures would need to be changed or it would be useless. Those changes could be done slowly overtime. But at least give us the interface and start addressing the real problem here.
As it is, we now filter through all source code searching for DateTime usage. Every instance is flagged as a potential bug (and usually is) needing special review. It is very timely and costly. Depreciating this poorly designed type would be very (*very*) appreciated. And depreciating it would also allow you to update signatures to use a new IDateTime interface.
I am also dissatisfied with TimeZoneInfo. It is a wrapper around OS timezone info. However, that OS support needs a manual update to be installed, and will need future updates as well. I can not assume that customers will have it or future updates installed. A better solution is to wrap something like
http://www.twinsun.com/tz/tz-link.htm . Caching the results for an appropriate amount of time should suffice. New data can be periodically downloaded. The user need not install anything, or even know anything about it. This is something that I can (and do) do on my own. However, integrating it into the framework library is difficult without any interfaces...
Please, design to interfaces. It would really solve so many issues.
A well detailed and explanatory post that adds clarity to the DateTime enhancement and will help us full understand and apply the new type where required.
I think that you have made the right design decisions - not to break with backwards compatibility and to leave the existing type in place to serve in most applications. The new type will be welcomed however by developers building systems that run across time zones.
This is one of the best posts I've ever read on MSDN blogs. It's extremely well written and informative. Thanks, Anthony!
FWIW, I totally agree that "DateTime is working just fine in the majority of existing uses". It works fine for my use cases, but it's good to know that the BCL team is working hard to handle other use cases as well.
>> Anthony Moore
Everyone here writes about IDateTime interface. BUT, everyone should be also aware that with current DateTime being Value Type, there would be a performance penalty caused by (un)boxing with methods expecting ref type (IDateTime) instead...
I share the question by an earlier poster:
"Are there any plans to support DateTimeOffset as a column type in SQL 2008?"
Any thoughts on this? If not, what is the preferred way of storing and converting back to DateTimeOffset from a SQL column?!
1. We will incrementally start to add new APIs that use the new type and will consider changing existing APIs where appropriate.
2. This is a good idea that we will definitely consider.
3. Not that I'm aware of. I think we always had planned on the new type being a value type. One of the main goals we had when designing DateTimeOffset was for it to be largely substitutable with DateTime.
1. There are no plans to add this to System.TypeCode. Could you explain your scenario for needing this?
2. DateTimeOffset is indeed a new column type in SQL Server 2008! We worked closely with the SQL team and both types are called DateTimeOffset and have the same internal layout.
3. DateTimeOffset will be supported in XSD schemas, although, I'm not sure whether or not this made it into .NET 3.5. Which System.IO classes are you referring to?
4. We've designed DateTimeOffset's members to be as similar to DateTime as possible, so it shouldn't be too hard to mass replace.
You can create custom TimeZoneInfo instances with the static TimeZoneInfo.CreateCustomTimeZone method filling it with data from a copy of the Olson time zone database if necessary.
As I mentioned above DateTimeOffset is indeed a new column type in SQL Server 2008! We worked closely with the SQL team and both type are called DateTimeOffset and have the same internal layout.
Justin Van Patten
CLR Program Manager
> BUT, everyone should be also aware
> that with current DateTime being
> Value Type, there would be a
> performance penalty caused by
>(un)boxing with methods expecting
> ref type (IDateTime) instead...
Of course performance is important. However, more important than that is correctability and lack of bugs. DateTime as is broken. Anyone suggesting otherwise either has not worked with the type for long or is in denial. The problems are clear and have been written about extensively. That is why various solutions are being attempted. However, none of those solutions integrate well into the existing framework. For that to happen, there are only two options: 1) inheritance or 2) interfaces. Option 1 is not really likely for technical reasons. Options 2, while potentially being less performant, offers a solution without sacrificing backward compatibility. For those who are happy with the current DateTime, they may continue to use it without any performance lose. (I truly think they will end up regretting it one day when the bugs become known, though.)
This is not a solution. It only changes the problem a little.
Justin Van Patten,
> You can create custom TimeZoneInfo
> instances with the static
> method filling it with data from a
> copy of the Olson time zone database
> if necessary.
I know. That was mentioned a few posts back. However, the real issue: how could it not be necessary? I can not assume that all users will have past and future updates installed. Therefore, I can not depend on the correct TZ information to be present on the system. Thus, I must ignore your data and always import the data via CreateCustomTimeZone.
Except for integration with TimeZoneInfo, I have been doing exactly that for a while. Once Orcas is released, I will need to port that over to your TimeZoneInfo implementation. Can the type be extended? Are there any interfaces to implement? Or will you tie my hands like with DateTime and absolutely force me to use CreateCustomTimeZone? I can probably guess the answers.
Magari non è uno dei massimi problemi per chi lavora in Europa e generalmente ha a che fare con un solo
Java had these problems 10 years ago, and you guys did not learn anything from it. They ended up deprecating most functionality in Date and moving it to Calendar. Why couldn't you have a DateTime object that optionaly has a null TimeZone property?
Great post and great title... ;)