Tom Archer's Blog : Visual C++

Are Hash Codes Unique?

tomarcher — Wed, 10 May 2006 08:02:00 GMT

While you will hear people state that hash codes generate a unique value for a given input, the fact is that, while difficult to accomplish, it is technically feasible to find two different data inputs that hash to the same value. However, the true determining factors regarding the effectiveness of a hash algorithm lie in the length of the generated hash code and the complexity of the data being hashed.

Let's first talk about the hash algorithms themselves. Hashing algorithms generate a fixed-length hash code regardless of the length of the input. For example, the MD5 hash function always generates hash codes that are 32 bytes in length, the SHA1 hash function generates 20-byte hash codes, SHA256 generates 256-bit (32 byte) hash codes, and so on. Therefore, since there are a limited range of possible values for a given hash code and an unlimited range of values to hash, it stands to reason that the length of the hash code generated with a given hash algorithm is directly related to the difficulty of finding two inputs that will generate the same hash code.

This is easy to prove. If n is the number of possible hash codes, we only need n + 1 distinct input values to prove that there is an overlap. Granted that for most hash functions, n is rather large so we can safely assume that for any meaningful input values, it would be very difficult to find another meaningful input that would give the same hash.

That brings me to the second point—the input value itself. It is assured that if the input is anything more involved than random data, then even if you were to find two inputs that generated the same hash value, the two inputs would have no semantic relationship to one another. This is especially evident if you are hashing text, because there are many more ways to produce gibberish than there are ways to produce meaningful words. In other words, it's extremely difficult to create random words, even harder to form sentences, and very unlikely that those sentences will form a paragraph or a document and still have it make sense.

As an example, let's say you're generating a hash code for a top-secret document that details the route of nuclear warheads for disposal. The chances of someone randomly generating words until they matched the hash code generated for the original document and producing a document that makes any sense are so small that you could accurately say it would be impossible.

Therefore, the fact that each possible hash code doesn't uniquely match an input value only comes into play when you are dealing with random (nonstructured) data. An example of this would be spyware-removal applications that determine if a file is spyware by hashing each file in a specified folder or drive and then comparing that hash value to a list of known spyware-file hash codes. In this case, relying solely on the hash code would be a mistake as the files being hashed are of varying lengths with many files having no semantic meaning (as the application is not determining the meaning of the data; only the hash code values). As a result, it would be highly recommended that these applications either match on file name before hashing the file's contents (which dramatically reduces the possibility of "false positive" results) or use a hash code with a very long output value - such as SHA256.

This brings us to the concept of password hashing. Instead of storing their users' passwords, some applications will store only the hash code for the password. When the user attempts to log into the system, the application hashes the user's input password and compares that hash value to what has been stored for the valid password. This is a very good technique because it means that the users' passwords are not stored and therefore - theoretically cannot be hacked as hash codes cannot be reverse-engineered to their pre-hashed values. However, assuming that a hacker got his hands on the password-hash file, a brute-force method could be employed by the hacker to continually generate hash codes until a code is generated that matches a hash code on the password list. The value the hacker used to generate the matching hash code could then be used to allow the hacker unauthorized access.

This is a very real risk as passwords generally have a fixed length (such as 6-10 characters). Therefore, the hacker doesn't need to generate as many hash codes as he would if attempting to regenerate the same hash code that was originally hashed for a much longer value. As a result, anyone using this technique to protect user passwords should go a step further by adding a salt, or static piece of data, to the input value before it is hashed. This would produce an almost fool-proof system as even if someone were to obtain a list of hashed passwords and were to produce a password that generated to a hash code in the list, this password would not work because when attempting to log in, the system would apply the salt and the resulting hash code would differ from that stored for the user. In other words, the hacker's input value would already be salted, but the system is expecting a non-salted value that it salts. As a result, the addition of the salt greatly increases the difficulty in cracking the passwords, because now the hacker would need to 1) steal the hash-code file, 2) know that a salt has been added, 3) know the value of the salt and 4) know exactly how the salt was added.

Therefore, while there isn't a one-to-one correlation between every possible hash code and every possible input value such that all combinations are guaranteed to be unique, hash codes are an extremely reliable means of protecting data integrity.

Related Reads: Keith Brown did a full article on securing the username token with WSE 2.0 that is a good read even if you're not doing Web Services as it talks about general security issues regarding the protection of user authentication information that is being sent over the wire.

Why MSDN publishes more Managed Code than Native Code Information

tomarcher — Sun, 26 Mar 2006 08:15:00 GMT

From time to time, I get queries like “Why is Microsoft forcing managed code on us?” or “Why is Microsoft abandoning native code?” and so on. As MSDN is a key Microsoft public interface to developers, I thought I’d answer that from my own perspective.

Being a Program Manager at MSDN offers me a unique position within Microsoft as my specific job (Content Strategist) is to publish content that balances the needs of marketing and the technical product teams with our own knowledge of the target audience and the technologies involved while aligning with Microsoft’s overall business goals. Needless to say, the job can be quite challenging and at times more political than I’d prefer as I’ve been a programmer since the early 80’s and sometimes long for the days of being lost in low-level code as opposed to endless meetings. However, at the same time, I do enjoy my job very much as it enables me to see the business from many different vantage points and to have a basic understanding of why we do some of the things we do. One of those issues is the whole “managed code vs. native code” issue regarding why we promote the former much more than the latter.

To start with, I submit to you that there is no black and white here. We’re talking about the amount of information on managed code and managed code tools relative to native code information. In terms of native code, there are over 7,000 new APIs in the Windows SDK and every two weeks, we publish a new chapter to the Windows Vista Developer Story – a 600+ page collection of information on native SDK development. In addition, as we get closer to releasing the Windows Vista and the new Windows SDK, you’ll begin to see even more native code articles. However, most of the native code information is centralized on the Windows Vista and Visual C++ Developer Centers with the majority of information being published by MSDN being focused on managed code. Hopefully, this post will explain why – at least from my point of view.

Marketing –At Microsoft we have hundreds of products, but it’s no surprise that the reason we remain the most profitable software company in the world is by virtue of selling two main products - Windows and Office. In addition, it’s critical for the long-term stability of Microsoft that we also have a major impact on the Web.

Everything else (especially development tools) is simply a means of accomplishing those goals. An example is the Windows SDK. It’s completely free to anyone that wants to download it as it serves the greater purpose of getting developers to write Windows applications, which in turn sells more copies of Windows. In addition, you frequently see us hold training seminars that, at best, break even and might even lose money. Once again, the goal isn’t making money from these products or functions. The goal is to get information into the hands of developers so that they write Windows applications – or Web applications using Microsoft technologies.

There are many more examples – such as the free Visual Studio Express products, free training, free technical articles on MSDN and so on, but you get the point. The focus is always things like, “What can we do to the O/S to enable developers to create the apps they want for their customers?”, “How can we make the development tools easier to use to lower the cost of delivering that software in a timely fashion?”, etc.

Having said that, the latest technologies that accomplish the goals of selling Windows, pushing the Web and making it easier for developers to write Windows applications, are things like Windows Presentation Foundation, Windows Communication Foundation, Windows Workflow, etc.

Therefore, most conversations between marketing and MSDN deal with focusing on these newer technologies – in the form of publishing technical articles and whitepapers written by evangelists, promoting training and seminars, providing demos and hands-on labs and so on.

Product Teams – This is an easy one as most likely if you’re reading my blog, you’re also a developer and realize that most developers want to work on the latest, coolest thing. Our dev teams are no different. Therefore, internally when our product team developers write for MSDN they’re generally writing about the latest technologies. In addition, the various levels of management for these product teams also focus on requesting that we more heavily promote the latest innovations and features of their products.

Target Audience – If you ever want the definitive answer for why we at Microsoft do something, follow the metrics. The Visual Studio and .NET Developer Centers are – by far - the two most popular Dev Centers (in terms of traffic and users). They are followed up by Windows Vista (which is gaining rapidly despite still being in beta) and VB.NET. The meaning is clear. Developers come to MSDN looking for information about managed code and Windows Vista.

Therefore, one way of answering the question of why we post so much managed code content (relative to native code) is via the old adage, "We don't make the news; we only report it." Like any other company we stay in business by meeting the needs of our customers. If customers weren't asking for and responding favorably to this, we’d be going in a different direction. Therefore, it’s simply inaccurate to say that we’re forcing anything on anyone. We’re the ones reacting to what the masses have requested and right now that’s managed code and tools for developing advanced UIs in the fasted time possible.

Free Microsoft Certification Exam (in Visual C++)

tomarcher — Thu, 26 Jan 2006 21:38:00 GMT

Sorry for the late notice, but I've just been informed that there's a free Visual C++ 2005 certification exam available (#71-526). While this is a "beta" exam and is only good until January 31, it does count towards certification in the same way as the final version of the exam. Note that the exam can be taken using Visual C++, Visual C# or Visual Basic - you specify your desired language when the exam begins.

The story here is that Microsoft has launched a "new generation of certifications" consisting of two distinct series of credentials for .NET Framework developers:

Microsoft Certified Technology Specialist (MCTS) - The MCTS credential highlights your ability to develop Windows, Web, or distributed applications using the .NET Framework 2.0 and Visual Studio 2005.
Microsoft Certified Professional Developer (MCPD) - The MCPD credential highlights your job role, featuring your specific area of expertise and allowing you to distinguish yourself as an expert in Windows Development, Web Application Development, or Enterprise Applications Development.

To explain how these exams fit together the bigger picture:

Exam 70-526 (the "final version" of this exam") is one of two exams required for the “Microsoft Certified Technology Specialist: .NET Framework 2.0 Windows Applications” (MCTS) credential. To achieve this credential you will also need to pass exam “70-536: TS: Microsoft .NET Framework 2.0 - Application Development Foundation”.

After achieving the MCTS credential for Windows applications, you will be able to achieve the “Microsoft Certified Professional Developer: Windows Developer” (MCPD) credential by passing one additional exam: “70-548 PRO: Designing and Developing Windows-Based Applications by Using the Microsoft .NET Framework.”

Special Notes

Please be aware of the following:

Promotional Code and Exam Number - The exam number is 71-526 and the promotional code is TSB526
Passing the exam - Candidates who take and achieve a passing score in the beta exam will receive a free exam voucher for any Microsoft Certification Exam. Vouchers will be sent after exam scores are tabulated.
Restrictions - The beta exams are not offered in China, India or Pakistan. Please don't waste your time sending me emails about this issue as I have no control over it.

Registering For and Taking the Exam

This is an in-person exam only (not an online exam). However, there is information online about the exam itself, including a self-test evaluation that you can take to measure your preparedness.

Exam Information

To register for the test in your area:

Thomson Prometric: (800) 755-EXAM (800-755-3926)
Pearson VUE: 800 TEST Registration (800-837-8734)

Outside the U.S./Canada, please visit the following Web sites for registration information:

Thomson Prometric (You must register with Thomson Prometric via telephone - (800) 755-EXAM (2926)
Pearson VUE

Free Visual Studio 2005, SQL Server 2005 and ASP.NET 2.0 Training!

tomarcher — Sun, 06 Nov 2005 22:23:00 GMT

This past Friday, we announced that our Training/Learning division is offering *free* training on several courses related to the November 7 launch of Visual Studio 2005, SQL Server 2005 and BizTalk Server 2006. From what I'm being told this training is only free until November 17, 2005 so you need to take advantage rather quickly.

Retrieving special folder locations the right way (and some history)

tomarcher — Tue, 01 Nov 2005 22:53:00 GMT

Raymond Chen has a great one page piece in this month's TechNet magazine about why you should use the Shell API to retrieve special folder location information (such as the Fonts folder), why the Shell Folder registry key should not be used and how that key came to be in the first place:

The Sad Story of the Shell Folders Key

Video from Product Team on new products

tomarcher — Thu, 27 Oct 2005 21:17:00 GMT

In the latest installment of "Microsoft After Dark", the product teams speak of the new innovations related to SQL Server 2005, Visual Studio 2005 and BizTalk Server 2006:

http://www.microsoft.com/winme/0510/25540/welcome.htm

Get Visual Studio 2005 today!!

tomarcher — Thu, 27 Oct 2005 19:48:00 GMT

While the official launch is still November 7, you can get Visual Studio 2005 today with an MSDN Subscription[1].

If you have an MSDN Subscription - if not, this is a great reason to get one! - you can obtain the final versions of the following products today:

Visual Studio 2005 Standard

Visual Studio 2005 Professional

Visual Studio 2005 Team Suite[2]

Visual Studio 2005 Roll based products (Architect, Dev, Test)

SQL 2005 Developer Edition

Visual SourceSafe 2005

Visio for Enterprise Architects

C++ tools for IA 64

Visual Studio tools for Office and Infopath

MSDN Library for VS 2005

[1] We will be posting only the CD versions of Visual Studio for download for the next few days , DVD editions will be added probably early next week along with the Enterprise, Standard, and Workgroup versions of SQL 2005.

[2] The following applies to the VSTS product:

• VST Suite is only available for download to VST Suite with MSDN Premium subscribers.
• Visual Studio 2005 Pro with MSDN Premium subscribers and VSTA/D/T with MSDN Premium subscribers have access to VST Suite trial as an upsell opportunity.
• VSPro with MSDN Professional, MSDN Operating Systems, and MSDN Library subscribers must get the VSTS Trial by other means.