Larry Osterman's WebLog

Glad I'm not the only one who was confounded by Pete's column. When he started into using gets(), I just gave up and assumed that it was a mis-dated April fool column.

Pete's statement about not worrying about gets() buffer overflows in a "suite of applications... with the assurance that designed-in assumptions about line lenghts are valid" and "...not something I stay up nights worrying about: My computer sits on my desk, and no malicious user can sneak in during the night and modify my saved files." strikes me as silly. If you're talking about a program you wrote to catalog your CD's or somesuch, that's fine. A program that's going to be running on computers that are not "your computer" can't act that way, though. Someone else _is_ going to stay up nights worrying about it-- or, rather, looking for it.

It seems like good practice to me to have every function check the validity of the caller's arguments, and to check the status returned by any functions it calls. The only time I'd deviate from this design would be when profiling shows a clear performance win. I'd change the function to perform validation "up front", and remove per-iteration validation. Of course, you must guarantee that there are no paths of execution that would bypass the "up front" validation, and as such, I'd consider the removal of parameter validation from a called function as grounds to inline that function in the caller, just to avoid a potential path of execution that lacked validation.

You're right, that article is pants. First, it's not clear that "safe" functions are any slower. Modern processors can schedule three instructions at the same time, especially trivia like using a register to keep track of the free space in a buffer. Second, if "safe" functions are too slow (implausible as this is), then use a better string representation. If you're checking all string lengths at the boundary then why not convert them to a better representation, such as counted strings, at the same time. This has the further advantage that there's no chance of unchecked strings slipping through to your potentially unsafe code. But your automotive analogy is poor. Most drivers have a certain level of acceptable risk. If you reduce their risk with safety features they'll drive more recklessly to increase the risk back to their acceptable level. So the gross effect of safety features is no improvement in driver safety coupled with increased risk for pedestrians, cyclists, etc. A net loss. Much research supports this.

I hate to say it, but Dr Dobbs lost me many year ago when the published an article about a new super fast sorting algorithm. What was the great advancement? Swapping pointers to elements instead of elements. Um... yeah... Makes me wonder if there is any type of review. It really isn't the articles that I know are stupid that scare me, it is the articles that I don't know that they are stupid.

You hit the nail on the head.  And it's an extremely wide-spread problem.  And, as you say, it's common to simply fall-back to "just design it correctly" and use of str*() is fine.  If that were the case we wouldn't have security issues with software.

The scary part is; there's 3rd party libraries modeled around the CRT--rife with unsized pointer arguments, meaning there's no possible way it can reliably ensure it's safe.

I agree with your criticisms of the article. By the way, though gets isn't the only problem, I was amazed to see that writeup. Around 10 years ago Doug Gwyn was defending use of gets in situations where he felt its use was sufficiently controlled. I was amazed then too. No one agreed with him, which of course wasn't amazing a bit. Now someone agrees with him. > (oh no, not another stupid automotive analogy). If you drive > safely, you don't need seat belts Yeah, there are people who say that, who really believe that as long as they drive safely there won't be some idiot coming the other way and suddenly making a right turn[*] right in front of you with no possibility for you to stop in time, there won't be some idiot barrelling through a red light after everyone else who has the red light already stopped and others who have the new green light already passed through, etc. [* In some countries think about that as a left turn instead.] Meanwhile did you read his footnotes? >> [3] When Borland was working on its first compiler that >> supported Windows, we had a mysterious crash in one of >> the programming examples from Microsoft's Windows SDK. >> The program ran fine with Microsoft's compiler and library, >> and crashed with ours. Since our compiler and library were >> incomplete, we naturally focused on them. But it turned out >> that the problem was in the example, which had a buffer >> overrun. We can STILL find problems like that, both in example code in MSDN articles and in some of the header files that Microsoft distributes as source. A few days ago for some reason I was reading part of winnt.h, two versions of which get included in programs build by Visual Studio 2005 for Windows Mobile 5 targets. I don't think I saw anything that would cause a buffer overrun, but some of the contents still sent chills up my spine. Then think about the stuff that Microsoft doesn't publish because it's likely even buggier than the stuff they do publish.

Great article Larry. The old saying 'don't believe everything you read' is quite true here. Without people like you willing to stick their head up, some of us less experienced programmers might follow their advice with the inevitable consequences.

OpenBSD guys created a safe strlcpy: http://www.gratisoft.us/todd/papers/strlcpy.html

This is offtopic, but your blog comments seem to randomly switch between supporting HTML syntax and not.

e.g. This should be a new paragraph (using <p>).

Why not use strcpy if I've proven that the parameters will not cause an overflow? Data that comes from file/network/user input can be anything and it has to be appropriately handled and checked. But internal (or processed by your carefully designed app) data can be guaranteed to be safe. In this case assert() is enough.

>one of the critical bugs in strncpy - strncpy isn't guaranteed to null terminate the output string

If it did there would be no way to detect the case when the destination is too small to hold the whole source.

If you're the world's best and safest driver, seatbelts protect you from all the morons out there. If you're the world's best and safest programmer, safe string copy functions protect your code from all the morons out there who accidentally work around your safety checks. And, also, if you're not quite as great as you think you are, they keep you from getting a whole lot of egg on your face. ;)

I think the real point of the article is being missed here. Surely everyone here would agree that formal proof of buffer overflow impossibility through static analysis and design is a MUCH better way to produce secure software than runtime buffer size checks? Just checking string bounds isn't always enough -- you have to make sure that a truncated string isn't used in a way that could cause a different vulnerability, or that throwing exceptions doesn't allow an easy way to cause a DoS, etc. I don't think bounds checking is a bad idea, as it definitely helps with robustness as well as security (think hardware errors!), but it's not a _great_ solution. For calls that are solely internal, it's a bit like speculatively adding try/catch. Personally, I would rather see more emphasis on language features and tools that help me declare and enforce guarantees at compile time, because that would help avoid a lot more problems than just buffer overflows. Also, I think the performance issue is a bit of a red herring, not because the checks are "free" -- they aren't, either in runtime or code size cost -- but because string operations themselves aren't cheap, and overuse of strings is a common performance problem.

Friday, October 06, 2006 3:03 AM by Phaeron

> formal proof of buffer overflow impossibility through static

> analysis and design is a MUCH better way to produce secure

> software than runtime buffer size checks?

That is 100% true.

For some kinds of programs you can do a formal proof.  How do you construct a formal proof?  In my experience it was done by humans applying logic to demonstrate that each fact is provable from previously proven facts.  In other words, humans were applying the same logic that humans apply in writing programs.  So the bug rate wasn't zero, the bug rate might potentially be geometrically lower (two simultaneous failures being geometrically less probable than one failure), but the actual results weren't that good.

In the general case there won't always be even a theoretical possibility of a formal proof.  It reduces to the halting problem.

Now, C strings are miserable.  Most of the C language combines the power of assembly language with the beauty of assembly language because it was invented as a substitute for assembly language.  Of course since being standardized it's not always possible to use it that way any more, but it still reflects the beauty of its origins (^_irony).  The strings section of the library seems to have been designed more hastily.  It took advantage of machine instructions which an assembly language programmer could code quickly for one kind of machine.  It didn't prepare for efficient but harder coding that would measure once before copying and concatenating and comparing several times.  C++, being object-oriented assembly assembly, derives with the same weak base, though it's a bit easier to make counted string libraries that will be easily usable by clients.

A few days ago I used _tcscpy safely.  I ran a few _tsclen calls, added the results, did a malloc, checked the result of malloc, and then if still running I copied each string to the appropriate portion of the resulting buffer.  Then I cringed at the thought of what will happen when someone copies and pastes the code without figuring out why it was OK this once.  I wished I could just use VB's & operator instead.

The real solution is to use C++

> The real solution is to use C++

Why stop here and not switch to C# or VB?

As I understand it, the main argument against to "Why stop here [C++] and not switch to C# or VB[.NET]?" is that C++ is still a LOT faster running...like 16 to 64 times faster...mostly due to the asbsence of strict variable typing and the direct access it affords to memory.  

That's how I used to think even just a few years ago, and I still do, to the extent that I think unmanaged code is still a hellalot better for, say, massive scientific simulations and the like, because it's still WAY WAY WAY faster, deny it all you like.

However, for pretty much any business-related project I could imagine, and most anything else non-scientific (or gaming), the hardware is now simply so fast that: who cares how much slower managed code executes?...it's just not generally an issue that it runs many times slower.  So, in the final analysis, I would never use C or even C++ for anything usual, especially if I needed the result to securly withstand public use.

"A few days ago I used _tcscpy safely.  I ran a few _tsclen calls, added the results, did a malloc, checked the result of malloc, and then if still running I copied each string to the appropriate portion of the resulting buffer."

Did you remember to check to for overflow on your addition?

He's not claiming that you should use assert(3) for parameter validation; indeed in that bit of the article he explicitly states that he wants to avoid doing parameter validation everywhere on efficiency grounds(!). The assert is just an expression of the "contract" of the function. Sure, the suggestions are pretty silly, but not really in the way you suggest, at least in that case.

If C# and C++ are too slow, then switch to Java.

I switched and never looked back.