I just realized I was so busy lately that I haven’t blogged for a while!
Here’s a quiz that left me clueless for some time (courtesy of our C# MVP Ahmed Ilyas):
using System;using System.Diagnostics; public class Examples{ public static void Main() { string stringToTest = "Hello"; Stopwatch equalsTimer = new Stopwatch(); equalsTimer.Start(); stringToTest.Equals("hello", StringComparison.OrdinalIgnoreCase); equalsTimer.Stop(); Console.WriteLine("Equals Timer: {0}", equalsTimer.Elapsed); Stopwatch compareTimer = new Stopwatch(); compareTimer.Start(); String.Compare(stringToTest, "hello", StringComparison.OrdinalIgnoreCase); compareTimer.Stop(); Console.WriteLine("Compare Timer: {0}", compareTimer.Elapsed); }}
On my machine, this prints out:
Equals Timer: 00:00:00.0009247 Compare Timer: 00:00:00.0000012
We looked at the source code of string.Equals and string.Compare and it was essentially the same (modulo very minor details which shouldn’t cause issues).
So what’s wrong? Why would the first call be 770 times slower than the second one? Jitting? No. Cache hit/miss? No.
After a while, we figured it out [UPDATE: So I thought!]. The first method is a virtual instance method, so a callvirt is emitted by the compiler:
callvirt instance bool [mscorlib]System.String::Equals(string, valuetype [mscorlib]System.StringComparison)
While the second method is a static one, so the call instruction is used instead:
call int32 [mscorlib]System.String::Compare(string, string, valuetype [mscorlib]System.StringComparison)
In this case, the method body was insignificant compared to the costs of doing virtual dispatch vs. a direct call. If you’d measure this in a loop of 1000000, the results will average out. So will they average out if you compare long strings, when the method body execution time dwarfs the call costs.
UPDATE: As always, Kirill jumps to conclusions too fast. Ahmed pointed out that if you swap the order of the calls, then the results are way different again! So it’s not the callvirt cost. Still puzzled, maybe it IS the JITter compiling the BCL code for the two method bodies.
Interesting...
Are the strings interned during the second method call? Just a pre-dinner guess...
No, probably not... It's most likely the methods being jitted...
That's not really a valid test at all. There's probably common code between both methods that is being JIT'd and processor caching taking place in the first call that drastically speed up the second call. If you don't want to add loops to smooth it out, you could just do Equals, Compare, Equals, Compare and only measure the last two.
My guess is the first call to String.Equals loads mscorlib.dll since that would be the first place mscorlib is actually needed. Try placing a call to String.Equals before starting the stopwatch.
Try doing something like 1000000 iterations for each call and then splitting the total time with the number of iterations. That way you the JIT will not impact your benchmark.
> maybe it IS the JITter compiling the BCL code for the two method bodies.
Framework code is NGENed though, so it doesn't need to be JITted.
It's easy to show that it's to do with caching, paging, JITting or something like that: instead of timing two different things, time the same thing twice - you'll still get the same results. My guess is that it's bringing the code into the CPU's L1 cache which is "slow". Of course, if you put in a loop of 10 million iterations or so in order to time a sensible length of operation, this difference vanishes.
Alternatively, you can call the method you're about to time *before* you start timing it as well... although in this case, it's all so fast that it's worth doing the same thing with the Stopwatch methods, to make sure the cost of bringing Stopwatch.Stop into the cache (or whatever's slowing it down) doesn't pollute the results.
When running the tests 100000000 times in a loop, the timings on my machine come out as 27.24s for equals and 25.80s for Compare, which is of course far from the performance difference that you get for the single instances. For such short methods, you cannot meaningfully micro-benchmark them by running them just once. Caches and all other kinds of stuff may interfere with your results.
What is interesting though, is that Compare seems to come out as a few percent faster in all of my tests. Why that is the case, I can only guess from looking at the BCL code in Reflektor. There, it seems that the code is identical with the exception of Equals using a length check to quickly return false for strings of different length. It then uses them same comparison routine as Compare and compares the result of that routine with 0, which probably adds another few cycles.
My guess is that the length check, which is not helping in the present example, is what is responsible for the difference in run time.
Having said all that, unless you have a really really tight loop that does nothing but billions of string comparisons, I would use what seems most natural in the program and not worry about the performance difference at all.
indeed. I was just curious to know what "difference" there would be with String.Equals and String.Compare and came around to finding out that both methods can have the same types of parameters, and was curious to know if there would be a difference in perf (I know its very minor minor perf difference - but you wouldn't believe some developers/clients who code....poorly!)
But it does show that String.Compare is slightly faster than String.Equals when you pass in the parameters as shown in the example.
This was one of those "right, lets investigate and learn" nights I had recently :-)
So maybe better practice would be to use String.Compare in some scenarios with the option of having culture sensitivity if required.
What an outrageous assumption that you can evaluate the performance of a method that takes 1ms to execute by executing it only once. I would not even trust the timer in that small time period because the cpu clock frequency gets varied on some cpus. stopwatch is relying on this.
Why are you outruling the jit? That is the most likely cause. A virtual call is only a few times more costly than a normal one. What an unlikely explanation.
Tobi - agreed that the CPU clock freq is varied on some CPU's. The real original problem was finding that there are 2 methods in the .NET Framework that allow us to do the same thing (but different return types), except one takes longer to execute than the other - this happened to be String.Compare that took longer to execute than String.Equals, giving it the same parameter values, (on the first hit) - one would have thought that it may be almost the same or equally the same execution time. Hope this cleans up some misconceptions.
I think it simply has to do with the GC logic. OrinalIgnoreCase needs to allocate a new string with upercase letters to do the case insensitive comparison. The GC needs to realloacte its Gen0 Heap quite some time to fine tune its size. After it has allocated enough Small object heaps the second test will run much faster since no reallocations are needed anymore.
You can check this by changing it to CompareOrdinal in the comparison.
Yours,
Alois Kraus
they are quite the same if you test on more iterations:
static void Main()
{
const string stringToTest = "Hello";
const int cycle = 100000000;
var equalsTimer = new Stopwatch();
equalsTimer.Start();
for (int i = 0; i < cycle; i++ )
stringToTest.Equals("hello", StringComparison.OrdinalIgnoreCase);
//String.Compare(stringToTest, "hello", StringComparison.OrdinalIgnoreCase);
}
equalsTimer.Stop();
Console.WriteLine("Equals: {0}", equalsTimer.Elapsed);
Console.ReadLine();
"The real original problem was finding that there are 2 methods in the .NET Framework that allow us to do the same thing..."
They most certainly do not do the same thing. Compare determines the relative order of two strings based on culture and casing rules. Equals simply determines if two strings are equivalent. For example, you can definitively return false immediately from Equals if two strings differ in length. But for Compare you still need to compare the characters until their order is determined.
yes thats right Josh. sorry, my mind has been everywhere. I know these 2 classes work differently but I meant in that when doing an ordinalignorecase on both methods, with the same test that it returns the evaluated result, in this case if they match then its a 0 for compare and true for equals. maybe im looking way way too much into this and over confusing myself, which I sholdn't do!