ASP.NET 2.0 and IIS 6.0

Don't use .NET System.Uri.UnescapeDataString in URL Decoding

Yangxin Deng — Thu, 09 Nov 2006 03:57:00 GMT

Don't use .NET System.Uri.UnescapeDataString in URL Decoding

URL Encoding should encode Space into "+" or "%20". URL Decoding should decode "+" or "20" into Space. However by design, System.Uri.UnescapeDataString doesn't decode "+" into Space.

The MSDN remark of Uri.UnescapeDataString says:

“Many Web browsers escape spaces inside of URIs into plus ("+") characters; however, the UnescapeDataString method does not convert plus characters into spaces because this behavior is not standard across all URI schemes.”

The issue will rise when your web application has query string like:

http:// www.ms.com/default?Comment=just+do+it

If you use System.Uri.UnescapeDataString to decode the query string value "just+do+it", the result is "just+do+it" instead of "just do it". When the downstream application need to URL encode the value again, it becomes "just%2bdo%2bit ". The final URL will looks like

http:// www.ms.com/default?Comment=just%2bdo%2bit

The spaces get lost and application could interpret the value as "just+do+it" instead of "just do it".

Detailed discussion:

RFC2396 defined reserved characters such as &, $, + and excluded characters such as space, %, < > must be escaped (URL encoded) when used as values in query string of URL in order to keep the original meaning of the character.

For example: to pass information such as

Products : Windows&Office Price: $200 Comment: In Stock Sign:+

The URL could be

http://www.ms.com/default.aspx?Products=Windows%26Office&Price=%24200&Comment=In%20Stock&sign=%2b

http://www.ms.com/default.aspx?Products=Windows%26Office&Price=%24200&Comment=In+Stock&sign=%2b

URL may be used as return URL value in other URL. In the case, the URL need to be encoded and already encoded characters will be double encoded.

http%3a%2f%2fwww.ms.com%2fdefault.aspx%3fProducts%3dWindows%2526Office%26Price%3d%2524200%26Comment%3dIn%2520Stock%26sign%3d%252b

http%3a%2f%2fwww.ms.com%2fdefault.aspx%3fProducts%3dWindows%2526Office%26Price%3d%2524200%26Comment%3dIn%2bStock%26sign%3d%252b

Characters	Single Encoded	Double Encoded
&	%26	%2526
$	%24	%2524
+	%2b	%252b
Space	%20, +	%2520, %2b
%	%25	%2525
<	%3c	%253c

Notice Space's single encoding can be "+" and double encoding can be "%2b" and + sign's single encoding is %2b.

If the function doesn't handle the encoding properly, the original meaning of the character could be lost in transaction.

The right encoding or decoding methods should do what the above table defines.

.NET encoding methods

Characters	HttpUtility.UrlEncode	System.Uri.EscapeDataString	System.Uri.EscapeUriString
&	%26	%26	&
$	%24	%24	$
+	%2b	%2B	+
Space	+	%20	%20
%	%25	%25	%25
<	%3c	%3C	%3C

Notice:

1. System.Uri.EscapeUriString doesn't encode RFC reserved characters

2. URLEncode encodes Space as "+" and EscapeDataString encode Space as "%20".

3. To encode the whole URL as return URL, EscapdeUriString should not be used.

.NET Methods	http://www.ms.com/default.aspx?Products=Windows%26Office&Price=%24200&Comment=In+Stock&sign=%2b
URLEncode	http%3a%2f%2fwww.ms.com%2fdefault.aspx%3fProducts%3dWindows%2526Office%26Price%3d%2524200%26Comment%3dIn%2bStock%26sign%3d%252b
EscapeDataString	http%3A%2F%2Fwww.ms.com%2Fdefault.aspx%3FProducts%3DWindows%2526Office%26Price%3D%2524200%26Comment%3DIn%2BStock%26sign%3D%252b
EscapdeUriString (not right)	http://www.ms.com/default.aspx?Products=Windows%2526Office&Price=%2524200&Comment=In+Stock&sign=%252b

.NET Methods	http://www.ms.com/default.aspx?Products=Windows%26Office&Price=%24200&Comment=In%20Stock&sign=%2b
URLEncode	http%3a%2f%2fwww.ms.com%2fdefault.aspx%3fProducts%3dWindows%2526Office%26Price%3d%2524200%26Comment%3dIn%2520Stock%26sign%3d%252b
EscapeDataString	http%3A%2F%2Fwww.ms.com%2Fdefault.aspx%3FProducts%3DWindows%2526Office%26Price%3D%2524200%26Comment%3DIn%2520Stock%26sign%3D%252b
EscapdeUriString (not right)	http://www.ms.com/default.aspx?Products=Windows%2526Office&Price=%2524200&Comment=In%2520Stock&sign=%252b

There are two decoding methods in .NET

Encoded Characters	HttpUtility.UrlDecode	System.Uri.UnescapeDataString
%26	&	&
%24	$	$
%2b	+	+
%20	Space	Space
+	Space	+
%25	%	%
%3c	<	<

Notice that UrlDecode UnescapeDataString decode "+" differently. This will cause problem when decoding return URL which contains double encoded Space as "%2b".

For example: "Comment%3dIn%2bStock" in encoded return URL should be double decoded into

Variable: "Comment" Value: "In Stock"

Call UrlDecode twice on it

"Comment%3dIn%2bStock" à "Comment=In+Stock" à "Comment=In Stock"

Call UnescapeDataString twice on it

"Comment%3dIn%2bStock" à "Comment=In+Stock" à "Comment=In+Stock"

The original string "In Stock" is broken by UnescapeDataString.

If the downstream application assumes the URL string had be restored to not encoded format "In Stock" and use it as input to encode it again, the single encoding will become

"Comment=In+Stock" à "Comment%3dIn%2bStock"

Instead of

"Comment=In Stock" à "Comment=In+Stock"

Conclusion:

Since an application has no control of its upstream (use input or config), it can only assume the right encoding is in the URL query string: Single encoded special character as query string parameter value. Especially the Space can be "+" or "%20". When the URL needs to used as return URL in query string, it must be encoded again. Space will be double encoded as "%2b" or %2520".

When the receiving application received the encoded URL, if it uses method like UnescapeDataString for decoding, the "%2b" will not decoded into Space, Instead it becomes "+" as final result.

Developer should avoid encoding Space into "+" or double encoded into "%2b". It is recommended that when encode URL use "System.Uri.EscapeDataString", when decode URL use " HttpUtility.UrlDecode"

Tester should ensure that

1. Reserved and Excluded characters as defined by RFC2396 should be singled encode when used as value in query string of URL as next table. (URL as links, config values or test values).

2. If the URL is used in return URL or value of another query string, the Reserved and Excluded characters should be doubled encoded as next table.

Characters	Single Encoded	Double Encoded
&	%26	%2526
$	%24	%2524
+	%2b	%252b
Space	%20, +	%2520, %2b
%	%25	%2525
<	%3c	%253c

Two test URL can be

http://www.ms.com/default.aspx?Products=Windows%26Office&Price=%24200&Comment=In%20Stock&sign=%2b

http://www.ms.com/default.aspx?Products=Windows%26Office&Price=%24200&Comment=In+Stock&sign=%2b

How to disable Managed Debug Assistants?

Yangxin Deng — Tue, 21 Mar 2006 21:11:00 GMT

When I wrote a C# program which calls a library which is a wrapper of COM ojbect and debugging with Visual Studio 2005, I always got the annoying Managed Debug Assistants popup complains about InvalidVariant.

InvalidVariant was detected
Message: An invalid VARIANT was detected during a conversion from an unmanaged VARIANT to a managed object. Passing invalid VARIANTs to the CLR can cause unexpected exceptions, corruption or data loss.

The MSDN article http://msdn2.microsoft.com/en-us/library/d21c150d.aspx gives 3 ways to disable the MDA but none of them works for me.

There is MSDN blog about MDA but none of them get the right solution about how to disable MDA. http://blogs.msdn.com/jmstall/archive/2005/11/10/introducing_mdas.aspx

Here is the right solution:

Bring up the Debug -> Exceptions dialog. Under Managed Debugging Assistants, uncheck the InvalidVariant MDA.

Refresh cache in CacheItemRemovedCallback can cause StackOverFlow in ASP.NET 2.0

Yangxin Deng — Wed, 21 Dec 2005 02:37:00 GMT

If you have code that refresh ASP.NET Cache using Cache.Insert in CacheItemRemovedCallback, it works under ASP.NET 1.1 but it may cause infinite loop under ASP.NET 2.0.

Create a ASP.NET 2.0 web form project using this code.

using System;
using System.Collections;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Web;
using System.Web.SessionState;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Web.UI.HtmlControls;
using System.Threading;
using System.Web.Caching;
namespace AspCacheTest
{

public partial class WebForm1 : System.Web.UI.Page
{
    // we use this isToSleep to create the race condition for first call ItemRemovedCallback
    // if recursive calls happens, all the subsequent calls will not be delayed the the sleep call.
    private static bool isToSleep = true;
    protected void Page_Load(object sender, EventArgs e)
{
    // This is request thread adding the item to the cache
    if (Cache.Get("Item01") == null)
        AddToCache("Item01", "Item01 Value");
}

private void AddToCache(string key, object value)
{
    WriteLog("a.PageLoad find : "+key+" is not in cache. Adding Item to Cache: " + key);
    DateTime exp = DateTime.Now.AddSeconds(10);
    // Using insert or add here shouldn't matter.
    Cache.Add(key, value, null, exp, Cache.NoSlidingExpiration, CacheItemPriority.Default,
    new CacheItemRemovedCallback(ItemRemovedCallback));
}

#region Web Form Designer generated code
override protected void OnInit(EventArgs e)
{
    //
    // CODEGEN: This call is required by the ASP.NET Web Form Designer.
    //
    InitializeComponent();
    base.OnInit(e);
}

/// <summary>
/// Required method for Designer support - do not modify
/// the contents of this method with the code editor.
/// </summary>
private void InitializeComponent()
{
}
#endregion

private void ItemRemovedCallback(string key, Object value, CacheItemRemovedReason reason)
{
    //
    // Here we are trying to simulate the race condition. The callback thread is put to sleep to give us
    // time to refresh the page and thus, add the same item from request thread. The static isToSleep is used
    // to turn off the Thread.Sleep() for subsequent callback executions in order to create the stack overflow quickly.
    //
    WriteLog("b.ItemRemovedCallback is called because Item: "+key+" " + reason.ToString());
    if (isToSleep)
    {
    isToSleep = false;
    WriteLog("c.ItemRemovedCallback is putting callback to sleep...");
    System.Threading.Thread.Sleep(10 * 1000);
}
    // This is callback thread "refreshing" the item in Cache.
    // Bug Repro:
    WriteLog("d.ItemRemovedCallback is inserting item : " + key);
    // Bug Fix: WriteLog("d.ItemRemovedCallback is adding item : " + key);
    DateTime exp = DateTime.Now.AddSeconds(10);
    // Bug Repro:
    Cache.Insert(key, value, null, exp, Cache.NoSlidingExpiration, CacheItemPriority.Default,
    new CacheItemRemovedCallback(ItemRemovedCallback));
    // Bug Fix: Use Cache.Add instead of Cache.Insert
    //Cache.Add(key, value, null, exp, Cache.NoSlidingExpiration, CacheItemPriority.Default,
    // new CacheItemRemovedCallback(ItemRemovedCallback));
}
private void WriteLog(string message)
{
    System.Diagnostics.Debug.WriteLine(message);
    System.Diagnostics.EventLog log = new System.Diagnostics.EventLog("Application");
    log.Source = "Application";
    log.WriteEntry(message, System.Diagnostics.EventLogEntryType.Warning, 1);
}
}
}

Steps to repro the loop:
1. Run the code in Visual Studio 2005 debugger.
2. Bring up the "Output Windows" to monitor the debugging info.
3. The Output Windows will output: a.PageLoad find : Item01 is not in cache. Adding Item to Cache: Item01 When the page is load.
4. Wait for about 30 seconds when next lines appear in the output window, click the refresh button of the page immediately.
b.ItemRemovedCallback is called because Item: Item01 Expired
c.ItemRemovedCallback is putting callback to sleep...
5. Notice the Output Windows will dump the debug info caused by the loop conditions.
a.PageLoad find : Item01 is not in cache. Adding Item to Cache: Item01
d.ItemRemovedCallback is inserting item : Item01
b.ItemRemovedCallback is called because Item: Item01 Removed
d.ItemRemovedCallback is inserting item : Item01
b.ItemRemovedCallback is called because Item: Item01 Removed
d.ItemRemovedCallback is inserting item : Item01
b.ItemRemovedCallback is called because Item: Item01 Removed
d.ItemRemovedCallback is inserting item : Item01
b.ItemRemovedCallback is called because Item: Item01 Removed
d.ItemRemovedCallback is inserting item : Item01
b.ItemRemovedCallback is called because Item: Item01 Removed
d.ItemRemovedCallback is inserting item : Item01
b.ItemRemovedCallback is called because Item: Item01 Removed
d.ItemRemovedCallback is inserting item : Item01
b.ItemRemovedCallback is called because Item: Item01 Removed
d.ItemRemovedCallback is inserting item : Item01
b.ItemRemovedCallback is called because Item: Item01 Removed
d.ItemRemovedCallback is inserting item : Item01 .................................................

Analysis:
1. Cached Item is added to the cache when the page is loaded.
2. Cache Item is expired after 10 seconds.
3. Item is removed from cache and CacheItemRemoved event is fired.
4. CacheItemRemovedCallBack is called.
5. CacheItemRemovedCallBack is put to sleep for 10 seconds to simulate the race condition.
6. Page is refreshed and Item is added back to cache at this time. (This is a simulation of race condition under multiple users environment).
7. Cache.Insert is called to add the item back to cache. What happens next is different under ASP.NET 1.1 and ASP.NET 2.0

Under 1.1:
8. Cache.Insert finds the item is already in the cache. It will remove the item, fire the CacheItemRemoved event and then insert the item.

Under 2.0:
8. Cache.Insert finds the item is already in the cache. It will fire the CacheItemRemoved event and CacheItemRemovedCallBack is called before the item is removed.
9. CacheItemRemovedCallBack will invoke the Cache.Insert again and Insert sees the item is still in the cache and jump back to step 8.

Resolution and Conclusion:
1. Use Cache.Add instead of Cache.Insert in the call back method will fix the problem.
2. Refresh cache in the CacheItemRemovedCallBack is not a good practice. Try to refresh the cache when the page is load.