I was able to cause this error in Internet Explorer 6 and Internet Explore 7.  I found that editing my registry entry so that the user agent was greater than 260 I could cause this error in jscript dialogs.  By removing entries to get below the size I could avoid the error.  This is the registry location for the user agent strings (see http://support.microsoft.com/kb/218933 for setting this using IEAK):

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\User Agent\Post Platform

Calls to the navigator object method userAgent was the source of the error.  I found that the size of the string to hold this value (internal to the Navigator object) is hard coded to a length of 260.  

update:  This affects three methods on the object:

appCodeName
appVersion
userAgent