One of my customers asked me how to mitigate this IE8's application compatibility issue.

Well, when their application tries to execute this code, it failes with 'invalide argument' error:

document.domain = somevalue;

Their web application works perfectly on IE6 and now they are considering moving to IE8.

Do you know why this fails?

This is explained in IE team blog as follows:

In Internet Explorer 7, the following set of calls would succeed:

// initial document.domain is app1.example.com
document.domain = "app1.example.com";  // 1. Domain property set to default value
document.domain = "example.com";       
// 2. “Loosen” domain
document.domain = "app1.example.com";          // 3. “Tighten” domain

In Internet Explorer 8 and other browsers, the 3rd assignment will throw an exception, because app1.example.com is not a suffix of the then-current value, example.com.

http://blogs.msdn.com/ie/archive/2008/09/02/ie8-security-part-vi-beta-2-update.aspx

In this case, I think the developer who understands the application needs to find how to mitigate this issue, considering using  postMessage() or XDomainRequest.