Another dirty trick played by an antivirus

Another dirty trick played by an antivirus

  • Comments 1

A PFE colleague from Italy, recently exposed the following situation in an internal distribution list:

From: <removed_to_protect_identities>
Sent: Wednesday, July 18, 2012 8:36 AM
To: <removed_to_protect_identities>
Subject: Report hanging when rendering on IE9

Hi there,

I am browsing a set of reports on RS2008 Sp3 with Internet Explorer 8 and IE9.

When browsing with IE8, no problem, reports complete in few seconds.

If I use IE9 the same reports hang, without completing rendering, without any error message. The SQL query completes correctly behind the scenes, but the reports never end. It is not possible to troubleshoot by querying the ExecutionLogStorage in the ReportServer db, since the reports do not end.

Is there any known issue about IE9?

Any help would be appreciated.

Thanks a lot

<removed_to_protect_identities>

MS Services Italy

I first suggested him to trace the HTTP messages exchanged between Internet Explorer and Reporting Services Web service in the scenario where the report was fully rendered and in the one that never completed, to determine whether or not the server was somehow sending back incomplete responses or some sort of malformed content that was causing that hanging like behavior on the client.

Indeed, when the hang occurred either Fiddler or the integrated IE developer tools let us realize the client had issued several GET requests to the server which were still Pending to fully receive a response.

clip_image001

With that, I requested the manual generation of a memory dump of the Reporting Services server process to analyze whether there was something pending to process from that interactive report request or not. A dump of the server process when the client showed that hanging state revealed the server had nothing pending to process. So, I requested the LogFile and it showed this:

rshost!rshost!12f8!07/19/2012-17:44:58:: e ERROR: HttpPipelineCallback::SendResponse(): failed writing response.

rshost!rshost!12f8!07/19/2012-17:44:58:: e ERROR: Failed with win32 error 0x10DD, pipeline=0x000000000064DD30.

0x10DD being ERROR_INVALID_OPERATION returned by HttpSrvConn::WriteAsync.

I correlated that error to the fact that customer had an antivirus installed in that machine and that some of their well-known intrusive components (like their buffer overflow protection module) were loaded into SSRS’ address space (see below), plus taking into consideration the server was in test (not in production yet) I suggested they should uninstall the antivirus, reboot the machine, and give it another try.

As most the time happens with this type of issues, customer refused to uninstall that piece of software. This time at least they accepted creating an image server in which they would also install the AV software but they would leave the AV components in an off state. Well, it’s better than nothing, but as explained to them it doesn’t guarantee that those AV components are not poking around even when their UI say they are “disabled”. AV or security software manufacturers tend to understand “Disabled” as a “I’ll continue with all my intrusive way of doing, only that if I detect something suspicious I won’t tell anyone. But I can keep being the cause of performance problems, memory leaks, or memory corruptions.“

00000000`6fa00000 00000000`6fa3f000   sophos_detoured_x64   (export symbols)       sophos_detoured_x64.dll

    Loaded symbol image file: sophos_detoured_x64.dll

    Image path: C:\Program Files (x86)\Sophos\Sophos Anti-Virus\sophos_detoured_x64.dll

    Image name: sophos_detoured_x64.dll

    Timestamp:        Fri Jan 14 21:09:51 2011 (4D30AD8F)

    CheckSum:         00044BBC

    ImageSize:        0003F000

    File version:     9.5.5.9574

    Product version:  9.5.5.0

    File flags:       0 (Mask 3F)

    File OS:          40004 NT Win32

    File type:        2.0 Dll

    File date:        00000000.00000000

    Translations:     0409.04b0

    CompanyName:      Sophos Plc

    ProductName:      Sophos Anti-Virus

    InternalName:     sophos_detoured.dll

    OriginalFilename: sophos_detoured.dll

    ProductVersion:   9.5.5

    FileVersion:      9.5.5.9574

    FileDescription:  Sophos Buffer Overrun Protection

    LegalCopyright:   © 1989-2011 Sophos Plc, www.sophos.com

00000000`75500000 00000000`75511000   swi_lsp_64   (export symbols)       swi_lsp_64.dll

    Loaded symbol image file: swi_lsp_64.dll

    Image path: C:\ProgramData\Sophos Web Intelligence\swi_lsp_64.dll

    Image name: swi_lsp_64.dll

    Timestamp:        Tue Sep 28 16:23:30 2010 (4CA1FA62)

    CheckSum:         000116BD

    ImageSize:        00011000

    File version:     1.0.6.0

    Product version:  9.5.4.0

    File flags:       2 (Mask 17) Pre-release

    File OS:          4 Unknown Win32

    File type:        2.0 Dll

    File date:        00000000.00000000

    Translations:     0409.04b0

    CompanyName:      Sophos Plc

    ProductName:      Sophos Anti-Virus

    InternalName:     swi_lsp

    OriginalFilename: swi_lsp.dll

    ProductVersion:   9.5.4.0

    FileVersion:      1.0.6.0 [2038726]

    FileDescription:  Sophos Web Intelligence

    LegalCopyright:   © 2010 Sophos Group, www.sophos.com

So, on they went and, on a new Windows Server, they installed Reporting Services and all the reports the customer used, only that in that new server they kept the antivirus services disabled.

Guess what? The issue did not show up.

However, as soon as they turned the antivirus on the hanging issue was systematically reproducible. They performed such tests many times confirming that the antivirus was the cause of those weird hangs.

They also noted that that issue appeared only when browsing the reports from the server itself. When the browsing was initiated from remote clients, that caused no problem, even if the AV was installed also on them.

It looks like it was a very specific combination of ingredients that caused the issue. Something along the lines of a data stream contained in that specific HTTP request with a very peculiar pattern one of the components of the antivirus didn’t like when it came from a request from the local host.

Once demonstrated, IT Administrators in that customer decided to configure their antivirus software so that it didn’t hooked into Reporting Services activities and inexplicably tampered with some extremely specific report requests.

No matter how strong your case against an antivirus component may be, convincing a customer that they must deactivate some parts or all of its components is a tough task. By default, people tends to over trust antivirus software above anything else.

We already had KB2033238 - You may experience performance and consistency issues with SQL Server when certain modules are loaded into SQL Server address space which warned of these types of problems, but we had never seen them in the context of Reporting Services up until now.

Leave a Comment
  • Please add 6 and 4 and type the answer here:
  • Post
  • Nice to know this Nacho. Thanks!

    I especially like your description of AV's disabled state! There is a reason KB309422 was extensively rewritten back in February.

Page 1 of 1 (1 items)