December, 2008

  • Never doubt thy debugger

    How to disable HTTP compression for specific file types?

    • 3 Comments

    The question arose from a customer whom had implemented an application to stream PDF files from ASP.NET and was also using HTTP compression to save bandwidth and improve download time; using IE 6, Adobe Reader failed to open the file with the following error message:

    Adobe Reader could not open ‘<name>.tmp’ because it is either not a supported file type or because the file has been damaged (for example, it was sent as an email attachment that wasn’t correctly decoded)

    The customer had no problems with Adobe Reader using IE7, but of course they could not force their customers to upgrade.

    I already worked on a few calls similar to this one in the past and I had the chance to dig into this issue with our Escalation Engineers (both from the Internet Explorer and IIS teams) and also with the Product Group; it turned out that there is a problem in the compression mechanism in IE until version 6, which basically affected the ability to successfully decompress the HTTP/Html stream received from an IIS server when using HTTP compression. At the time there was also a minor issue on the server side of compression, but it has been fixed. The point is that for Internet Explorer 6, the Product Group decided they could not fix this problem because there is too much code to change, retest and redistribute (this is involving some core components in IE which are also widely used across the Operating System, which means a code change there, needs a full re-test of IE and Windows codebase and is way too risky in terms of the possibility to introduce other bugs in other core OS components).

    The latest information I had at the time was that the issue would have been fixed in Internet Explorer 7 since the Product Group had a massive code-rewrite; this was confirmed by the fact that the customer was not able to reproduce the problem on IE7.

    Well, we have a couple of options: disable HTTP compression for specific file types as described in Customizing the File Types IIS Compresses (IIS 6.0) (but this means you could have to add more file types over time, and you cannot controls specific file names, but the exclusion will apply to all files with the specified extensions) or store all the files you do not want to be compressed in a specific folder on your server, and then edit the metabase to tell IIS to not compress the content of that folder.
    To disable static compression for only a single directory, first enable global static compression (if it is disabled) and then disable static compression at that directory. For example, to enable static compression for a directory at http://www.mysite.com/Home/StyleSheets, run the following steps:

    1. Enable global static compression by executing the following command at a command prompt:
          adsutil set w3svc/filters/compression/parameters/HcDoStaticCompression true
    2. Disable static compression at this directory by executing the following command at a command prompt:
          adsutil set w3svc/1/root/Home/StyleSheets/DoStaticCompression false

    There is another nice post on this matter at IIS HTTP Compression and Streaming PDF's: Don't do it.

    Anyway note that this approach can he used if you are loading binary content from disk. If you load the file from disk then use BinaryWrite or Stream* classes to stream to transfer (force a download) the binary content from IIS to the client, then you should not be affected by the bug as long as modifying the metabase as explained in my previous email you exclude one (or more) specific folder from HTTP compression and your PDF files are loaded from there. Of course this implies that you can control your application, where and how PDF files or other binary content is stored and loaded from.

    If instead you extract some data from a database, create a PDF file on the fly in memory and then stream is to the client (even if using the techniques explained above) then it is not possible to tell IIS to not use HTTP compression for content which is created and manipulated all in memory, since even if we can exclude file types from HTTP compression, at application level we are working with a memory stream which is not a file type, we qualify it only when adding the relevant HTTP header value before actually writing the stream to the outgoing output for the client which is too late...

     

    Carlo

    Quote of the day:

    It is a good rule in life never to apologize. The right sort of people do not want apologies, and the wrong sort take a mean advantage of them – P. G. Wodehouse

  • Never doubt thy debugger

    ViewState validation troubles?

    • 2 Comments

    If you work with web sites in a complex/multi server environment you might be familiar with this error and as the error message itself suggest, the first thing to check if the machineKey value which must be the same across all of the involved server.

    Anyway every now and then it happens that you might get this same error even if you are not in a NBL/Cluter environment (Tess and Tom have blogged about this here, here and here) or if you are absolutely sure you have properly configured the machineKey value. So what else can lease you to this state?

    If you want some quick  background you have a look at my post here; basically we have the following (very raw) steps:

    • Begin processing the new request
    • Load the “old” state (ViewState and ControlState) received with the posted values
    • Process the posted data (the actual form values) and raise the “change” events to change control state and do the real page work
    • Save the new state (again, ViewState and ControlState)
    • Send the result back to the client

    It is important to note that if you dynamically add controls at runtime on your page (i.e. you do not only rely on the page markup to create your layout) it is vital that the controls are added always in the same order or the viewstate validation will fail and you’ll get the error.

    Some debugging

    We got a dump of the problem; unfortunately in SOS for .NET 2.0 we miss some commands we have in SOS for .NET 1.1, but even if with a bit more typing and digging we can get the same results. The next logical step is to have a look at the exceptions on the managed heap:

    0:033> !dumpheap -type Exception -stat
    ------------------------------
    Heap 0
    total 2 objects
    ------------------------------
    Heap 1
    total 4 objects
    ------------------------------
    Heap 2
    total 0 objects
    ------------------------------
    Heap 3
    total 7 objects
    ------------------------------
    Heap 4
    total 9 objects
    ------------------------------
    Heap 5
    total 0 objects
    ------------------------------
    Heap 6
    total 0 objects
    ------------------------------
    Heap 7
    total 4 objects
    ------------------------------
    total 26 objects
    Statistics:
          MT    Count    TotalSize Class Name
    79103ca4        2           24 System.Text.DecoderExceptionFallback
    79103c58        2           24 System.Text.EncoderExceptionFallback
    502254bc        2           40 System.ServiceModel.Diagnostics.ExceptionUtility
    790fe17c        1           72 System.ExecutionEngineException
    790fe0e0        1           72 System.StackOverflowException
    790fe044        1           72 System.OutOfMemoryException
    6640ab48        1          104 System.Web.UI.ViewStateException
    20f199b4        1          112 Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException
    790fe284        2          144 System.Threading.ThreadAbortException
    663dc0d4        2          168 System.Web.HttpException
    79118794       11          352 System.UnhandledExceptionEventHandler

    We have a ViewStateException so for sure it worth digging into it using the MetodTable:

    0:033> !dumpheap -mt 6640ab48        
    ------------------------------
    Heap 0
     Address       MT     Size
    total 0 objects
    ------------------------------
    Heap 1
     Address       MT     Size
    042c74dc 6640ab48      104     
    total 1 objects
    ------------------------------
    Heap 2
     Address       MT     Size
    total 0 objects
    ------------------------------
    Heap 3
     Address       MT     Size
    total 0 objects
    ------------------------------
    Heap 4
     Address       MT     Size
    total 0 objects
    ------------------------------
    Heap 5
     Address       MT     Size
    total 0 objects
    ------------------------------
    Heap 6
     Address       MT     Size
    total 0 objects
    ------------------------------
    Heap 7
     Address       MT     Size
    total 0 objects
    ------------------------------
    total 1 objects
    Statistics:
          MT    Count    TotalSize Class Name
    6640ab48        1          104 System.Web.UI.ViewStateException
    Total 1 objects
    
    
    0:033> !do 042c74dc 
    Name: System.Web.UI.ViewStateException
    MethodTable: 6640ab48
    EEClass: 6640aad0
    Size: 104(0x68) bytes
     (C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
    Fields:
          MT    Field   Offset                 Type VT     Attr    Value Name
    790fd8c4  40000b5        4        System.String  0 instance 042cbd04 _className
    7910ebc8  40000b6        8 ...ection.MethodBase  0 instance 00000000 _exceptionMethod
    790fd8c4  40000b7        c        System.String  0 instance 00000000 _exceptionMethodString
    790fd8c4  40000b8       10        System.String  0 instance 00000000 _message
    79116114  40000b9       14 ...tions.IDictionary  0 instance 00000000 _data
    790fdf04  40000ba       18     System.Exception  0 instance 042c73c8 _innerException
    790fd8c4  40000bb       1c        System.String  0 instance 00000000 _helpURL
    790fd0f0  40000bc       20        System.Object  0 instance 00000000 _stackTrace
    790fd8c4  40000bd       24        System.String  0 instance 00000000 _stackTraceString
    790fd8c4  40000be       28        System.String  0 instance 00000000 _remoteStackTraceString
    79102290  40000bf       34         System.Int32  1 instance        0 _remoteStackIndex
    790fd0f0  40000c0       2c        System.Object  0 instance 00000000 _dynamicMethods
    79102290  40000c1       38         System.Int32  1 instance -2146233088 _HResult
    790fd8c4  40000c2       30        System.String  0 instance 00000000 _source
    791016bc  40000c3       3c        System.IntPtr  1 instance        0 _xptrs
    79102290  40000c4       40         System.Int32  1 instance -532459699 _xcode
    7910be50  4002542       60       System.Boolean  1 instance        1 _isConnected
    790fd8c4  4002543       44        System.String  0 instance 042ca6b4 _remoteAddr
    790fd8c4  4002544       48        System.String  0 instance 042ca8cc _remotePort
    790fd8c4  4002545       4c        System.String  0 instance 042bd764 _userAgent
    790fd8c4  4002546       50        System.String  0 instance 042c16ec _persistedState
    790fd8c4  4002547       54        System.String  0 instance 042bd62c _referer
    790fd8c4  4002548       58        System.String  0 instance 042bbe48 _path
    790fd8c4  4002549       5c        System.String  0 instance 042cb178 _message
    7910be50  400254a       61       System.Boolean  1 instance        1 _macValidationError

    The _message field is as follows:

    0:033> !do 042cb178 
    Name: System.String
    MethodTable: 790fd8c4
    EEClass: 790fd824
    Size: 2104(0x838) bytes
     (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
    String: Invalid viewstate. 
        Client IP: <ip_address>
        Port: 1449
        User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.04506.30)
        ViewState: /wEPDwUJNDY5NzIzNzI1D2QWAgIDD2QWAgIBDxQrAAQPFg4eDkFzeW5jUmVuZGVyaW5nZx4UU2hvd1Byb21wdEFyZWFCdXR0b25nHgtab29tUGVyY2VudAJkHghab29t
    TW9kZQsphgFNaWNyb3NvZnQuUmVwb3J0aW5nLldlYkZvcm1zLlpvb21Nb2RlLCBSZXBvcnRpbmdTZXJ2aWNlc1dlYlNlcnZlciwgVmVyc2lvbj05LjAuMjQyLjAsIEN1bHR1cmU9bmV1
    dHJhbCwgUHVibGljS2V5VG9rZW49ODk4NDVkY2Q4MDgwY2M5MQIeFERvY3VtZW50TWFwQ29sbGFwc2VkaB4OUmVuZGVyaW5nU3RhdGULKZIBTWljcm9zb2Z0LlJlcG9ydGluZy5XZWJGb
    3Jtcy5SZXBvcnRSZW5kZXJpbmdTdGF0ZSwgUmVwb3J0aW5nU2VydmljZXNXZWJTZXJ2ZXIsIFZlcnNpb249OS4wLjI0Mi4wLCBDdWx0dXJlPW5ldXRyYWwsIFB1YmxpY0tleVRva2Vu
    PTg5ODQ1ZGNkODA4MGNjOTECHhBWMVN0eWxlU2hlZXROYW1lZGQoKVhTeXN0ZW0uR3VpZCwgbXNjb3JsaWIsIFZlcnNpb249Mi4wLjAuMCwgQ3VsdHVyZT1uZXV0cmFsLCBQdWJsaWNL
    ZXlUb2tlbj1iNzdhNWM1NjE5MzRlMDg5JGI2MzVjZjFkLTg4MGYtNDEyMC05Y2RjLTk2ZTE0ZGViYzRiMQIBFCsAARQrAAICARQrAAIFGGx3Y2d4MGIzZGI1MzB4NTVsY2F2eXA1NQIB
    FgpmDw8WAh4LSG...
    Fields: MT Field Offset Type VT Attr Value Name 79102290 4000096 4 System.Int32 1 instance 1044 m_arrayLength 79102290 4000097 8 System.Int32 1 instance 1043 m_stringLength 790ff328 4000098 c System.Char 1 instance 49 m_firstChar 790fd8c4 4000099 10 System.String 0 shared static Empty >> Domain:Value 000d49f0:790d884c 00101438:790d884c 1f7865b8:790d884c << 7912dd40 400009a 14 System.Char[] 0 shared static WhitespaceChars >> Domain:Value 000d49f0:0a2203f4 00101438:0a2244e8 1f7865b8:0226db3c <<

    The ViewState in this error message is truncated, but we can get the entire string from _persistedState:

    0:033> !do 042c16ec 
    Name: System.String
    MethodTable: 790fd8c4
    EEClass: 790fd824
    Size: 10162(0x27b2) bytes
     (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
    String: /wEPDwUJNDY5NzIzNzI1D2QWAgIDD2QWAgIBDxQrAAQPFg4eDkFzeW5jUmVuZGVyaW5nZx4UU2hvd1Byb21wdEFyZWFCdXR0b25nHgtab29tUGVyY2VudAJkHghab29tTW9kZ
    QsphgFNaWNyb3NvZnQuUmVwb3J0aW5nLldlYkZvcm1zLlpvb21Nb2RlLCBSZXBvcnRpbmdTZXJ2aWNlc1dlYlNlcnZlciwgVmVyc2lvbj05LjAuMjQyLjAsIEN1bHR1cmU9bmV1dHJhbC
    wgUHVibGljS2V5VG9rZW49ODk4NDVkY2Q4MDgwY2M5MQIeFERvY3VtZW50TWFwQ29sbGFwc2VkaB4OUmVuZGVyaW5nU3RhdGULKZIBTWljcm9zb2Z0LlJlcG9ydGluZy5XZWJGb3Jtcy5
    SZXBvcnRSZW5kZXJpbmdTdGF0ZSwgUmVwb3J0aW5nU2VydmljZXNXZWJTZXJ2ZXIsIFZlcnNpb249OS4wLjI0Mi4wLCBDdWx0dXJlPW5ldXRyYWwsIFB1YmxpY0tleVRva2VuPTg5ODQ
    1ZGNkODA4MGNjOTECHhBWMVN0eWxlU2hlZXROYW1lZGQoKVhTeXN0ZW0uR3VpZCwgbXNjb3JsaWIsIFZlcnNpb249Mi4wLjAuMCwgQ3VsdHVyZT1uZXV0cmFsLCBQdWJsaWNLZXlUb2t
    lbj1iNzdhNWM1NjE5MzRlMDg5JGI2MzVjZjFkLTg4MGYtNDEyMC05Y2RjLTk2ZTE0ZGViYzRiMQIBFCsAARQrAAICARQrAAIFGGx3Y2d4MGIzZGI1MzB4NTVsY2F2eXA1NQIBFgpmDw8
    WAh4LSGFzQ29udHJvbHNnZBYEZg8PFgIeDU9uQ2xpZW50Q2xpY2sF/wFyZXR1cm4gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoJ1JlcG9ydFZpZXdlckNvbnRyb2xfY3RsMDAnKS5QYXJ
    hbWV0ZXJzQ29udHJvbGxlci5WYWxpZGF0ZUhhc1ZhbHVlKCdSZXBvcnRWaWV3ZXJDb250cm9sX2N0bDAwX2N0bDAzX3R4dFZhbHVlJywgJycsICdQbGVhc2UgZW50ZXIgYSB2YWx1ZSBm
    b3IgdGhlIHBhcmFtZXRlciBcJ86az4nOtM65zrrPjM+CIM6Rzq/PhM63z4POt8+CXCcuICBUaGUgcGFyYW1ldGVyIGNhbm5vdCBiZSBibGFuay4nKTtkZAIDD2QWAgIBDw8WAh4EVGV4d
    AUPNjQ4MTAxMTA4MDI0MTY1ZGQCAQ9kFg5mD2QWAmYPD2QWAh4Hb25jbGljawWoAWRvY3VtZW50LmdldEVsZW1lbnRCeUlkKCdSZXBvcnRWaWV3ZXJDb250cm9sJykuQ2xpZW50Q29udH
    JvbGxlci5TZXREb2NNYXBWaXNpYmlsaXR5KCFkb2N1bWVudC5nZXRFbGVtZW50QnlJZCgnUmVwb3J0Vmlld2VyQ29udHJvbCcpLkNsaWVudENvbnRyb2xsZXIuSXNEb2NNYXBWaXNpYmx
    lKCkpOxYCZg8PFgYeBVdpZHRoGwAAAAAAADBAAQAAAB4GSGVpZ2h0GwAAAAAAADBAAQAAAB4EXyFTQgKAAxYCHgVzdHlsZQUMcGFkZGluZzoycHg7ZAIBD2QWCmYPZBYCAgEPZBYCZg8P
    FgYfCxsAAAAAAAAwQAEAAAAfDBsAAAAAAAAwQAEAAAAfDQKAAxYCHw4FDHBhZGRpbmc6MnB4O2QCAQ9kFgICAQ9kFgJmDw8WBh8LGwAAAAAAADBAAQAAAB8MGwAAAAAAADBAAQAAAB8NA
    oADFgIfDgUMcGFkZGluZzoycHg7ZAICDw8WBB8JZR4HRW5hYmxlZGgWAh4Kb25rZXlwcmVzcwWNA2lmIChldmVudC5rZXlDb2RlID09IDEwIHx8IGV2ZW50LmtleUNvZGUgPT0gMTMpI
    HsNCnZhciBwYWdlTnVtYmVyID0gcGFyc2VJbnQodGhpcy52YWx1ZSwgMTApOw0KaWYgKGlzTmFOKHBhZ2VOdW1iZXIpIHx8IHBhZ2VOdW1iZXIgPCAxIHx8IHBhZ2VOdW1iZXIgPiBkb
    2N1bWVudC5nZXRFbGVtZW50QnlJZCgnUmVwb3J0Vmlld2VyQ29udHJvbCcpLkNsaWVudENvbnRyb2xsZXIuVG90YWxQYWdlcykNCiAgICBhbGVydCgnRW50ZXIgYSB2YWxpZCBwYWdlI
    G51bWJlcicpOw0KZWxzZQ0KICAgIGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKCdSZXBvcnRWaWV3ZXJDb250cm9sJykuQ2xpZW50Q29udHJvbGxlci5BY3Rpb25IYW5kbGVyKCdQYWdlT
    mF2JywgcGFnZU51bWJlcik7OzsgcmV0dXJuIGZhbHNlO31kAgUPZBYCAgEPZBYCZg8PFgYfCxsAAAAAAAAwQAEAAAAfDBsAAAAAAAAwQAEAAAAfDQKAAxYCHw4FDHBhZGRpbmc6MnB4O
    2QCBg9kFgICAQ9kFgJmDw8WBh8LGwAAAAAAADBAAQAAAB8MGwAAAAAAADBAAQAAAB8NAoADFgIfDgUMcGFkZGluZzoycHg7ZAICD2QWAmYPZBYCAgEPZBYCZg8PFgYfCxsAAAAAAAAwQ
    AEAAAAfDBsAAAAAAAAwQAEAAAAfDQKAAxYCHw4FDHBhZGRpbmc6MnB4O2QCAw9kFgJmDxAPZBYEHghvbmNoYW5nZQVgZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoJ1JlcG9ydFZpZXdlc
    kNvbnRyb2wnKS5DbGllbnRDb250cm9sbGVyLlNldFpvb20oZXZlbnQuc3JjRWxlbWVudC52YWx1ZSk7HghkaXNhYmxlZAUIZGlzYWJsZWRkFgECBWQCBA9kFgJmDw8WAh8PaBYEHhBvb
    nByb3BlcnR5Y2hhbmdlBaMCaWYgKGV2ZW50LnByb3BlcnR5TmFtZSA9PSAndmFsdWUnKQ0Kew0KICAgIGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKCdSZXBvcnRWaWV3ZXJDb250cm9sX
    2N0bDAxX2N0bDA0X2N0bDAxJykuQ29udHJvbGxlci5TZXRWaWV3ZXJMaW5rQWN0aXZlKHRoaXMudmFsdWUgIT0gbnVsbCAmJiB0aGlzLnZhbHVlICE9ICcnKTsNCiAgICBkb2N1bWVud
    C5nZXRFbGVtZW50QnlJZCgnUmVwb3J0Vmlld2VyQ29udHJvbF9jdGwwMV9jdGwwNF9jdGwwMycpLkNvbnRyb2xsZXIuU2V0Vmlld2VyTGlua0FjdGl2ZShmYWxzZSk7DQp9HxAF4QFpZ
    iAoZXZlbnQua2V5Q29kZSA9PSAxMCB8fCBldmVudC5rZXlDb2RlID09IDEzKSB7ZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoJ1JlcG9ydFZpZXdlckNvbnRyb2wnKS5DbGllbnRDb250c
    m9sbGVyLkFjdGlvbkhhbmRsZXIoJ1NlYXJjaCcsIGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKCdSZXBvcnRWaWV3ZXJDb250cm9sX2N0bDAxX2N0bDA0X2N0bDAwJykudmFsdWUpOzsgc
    mV0dXJuIGZhbHNlO31kAgcPZBYCZg9kFgICAQ9kFgJmDw8WBh8LGwAAAAAAADBAAQAAAB8MGwAAAAAAADBAAQAAAB8NAoADFgIfDgUMcGFkZGluZzoycHg7ZAIID2QWAmYPZBYCAgEPZ
    BYCZg8PFgYfCxsAAAAAAAAwQAEAAAAfDBsAAAAAAAAwQAEAAAAfDQKAAxYCHw4FDHBhZGRpbmc6MnB4O2QCBg8WAh4FVmFsdWUFATFkAgcPFgIfFAUFZmFsc2VkAggPFgIfFAUFZmFsc
    2VkGAEFHl9fQ29udHJvbHNSZXF1aXJlUG9zdEJhY2tLZXlfXxYQBStSZXBvcnRWaWV3ZXJDb250cm9sJGN0bDAxJGN0bDAwJGN0bDAwJGN0bDAwBTFSZXBvcnRWaWV3ZXJDb250cm9sJ
    GN0bDAxJGN0bDAxJGN0bDAwJGN0bDAwJGN0bDAwBTFSZXBvcnRWaWV3ZXJDb250cm9sJGN0bDAxJGN0bDAxJGN0bDAwJGN0bDAxJGN0bDAwBTFSZXBvcnRWaWV3ZXJDb250cm9sJGN0b
    DAxJGN0bDAxJGN0bDAxJGN0bDAwJGN0bDAwBTFSZXBvcnRWaWV3ZXJDb250cm9sJGN0bDAxJGN0bDAxJGN0bDAxJGN0bDAxJGN0bDAwBTFSZXBvcnRWaWV3ZXJDb250cm9sJGN0bDAxJ
    GN0bDAxJGN0bDA1JGN0bDAwJGN0bDAwBTFSZXBvcnRWaWV3ZXJDb250cm9sJGN0bDAxJGN0bDAxJGN0bDA1JGN0bDAxJGN0bDAwBTFSZXBvcnRWaWV3ZXJDb250cm9sJGN0bDAxJGN0b
    DAxJGN0bDA2JGN0bDAwJGN0bDAwBTFSZXBvcnRWaWV3ZXJDb250cm9sJGN0bDAxJGN0bDAxJGN0bDA2JGN0bDAxJGN0bDAwBTFSZXBvcnRWaWV3ZXJDb250cm9sJGN0bDAxJGN0bDAyJ
    GN0bDAwJGN0bDAwJGN0bDAwBTFSZXBvcnRWaWV3ZXJDb250cm9sJGN0bDAxJGN0bDAyJGN0bDAwJGN0bDAxJGN0bDAwBStSZXBvcnRWaWV3ZXJDb250cm9sJGN0bDAxJGN0bDA2JGN0b
    DAwJGN0bDAwBTFSZXBvcnRWaWV3ZXJDb250cm9sJGN0bDAxJGN0bDA3JGN0bDAwJGN0bDAwJGN0bDAwBTFSZXBvcnRWaWV3ZXJDb250cm9sJGN0bDAxJGN0bDA3JGN0bDAwJGN0bDAxJ
    GN0bDAwBTFSZXBvcnRWaWV3ZXJDb250cm9sJGN0bDAxJGN0bDA4JGN0bDAwJGN0bDAwJGN0bDAwBTFSZXBvcnRWaWV3ZXJDb250cm9sJGN0bDAxJGN0bDA4JGN0bDAwJGN0bDAxJGN0b
    DAwX2RaxoBglgh8qbVp7F4IKaK6C+A=
    Fields: MT Field Offset Type VT Attr Value Name 79102290 4000096 4 System.Int32 1 instance 5073 m_arrayLength 79102290 4000097 8 System.Int32 1 instance 5072 m_stringLength 790ff328 4000098 c System.Char 1 instance 2f m_firstChar 790fd8c4 4000099 10 System.String 0 shared static Empty >> Domain:Value 000d49f0:790d884c 00101438:790d884c 1f7865b8:790d884c << 7912dd40 400009a 14 System.Char[] 0 shared static WhitespaceChars >> Domain:Value 000d49f0:0a2203f4 00101438:0a2244e8 1f7865b8:0226db3c <<

    There is also an _innerException we can look at:

    0:033> !do 042c73c8 
    Name: System.Web.HttpException
    MethodTable: 663dc0d4
    EEClass: 663dc05c
    Size: 84(0x54) bytes
     (C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
    Fields:
          MT    Field   Offset                 Type VT     Attr    Value Name
    790fd8c4  40000b5        4        System.String  0 instance 042cbd58 _className
    7910ebc8  40000b6        8 ...ection.MethodBase  0 instance 00000000 _exceptionMethod
    790fd8c4  40000b7        c        System.String  0 instance 00000000 _exceptionMethodString
    790fd8c4  40000b8       10        System.String  0 instance 042c745c _message
    79116114  40000b9       14 ...tions.IDictionary  0 instance 00000000 _data
    790fdf04  40000ba       18     System.Exception  0 instance 00000000 _innerException
    790fd8c4  40000bb       1c        System.String  0 instance 00000000 _helpURL
    790fd0f0  40000bc       20        System.Object  0 instance 042c74a0 _stackTrace
    790fd8c4  40000bd       24        System.String  0 instance 00000000 _stackTraceString
    790fd8c4  40000be       28        System.String  0 instance 00000000 _remoteStackTraceString
    79102290  40000bf       34         System.Int32  1 instance        0 _remoteStackIndex
    790fd0f0  40000c0       2c        System.Object  0 instance 00000000 _dynamicMethods
    79102290  40000c1       38         System.Int32  1 instance -2147467259 _HResult
    790fd8c4  40000c2       30        System.String  0 instance 00000000 _source
    791016bc  40000c3       3c        System.IntPtr  1 instance        0 _xptrs
    79102290  40000c4       40         System.Int32  1 instance -532459699 _xcode
    79102290  4001029       48         System.Int32  1 instance        0 _httpCode
    663e49c4  400102a       44 ...eb.ErrorFormatter  0 instance 00000000 _errorFormatter
    79102290  400102b       4c         System.Int32  1 instance        0 _webEventCode
    
    
    0:033> !do 042c745c 
    Name: System.String
    MethodTable: 790fd8c4
    EEClass: 790fd824
    Size: 66(0x42) bytes
     (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
    String: Unable to validate data.
    Fields:
          MT    Field   Offset                 Type VT     Attr    Value Name
    79102290  4000096        4         System.Int32  1 instance       25 m_arrayLength
    79102290  4000097        8         System.Int32  1 instance       24 m_stringLength
    790ff328  4000098        c          System.Char  1 instance       55 m_firstChar
    790fd8c4  4000099       10        System.String  0   shared   static Empty
        >> Domain:Value  000d49f0:790d884c 00101438:790d884c 1f7865b8:790d884c <<
    7912dd40  400009a       14        System.Char[]  0   shared   static WhitespaceChars
        >> Domain:Value  000d49f0:0a2203f4 00101438:0a2244e8 1f7865b8:0226db3c <<

    So we know in this case we are unable to validate the ViewState; we can also paste the ViewState string from _persistedState in ViewState Decoder

    ViewState Decoder

    The “Character Count” value is just a bit more than 5.000, not much for the complex pages we were looking at… Depending on their configuration, firewall and/or proxies software can truncate fields and values exceeding a certain amount of bytes, and this is another known cause for InvalidViewstateExceptions. Who can we quickly check?

    A network trace can help, but as an empiric test we can use LogParser on IIS logs. These are the maximum, average and minimum amount of bytes send from the server as a result of a GET to an aspx page:

    logparser 
        "select max(sc-bytes), avg(sc-bytes), min(sc-bytes) from *.log 
        where to_lowercase(extract_extension(cs-uri-stem))='aspx' and to_lowercase(cs-method)='get'" -i:iisw3c
    
    MAX(ALL sc-bytes) AVG(ALL sc-bytes) MIN(ALL sc-bytes)
    ----------------- ----------------- -----------------
    72756             44009             448

    And these are the same values, calculated on the bytes posted from the clients:

    logparser 
        "select max(cs-bytes), avg(cs-bytes), min(cs-bytes) from *.log 
        where to_lowercase(extract_extension(cs-uri-stem))='aspx' and to_lowercase(cs-method)='post'" -i:iisw3c
    
    MAX(ALL cs-bytes) AVG(ALL cs-bytes) MIN(ALL cs-bytes)
    ----------------- ----------------- -----------------
    11662             4620              819
    Uhm… the server is sending an average of about 44 Kb and the clients are posting an average of less than 5 Kb… also the maximum values are quite different. Not a crushing proof but still something to keep in mind…

    A quick check on customer’s network revealed they actually had configured (I think for security reasons) a maximum allowed size for HTML incoming fields and due to some policies and constraints they had, we could not increase that value. Ok, now what?

    Enter MaxPageStateFieldLength

    Luckily, we can configure the MaxPageStateFieldLength property exactly for this purpose:

    When the MaxPageStateFieldLength property is set to a positive number, the view state sent to the client browser is broken into multiple hidden fields, and each field's value is less than the size specified in the MaxPageStateFieldLength property.

    Setting the MaxPageStateFieldLength property to a negative number (the default) indicates that the view-state field should not be separated into chunks. Setting the MaxPageStateFieldLength to a small number may result in poor performance.

    Set the value of the MaxPageStateFieldLength property in the pages element of the Web.config file.

    We set the values to something less than 5 Kb and the problem magically disappeared smile_regular

     

    Carlo

    Quote of the day:
    Have a heart that never hardens, and a temper that never tires, and a touch that never hurts -- Charles Dickens
  • Never doubt thy debugger

    “Padding is invalid an cannot be removed” or “WebPartManager is undefined”

    • 3 Comments

    I saw this error twice very recently and searching the Internet remarkably I’ve not been able to find any good explanation for it, so I hope this can be helpful in case you’ll get in trouble.

    Web Farm?

    The first occurrence happened with an application hosted on a web farm, and in IE the users were getting the error “WebPartManager is undefined”. If you take a look at the HTTP traffic with Fiddler (or you look into the IIS logs) and you see you are getting error 500 on WebResource.axd, then have a look at the “TextView” tab and other tabs to carefully inspect the request you get from the server. That way, we found the real error message, which was:

    Padding is invalid and cannot be removed.

    Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

    Exception Details: System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.

    Source Error: An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

    Stack Trace:
    [CryptographicException: Padding is invalid and cannot be removed.]
    System.Security.Cryptography.RijndaelManagedTransform.DecryptData(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount, Byte[]& outputBuffer, Int32 outputOffset, PaddingMode paddingMode, Boolean fLast) +1455156
    System.Security.Cryptography.RijndaelManagedTransform.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount) +306
    System.Security.Cryptography.CryptoStream.FlushFinalBlock() +30 System.Web.Configuration.MachineKeySection.EncryptOrDecryptData(Boolean fEncrypt, Byte[] buf, Byte[] modifier, Int32 start, Int32 length, Boolean useValidationSymAlgo) +159
    System.Web.UI.Page.DecryptString(String s) +79
    System.Web.Handlers.AssemblyResourceLoader.System.Web.IHttpHandler.ProcessRequest(HttpContext context) +211
    System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +303
    System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +64

    The validationKey and decryptionKey for MachineKey were correctly set to a static value shared across all the servers on the Web Farm, but the decryption attribute was still set to “Auto”; the problem was resolved changing the decryption value to “EAS” on all servers.

    Hardcoded links to resources

    In this case we were randomly getting the following (SSL was involved here, but is should not really make a difference):

    Padding is invalid and cannot be removed.

    Url : http://MyWebSite.com/App_Themes/Default/WebResource.axd?d=h5orERdAXqJwNOs03yyPelp7bMUUMOtSL2yUIRd-eh2-cuX6WPyygZ3af2jYqwPf0GxIQ9SNZIrG5n9i8AgXUw2&t=633059379620000000

    Script : /App_Themes/Default/ WebResource.axd

    Server.GetLastError.ToString: System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed
    at System.Security.Cryptography.RijndaelManagedTransform.DecryptData(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount, Byte[]& outputBuffer, Int32 outputOffset, PaddingMode paddingMode, Boolean fLast)
    at System.Security.Cryptography.RijndaelManagedTransform.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount)
    at System.Security.Cryptography.CryptoStream.FlushFinalBlock()
    at System.Web.Configuration.MachineKeySection.EncryptOrDecryptData(Boolean fEncrypt, Byte[] buf, Byte[] modifier, Int32 start, Int32 length, Boolean useValidationSymAlgo) at System.Web.UI.Page.DecryptString(String s)
    at System.Web.Handlers.AssemblyResourceLoader.System.Web.IHttpHandler.ProcessRequest(HttpContext context)
    at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExeutionStep.Execute()
    at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

    As you can see the stack of the error is the same. But this time we were on a single server (no Web Farm and the customer was not using Web Garden) and even if the customer had tried using a fixed MachineKey value, it did not help. Also article A System.Security.Cryptography.CryptographicException exception occurs when you try to use the RijndaelManaged class to decrypt data did not help because the customer was not working with encryption in his code.

    The problem here was one of the CSS the customer was using, here is the relevant excerpt:

    cursor:pointer; width:15px; height:15px; float:left; background-repeat:no-repeat;background-position:50%;
    background-image:url(WebResource.axd?d=h5orERdAXqJwNOs03yyPelp7bMUUMOtSL2yUIRd-e h2-cuX6WPyygZ3af2jYqwPfNQQAbG75SUCIun2YdKD8uQ2&t=633059379620000000);

    As you can see the background image URL is a hardcoded WebResource.axd and is exactly the same one logged in the error…

    URLs to web resources should not be hardcoded but rather we must use GetWebResourceUrl instead.

     

    Carlo

    Quote of the day:
    Everything is vague to a degree you do not realize till you have tried to make it precise. - Bertrand Russell
  • Never doubt thy debugger

    WebResource.axd or ScriptResource.axd not working

    • 3 Comments

    Http compression with client-side scripting should be handled with care. The problem can have different symptoms and manifest in different ways, but essentially it has a common root cause and I have already discussed some of the aspects (for example see here and here). This time the customer whom reported the problem was using Forms Authentication in his application so the first time an unauthenticated user browsed it, he was redirected to the login page configured in web.config. So far so good.

    In this case the problem was a javascript error notified by the standard yellow icon in the bottom left corner of IE; the error was an “Object expected” on “WebForm_AutoFocus()” which is a method generated by ASP.NET as a result of an object.SetFocus() call on the server-side.

    Some quick theory

    WebResource.axd and ScriptResource.axd are Http Handlers used by ASP.NET and Ajax to add client-side scripting (usually javascript) to the outgoing web page for example to have client-side input validation, set the focus on a specific control (as in this example) etc… Note that if you search your disk for .axd files, you’ll not find them; they are created on the fly in memory and executed from there.

    Well, this is not completely true… if you want to have a look at them you can search the IE temporary cache on your client, you’ll likely find quite a few instances of WebResource.axd and ScriptResource.axd, but you’ll not find them on the server to be used by IIS.

    Troubleshooting

    At this point I already had some suspects but still no real clue, so let’s go hunting for them.

    First thing is to check if you are using HTTP compression: if you do, test if disabling it resolves the problem:

    HTTP compression in IIS 6.0

     

    Another possibility is that the application mappings are not configured correctly (try with “aspnet_regiis –i”) or for some reason for the extension .axd the flag “Verify that file exists” flag has been set: if it is, remove it.

    Verify that file exists for .axd

     

    Anyway everything looked good on the faulting server so far… So next step is to have a look at the IIS logs.

    Using LogParser you can quickly check what IIS returns when it comes to serve WebResource.axd (or the handler you’re looking for):

    logparser 
        "select distinct sc-status, count(*) as hits from *.log 
        where index_of(to_lowercase(cs-uri-stem), 'webresource.axd') > 0 
        group by sc-status order by hits desc" -i:iisw3c

    Result? Every time WebResource.axd is requested, IIS returns a 404 (File not Found):

    sc-status hits 
    --------- ----
    404       306

     

    This time let’s check the Wildcard application mappings… If you have your IIS Manager at hand, give a quick look at the following screenshot:

    Wildcard application maps in IIs 6

    Otherwise if you are like me and have only the IIS metabase at hand, you can have a look at the ScriptMaps elements you have in there:

    ScriptMaps=".ad,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .adprototype,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .asa,C:\WINDOWS\system32\inetsrv\asp.dll,5,GET,HEAD,POST,TRACE
                .asax,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .ascx,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .ashx,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,1,GET,HEAD,POST,DEBUG
                .asmx,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,1,GET,HEAD,POST,DEBUG
                .asp,C:\WINDOWS\system32\inetsrv\asp.dll,5,GET,HEAD,POST,TRACE
                .aspx,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,1,GET,HEAD,POST,DEBUG
                .axd,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,1,GET,HEAD,POST,DEBUG
                .browser,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .cd,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .cdx,C:\WINDOWS\system32\inetsrv\asp.dll,5,GET,HEAD,POST,TRACE
                .cer,C:\WINDOWS\system32\inetsrv\asp.dll,5,GET,HEAD,POST,TRACE
                .compiled,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .config,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .cs,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .csproj,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .dd,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .exclude,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .idc,C:\WINDOWS\system32\inetsrv\httpodbc.dll,5,GET,POST
                .java,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .jsl,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .ldb,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .ldd,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .lddprototype,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .ldf,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .licx,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .master,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .mdb,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .mdf,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .msgx,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,1,GET,HEAD,POST,DEBUG
                .refresh,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .rem,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,1,GET,HEAD,POST,DEBUG
                .resources,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .resx,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .sd,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .sdm,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .sdmDocument,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .shtm,C:\WINDOWS\system32\inetsrv\ssinc.dll,5,GET,POST
                .shtml,C:\WINDOWS\system32\inetsrv\ssinc.dll,5,GET,POST
                .sitemap,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .skin,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .soap,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,1,GET,HEAD,POST,DEBUG
                .stm,C:\WINDOWS\system32\inetsrv\ssinc.dll,5,GET,POST
                .svc,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,1,GET,HEAD,POST,DEBUG
                .vb,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .vbproj,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .vjsproj,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                .vsdisco,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,1,GET,HEAD,POST,DEBUG
                .webinfo,c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll,5,GET,HEAD,POST,DEBUG
                *,C:\Inetpub\ISAPI\NoSQLInjection.dll,4,All"

    Pay attention to the last entry:

    *,C:\Inetpub\ISAPI\NoSQLInjection.dll,4,All

    What is important here is the “Flags” part: “4” means “The server attempts to access the PATH_INFO portion of the URL, as a file, before starting the scripting engine. If the file can't be opened, or doesn't exist, an error is returned to the client”.

    Here we are again… Check and clear the flag “Verify that file exists” for your wildcard application mapping. The point is that a wildcard ISAPI extension intercepts all requests and is executed as the first step for every of them; since WebResource.axd does not exist as a file on disk, IIS will not find it, will return a 404 error and will abort the request without passing it to aspnet_isapi.dll for further handling.

    Verify that file exists for wildcard app maps

    Conclusion

    This error can con in different forms, sometimes obvious and sometimes not so obvious; but when “dynamic” resources come into the field, spending a couple of minutes to take case of HTTP compression and Application Mappings configuration can save you from complains by your users and hours of troubleshooting and headaches…

     

    Carlo

    Quote of the day:
    The folly of mistaking a paradox for a discovery, a metaphor for a proof, a torrent of verbiage for a spring of capital truths, and oneself for an oracle, is inborn in us. - Paul Valery
  • Never doubt thy debugger

    How much garbage to you still have on the managed heap?

    • 0 Comments

    When you’re focused on a specific problem, you’re sunk into it and you’re at a dead end, you have to take a step back to see the whole picture and breath again before being able to continue; often discussing with a colleague whom still has a “fresh mind” and doesn’t know anything about the specific problem and the troubleshooting you’ve already done can give nice suggestions and tips about next steps and a different approach to the problem. This is what happened a couple of days ago with Doug while we were discussing a memory leak issue I’m currently working on.

    This seems to be a complex and a bit confusing one (I do not want to dig into the details for post, I’ll likely talk again in details when the problem will be resolved smile_nerd); anyway a question that should always come to mind dealing with memory leaks is: how much of the memory I see still allocated in a dump is really valid and how much is garbage which our friendly Garbage Collector will remove next time it will come into the game?

    Well… a nice solution is to have a very simple “service” page to trigger a full Garbage Collection when we need it (this is only for troubleshooting, do not call it too often if you do not want bad performance issues because the GC will be continuously triggered!). The page can be as simple as the following 4 lines of code, for example in the Page_Load event or in a Button_Click handler:

    GC.Collect();
    GC.WaitForPendingFinalizers();
    GC.Collect();
    Response.Write("Full collection run on " + DateTime.Now.ToString());

    You can run it when the memory utilization is high and when the page returns you can then capture a hang dump with your favorite tool (see here if you are not an expert and want to do it yourself); at that point you can be sure that what will be left on the managed heap will be valid objects, still rooted somewhere in memory and not garbage. If you repeat the steps 3-4 times when the memory will be grown of another 100-120 Mb you’ll finally have a series of 4 dumps taken at different times to compare and play with (or ready to be sent to CSS smile_regular).

     

    Carlo

    Quote of the day:
    The effort to understand the universe is one of the very few things that lifts human life a little above the level of farce, and gives it some of the grace of tragedy. - Steven Weinberg
Page 1 of 1 (5 items)