David Kline : Networking && Web Services

Exploring The NetCF WebCrawler Sample III - Visual Studio 2005 Beta 2

DavidKlineMS — Wed, 27 Apr 2005 07:30:00 GMT

In the first part of this series, I discussed how the WebCrawler uses an HTTP HEAD request to determine if the URL can be crawled (points to an HTML page). The Visual Studio.NET 2003 version of the WebCrawler used a very simple check of the Content-Type header (equal to "text/html").

When I was reviewing the sample for the Visual Studio 2005 Beta 2 release, I ran the WebCrawler under the debugger and noticed that some URLs were failing unexpectedly. While stepping through the Crawler.PageIsHtml method, I noticed that one of the failing sites were specifying the text encoding (ex: "text/html; charset=utf-8") as part of the content type. Because of the WebCrawler's very simple approach to checking the URL's content type, this site was not being crawled.

To fix this, I changed the content type comparison to use String.StartsWith instead of String.Equals. After making this change, the previously failing URL was correctly identified and the WebCrawler successfully crawled the page.

The code fragment, below, is an excerpt from the Crawler.PageIsHtml method and shows the updated content type check. For a complete source listing, please consult the files that were installed with Visual Studio .NET 2005 / .NET Framework SDK Beta 2.


// check the content type
string contentType = headers["Content-type"];
if (contentType != null)
{
    contentType = contentType.ToLower(CultureInfo.InvariantCulture);
    if (contentType.StartsWith(TypeHTML))
    {
        isHtml = true;
    }
}

For reference, the original code looked like this:


// check the content type
string contentType = headers["Content-type"];
if(contentType != null)
{
    contentType = contentType.ToLower();
    if(contentType.Equals(TypeHTML))
    {
        isHtml = true;
    }
}

Until next time,
-- DK

Disclaimer(s):
This posting is provided "AS IS" with no warranties, and confers no rights.
Some of the information contained within this post may be in relation to beta software. Any and all details are subject to change.

Exploring the NetCF Web Crawler sample (Visual Studio .NET 2003) - Part II: Updating the UI

DavidKlineMS — Mon, 21 Mar 2005 23:45:00 GMT

A couple of weeks ago, I discussed link tracking in the NetCF Web Crawler sample from Visual Studio .NET 2003. Today, I would like to use the Web Crawler sample to discuss updating an application's user interface from a worker thread.

When updating user interface elements, it is very important to perform the tasks while running in the context of the thread that created the controls. One side effect of attempting to modify controls from a different thread are application hangs. This is most often seen when attempting to update a control (ex: modify it's Text property) in a asynchronous method callback. While callback methods typically live in the class that owns the controls, they are most often called by a thread that was created by another piece of code (the NetCF HTTP client class library, for example). When UI updates are attempted from this callback, application hangs may occur.

The way to avoid these side effects is to marshal the callback calls to your user interface's thread context. This can be achived using the Invoke method of the System.Windows.Forms.Control class. The Web Crawler example uses this technique to provide runtime status to the user.

NOTE: The code in this post has been edited for clarity and to reduce size. For full sample code, please consult the files that were installed with Visual Studio .NET 2003 (\Program Files\Microsoft Visual Studio .NET 2003\CompactFrameworkSDK\v1.0.5000\Windows CE\Samples\VC#\Pocket PC\WebCrawler).

Events and event handlers
The Web Crawler's Crawler class supports a number of events for which that clients can register handlers. To keep my example as clear as possible, I am going to focus on a very simple case -- the PageFoundEvent.

The WebCrawler.Crawler object exposes a public EventHandler field for the PageHandledEvent as shown below:

public EventHandler PageFoundEvent;

The client (WebCrawler.MainForm) registers it's event hander:

 // from startButton_Click
this.crawler = new Crawler(startingPage,
                           noProxy);
this.crawler.PageFoundEvent += new EventHandler(this.HandlePageFoundEvent);

The event handler implementation also lives in the client (WebCrawler.MainForm):

 private void HandlePageFoundEvent(object sender, EventArgs e)
{
    if(CustomInvokeRequired())
    {
        this.linkCount.Invoke(new EventHandler(this.HandlePageFoundEvent));
        return;
    }
            
    try
    {
        this.linkCount.Text = string.Format("{0}", Int32.Parse(this.linkCount.Text)+1);
    }
    catch(FormatException)
    { }
}

The HandlePageFoundEvent method first checks to see if the call was made in the context of the user interface thread by calling CustomInvokeRequired() (more on this soon). If not (returns true), it calls the linkCount object's Invoke method to marshal the call to the user interface thread. The HandlePageFoundEvent method is then re-called on the user interface thread. CustomInvokeRequired() is called (returns false) and the method updates the linkCount control's Text property.

Determining if thread marshaling is required
I mentioned earlier that I would go into more detail on the web crawler's CustomInvokeRequired method (WebCrawler.MainForm). At startup, the web crawler's MainForm object stores it's thread in a private member variable:

private readonly System.Threading.Thread FormThread = System.Threading.Thread.CurrentThread;

This member is marked as readonly so that it cannot be modified.

When called, the CustomInvokeRequired method checks to see if the current thread is the same as the FormThread (the thread that owns the user interface):

if(this.FormThread.Equals(System.Threading.Thread.CurrentThread))

If the current thread is the user interface thread (this.FormThread.Equals(Thread.CurrentThread)), there is no marshaling required and CustomInvokeRequired returns false. Otherwise, true is returned indicating that thread marshaling is needed.

Using this technique, the web crawler's worker thread can keep the user informed, and not cause any undesirable side effects.

Until next time,
-- DK

Disclaimer(s):
This posting is provided "AS IS" with no warranties, and confers no rights.

Exploring the NetCF WebCrawler sample (Visual Studio .NET 2003)

DavidKlineMS — Tue, 08 Mar 2005 18:17:00 GMT

Every so often, I talk to people about the NetCF WebCrawler sample that shipped as part of Visual Studio .NET 2003 (\Program Files\Microsoft Visual Studio .NET 2003\CompactFrameworkSDK\v1.0.5000\Windows CE\Samples\VC#\Pocket PC\WebCrawler). One of the things I get asked about is how the sample keeps track of the pages it has visited and when it knows to stop. The way the WebCrawler sample handles this is in how it uses the StatusCode member of the LinkInfo class.

The LinkInfo.StatusCode member is declared as being of type HttpStatusCode. HttpStatusCode is an enum defined in the System.Net namespace. Whenever a link is discovered, the WebCrawler creates an instance of LinkInfo and stores the Url in the LinkPath member. It also sets the value of StatusCode to 0. As shown in the code fragment, below, 0 is being used to indicate that a connection to this Url has not been attempted. There are also two other special values noted: -1 and -2, which denote application specific special case values.

NOTE: The code in this post has been edited for clarity and to reduce size. For full sample code, please consult the files that were installed with Visual Studio .NET 2003.

 public class LinkInfo
{
    /// The link's address (ex: http://www.microsoft.com)
    public readonly string LinkPath;
  
    /// HttpStatusCode received when we attempted to
    /// connect to the link
    /// NOTE: Other possible values
    ///  0 == no connection attempted
    /// -1 == generic failure
    /// -2 == link does not point to html data
    public HttpStatusCode StatusCode;
}

Each LinkInfo object that is created by the WebCrawler (in WebCrawler.Crawler.GetPageLinks) is stored for future reference -- the WebCrawler sample uses a Hashtable for storage. Each Url found by the WebCrawler is stored only once. If the same Url is identified later in the crawl, the new instance is discarded. This helps to avoid circular crawls (a page that links to another, which links back to the first).

 // add the link
links.Add(linkString, new LinkInfo(linkString, (HttpStatusCode)0));

In the above, you can see the use of 0 (not visited) for the LinkInfo.StatusCode value. As mentioned before, System.Net.HttpStatusCode is an enum and since enums are defined as being of an integer type (ex: short, int, etc), integer values can be cast to the enum type (in this case HttpStatusCode).

Later in the crawl, the discovered links are read from the Hashtable and if their StatusCode is equal to 0, an HTTP header request is issued (in WebCrawler.Crawler.PageIsHtml):

 // create the web request
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(pageAddress);
                
//  get headers only
req.Method = "HEAD";
                
// make the connection
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
 // read the headers
WebHeaderCollection headers = resp.Headers;
                
// check the content type
string contentType = headers["Content-type"];
if(contentType != null)
{
    contentType = contentType.ToLower();
    if(contentType.Equals(TypeHTML))
    {
        isHtml = true;
    }
}
 // get the status code (should be 200)
status = resp.StatusCode;
                
// close the connection
resp.Close();

In the above example TypeHTML is defined as being "text/html".

The result of the header request (HttpWebRequest.StatusCode) is stored in the link's LinkInfo.StatusCode field. If the request fails, the exception handler (not shown) stores the value of WebException.Response.StatusCode or -1 (generic error) if the exception is anything other than a WebException. If the Content-Type header reports anything other than "text/html", the caller sets LinkInfo.Status code to -2 (defined in LinkInfo as denoting a link to non-HTML data).

Once it has been determined that the Url points to HTML, the crawler requests the full page, stores the StatusCode and if successful searches for additional links (in WebCrawler.Crawler.Crawl). As you can se below, the WebCrawler sample supports links that are found in the a, frame, area and link HTML tags.

 string pageData = "";
li.StatusCode = GetPageData(ref pageUri,
                            out pageData);
// if we successfully retrieved the page data
if(HttpStatusCode.OK == li.StatusCode)
{
    // <a href=
    GetPageLinks(pageUri,
                 pageData,
                 "a",
                 "href",
                 found);
    // <frame src=
    GetPageLinks(pageUri,
                 pageData,
                 "frame",
                 "src",
                 found);
    // <area href=
    GetPageLinks(pageUri,
                 pageData,
                 "area",
                 "href",
                 found);
    // <link href=
    GetPageLinks(pageUri,
                 pageData,
                 "link",
                 "href",
                 found);
}

Once the crawler finds that there are no more unvisited links (LinkInfo.StatusCode == 0) in it's Hashtable, it stops.

It is important to note that the WebCrawler sample keeps a master Hashtable (not shown in the above exampled) that holds all of the Urls that it discovers (regardless of whether or not it was able to successfully visit them) as well as a temporary Hashtable used while collecting links from a Url. This technique is used to avoid modifying the list of discovered links while iterating through them. Once all links have been discovered, the temporary Hashtable is merged with the master Hashtable and is discarded, to be cleaned up by the Garbage Collector.

As we have seen, the WebCrawler uses it's master Hashtable to keep track of the Urls it has found and uses the data stored in the Urls associated LinkInfo object to determine if it has previously visited the link. Like I mentioned earlier, this post touches on a very small part of the WebCrawler sample code -- for the complete sample, please consult your Visual Studio .NET 2003 installation (\Program Files\Microsoft Visual Studio .NET 2003\CompactFrameworkSDK\v1.0.5000\Windows CE\Samples\VC#\Pocket PC\WebCrawler).

Until next time,
-- DK

Disclaimer(s):
This posting is provided "AS IS" with no warranties, and confers no rights.

Accessing a WebService's WebRequest and WebResponse in NetCF

DavidKlineMS — Tue, 01 Feb 2005 04:34:00 GMT

One common question regarding the .NET Compact Framework involves the availability of the WebRequest and WebResponse objects used by the SOAP client when making a web service call. If you are using version 1 of the .NET Compact Framework (1.0.2268.0), these objects are not available to your code. Since the release of Service Pack 1 (1.0.3111.0), it is become possible to gain access to these objects.

Why access the underlying request and response?

There are a number of reasons why access to the WebRequest and/or WebResponse is useful. Custom headers can be used for simple client identification, cookies can be implemented, etc. I'm sure many other interesting scenarios will come to mind.

A few notes...

Most web service client proxy classes are autogenerated (either by WSDL.exe or Add Web Reference in Visual Studio .NET). Changes to the generated code files will be lost the next time they are regenerated (ex: "Update web reference" in Visual Studio).
GetWebRequest and GetWebResponse are protected methods on WebClientProtocol (a parent class to SoapHttpClientProtocol) and are called by the SOAP client class library, not by application code.

Keeping the above in mind, I recommend implementing your customizations in a derived class. Reference your derived class in your application, instead of the generated one. This way, you can update your web service client proxy as often as you need, without risk of losing your custom code.

Below is a simple example that sets a custom HTTP header on the underlying WebRequest.

Web service client proxy
public class MyWebService : System.Web.Services.Protocols.SoapHttpClientProtocol { { // autogenerated code will be in here }

Derived proxy class
public class MyCustomWebService : MyWebService { private String m_HeaderName; private String m_HeaderValue;

     //----------------
    // GetWebRequest
    //
    // Called by the SOAP client class library
    //----------------   
    protected override WebRequest GetWebRequest(Uri uri)
    {
        // call the base class to get the underlying WebRequest object
        HttpWebRequest req = (HttpWebRequest)base.GetWebRequest(uri);
    
        if(null != this.m_HeaderName)
        {
            // set the header
            req.Headers.Add(this.m_HeaderName, this.m_HeaderValue);
        }       
    
        // return the WebRequest to the caller
        return (WebRequest)req;
    }
     //----------------
    // SetRequestHeader
    //
    // Sets the header name and value that GetWebRequest will add to the
    // underlying request used by the SOAP client when making the 
    // we method call
    //----------------   
    public void SetRequestHeader(String headerName, String headerValue)
    {
        this.m_HeaderName = headerName;
        this.m_HeaderValue = headerValue;
    }   
}

Simple client application
class MyApplication { public static void Main() { // create the web service client object MyCustomWebService service = new MyCustomWebService();

         // set the custom header
        service.SetRequestHeader("ACustomHeader", "Go Huskies!");
         // call the desired web service method        
        Object result = service.ExampleMethod( ... );        
    }
}

GetWebResponse can be overriden similarly. As you can see, it is easy to gain access to the underlying WebRequest and WebResponse objects. Please remember that GetWebRequest and GetWebResponse are not called by application code -- they are methods you implement for the SOAP client to call.

Have fun!
-- DK

Disclaimer(s):
This posting is provided "AS IS" with no warranties, and confers no rights.

[Edit: fix version number]

Determining the IP address of your device

DavidKlineMS — Tue, 18 Jan 2005 21:27:00 GMT

Device's do not typically register a name with the network. This means you cannot just use a name to connect to or ping the device. Back in July 2004, I posted a snippet that shows all of the IP addresses (IPv4 and IPv6) that equate to "localhost". Today, I am posting a slightly modified version that displays the IP addresses that you can use to connect to your device.

Please note: I intentionally did not make a user interface to keep the snippet as small as possible. Also, since many devices do not come with a command prompt, I did not use Console.WriteLine.

 using System;
using System.Net;
using System.Windows.Forms;
 class WhoAmI
{
    public static void Main()
    {
        try
        {
            IPHostEntry he = Dns.Resolve(Dns.GetHostName());
            foreach(IPAddress addr in he.AddressList)
            {
                MessageBox.Show(addr.ToString());
            }
        }
        catch(Exception e)
        {
            MessageBox.Show(String.Format("Caught: {0}\r\n{1}", e.GetType().ToString(), e.Message));
        }
    }
}

That's all there is to it! I use code based on the above when running the Windows Mobile emulator as a standalone application (with ethernet enabled).

Enjoy!
-- DK

Disclaimer(s):
This posting is provided "AS IS" with no warranties, and confers no rights.

Determining the type of data pointed to by a Url - VB.NET version

DavidKlineMS — Wed, 08 Dec 2004 01:47:00 GMT

Earlier today, I posted an example of some Visual C# code that checks the type of the data pointed to by a Url. I was thinking that it might be nice to post a Visual Basic .NET equivalent...

This version is a direct port of the Visual C# code and comes with the same caveat:

"Please note: To keep this example as small as possible, only minimal error checking is performed - any real-world implementation would need to do much more than what I show here."

 Imports System
Imports System.Net
 module Module1
     sub Main(args as String())
    
        if(args.Length <> 1) then
            Console.WriteLine("Please specify a url path.")
            return
        end if
    
        ' display the content type for the url
        Dim url as String = args(0)
        Console.WriteLine(String.Format("Url : {0}", url))
        Console.WriteLine(String.Format("Type: {0}", GetContentType(url)))
     end sub   
    
    function GetContentType(url as String) as String
         Dim response as HttpWebResponse = nothing
        Dim contentType as String = ""
        
        try
        
            ' create the request
            Dim request as HttpWebRequest = CType(WebRequest.Create(url), HttpWebRequest)
             ' instruct the server to return headers only
            request.Method = "HEAD"
             ' make the connection
            response = CType(request.GetResponse(), HttpWebResponse)
             ' read the headers
            Dim headers as WebHeaderCollection = response.Headers
             ' get the content type
            contentType = headers("Content-type")
         catch e as WebException
        
            ' we encountered a problem making the request
            '  (server unavailable (404), unauthorized (401), etc)
            response = CType(e.Response, HttpWebResponse)
             ' return the message from the exception
            contentType = e.Message
         catch e as NotSupportedException
        
            ' this will be caught if WebRequest.Create encounters a uri
            '  that it does not support (ex: mailto)
             ' return a friendly error message
            contentType = "Unsupported Uri"
         catch e as UriFormatException
         
            ' the url is not a valid uri
            
            ' return a friendly error message
            contentType = "Malformed Uri"
         finally
        
            if not (response is nothing) then
                response.Close()               
            end if
            
        end try
               
        return contentType
        
    end function
    
end module

As with the Visual C# version, the above code can be compiled and run on either the .NET Framework or the .NET Compact Framework.

Take care,
-- DK

Disclaimer(s):
This posting is provided "AS IS" with no warranties, and confers no rights.

Determining the type of data pointed to by a Url

DavidKlineMS — Tue, 07 Dec 2004 21:25:00 GMT

Have you ever wanted to know what type of file was being pointed to by a given url before clicking the link? Maybe you are writing an application that needs to filter out certain types of links. A web crawler is a good example of an application which needs to do such link filtering (skip links to graphics, audio, zip files, etc).

In order to check the type of data pointed to by a url, you are going to need to issue a request to the server. Normally, this involves receiving the entire page or file at that location. This can be a time consuming proposition, especially over slow network connections, and defeats the purpose of allowing your application to filter out undesired links.

The solution is to issue a request to the server, asking only for the HTTP headers. This "HEAD" request is small, fast (does not transfer file contents) and provides you with exactly the data your application needs. While there are plenty of interesting headers, the header we are interested in today is "Content-type".

Below is a simple console application that takes a url path and displays the value of the content-type header. Please note: To keep this example as small as possible, only minimal error checking is performed - any real-world implementation would need to do much more than what I show here.

 using System;
using System.Net;
 class ContentType
{
    public static void Main(String[] args)
    {
        if(args.Length != 1)
        {
            Console.WriteLine("Please specify a url path.");
            return;
        }
         // display the content type for the url
        String url = args[0]; 
        Console.WriteLine(String.Format("Url : {0}", url));
        Console.WriteLine(String.Format("Type: {0}", GetContentType(url)));
    }
     private static String GetContentType(String url)
    {
        HttpWebResponse response = null;
        String contentType = "";
         try
        {
            // create the request
            HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
             // instruct the server to return headers only
            request.Method = "HEAD";
             // make the connection
            response = request.GetResponse() as HttpWebResponse;
             // read the headers
            WebHeaderCollection headers = response.Headers;
             // get the content type
            contentType = headers["Content-type"];
        }
        catch(WebException e)
        {
            // we encountered a problem making the request
            //  (server unavailable (404), unauthorized (401), etc)
            response = e.Response as HttpWebResponse;
             // return the message from the exception
            contentType = e.Message;
        }
        catch(NotSupportedException)
        {
            // this will be caught if WebRequest.Create encounters a uri
            //  that it does not support (ex: mailto)
             // return a friendly error message
            contentType = "Unsupported Uri";
        }
        catch(UriFormatException)
        {
            // the url is not a valid uri
            
            // return a friendly error message
            contentType = "Malformed Uri";
        }
        finally
        {
            // make sure the response gets closed
            //  this avoids leaking connections
            if(response != null)
            {
                response.Close();
            }
        }
         return contentType;
    }
}

The above code can be compiled and run on either the .NET Framework or the .NET Compact Framework.

Here's a sampling of the content types I received when running the above application against a handful of urls:

text/html
text/html; charset=utf-8
image/gif
application/octet-stream
text/plain

Enjoy!
-- DK

Disclaimer(s):
This posting is provided "AS IS" with no warranties, and confers no rights.

Simpifying socket exception handling

DavidKlineMS — Fri, 05 Nov 2004 05:07:00 GMT

Remembering numeric error codes is hard. Unless the error handling code is well documented, code reviews take longer and maintenance is more difficult as you may need to consult reference manuals, native code header files or online help frequently.

Socket errors are particularly hard for me to remember. To help myself out, I have collected Windows Sockets error codes into a couple of enums.

The first enum, SocketErrorCodes uses friendly descriptions based on the following Microsoft Support article . For detailed descriptions of the error codes, please refer to this article or a Windows Sockets reference.

 public enum SocketErrorCodes
{
    InterruptedFunctionCall         = 10004,
    PermissionDenied                = 10013,
    BadAddress                      = 10014,
    InvalidArgument                 = 10022,
    TooManyOpenFiles                = 10024,
    ResourceTemporarilyUnavailable  = 10035,
    OperationNowInProgress          = 10036,
    OperationAlreadyInProgress      = 10037,
    SocketOperationOnNonSocket      = 10038,
    DestinationAddressRequired      = 10039,
    MessgeTooLong                   = 10040,
    WrongProtocolType               = 10041,
    BadProtocolOption               = 10042,
    ProtocolNotSupported            = 10043,
    SocketTypeNotSupported          = 10044,
    OperationNotSupported           = 10045,
    ProtocolFamilyNotSupported      = 10046,
    AddressFamilyNotSupported       = 10047,
    AddressInUse                    = 10048,
    AddressNotAvailable             = 10049,
    NetworkIsDown                   = 10050,
    NetworkIsUnreachable            = 10051,
    NetworkReset                    = 10052,
    ConnectionAborted               = 10053,
    ConnectionResetByPeer           = 10054,
    NoBufferSpaceAvailable          = 10055,
    AlreadyConnected                = 10056,
    NotConnected                    = 10057,
    CannotSendAfterShutdown         = 10058,
    ConnectionTimedOut              = 10060,
    ConnectionRefused               = 10061,
    HostIsDown                      = 10064,
    HostUnreachable                 = 10065,
    TooManyProcesses                = 10067,
    NetworkSubsystemIsUnavailable   = 10091,
    UnsupportedVersion              = 10092,
    NotInitialized                  = 10093,
    ShutdownInProgress              = 10101,
    ClassTypeNotFound               = 10109,
    HostNotFound                    = 11001,
    HostNotFoundTryAgain            = 11002,
    NonRecoverableError             = 11003,
    NoDataOfRequestedType           = 11004
}

The second, WinSockErrorCodes, uses the symbolic names you see in native WinSock application code.

 public enum WinSockErrorCodes
{
    WSAEINTR           = 10004,
    WSAEACCES          = 10013,
    WSAEFAULT          = 10014,
    WSAEINVAL          = 10022,
    WSAEMFILE          = 10024,
    WSAEWOULDBLOCK     = 10035,
    WSAEINPROGRESS     = 10036,
    WSAEALREADY        = 10037,
    WSAENOTSOCK        = 10038,
    WSAEDESTADDRREQ    = 10039,
    WSAEMSGSIZE        = 10040,
    WSAEPROTOTYPE      = 10041,
    WSAENOPROTOOPT     = 10042,
    WSAEPROTONOSUPPORT = 10043,
    WSAESOCKTNOSUPPORT = 10044,
    WSAEOPNOTSUPP      = 10045,
    WSAEPFNOSUPPORT    = 10046,
    WSAEAFNOSUPPORT    = 10047,
    WSAEADDRINUSE      = 10048,
    WSAEADDRNOTAVAIL   = 10049,
    WSAENETDOWN        = 10050,
    WSAENETUNREACH     = 10051,
    WSAENETRESET       = 10052,
    WSAECONNABORTED    = 10053,
    WSAECONNRESET      = 10054,
    WSAENOBUFS         = 10055,
    WSAEISCONN         = 10056,
    WSAENOTCONN        = 10057,
    WSAESHUTDOWN       = 10058,
    WSAETIMEDOUT       = 10060,
    WSAECONNREFUSED    = 10061,
    WSAEHOSTDOWN       = 10064,
    WSAEHOSTUNREACH    = 10065,
    WSAEPROCLIM        = 10067,
    WSASYSNOTREADY     = 10091,
    WSAVERNOTSUPPORTED = 10092,
    WSANOTINITIALIZED  = 10093,
    WSAEDISCON         = 10101,
    WSATYPE_NOT_FOUND  = 10109,
    WSAHOST_NOT_FOUND  = 11001,
    WSATRY_AGAIN       = 11002,
    WSANO_RECOVERY     = 11003,
    WSANO_DATA         = 11004
}

I find the descriptive enum (SocketErrorCodes) most useful when I am writing new code. The WinSockErrorCodes enum is handy when porting native code to .NET. With either enum, I can simply cast the Int32 found in SocketException.ErrorCode to the appropriate enum to make my code easier to read and maintain. For example:

 try
{
    // code that causes a SocketException
}
catch(SocketException se)
{
    SocketErrorCodes errorCode = (SocketErrorCodes)se.ErrorCode;
    
    switch(errorCode)
    {
        case SocketErrorCodes.PermissionDenied:
            // error handling
            break;
         case SocketErrorCodes.AddressInUse:
            // error handling
            break;
         // etc..
    }
}

As mentioned earlier, the data used to create the above enums can be found here . Of course, any mistakes you may find here are entirely mine. When in doubt, please consult the support article or a Windows Sockets reference.

Until next time,
-- DK

[Edit: fix typeo]

Disclaimer(s):
This posting is provided "AS IS" with no warranties, and confers no rights.

Simple object remoting using the XmlSerializer

DavidKlineMS — Thu, 28 Oct 2004 04:02:00 GMT

Today, I'm going to talk about how you can use the XmlSerializer, TcpClient and TcpListener classes to create a simple remoting layer to send your custom objects from one device to another. This post requires the .NET Compact Framework version 2 (beta 1) or the .NET Framework for desktop PCs.

Since NetCF doesn't support the TCP Remoting classes, you will need to setup the network connection yourself. The serializer handles all of the data transfer. You can use this technique to send data between any combination of systems which support the XmlSerializer and network connectivity.

As you can see from the code below, there isn't too much to making a simple connection, though a real application will need to be more robust and will typically need to handle bi-directional data flow.

Let's take a look at simple client and server code that remotes a custom object between devices.

Object
The only requirements on the object is that it is serializable. This means that it contains a nullary constructor and all of the public fields are either read/write or are a collection (supporting the Add method). The example we will be using is a very simple object containing a name, a team name and a player's jersey number.
public class Player { public String Name; pulbic String Team; public Int32 Number; }

Client
The client application instantiates the Player object and populates it's data fields.
Player p = new Player(); p.Name = "David"; p.Team = "NetCF"; p.Number = 43;

Next, it creates an XmlSerializer object, informing the serializer of the type of data which will be sent.
XmlSerializer xs = new XmlSerializer(typeof(Player));

Once we have our serializer, we create a TcpClient object, get connected to the remote server and acquire the stream to be used to communicate with the server.
TcpClient client = new TcpClient(); client.Connect(serverAddress, serverPort); Stream stream = client.GetStream();
Now, we can serialize our Player object. Since the stream we provide the serializer is a network stream, the data is sent to the server when the serializer writes the XML.
xs.Serialize(stream, p);

The rest of our client is housekeeping. We need to be sure to close our stream and client objects.
stream.Close(); client.Close();

Server
Now let's take a look at the server code.
Player p;

As with the client, we need to create an XmlSerializer object, informing the serializer of the type of data which will be received.
XmlSerializer xs = new XmlSerializer(typeof(Player));

Next, a listener needs to be created and started.
TcpListener listener = new TcpListener(port); listener.Start();

The listener accepts the incoming connection.
TcpClient client = listener.AcceptTcpClient();

And we can get the data stream from the client which was created by the call to AcceptTcpClient.
Stream stream = client.GetStream();
Now we can deserialize the Player object sent by the remote client. Like the client, the stream used for deserialization is a network stream. This causes the data to be received when the serializer reads the XML.
p = (Player)xs.Deserialize(stream);

Again, we need to perform some housekeeping. We need to be sure to close our stream, client and listener objects.
stream.Close(); client.Close(); listener.Close();

And we can display the received object.
MessageBox.Show(String.Format("Name: {0}\r\nTeam: {1}\r\nNumber: {2}", p.Name, p.Team, p.Number));

As mentioned earlier, the above is a very simple example of how to remote a data object using the XmlSerializer. The networking code for real world applications will likely be significantly more involved and handle being sent and received by both devices.

Enjoy!
-- DK

[Edited to fix formatting]
[Edited to fix typo]

Disclaimers:
This posting is provided "AS IS" with no warranties, and confers no rights.
Some of the information contained within this post may be in relation to beta software. Any and all details are subject to change.

WebExceptions, ServicePoints and Connection Limits

DavidKlineMS — Sat, 11 Sep 2004 03:49:00 GMT

Today's topic involves WebExceptions and their proper handling. Along the way, we'll talk a little about HttpWebResponse, ServicePoint and connection limits.

Whenever you write an application that communicates on the web, you inevitably run into connection issues (site moved, incorrect access credentials, insufficient privileges to view the site, etc). .NET applications inform you of these issues by throwing a WebException when you attempt to establish your connection (by calling HttpWebRequest.GetResponse()).

The handling of a WebException requires a bit of care. This brings us to the topic of resonses and connection limits.

The .NET Compact Framework has a default connection limit, per ServicePoint (more on these later), of two (2). What this means is that you can only have two open connections on a given ServicePoint at the same time.

You may be asking what constitutes a ServicePoint. ServicePoints are made up of the scheme (ex: http), the host (ex: www.microsoft.com) and the port (ex:80). Paths, query strings, etc are not taken into consideration as part of the ServicePoint. This means that if you connect to both "folderA/page1.htm" and "folderB/page2.htm" on the same server, using the same port, you will have consumed the maximum number of connections on your service point (http://www.microsoft.com:80, in our example).

While you can increase the connection limit using the following,

// set connection limit to 5 ServicePointManger.DefaultConnectionLimit = 5;

I only recommend doing this if the users of your application must have more than two connections to any given server open at the same time.

Which brings us back to handling WebExceptions...

When your application catches a WebException, the first thing you should do is extract the response:

try { // http connection code } catch(WebException we) { // check the response HttpWebResponse response = we.Response as HttpWebResponse;

// the rest will be filled in below }

If the response is not null (Nothing in VisualBasic.NET), you can extract more information regarding the reason for the exception (from the HttpWebResponse.StatusCode property). Once you have the data, be sure to close the response object. This will release the connection and allow you to make another connection to the same server,

try { // http connection code } catch(WebException we) { // check the response HttpWebResponse response = we.Response as HttpWebResponse;

     if(null != response)
    {
        // get the reason for the exception
        HttpStatusCode status = response.StatusCode;
        String description = response.StatusDescription;
    
        // close the response
        response.Close();
    }

// do any further exception processing here }

Handling web exceptions in this way allows your application to take recovery steps (ex: re-prompt for credentials and retry the connection) and/or to display connection failure information to your customers.

Earlier, I mentioned that you can change the connection limit. If you are going to change the default value, please note that any active service points will not abide by the new limit. To ensure that all service points use the desired limit, you should modify the connection prior to making any connections (ex: at application startup).

Take care,
-- DK

[Edit: Fixed formatting && add missing null check]

Disclaimers:
This posting is provided "AS IS" with no warranties, and confers no rights.
Some of the information contained within this post may be in relation to beta software. Any and all details are subject to change.

NetCF application hanging when remote server disconnects from the network?

DavidKlineMS — Thu, 29 Jul 2004 22:55:00 GMT

I've been seeing a few reports of NetCF web service applications hanging when the target server disconnects from the network. If you are experiencing this, I may have a solution for you.

What we have found is that if the server disconnects before sending the initial HTTP response to the client (maybe it never actually received the HTTP request, or failed just prior to sending the response), NetCF (v1 and v2 beta1) will wait indefinitely for the response to be received.

Many of you may note that there is a Timeout property on the SoapHttpClientProtocol class (the parent class of each of your web service client proxy classes). This timeout does not come into play here (it applies to individual reads and writes between the client and server after the client receives the initial server response -- it does not specify the maximum amount of time for the call to complete).

Please note that since mobile devices typically operate on slow connections, timing out the entire request would likely cause most, if not all web service calls to fail and that sometimes your application may look hung and then eventually, start responding again (if the server was just really slow in responding).

If you are encountering this issue, the following workaround may be helpful. Please note that while this code utilizes web services, the workaround applies equally to the HTTP client (HttpWebRequest). Also note that this code implements a timeout on the complete method call and the timer will need to wait long enough to receive all of the expected data from the server (I recommend a few minutes in most cases).

orignal code
MyService service = new MyService();
service.Method( /* args */ );

workaround code
System.Threading.Timer timer;
Int32 timeout = 60000;
MyService service = new MyService();

private void CallService()
{
    try
    {
        // set the timeout to longer than you reasonably expect a user to wait for the call to complete
        this.timeout = 600000;   // NOTE: this is a really long timeout

        // create the timer
        this.timer = new System.Threading.Timer(new TimerCallback(TimeoutHandler),
                                               null,
                                               0,
                                               this.timeout);

        // call the web service method
        this.service.Method( /* args */ );
    }
    catch(Exception e)
    {
        // dispose of the timer
        this.timer.Dispose();

        // handle failure
        // TODO: perform any required cleanup and/or user notification, etc
    }
}

private void TimeoutHandler(Object obj)
{
   // we have waited long enough for the server to respond
   // abort the call
   this.service.Abort();
}

Hope this helps!
-- DK

NTLM authentication support in NetCF

DavidKlineMS — Mon, 19 Jul 2004 06:50:00 GMT

Many folks have asked the .NET Compact Framework team when our HTTP client was going to support NTLM authentication. For version 1 (and all service packs) our answer was: 'Sorry, we do not support NTLM. You may wish to locate a third party solution or write the auth code yourself...'. Not too cool.

With the beta 1 release of NetCF version 2, NTLM support is here!

If you have previously protected your web site / service using Digest authentication, all you need to do is enable NTLM on your server, install NetCF v2 on your clients and you'll authenticate using NTLM. As long as your client application collects credentials from the user and places them into a NetworkCredential object, you're set. The HTTP client will authenticate using whichever is the most secure method (Negotiate, NTLM, Digest, Basic) supported by the server. The code fragment, below, shows the construction and use of a NetworkCredential object.

using System.Net;

public void SetCredentials(HttpWebRequest request,
                           String userName,
                           String password,
                           String domain)
{
    // construct the credentials object
    NetworkCredential credentials = new NetworkCredential(userName,
                                                          password,
                                                          domain);

// set the credentials on the request
request.Credentials = credentials;
}

Nothing to it! Your client will now authenticate to your server using the most secure method available.

Please note, if your device has NetCF v1 and v2 installed, you must target your application to version 2 for NTLM authentication to occur. This can be done by:

Building your application against the v2 beta
Uninstall v1 from your device (not possible on PPC 2003 devices, as v1 is installed in ROM)
Construct an application configuration file for your executable, instructing it to run against the v2 beta

Option 1 is the simplest and ensures that your application always has NTLM support available.
Options 2 and 3 provide NTLM support for existing applications by retargeting them to the NetCF v2 runtime. (I will talk more about application configuration file support in NetCF v2 in a future post.)

-- DK

IPv6 and the NetCF v2 beta

DavidKlineMS — Tue, 06 Jul 2004 23:17:00 GMT

One of the new features of the .NET Compact Framework v2 beta is support for IPv6. Unlike the full .NET Framework, you do not need to explicitly enable IPv6 in NetCF -- if your device has support for IPv6 installed, it will be available to NetCF applications.

On devices supporting both IPv6 and IPv4 addressing, when resolving a host (localhost, in this example) the IPv6 addresses are always listed first in the resulting IPHostEntry.AddressList field. The code below resolves the local machine (via "localhost") and displays the addresses as they appear in the IPHostEntry.AddressList field:

using System;
using System.Net;

public class TestClass
{
    public static void Main()
    {
        try
        {
            IPHostEntry hostent = Dns.Resolve("localhost");
            foreach(IPAddress addr in hostent.AddressList)
            {
                Console.WriteLine(String.Format("{0} [{1}]",
                                                addr.ToString(),
                                                addr.AddressFamily.ToString()));
            }
        }
        catch(Exception e)
        {
            Console.WriteLine(String.Format("Caught: {0}", e.GetType().ToString()));
            Console.WriteLine(e.Message);
        }
    }
}

Output from above sample code:
::1 [InterNetworkV6]
127.0.0.1 [InterNetwork]

Of course, resolving localhost is not very interesting -- replacing "localhost" with my machine name yields a better example (actual addresses have been removed):
<<IPv6 over IPv4>> [InterNetworkV6]
<<IPv6 #1>> [InterNetworkV6]
<<IPv6 #2>> [InterNetworkV6]
<<IPv4 #1>> [InterNetwork]
<<IPv4 #2> [InterNetwork]

Note: The above system has two network adapters.

This behavior effects how the HTTP client attempts to communicate with a remote server -- if the remote machine supports IPv6, the v6 address will be tried first. If you wish to force the HTTP client to use an IPv4 address, you will need to resolve the host yourself and then replace the host name with it's corresponding v4 address.

Until next time.
-- DK

Disclaimers:
This posting is provided "AS IS" with no warranties, and confers no rights.
The information contained within this post is in relation to beta software. Any and all details are subject to change.

Why do I get SocketExceptions on NetCF when setting some socket options?

DavidKlineMS — Fri, 25 Jun 2004 19:16:00 GMT

I am frequently asked why NetCF applications encounter a SocketException when attempting to use some socket options, when applications written to the full .NET Framework do not. The companion question is invariably "Is this a bug in NetCF?".

If the result is WSAENOPROTOOPT (10042), what we are seeing is limitations of the operating system -- in this case, Windows CE -- so the answer is “no, this is not a bug, it is a limiation of the underlying operating system.“

Since the beginning, the .NET Compact Framework was designed to be portable -- run on more than one operating system. Because of this, we did not limit the available options in the SocketOptionNames enum. When calling Socket.GetSocketOption() or Socket.SetSocketOption(), the request is forwarded on to the operating system to do the work. If the result is a failure, the return code is wrapped into a SocketException and thrown.

The list of socket options which you can expect to throw, as described above, can be seen below (taken from the Windows CE API reference).

GetSocketOption
BSD options not supported for getsockopt are as follows.

Value	Type	Description
SO_RCVLOWAT	int	Retrieves recv low watermark.
SO_RCVTIMEO	int	Retrieves recv time-out value.
SO_SNDLOWAT	int	Retrieves send low watermark.
SO_SNDTIMEO	int	Retrieves time-out value for send.
TCP_MAXSEG	int	RetrievesTCP maximum-segment size.

Note: The above table taken from the MSDN documentation for getsockopt.

SetSocketOption
The following table shows BSD options not supported for setsockopt .

Value	Type	Description
SO_ACCEPTCONN	BOOL	Sets socket listening.
SO_RCVLOWAT	int	Sets recv low watermark.
SO_RCVTIMEO	int	Sets time-out for recv.
SO_SNDLOWAT	int	Sets send low watermark.
SO_SNDTIMEO	int	Sets time-out value for send.
SO_TYPE	Int	Sets socket type.

Note: The above table taken from the MSDN documentation for setsockopt.

If you are writing portable code (runs on both the .NET Framework and the .NET Compact Framework), you will need to handle this exception and change your application behavior as needed. For example, you may wish to disable controls related to expert socket features (ex: Receive Timeout). The code fragment below shows

using System.Net;
using System.Net.Sockets;

class SocketHelpers
{
   bool ReceiveTimeoutSupported(Socket socket)
   {
      try
      {
         // try to get ReceiveTimeout option (this will get the default value)
         Int32 timeout = (Int32)socket.GetSocketOption(SocketOptionLevel.Socket,
                                                       SocketOptionName.ReceiveTimeout);

         // try to set what we received (we don't want to change the default value)
         socket.SetSocketOption(SocketOptionLevel.Socket,
                                SocketOptionName.ReceiveTimeout,
                                timeout);

         // if we get here, ReceiveTimeout is supported
         return true;
      }
      catch(SocketException e)
      {
         // check to see if the selected option is supported
         if(10042 != e.ErrorCode)   // WSAENOPROTOOPT
         {
            // ReceiveTimeout not supported by operating system
            return false;
         }
         else
         {
            // no, let the caller handle the exception
            throw;
         }
      }
   }
}

I hope this helps clear up some of the confusion around socket options and the .Net Compact Framework.

-- DK

Disclaimer:
This posting is provided "AS IS" with no warranties, and confers no rights.

Increasing web service client performance

DavidKlineMS — Fri, 18 Jun 2004 21:03:00 GMT

As the demand for web enabled smart devices grows, more and more mobile applications will be using web services. Many of the folks who are already using web services with the .NET Compact Framework (NetCF) have come to me with performance issues. More often than not, the problem boils down to what happens when NetCF sees the first web method call on an instance of the service object.

When the first web method on a service object is called, NetCF uses Reflection to examine the service's proxy (to identify methods, headers, properties, etc). Unlike the full .NET Framework, NetCF (for working set size reasons) does not cache the results of this examination. Because of this, NetCF applications incur a performance penalty if they use multiple instances of the same service. The code below illustrates an application demonstrating this performance hit.

class SlowerWebServicePerformance
{
    public static void Main()
    {
        // application setup

        foreach(String name in Friends)
        {
            String phoneNumber = CallWebService(name);

// process / display the received data
}

// application cleanup
}

    public static string CallWebService(String name)
    {
        // create new instance of the web service proxy object
        PhoneBookService service = new PhoneBookService();

        // call the desired web method
        // proxy reflection occurs here
        return service.LookupPhoneNumber(name);
    }
}

In the above example, each call to CallWebService creates a new instance of the fictitios PhoneBookService object with each call to the LookupPhoneNumber method causing NetCF to reflect over the service proxy code. In this example, users with a fewer friends are better off than those with more -- at least as far as application performance goes.

To minimize the effects of this issue, applications can create a class global instance of their web service object and make a simple call to it (check version, etc) during the startup code. The code below is a re-write of the previous example, this time using a class global service object.

class FasterWebServicePerformance
{
private static PhoneBookService service = new PhoneBookService();

    public static void Main()
    {
        // call a simple web service method to "prime the pump"
        // proxy reflection occurs here
        service.GetVersion();

// application setup

        foreach(String name in Friends)
        {
            String phoneNumber = CallWebService(name);

// process / display the received data
}

// application cleanup
}

    public static string CallWebService(String name)
    {
        // call the desired web method
        // proxy reflection does not occur
        return service.LookupPhoneNumber(name);
    }
}

As you can see from the second implementation, the Main method makes a call to the service's GetVersion method so that the reflection occurs exactly once during the course of the application. The data received from this call is not relevant here, since we merely wish to “prime the pump“. With this change, the penalty for having more friends is gone.

Please keep in mind, when writing your applications in this manner, any headers required by the web service are applied to all method calls, so do not modify them while a web method call is in progress (or your calls may fail based on bad header data). While this applies to asynchronous and multi-threaded applications, it's still a good idea to keep it in mind whenever working with web services. Since NetCF's web service client classes are thread safe, so you can feel free to pass your class global service instance to child threads -- provided that you remember the previous statement.

-- DK

[Edit: categorization]

Disclaimer:
This posting is provided "AS IS" with no warranties, and confers no rights.