Online Store Automation - Populating Our Store (Part 2)

Going back to my previous post, I wanted to add more details on how webpages can be extracted, especially in the case where no login is required.

If there was no need for a web-based login for a page, I could have extracted the content of a page without the need for a WebBrowser control. The following code takes a web page based on a URI and extracts its contents into a string.

1:         
    static public string Extract(string URI, int MaxRedirects, System.Net.NetworkCredential credentials)
2:         {
3:             WebRequest myWebRequest = CreateRequest(URI, MaxRedirects);
4:             myWebRequest.Credentials = credentials;
5:             WebResponse myWebResponse = myWebRequest.GetResponse();
6:  
7:             Stream ReceiveStream = myWebResponse.GetResponseStream();
8:             Encoding encode = Encoding.GetEncoding("utf-8");
9:             StreamReader readStream = new StreamReader(ReceiveStream, encode);
10:             Char[] read = new Char[256];
11:             int count = readStream.Read(read, 0, 256);
12:             string html = "";
13:             while (count > 0)
14:             {
15:                 // Dump the 256 characters on a string and display the string
    onto the console.
16:                 html += new string(read, 0, count);
17:                 count = readStream.Read(read, 0, 256);
18:             }
19:  
20:             readStream.Close();
21:             ReceiveStream.Close();
22:             myWebResponse.Close();
23:  
24:             return html;
25:         }

The code for the "CreateRequest" method is as follows:

1:         
    static public WebRequest CreateRequest(string URI)
2:         {
3:             HttpWebRequest myWebRequest = WebRequest.Create(URI) as HttpWebRequest;
4:             myWebRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
    5.0; .NET CLR 1.1.4433)";
5:             myWebRequest.KeepAlive = false;
6:             myWebRequest.AllowAutoRedirect = false;
7:             return myWebRequest;
8:         }

This is pretty straightforward but thought I should share this for completness.