Media And Microcode : Scripting The Web

Microcode: PowerShell Scripting Tricks: Scripting the Web (Part 3) (Resolve-Link, Get-WebPageLink)

JamesBrundage — Fri, 12 Dec 2008 12:59:00 GMT

The first post in this series was learning to crawl. I introduced Get-Web, which allows you to use System.Net.Webclient to download web sites in a variety of ways. The next post was learning to walk. I showed us Get-MarkupTag, which helps coerce parts of the web into XML. Now we can start to really have some fun with the data and run wild.

Pulling out semi-structured data is one thing, but it’s important to be able to pull out more complex information as well. One interesting case is pulling out all of the links from a webpage. This task breaks down into four smaller tasks:

Downloading the page (done with Get-Web)
Getting the <a> tags in a meaningful way (done with Get-MarkupTag)
Extracting out the href attribute
Determining if the link is relative or absolute

To determine if the link is relative or absolute, I made a Resolve-Link function. It takes a base url (e.g http://www.foo.com/blah/blah.asp) and a link found on it, and returns the real item it resolves to. It optionally returns a property bag with the type of link and the resolved link.

Here’s Resolve-Link:

function Resolve-Link([Uri]$uri, [string]$link, [switch]$returnLinkType) { #.Synopsis # Resolves a relative or absolute link to an absolute url #.Description # Takes a uri and a link to a page and returns the absolute url, or # optionally returns a property bag with the link type # (absolute, relative, or host relative) and the link #.Parameter uri # The uri the link is located on #.Parameter link # The original link text #.Parameter returnLinkType # The return link type #.Example # Resolve-Link http:/www.microsoft.com/ /technet/scriptcenter if ($link.StartsWith("/")) { # Relative to Host site if ($returnLinkType) { return New-Object Object | Add-Member NoteProperty Type "Host Relative" -PassThru | Add-Member NoteProperty Link ([uri]"$($uri.Scheme)://$($uri.DnsSafehost)$($link)") -PassThru } return "$($uri.Scheme)://$($uri.DnsSafehost)$($link)" } else { if ($link.StartsWith("$($uri.Scheme)://")) { # Absolute Link if ($returnLinkType) { return New-Object Object | Add-Member NoteProperty Type "Absolute" -PassThru | Add-Member NoteProperty Link ([uri]$link) -PassThru } return $link } else { # Relative link $realLink = $uri.AbsoluteUri.Substring(0, $uri.AbsoluteUri.LastIndexOf("/")) + "/$link" if ($returnLinkType) { return New-Object Object | Add-Member NoteProperty Type "Relative" -PassThru | Add-Member NoteProperty Link ([uri]$realLink) -PassThru } return $realLink } } }

Once Resolve-Link was written, making Get-WebPageLink is an incredible snap. It’s below, and it actually takes only 3 lines to do the real work and 11 lines to explain the work and give examples.

function Get-WebPageLink($url) { #.Synopsis # Returns all of the links within a webpage #.Description # Resolves all <A> references and returns a property bag with # the text contained in the link, the page the link came from, # and the type of link returned (absolute, host relative, or relative) #.Parameter urltp # The page to get links from #.Example # Get-WebPageLink http://blogs.msdn.com/ Get-MarkupTag a (Get-Web $url) | Foreach-Object { Resolve-Link $url $_.Xml.Href -returnLinkType | Add-Member NoteProperty Text $_.Xml."#text" -PassThru } }

Go ahead and give Get-WebpageLink a whirl:

Get-WebpageLink http://blogs.msdn.com

Ready for some real fun? Remember way back when I did a post about getting RSS feeds in PowerShell with Microsoft.FeedsManager (Get-Feed). If you have that script handy, go ahead and check out this one liner that will refresh every RSS item you’ve got and extract out all of the links from it.

Get-Feed -recurse -articles | Foreach-Object { Get-WebPageLink $_.Link }

That particular command line can take a while, depending on how many blogs you subscribe to, but it gives you a brand new view on blogs (as a simmering stew of scripts, rather than just text to be read and comprehended).

There’s more fun to come in unlocking the web, but these two scripts should get you started in extracting a little more into the wild world of the web.

Hope this Helps,

James Brundage [MSFT]

Microcode: PowerShell Scripting Tricks: Scripting The Web (Part 2) (Get-MarkupTag)

JamesBrundage — Mon, 08 Dec 2008 11:17:00 GMT

The first post about scripting the was a lot of waxing philosophical but little about how to extract data and give it form. There are several approaches, with various difficulties. I could build a full HTML parser and walk though object models, or I could use the object model of IE. Since I personally like to minimize dependencies, I’ve chosen to use System.Net.Webclient to download the webpage as text rather than use Internet Explorer and get it through a proper object model. This means I will either have to write an HTML parser, or I’ll have to nudge HTML into something more useful to avoid writing a full parser.

HTML is preciously close to XML, and XML is something that PowerShell supports quite well, so e IE sI decided to try to nudge HTML into XML. I wrote a function, Get-MarkupTag, which will extract out the text for a markup tag (e.g. <a>) and attempt to coerce it into XML.

It’s not a perfect approach, because there several things that are legal in HTML but not in XML. Here’s an inventory of every curveball I’ve hit so far.

Unclosed tags like <IMG>:

Some tags in HTML, like <IMG> are unmatched. I’ll first have to identify all of the unmatched tags and coerce them into property closed xml (e.g. ensure that <IMG is <IMG />). For things like <BR> or <HR> this is easy, because attributes are rare, but for IMG, getting this right is critically important, because it’s how you download images.

HTML Escape Sequences end up as XML unrecognized entities:

There’s a lot of ways to embed special characters within HTML that get recognized by the parser as XML entities. The most common example is   (the explicit space), but foreign currencies often show up in this format as well. I found a complete list online that I could extract out all of the sequences from, and I embedded a hashtable within to hold each item. Then I walked through the hashtable and replaced the escape sequences with their real value. However, since the entity is legal in any since the entity is legal in any case (e.g. &nsbp; is the same as &NBSP; to an HTML parser), I had to write a quick case insensitive replace.

Unquoted Attributes:

Most browsers will accept HTML attributes without quotes, e.g. <a href=www.microsoft.com>, but XML can handle this. Once I’ve got figured out the text in each tag, I need to check each attribute within that tag to ensure that the attributes are quoted.

Nested tags:

One of the more interesting parts of parsing HTML was nested tags. In order to match nested tags, what I did was use two regular expressions to identify all of the matches. While PowerShell has a –match operator, –match doesn’t work for multiline strings. So what I did was use New-Object to create the Regular Expression for extracting out a tag and then stored the result in a variable.

If the number of start tags was equal to the number of end tags, I assumed that the tags are balanced. If they’re odd, I’ll only deal with the start tags instead. If I need to determine which start tag matches which end tag, I need to use a data structure called a stack. Luckily, .NET comes with one (System.Collections.Stack). A stack in code is just like a stack in real life. You can put something on the top of the stack (Push), see what’s on top of the stack (Peek), and pull off the top of of the stack (Pop).

If I put start tags and end tags in one list, and sort them by the order that they occurred (in PowerShell, this is $list1 + $list2 | Sort-Object PropertyName), then I can simply walk through the list, pushing the start tags onto the stack and popping off end tags as I encounter them.

I don’t believe that I’ve taken care of all of the possible pains that exist (for instance, I know that georgraphic coordinates on wikipedia do not coerce into XML at this point. However, it’s a good start and it can yield some quite interesting results.

Since Get-MarkupTag involves a lot of escape sequences, which lends to very annoying blog formatting, I’m going to attach the script instead of embedding it. You can download it here.

If you look at the code in Get-MarkupTag, close to the end, you’ll see a trap statement. This means that there’s any error coercing the chunk of HTML into XML will be swallowed into the verbose log. If you want to examine why you can’t use the XML, simply set $verbosePreference to “continue” and you will be able to see the errors produced trying to convert the tag into XML. This is an example of a technique I call error redirection.

One of the kind of cool things you can do with it is extract the individual rows from ConvertTo-HTML:

$text = Get-ChildItem | Select Name, LastWriteTime | ConvertTo-HTML | Out-String Get-MarkupTag "tr" $text

It's also possible to extract out all of the pages www.microsoft.com links to:

$microsoft= (New-Object Net.Webclient).DownloadString("http://www.microsoft.com/") Get-MarkupTag "a" $microsoft | % { $_.Xml.Href }

What this fairly slightly quick and dirty approach to extracting out HTML gives you is a fairly simply way to start giving the data form. With XML you get the syntactic sugar PowerShell pours on XML, and you also get the power of XPath, and from that, you can start turning the HTML into something more interesting.

In the next post, I’ll look at some of the things you can do with this new toy.

Try it out on a few sites.

Hope this helps,

James Brundage [MSFT]

Microcode: PowerShell Scripting Tricks: Scripting The Web (Part 1) (Get-Web)

JamesBrundage — Mon, 01 Dec 2008 06:13:06 GMT

Several of the last posts have tackled how to take the wild world of data and start to turn it into PowerShell objects, so that it’s easier to make heads or tails out of it. Once all of that data is in a form that PowerShell can use more effectively in the object pipeline, you can use PowerShell to slice and dice that data in wonderfully effective ways.

Data mining is a strange art that I find scripting languages normally are a little better at than compiled languages. I find that PowerShell can be incredibly useful for data miners for two important reasons. First, it can pull upon an unprecedented array of technologies (COM, WMI, .NET, SQL, Regular Expressions, Web Service, Command Line) in order to get structured data. Second, and more importantly, since you can easily create objects on the fly in PowerShell, and since PowerShell has wonderful string processing, you can often use PowerShell to extract structure data out of unstructured data.

Being able to pull data out of the mist and give it form is a very valuable skill, because people do not think in structured data. While it might be useful to the computing world to have most information in structured data, most people disseminating information don’t think or record their thoughts with rigorous structure. However, many people do record their thoughts in semi-rigorous structure, like the sentences and paragraphs you’re reading now.

The fact of the matter is that tons of data floats on the web requiring a very little bit of work and a small amount of art to extract it. This is because the web that people record their thoughts in is largely in HTML, and so, it is possible to learn a few ways to pull the data from the little structure that exists.

The first piece of the toolkit to extract out data from the Web is a function I’ve called Get-Web. Get-Web will simply download web pages, and it wraps part of the System.Net.Webclient object.

Using WebClient has pros and cons. The biggest pro is that it relies on .NET, rather than on a particular browser, which means that you can use it without IE. The biggest con is that a lot of web pages do checks on the browser in order to change how they display.

Get-Web is below. As with Get-HashtableAsObject, I’m using comments to declare some nifty inline help.

function Get-Web($url, 
    [switch]$self,
    $credential, 
    $toFile,
    [switch]$bytes)
{
    #.Synopsis
    #    Downloads a file from the web
    #.Description
    #    Uses System.Net.Webclient (not the browser) to download data
    #    from the web.
    #.Parameter self
    #    Uses the default credentials when downloading that page (for downloading intranet pages)
    #.Parameter credential
    #    The credentials to use to download the web data
    #.Parameter url
    #    The page to download (e.g. www.msn.com)    
    #.Parameter toFile
    #    The file to save the web data to
    #.Parameter bytes
    #    Download the data as bytes   
    #.Example
    #    # Downloads www.live.com and outputs it as a string
    #    Get-Web http://www.live.com/
    #.Example
    #    # Downloads www.live.com and saves it to a file
    #    Get-Web http://wwww.msn.com/ -toFile www.msn.com.html
    $webclient = New-Object Net.Webclient
    if ($credential) {
        $webClient.Credential = $credential
    }
    if ($self) {
        $webClient.UseDefaultCredentials = $true
    }
    if ($toFile) {
        if (-not "$toFile".Contains(":")) {
            $toFile = Join-Path $pwd $toFile
        }
        $webClient.DownloadFile($url, $toFile)
    } else {
        if ($bytes) {
            $webClient.DownloadData($url)
        } else {
            $webClient.DownloadString($url)
        }
    }
}

To walk through a few examples of Get-Web, simply point it to any webpage.

	Get-Web http://en.wikipedia.org/

To save a page to disk

	Get-Web http://www.msn.com/ -toFile www.msn.com.html

Just downloading the data is only the first step. All being able to download web data gives you is a way to get the mist into a bottle, but it doesn’t help you give it form. The next piece will cover how to pull the data out of the web and into PowerShell.

Hope this helps,

James Brundage [MSFT]