Media And Microcode : Get-MarkupTag

Microcode: PowerShell Scripting Tricks: Scripting the Web (Part 3) (Resolve-Link, Get-WebPageLink)

JamesBrundage — Fri, 12 Dec 2008 12:59:00 GMT

The first post in this series was learning to crawl. I introduced Get-Web, which allows you to use System.Net.Webclient to download web sites in a variety of ways. The next post was learning to walk. I showed us Get-MarkupTag, which helps coerce parts of the web into XML. Now we can start to really have some fun with the data and run wild.

Pulling out semi-structured data is one thing, but it’s important to be able to pull out more complex information as well. One interesting case is pulling out all of the links from a webpage. This task breaks down into four smaller tasks:

Downloading the page (done with Get-Web)
Getting the <a> tags in a meaningful way (done with Get-MarkupTag)
Extracting out the href attribute
Determining if the link is relative or absolute

To determine if the link is relative or absolute, I made a Resolve-Link function. It takes a base url (e.g http://www.foo.com/blah/blah.asp) and a link found on it, and returns the real item it resolves to. It optionally returns a property bag with the type of link and the resolved link.

Here’s Resolve-Link:

function Resolve-Link([Uri]$uri, [string]$link, [switch]$returnLinkType) { #.Synopsis # Resolves a relative or absolute link to an absolute url #.Description # Takes a uri and a link to a page and returns the absolute url, or # optionally returns a property bag with the link type # (absolute, relative, or host relative) and the link #.Parameter uri # The uri the link is located on #.Parameter link # The original link text #.Parameter returnLinkType # The return link type #.Example # Resolve-Link http:/www.microsoft.com/ /technet/scriptcenter if ($link.StartsWith("/")) { # Relative to Host site if ($returnLinkType) { return New-Object Object | Add-Member NoteProperty Type "Host Relative" -PassThru | Add-Member NoteProperty Link ([uri]"$($uri.Scheme)://$($uri.DnsSafehost)$($link)") -PassThru } return "$($uri.Scheme)://$($uri.DnsSafehost)$($link)" } else { if ($link.StartsWith("$($uri.Scheme)://")) { # Absolute Link if ($returnLinkType) { return New-Object Object | Add-Member NoteProperty Type "Absolute" -PassThru | Add-Member NoteProperty Link ([uri]$link) -PassThru } return $link } else { # Relative link $realLink = $uri.AbsoluteUri.Substring(0, $uri.AbsoluteUri.LastIndexOf("/")) + "/$link" if ($returnLinkType) { return New-Object Object | Add-Member NoteProperty Type "Relative" -PassThru | Add-Member NoteProperty Link ([uri]$realLink) -PassThru } return $realLink } } }

Once Resolve-Link was written, making Get-WebPageLink is an incredible snap. It’s below, and it actually takes only 3 lines to do the real work and 11 lines to explain the work and give examples.

function Get-WebPageLink($url) { #.Synopsis # Returns all of the links within a webpage #.Description # Resolves all <A> references and returns a property bag with # the text contained in the link, the page the link came from, # and the type of link returned (absolute, host relative, or relative) #.Parameter urltp # The page to get links from #.Example # Get-WebPageLink http://blogs.msdn.com/ Get-MarkupTag a (Get-Web $url) | Foreach-Object { Resolve-Link $url $_.Xml.Href -returnLinkType | Add-Member NoteProperty Text $_.Xml."#text" -PassThru } }

Go ahead and give Get-WebpageLink a whirl:

Get-WebpageLink http://blogs.msdn.com

Ready for some real fun? Remember way back when I did a post about getting RSS feeds in PowerShell with Microsoft.FeedsManager (Get-Feed). If you have that script handy, go ahead and check out this one liner that will refresh every RSS item you’ve got and extract out all of the links from it.

Get-Feed -recurse -articles | Foreach-Object { Get-WebPageLink $_.Link }

That particular command line can take a while, depending on how many blogs you subscribe to, but it gives you a brand new view on blogs (as a simmering stew of scripts, rather than just text to be read and comprehended).

There’s more fun to come in unlocking the web, but these two scripts should get you started in extracting a little more into the wild world of the web.

Hope this Helps,

James Brundage [MSFT]

Microcode: PowerShell Scripting Tricks: Scripting The Web (Part 2) (Get-MarkupTag)

JamesBrundage — Mon, 08 Dec 2008 11:17:00 GMT

The first post about scripting the was a lot of waxing philosophical but little about how to extract data and give it form. There are several approaches, with various difficulties. I could build a full HTML parser and walk though object models, or I could use the object model of IE. Since I personally like to minimize dependencies, I’ve chosen to use System.Net.Webclient to download the webpage as text rather than use Internet Explorer and get it through a proper object model. This means I will either have to write an HTML parser, or I’ll have to nudge HTML into something more useful to avoid writing a full parser.

HTML is preciously close to XML, and XML is something that PowerShell supports quite well, so e IE sI decided to try to nudge HTML into XML. I wrote a function, Get-MarkupTag, which will extract out the text for a markup tag (e.g. <a>) and attempt to coerce it into XML.

It’s not a perfect approach, because there several things that are legal in HTML but not in XML. Here’s an inventory of every curveball I’ve hit so far.

Unclosed tags like <IMG>:

Some tags in HTML, like <IMG> are unmatched. I’ll first have to identify all of the unmatched tags and coerce them into property closed xml (e.g. ensure that <IMG is <IMG />). For things like <BR> or <HR> this is easy, because attributes are rare, but for IMG, getting this right is critically important, because it’s how you download images.

HTML Escape Sequences end up as XML unrecognized entities:

There’s a lot of ways to embed special characters within HTML that get recognized by the parser as XML entities. The most common example is   (the explicit space), but foreign currencies often show up in this format as well. I found a complete list online that I could extract out all of the sequences from, and I embedded a hashtable within to hold each item. Then I walked through the hashtable and replaced the escape sequences with their real value. However, since the entity is legal in any since the entity is legal in any case (e.g. &nsbp; is the same as &NBSP; to an HTML parser), I had to write a quick case insensitive replace.

Unquoted Attributes:

Most browsers will accept HTML attributes without quotes, e.g. <a href=www.microsoft.com>, but XML can handle this. Once I’ve got figured out the text in each tag, I need to check each attribute within that tag to ensure that the attributes are quoted.

Nested tags:

One of the more interesting parts of parsing HTML was nested tags. In order to match nested tags, what I did was use two regular expressions to identify all of the matches. While PowerShell has a –match operator, –match doesn’t work for multiline strings. So what I did was use New-Object to create the Regular Expression for extracting out a tag and then stored the result in a variable.

If the number of start tags was equal to the number of end tags, I assumed that the tags are balanced. If they’re odd, I’ll only deal with the start tags instead. If I need to determine which start tag matches which end tag, I need to use a data structure called a stack. Luckily, .NET comes with one (System.Collections.Stack). A stack in code is just like a stack in real life. You can put something on the top of the stack (Push), see what’s on top of the stack (Peek), and pull off the top of of the stack (Pop).

If I put start tags and end tags in one list, and sort them by the order that they occurred (in PowerShell, this is $list1 + $list2 | Sort-Object PropertyName), then I can simply walk through the list, pushing the start tags onto the stack and popping off end tags as I encounter them.

I don’t believe that I’ve taken care of all of the possible pains that exist (for instance, I know that georgraphic coordinates on wikipedia do not coerce into XML at this point. However, it’s a good start and it can yield some quite interesting results.

Since Get-MarkupTag involves a lot of escape sequences, which lends to very annoying blog formatting, I’m going to attach the script instead of embedding it. You can download it here.

If you look at the code in Get-MarkupTag, close to the end, you’ll see a trap statement. This means that there’s any error coercing the chunk of HTML into XML will be swallowed into the verbose log. If you want to examine why you can’t use the XML, simply set $verbosePreference to “continue” and you will be able to see the errors produced trying to convert the tag into XML. This is an example of a technique I call error redirection.

One of the kind of cool things you can do with it is extract the individual rows from ConvertTo-HTML:

$text = Get-ChildItem | Select Name, LastWriteTime | ConvertTo-HTML | Out-String Get-MarkupTag "tr" $text

It's also possible to extract out all of the pages www.microsoft.com links to:

$microsoft= (New-Object Net.Webclient).DownloadString("http://www.microsoft.com/") Get-MarkupTag "a" $microsoft | % { $_.Xml.Href }

What this fairly slightly quick and dirty approach to extracting out HTML gives you is a fairly simply way to start giving the data form. With XML you get the syntactic sugar PowerShell pours on XML, and you also get the power of XPath, and from that, you can start turning the HTML into something more interesting.

In the next post, I’ll look at some of the things you can do with this new toy.

Try it out on a few sites.

Hope this helps,

James Brundage [MSFT]