Media And Microcode : Get-Feed

Microcode: PowerShell Scripting Tricks: Scripting the Web (Part 3) (Resolve-Link, Get-WebPageLink)

JamesBrundage — Fri, 12 Dec 2008 12:59:00 GMT

The first post in this series was learning to crawl. I introduced Get-Web, which allows you to use System.Net.Webclient to download web sites in a variety of ways. The next post was learning to walk. I showed us Get-MarkupTag, which helps coerce parts of the web into XML. Now we can start to really have some fun with the data and run wild.

Pulling out semi-structured data is one thing, but it’s important to be able to pull out more complex information as well. One interesting case is pulling out all of the links from a webpage. This task breaks down into four smaller tasks:

Downloading the page (done with Get-Web)
Getting the <a> tags in a meaningful way (done with Get-MarkupTag)
Extracting out the href attribute
Determining if the link is relative or absolute

To determine if the link is relative or absolute, I made a Resolve-Link function. It takes a base url (e.g http://www.foo.com/blah/blah.asp) and a link found on it, and returns the real item it resolves to. It optionally returns a property bag with the type of link and the resolved link.

Here’s Resolve-Link:

function Resolve-Link([Uri]$uri, [string]$link, [switch]$returnLinkType) { #.Synopsis # Resolves a relative or absolute link to an absolute url #.Description # Takes a uri and a link to a page and returns the absolute url, or # optionally returns a property bag with the link type # (absolute, relative, or host relative) and the link #.Parameter uri # The uri the link is located on #.Parameter link # The original link text #.Parameter returnLinkType # The return link type #.Example # Resolve-Link http:/www.microsoft.com/ /technet/scriptcenter if ($link.StartsWith("/")) { # Relative to Host site if ($returnLinkType) { return New-Object Object | Add-Member NoteProperty Type "Host Relative" -PassThru | Add-Member NoteProperty Link ([uri]"$($uri.Scheme)://$($uri.DnsSafehost)$($link)") -PassThru } return "$($uri.Scheme)://$($uri.DnsSafehost)$($link)" } else { if ($link.StartsWith("$($uri.Scheme)://")) { # Absolute Link if ($returnLinkType) { return New-Object Object | Add-Member NoteProperty Type "Absolute" -PassThru | Add-Member NoteProperty Link ([uri]$link) -PassThru } return $link } else { # Relative link $realLink = $uri.AbsoluteUri.Substring(0, $uri.AbsoluteUri.LastIndexOf("/")) + "/$link" if ($returnLinkType) { return New-Object Object | Add-Member NoteProperty Type "Relative" -PassThru | Add-Member NoteProperty Link ([uri]$realLink) -PassThru } return $realLink } } }

Once Resolve-Link was written, making Get-WebPageLink is an incredible snap. It’s below, and it actually takes only 3 lines to do the real work and 11 lines to explain the work and give examples.

function Get-WebPageLink($url) { #.Synopsis # Returns all of the links within a webpage #.Description # Resolves all <A> references and returns a property bag with # the text contained in the link, the page the link came from, # and the type of link returned (absolute, host relative, or relative) #.Parameter urltp # The page to get links from #.Example # Get-WebPageLink http://blogs.msdn.com/ Get-MarkupTag a (Get-Web $url) | Foreach-Object { Resolve-Link $url $_.Xml.Href -returnLinkType | Add-Member NoteProperty Text $_.Xml."#text" -PassThru } }

Go ahead and give Get-WebpageLink a whirl:

Get-WebpageLink http://blogs.msdn.com

Ready for some real fun? Remember way back when I did a post about getting RSS feeds in PowerShell with Microsoft.FeedsManager (Get-Feed). If you have that script handy, go ahead and check out this one liner that will refresh every RSS item you’ve got and extract out all of the links from it.

Get-Feed -recurse -articles | Foreach-Object { Get-WebPageLink $_.Link }

That particular command line can take a while, depending on how many blogs you subscribe to, but it gives you a brand new view on blogs (as a simmering stew of scripts, rather than just text to be read and comprehended).

There’s more fun to come in unlocking the web, but these two scripts should get you started in extracting a little more into the wild world of the web.

Hope this Helps,

James Brundage [MSFT]

Microcode: Scripting RSS Feeds with PowerShell and Microsoft.FeedsManager

JamesBrundage — Tue, 11 Nov 2008 04:34:53 GMT

PowerShell's an amazing glue language. It can help you bring code from all corners of the earth into one environment, and then you can custom the code to be more to your liking. While the last several entries of my blog have spent time looking at what we can glue together with .NET, this one shows you how you can bring COM into the mix.

Internet Explorer has an RSS feed manager which you can script in .NET. While the command line might not be the ideal spot to read your blogs, it's possible to make a much nicer front end by scripting WPF & PowerShell, and having a quick and easy API to get at your RSS feeds makes building applications that use RSS feeds as a way to synchronize information between multiple machines.

The object that makes this all work is Microsoft.FeedsManager, and the cmdlet that makes working with it possible is New-Object. By default, New-Object will create a new .NET object from the classes that are currently loaded by .NET (for more on this, see my previous posts about Get-Type and Get-Assembly). While .NET is a fairly easy world to explore, COM is considerably more tricky.

Luckily, COM objects are the main backbone of VBScript. This is an important point for VBScripters trying to learn PowerShell: while the world of .NET may be new and exotic, and Cmdlets might be cool, you don't have to drop your VBScript knowledge at the door. While some of a VBScript (e.g. strings) will require new learning in PowerShell, the vast majority of it doesn't.

Microsoft.FeedsManager is one of these examples. I found out about this object by looking at the source code for the RSS reader vista sidebar gadget, and, once I knew its name, I knew how to get at it in PowerShell. Even though I was looking at javascript, I knew that there was a scriptable COM object there to help me out, and I knew how to bring it into PowerShell.

To create a Microsoft.FeedsMananger, simply use this one liner:

New-Object -comObject Microsoft.FeedsManager

Since the root folder looks promising, let's explore that:

(New-Object -comObject Microsoft.FeedsManager).RootFolder

While I can just save this into a variable and script it like I were still in VBScript land (e.g. $feeds = New-Object -comObject Microsoft.FeedsManager, $feeds.RootFolder), I far prefer the parlance of PowerShell, so I'll make a quick Get-Feed function to explore feeds more easily. I'll accept a wildcard for the name or title of the feed, and, since the feeds can have an arbitrary number of subfolders, I'll add a -recurse parameter. In order to make -recurse work elegantly, I'll use a short recursive script and I'll add a $folder parameter, which can be null and will be assumed to be the root folder. Finally, since I'll often want to read the items more than the feed, I'll add a switch parameter to extract the articles.

function Get-Feed($feed = "*", $folder, [switch]$recurse, [switch]$articles) {
    if (! $folder) {
        $feedsManager = New-Object -ComObject Microsoft.FeedsManager
        $folder = $feedsManager.RootFolder
    }
    $folder.Feeds |
        Where-Object {
            ($_.Title -like "$feed") -or ($_.Name -like "$feed")
        } | Foreach-Object {
            if ($articles) { $_.Items } else { $_ }
        }
    if ($recurse) {
        $folder.Subfolders | Foreach-Object {
            Get-Feed $feed $_ -recurse -articles:$articles
        }
    }
}

Once I've defined Get-Feed, I can use it to search my feeds through PowerShell. This one liner will get my all my feeds, sort them by name, and display just the Name and URL:

Get-Feed -recurse | Sort-Object Name | Select-Object Name, URL

To see all of the feed items sort them by when they were published, and display the post title, the blog title, and the publish date, you can use this pipeline:

Get-Feed -articles -recurse |
Sort-Object PubDate -descending |
Select-Object Title, @{Name='Blog'; Expression={$_.Parent.Title }}, PubDate

I promise that we can do much more with RSS and PowerShell, but hopefully this post will get you started down the path of using the FeedsManager from PowerShell.

Hope this helps,

James Brundage [MSFT]

Module Name	Scripts
RSS	Get-Feed