Merry Christmas From PowerShell: The CodeDownloader Module

Merry Christmas From PowerShell: The CodeDownloader Module

  • Comments 3

Twas the night before Christmas, and all through the net
PowerShell lovers were wondering exactly what they might get
Their readers were ready, their minds were aware
That more joy of CTP3 would soon be there
A cmdlet, a function?  What has the PowerShell team done?
How about a whole module, to share scripts with everyone?
Some pluggable functions, to get code to share
Scripts from Write-CommandBlogPost, from PoshCode, from Everywhere
And what on the morning of Christmas did they see?
But eight advanced functions, to get code from you or me.
To rhyme more would be wrong, so I'll just explain.
I'll introduce the functions, and call them by name.
Get-MarkupTag, Get-CommandPlugin, and Resolve-ShortcutFile you've seen
But the CodeDownloader module will make your eyes gleam
Bookmark some favorite posts, or something on Posh-Code
Put them in your favorites directory, and into a module they'll go
Import-ModuleFromWeb on a directory of lines, and down comes the script
Into a module named anything you see fit.
And if your blog doesn't work with the toy
Then write a plugin for Get-Code, and share in the joy.
While rhyming is cute, for explanations it won't do
So here's each script from the module, for every one of you.
And if copying and pasting is too much to ask
Download the attached zip file, and you're about done with your task.

  1. Get the Get-MarkupTag, Resolve-ShortcutFile, and Get-CommandPlugin from the earlier blog posts
  2. Download the zip file.
  3. Right click properties, click "unblock".
  4. Extract the Zip file to $env:UserProfile\Documents\WindowsPowerShell\Modules\CodeDownloader
  5. Run Import-Module CodeDownloader

Add some of the recent blog posts to your favorites, or some scripts from poshcode, and try using Import-ModuleFromWeb to import some scripts from the web into a module.


Import-ModuleFromWeb


Detailed Description:

Takes a base directory containing directories full of links to post.
Each directory containing links is made into a module using Get-Code
Extracts the link from a .url file and then passes the links to Get-Code
(a pluggable function, please try writing one of your own).
Get-Code extracts out code snippets from the web pages.
Each function is placed in its own file and a module is generated that dot sources
each function.

 


Here's Import-ModuleFromWeb:

function Import-ModuleFromWeb {
           
    <#
    .Synopsis
        Takes a directory of web links and outputs 
    .Description
        Takes a base directory containing directories full of links to post.
        Each directory containing links is made into a module using Get-Code
        Extracts the link from a .url file and then passes the links to Get-Code
        (a pluggable function, please try writing one of your own).
        Get-Code extracts out code snippets from the web pages.
        Each function is placed in its own file and a module is generated that dot sources
        each function.
    .Example        
        Get-ChildItem $env:UserProfile\Favorites\Code\PowerShell\Modules | New-ModuleFromWeb -force
    .Link
        Resolve-ShortcutFile
    .Link
        Get-Code
    .Parameter directory
        The name of the directory full of links to download and import
    .Parameter force
        If this is not set, will not overwrite an existing module directory
    .Parameter importToSession
        If this is set, Import-Module will be called on the newly created module
    #>
    param(
    [Parameter(
        ValueFromPipeline=$true,
        ValueFromPipelineByPropertyName=$true,
        Position = 0)]
    [Alias("FullName")]
    [string]
    $directory,
    
    [switch]$force,
    
    [switch]$importToSession
    )
begin {

        if (-not (Get-Command Resolve-ShortcutFile -errorAction SilentlyContinue)) { 
            throw $error[0]
        }
        if (-not (Get-Command Get-Code -errorAction SilentlyContinue)) { 
            throw $error[0]
        }
    
}
process {

        $item = Get-Item $directory
        if ($item.PSIsContainer) {
            $moduleName = $item.Name
            if (Test-Path $env:UserProfile\Documents\WindowsPowerShell\Modules\$moduleName) {
                if (-not $force) {
                    Write-Error "$moduleName already exists, use -force to overwrite"
                    return
                }
            }
            $modulePath = "$env:UserProfile\Documents\WindowsPowerShell\Modules\$moduleName"
            $folder = New-Item $modulePath -force -type Directory
            $moduleText = ""
            Get-ChildItem $item -Recurse -Filter *.url |
                Resolve-ShortcutFile | 
                Get-Code | Foreach-Object {
                    $functionFile = $_.FunctionName + '.ps1'
                    $_.ScriptBlock | Out-File (Join-Path $modulePath $functionFile) -Force
                    $moduleText += ". `$psScriptRoot\$functionFile                                                            
"
                }
            $moduleText | Out-File (Join-Path $modulePath "$moduleName.psm1") -Force
            if ($importToSession) {
                Import-Module $moduleName -Force
            }
        }
    
}

}
    

 

 

Automatically generated with Write-CommandBlogPost


Import-CodeSnippet


Detailed Description:

Takes a block of text containing a single PowerShell function definition (e.g. function foo($bar) { $bar } )
and extracts out the name of the function and returns a script block that can be used to define the function.
The function will not be registered by using Import-CodeSnippet.
To register functions with Import-CodeSnippet use:
Import-CodeSnippet | Foreach-Object { . $_.ScriptBlock }

 


Here's Import-CodeSnippet:

function Import-CodeSnippet {
           
    <#
    .Synopsis
        Parses text to extract out a function name, and returns the script block to create the function
    .Description
        Takes a block of text containing a single PowerShell function definition (e.g. function foo($bar) { $bar } )
        and extracts out the name of the function and returns a script block that can be used to define the function.
        The function will not be registered by using Import-CodeSnippet.
        To register functions with Import-CodeSnippet use:
            Import-CodeSnippet | Foreach-Object { . $_.ScriptBlock }
    .Parameter text
        A block of text containing a single PowerShell function definition        
    .Example
        $text = 'function foo() { }'
        $text | Import-CodeSnippet
    #>
    param(
        [Parameter(Mandatory=$true,
            Position=0,
            ValueFromPipeline=$true,
            ValueFromPipelineByPropertyName=$true)]
        [Alias("ScriptBlock")]
        [string]$text
    )
process {

        if (-not $text) { return } 
        # PowerShell can now create generic types.
        # We'll use this one to collect the parse errors.
        $err = New-Object System.Collections.ObjectModel.Collection`[System.Management.Automation.PSParseError]
        $tokens = [Management.Automation.PSParser]::Tokenize($text, [ref]$err)
        foreach ($e in $err) {
            Write-Error -message $e.Message `
              -category ParserError `
              -errorID $e.Message `
              -targetObject $e.Token
        }
        $c = 0;
        $tokens | Where-Object {
            $_.Type -eq "Keyword" -and $_.Content -eq "Function"
            $c++
        } | Foreach-Object {
            $tokens[$c].Content | Select-Object @{
                Name='FunctionName'
                Expression={$_}
            }, @{
                Name='ScriptBlock'
                Expression={[ScriptBlock]::Create($text)}
            }
        }    
    
}

}
    

 

 

Automatically generated with Write-CommandBlogPost


Get-Code


Detailed Description:

Get-Code is a pluggable function. It looks for all other functions named GetCodeFrom*
that have a parameter, Text, and calls each of these functions with a block of text in order
to extract code any number of possible sources. If a Get-Code plugin does not find code within the text
it should return nothing. If code is found, it should return the chunk of text containing the function
declaration. This text will then be passed to Import-CodeSnippet, which will return the name of the function
and a script block to declare the function.

 


Here's Get-Code:

function Get-Code {
           
    <#
    .Synopsis
        Uses plugins to Get-Code to extract code from a web page of block of text.
    .Description
        Get-Code is a pluggable function.  It looks for all other functions named GetCodeFrom*
        that have a parameter, Text, and calls each of these functions with a block of text in order
        to extract code any number of possible sources.  If a Get-Code plugin does not find code within the text
        it should return nothing. If code is found, it should return the chunk of text containing the function
        declaration.  This text will then be passed to Import-CodeSnippet, which will return the name of the function
        and a script block to declare the function.
    .Parameter url
        If url is specified, the URL will be downloaded and the text from the URL will be passed to Get-Code
    .Parameter text 
        The block of text       
    #>
    [CmdletBinding(DefaultParameterSetName="Url")]
    param(
    [Parameter(
        ParameterSetName="Url",
        Mandatory=$true,
        ValueFromPipelineByPropertyName=$true)]
    [string]
    $url,
    [Parameter(
        ParameterSetName="Text",
        Mandatory=$true,
        ValueFromPipelineByPropertyName=$true)]
    [Alias("ScriptBlock")]
    $text        
    )
begin {

        if (-not (Get-Command Get-CommandPlugin -errorAction SilentlyContinue)) { 
            throw $error[0]
        }   
        if (-not (Get-Command Import-CodeSnippet -errorAction SilentlyContinue)) { 
            throw $error[0]
        }    
    
}
process {

    if ($psCmdlet.ParameterSetName -eq "Url") {
        Write-Progress "Downloading Data" $url
        $downloadedText = (New-Object Net.Webclient).DownloadString($url)
        & $myInvocation.MyCommand -Text $downloadedText
        return
    }
    $myInvocation.MyCommand |
        Get-CommandPlugin -preposition "From" -parameters "text" |
        Foreach-Object {
            Write-Progress "Attempting to extract code" "Using $_"            
            $extractedText = & $_ @psBoundParameters
            if ($extractedText) {
                $extractedText | Import-CodeSnippet
            }
        }
    
}

}
    

 

 

Automatically generated with Write-CommandBlogPost


Get-CodeFromPoshCode


Detailed Description:

Takes text downloaded from a posting on the PowerShell Code Repository (http://www.poshcode.org)
and extracts out the code segment.

 


Here's Get-CodeFromPoshCode:

function Get-CodeFromPoshCode {
           
    <#
    .Synopsis
        Takes text containing a web page downloaded from Poshcode.org and extracts out the function contained on the page
    .Description
        Takes text downloaded from a posting on the PowerShell Code Repository (http://www.poshcode.org) 
        and extracts out the code segment.
    .Parameter text
        The text of the file
    .Example
        $text = (New-Object Net.Webclient).DownloadString("http://www.poshcode.org/735")
        Get-CodeFromPoshCode $text
    .Link
        Get-Code        
    #>
    param(
        [Parameter(Mandatory=$true, Position=0)]
        [string]
        $text
    )
process {

        if (-not (Get-Command Get-MarkupTag -errorAction SilentlyContinue)) { 
            throw $error[0]
        }        
        Get-MarkupTag textArea $text | Where-Object {
            $_.Xml.Id -eq "Code"
        } | Foreach-Object { 
            $_.Xml.'#text'
        }
    
}

}
    

 

 

Automatically generated with Write-CommandBlogPost


Get-CodeFromCommandBlogPost


Detailed Description:

Takes text downloaded from a blog post generated with Write-CommandBlogPost
(http://blogs.msdn.com/powershell/archive/2008/12/24/write-commandblogpost.aspx)
and extracts out the code segments.

 


Here's Get-CodeFromCommandBlogPost:

function Get-CodeFromCommandBlogPost {
           
    <#
    .Synopsis
        Takes text containing a web page downloaded from a blog post written with Write-CommandBlogPost and extracts out the function definitions
    .Description
        Takes text downloaded from a blog post generated with Write-CommandBlogPost
        (http://blogs.msdn.com/powershell/archive/2008/12/24/write-commandblogpost.aspx) 
        and extracts out the code segments.
    .Parameter text
        The text of the file
    .Example
        $text = (New-Object Net.Webclient).DownloadString("http://blogs.msdn.com/powershell/archive/2008/12/24/write-commandblogpost.aspx")
        Get-CodeFromPoshCode $text
    .Link
        Get-Code
    #>
    param(
        [Parameter(Mandatory=$true, Position=0)]
        [string]
        $text
    )
if (-not (Get-Command Get-MarkupTag -errorAction SilentlyContinue)) { 
        throw $error[0]
    }        
    Get-MarkupTag pre $text | Where-Object {
        $_.Xml.Class -eq "CmdletDefinition"
    } | Foreach-Object { 
        $_.Xml.'#text'
    }

}
    

 

 

Automatically generated with Write-CommandBlogPost

 

Hope this Helps, and Happy Holidays :

James Brundage [MSFT]

Attachment: CodeDownloader.zip
Leave a Comment
  • Please add 2 and 1 and type the answer here:
  • Post
  • You know PoshCode already has functions for v1 and v2 CTP3 right?

    You can *search* and download, and even upload, etc.  And you don't need to be webscraping to download anyway. If you have a URL like http://poshcode.org/750 ... just insert a "/get" and we'll serve you a PS1 script file: http://poshcode.org/get/750

  • On our internal discussion list, someone just asked about how to pass switch parameters from one function

  • @jaykul.  It's great that you can download, and this could be used to simplify Get-CodeFromPoshCode a little, but the idea is to make a flexible way to download PowerShell scripts from any source.  While an alternative URL will let me download the file, the point is to extract out code from many different sources and put them in a module.

    If I used that special convention just for poshcode, then I would end up having Get-Code download the web page once for each plugin, instead of once for all plugins.  Scraping poshcode is very simple and it's quicker to fail at scraping one source than to download N times (once per plugin).

    James

Page 1 of 1 (3 items)