Interactive Media Manager Blog : metadata

How to use inferencing with the sw_Sparql stored procedure and how to get the InferenceIndex for a specific ontology in the database

johndeu — Mon, 08 Dec 2008 23:22:48 GMT

It’s often very useful when creating custom SPARQL queries against the IMM Metadata Store using the sw_Sparql stored procedure to pull in some “inferred” metadata from the ontologies that you have imported. For example, you may have ingested a bunch of metadata into the IMM data table that stores the “facts” but you want to query back for information that is generated dynamically based on you ontology definitions.

One simple example of this is you may have saved a VideoItem in the database with it’s rdf:type set to imm:VideoItem, but you never provided the metadata to explain that this item is also derived from imm:MediaItem. That information is provided in the IMM core ontology. To write a query for that directly against the sw_sparql stored procedure in the database, you need to know two well known graph URI’s.

The first one is the URI to the IMM data graph which stores all of the known “facts". Note that the “IMM” in this URI is case sensitive!

http://schemas.microsoft.com/IMM/data

The next URI you will need is the one for the inferred metadata generated from your imported ontology.

If I execute the following query in SQL Server Management Studio against the IMMMetadata database, it will select all the VideoItems from the “facts” table as you can see in the FROM clause. If you have the Litware sample data installed, you should get some results back. If you look at the Litware rdf data, you will notice that we have set the type of these objects to VideoItem already, so the “facts” table contains their types.

     exec sw_Sparql '

    PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

    PREFIX dii: <urn:mpeg:mpeg21:2002:01-DII-NS#>

    PREFIX did: <urn:mpeg:mpeg21:2002:02-DIDMODEL-NS#>

    PREFIX imm: <http://schemas.microsoft.com/imm/2.0/core/>

    SELECT ?s 

    FROM <http://schemas.microsoft.com/IMM/data>

    WHERE {  

        ?s rdf:type imm:VideoItem.

    }',

    1

If I execute this query, which is the same as above except now this time I am asking for a type that is in the IMM core Ontology, but not declared in the facts that I imported – it will return no results. Notice that the only change I made is the name of the type.

    exec sw_Sparql '

    PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

    PREFIX dii: <urn:mpeg:mpeg21:2002:01-DII-NS#>

    PREFIX did: <urn:mpeg:mpeg21:2002:02-DIDMODEL-NS#>

    PREFIX imm: <http://schemas.microsoft.com/imm/2.0/core/>

    SELECT ?s 

    FROM <http://schemas.microsoft.com/IMM/data>

    WHERE {  

        ?s rdf:type imm:MediaItem.

    }',

    1

So, how do I get back these “inferred” types in a query. First we need to know the graph name that was generated for us when we imported our Ontology. IMM automatically creates an inference graph when you import the core and any custom ontology. We automatically add those inference graphs if you pass in the URI to to your ontology in the GetResource or Query methods on the web service.

In order to get back this information from the database you can execute the following SQL script which will return back a list of the known ontologies that are imported into IMM, as well as an ID that is used to identify their inference graph name.

    exec sw_sparql '

    SELECT ?s ?o 

    FROM <http://schemas.microsoft.com/IMM/ontology>

    WHERE { graph ?g { ?s <http://schemas.microsoft.com/imm/2.0/core/InferenceIndex> ?o. }}

    order by ?o',

    1

Now, we can rewrite the script above to add a extra FROM clause that uses this inference index to point to the correct inference graph.
Look at the results of the above query and locate the Ontology that you are interested in using. Append the InferenceIndex that is returned in column 2 of the results to the following URI “http://schemas.microsoft.com/IMM/inference_” and you will get the URI for that inference graph.

For example the results from my database contains the index number “8” for the IMM Core ontology, so my resulting graph URI would be “http://schemas.microsoft.com/IMM/inference_8”.

My new query with ontology inferencing enabled would then look like this:

    exec sw_Sparql '

    PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

    PREFIX dii: <urn:mpeg:mpeg21:2002:01-DII-NS#>

    PREFIX did: <urn:mpeg:mpeg21:2002:02-DIDMODEL-NS#>

    PREFIX imm: <http://schemas.microsoft.com/imm/2.0/core/>

    SELECT ?s 

    FROM <http://schemas.microsoft.com/IMM/data>

    FROM <http://schemas.microsoft.com/IMM/inference_8>

    WHERE {  

        ?s rdf:type imm:MediaItem.

    }',

    1

If you execute this against the Litware sample metadata you will now get back the same (or more if you have other items that are derived from MediaItem) number of results as in the first query.

John Deutscher
Group Program Manager
Microsoft
johndeu@microsoft.com

Updated MetadataPolicy.xml Configuration File

johndeu — Wed, 05 Nov 2008 01:26:57 GMT

Today we posted an updated MetadataPolicy.xml configuration file to support more image metadata fields when using the IMM Media Services to extract metadata from a JPEG or TIFF image. Download the latest cersion of the file and replace the existing one in the folder:

%Program Files%/Microsoft Interactive Media Manager\Microsoft Interactive Media Manager\Media Services\Metadata

The updates include support for more EXIF, TIFF, and XMP metadata fields that are commonly found.

It’s very useful to spend a few moments to understand how the metadata policy file works, if you need to extend it to extract metadata fields that may not be standard. You can easily edit this file and just save it back and it will be used immediately on the next query to the Image endpoint’s GetMetadata method.

The MetadataPolicy.xml file is modeled after the default Vista metadata policy in how it chooses what metadata to parse from an image file. You will notice that it is grouped into Policy Groups. Each Group defines a prioritized list of metadata paths to attempt to extract metadata from a file. IMM MediaServices uses this policy file to determine which path to attempt first and once it gets a hit, it will stop processing that Policy Group and return the results in RDF format using the property name defined in the immPredicate attribute of the Policy element.

There are lots of tools out there to use to view the existing metadata on your image files and help determine which ones to add to the policy document. Here’s a couple of my favorite tools.

WIC EXPLORER

A great tool for helping out with image metadata and our MetadataPolicy file is the WIC Explorer which is in the Windows Imaging Component Code Samples and Tools. You can download it from here.

The WIC Explorer lets you view a tree of the metadata extracted in the files. You can use the “Metadata Values” decimal (in Green below) to extract the metadata by setting the appropriate path in MetadataPolicy.xml file.

EXIF TOOL

Another handy tool is ExifTool by Phil Harvey. It is great for getting a dump from a file of available metadata in XMP format. http://www.sno.phy.queensu.ca/~phil/exiftool/

Drop it in your Windows folder, rename it to just “exiftool.exe” and run it like this to get an .xmp file in RDF format. Great for debugging.

c:\Content>exiftool 068.JPG -o -s %d%f.xmp

Also, just run the console like this to get a dump of metadata and field names.

c:\Content>exiftool 068.JPG –s

Updating Ontology to Support New Properties

Once you have located and added the new properties to the MetadataPolicy.xml file you will need to update your Owl Ontology’s ImageEssenceFormat class to have the new property restrictions. In the current version of the IMM 2.0 Core Ontology, only the following properties are defined on the ImageEssenceFormat. If you want to use other properties in your code, you will need to add more property restrictions to this class and then regenerate your own custom ontology assemblies for use in .NET development.

Default Properties of ImageEssenceFormat on the IMM Core 2.0 Ontology.

imm:ResolutionX
imm:ResolutionY
imm:Codec
imm:BitDepth
MicrosoftPhoto:CameraSerialNumber
MicrosoftPhoto:DateAcquired
MicrosoftPhoto:FlashManufacturer
MicrosoftPhoto:FlashModel
MicrosoftPhoto:LastKeywordIPTC
MicrosoftPhoto:LastKeywordXMP
MicrosoftPhoto:LensManufacturer
MicrosoftPhoto:LensModel
MicrosoftPhoto:Rating
MicrosoftPhoto:RatingPercent
dc:Subject

To extend the ImageEssenceFormat Class, you simply create a new custom ontology (or if you already defined one use that one), and then you add restrictions to the ImageEssenceFormat class.

First open or create a new OWL document. Define a new Ontology element and import the IMM Core 2.0 ontology from the web.

<?xml version="1.0" encoding="UTF-8"?>
<rdf:RDF xmlns:owl=http://www.w3.org/2002/07/owl# 
 xmlns:rdf=http://www.w3.org/1999/02/22-rdf-syntax-ns#
 xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#">
    <rdf:Description rdf:about="foo:MyCustomOntology">
        <rdf:type>
            <rdf:Description rdf:about="http://www.w3.org/2002/07/owl#
Ontology"/>
        </rdf:type>
        <rdfs:comment>My Custom Ontology example for extending 
ImageEssenceFormat</rdfs:comment>
        <owl:imports>
            <rdf:Description rdf:about="http://schemas.microsoft.com/imm/2.0/core/"/>
        </owl:imports>
        <owl:versionInfo>1.0</owl:versionInfo>
    </rdf:Description>
</rdf:RDF>

First you need to define the new property that you want to add. In this case, I am adding the XMP Rating property as a new DatatypeProperty of type integer.

<rdf:Description rdf:about="http://ns.adobe.com/xap/1.0/Rating">

        <rdf:type rdf:resource="http://www.w3.org/2002/07/owl#
DatatypeProperty"/>

        <rdfs:comment>A number that indicates a document’s status relative to

other documents, used to organize documents in a file

browser. Values are user-defined within an application defined

range.</rdfs:comment>

        <rdfs:label>Rating</rdfs:label>

        <rdfs:range rdf:resource="http://www.w3.org/2001/XMLSchema#integer"/>

</rdf:Description>

Next you need to add the property as a restriction on the ImageEssenceFormat class.

<rdf:Description rdf:about="http://schemas.microsoft.com/imm/2.0/core/
ImageEssenceFormat">

    <rdfs:subClassOf>

            <rdf:Description>

                <owl:maxCardinality rdf:datatype="http://www.w3.org/
2001/XMLSchema#nonNegativeInteger">1</owl:maxCardinality>

                <owl:onProperty>

                    <rdf:Description rdf:about="http://ns.adobe.com/
xap/1.0/Rating"/>

                </owl:onProperty>

                <rdf:type>

                    <rdf:Description rdf:about="http://www.w3.org/2002/
07/owl#Restriction"/>

                </rdf:type>

            </rdf:Description>

        </rdfs:subClassOf>

</rdf:Description>

Now, if I regenerate my ontology class using the Visual Studio tools, I will have a new Rating property on my ImageEssenceFormat class. I can now deserialize the RDF/XML from the GetMetadata call to Media Services and it will populate this new Rating property if the image has an XMP Rating value stored on it.

Advanced SPARQL in IMM

derbaron — Tue, 28 Oct 2008 15:02:00 GMT

Today I would like to profile an advanced SPARQL query, break it down, and explain how each part works.

The Query:

with (
      select ?entityIdIn ?highestTypeOut
      {
            { ?entityIdIn rdf:type ?highestTypeOut.  }
            not() {
               ?entityIdIn rdf:type ?hType.
               ?hType rdfs:subClassOf ?highestTypeOut.
            }
      } as :getMostSpecificType
)

SELECT ?title ?rdfType
WHERE
{     
      ?entityId ?predicate ?title.    
      { :getMostSpecificType(?entityId, ?rdfType) }
      FILTER(:contains(?title, 'Superman'))
}

Overview

In essence, this query is searching every entity in the store and returns those entities where a property contains the word “Superman.” It also finds and returns the most specific type of each result. The purpose of this query is simple: to allow a user to search for a single term and find any resultant entities, regardless of their type or their schema.

In the results below, a number of different entities are returned: videos, clips, containers, and even resources.

Inference Rules

The “with” clause provides the ability to specify custom inference rules within a SPARQL query. In this case, we have a named rule that returns the most derivative (specific) type of the supplied entity. Since SPARQL variables either act as constraints or as variables depending on whether they are bound or unbound, I have suffixed “in” and “out” to make it clear how each is expected to be used. Calling the rule is simple. It is important to note that the results of this rule are not optional. Therefore, all returned entities must have at least one rdf:type.

{ :getMostSpecificType(?entityId, ?rdfType) }

Negation

The idea of negation is simple. The solution is the result of a graph pattern which is negatively constrained against the results of another graph pattern.

{ ?entityIdIn rdf:type ?highestTypeOut. }
not() {
?entityIdIn rdf:type ?hType.
?hType rdfs:subClassOf ?highestTypeOut.
}

Expressed in English: “Any triple where the subject is ?entityIdIn and the predicate is rdf:type EXCEPT where the same triple exists in the following graph pattern: any triple where the subject is ?entityIdIn and the predicate is rdf:type AND the resultant type is a subclass of another resultant type.” Whew. Put more simply: “Bind ?highestTypeOut to the rdf:type which has no derivative types.” This is an example of using negation to constrain the results of a graph pattern.

Full Text Search

The :contains function provides support for SQL full-text queries in a SPARQL filter. This is the most performant method of querying for specific text. In addition to simple word matching seen in the query above, advanced features of full-text can be used. For example, we can search for different inflections of the word “ship” by using the following:

FILTER(:contains(?title, 'FORMSOF(INFLECTIONAL, \"ship\")'))

This will return results with values such as “ships,” “shipped,” and “shipping.”

Other Thoughts

As mentioned in my previous post, the “rdf” and “rdfs” prefixes used in this query are handled automatically by IMM. Additionally, this query uses the ontology graph to determine class derivation. An enterprising developer could utilize more facts from the ontology to enhance the results of the query. For example, he/she could return the friendly name (rdfs:label) for a type instead of the type’s URI.

IMM's Metadata Model: a 2min. 18sec. vidcast

mkashman — Wed, 20 Aug 2008 22:28:00 GMT

IMM is designed to enable users who work with rich media to ingest, manage, edit, and distribute content. While IMM is based on Office SharePoint Server 2007 and offers collaborative, enterprise content management, and search capabilities, IMM is designed specifically to work with media files, and has been modified to integrate with the wider media industry. Key to this is the addition of the IMM Semantic Metadata Store. IMM uses Semantic Web technologies to express, interchange, store and search media asset metadata. This is because RDF and OWL enable developers and users to add business specific and technical information to media assets. In the below vidcast, John Deutscher - Group Program Manager, spends a few minutes drilling into the IMM metadata model and reasons behind the decision to use the RDF standard: