Aim Mangkorntong's Web Log

Oslo Repository in May CTP

2009-06-26T12:27:00Z

May CTP was an exciting milestone for Oslo Repository. We successfully integrated Repository with the M tool chain. All of main shapes and instances (i.e. tables, views, functions, locale instances) in Oslo Repository are now defined in M Schema instead of T-SQL. The remaining T-SQL implementation of Oslo Repository was split into pre and post SQL scripts following the MX convention so that, we can compile everything into the Repository.mx image and install it onto the SQL Server using MX. The diagram below describes the two steps that you would need to perform now in order to create a repository.

Repository.mx consists of 3 modules that are mapped to the existing Repository schemas: Repository, System.Globalization and Repository.Item. You can create an M model referencing all items defined in these 3 modules. However, there are two issues below that are worth taking note of.

- No cross image label references: Currently , you cannot reference any labeled instances that are in a different image. For example, as a part of the Repository installation, default folder M instances were created (Repository, Applications, etc.). You will not be able to reference those folders with FoldersTable.Repository.

- No instance creation against views: Currently, you can only create M instances against extents and not computed values (i.e. views). For example, when you create a Folder, you would actually have to create an instance of "FoldersTable" rather than "Folders". This may cause your model to behave incorrectly if your also create a view with triggers on top of your model.

Repository.mx is a TSQL10 image and therefore compiling /t:Repository image referencing the Repository image is not supported (You would get some warnings if you try to do this but they can be ignored). To get the same behavior as /t:Repository, we recommend you to create an M model with a view and then make use of the stored procedure [Repository.Item].[AddStandardPatterns] to obtain all repository patterns. You can also inject each repository pattern separately through the following stored procedures: [Repository.Item].[CreateIdSequence], [Repository.Item].[AddFolderForeighKey], [Repository.Item].[AddSecurityViewsInsteadOfTriggers], [Repository.Item].[AddAuditing], [Repository.AddChangeTracking].

Managing Secured Resource Permissions in "Oslo" Repository

2009-04-12T05:16:00Z

What security model have we adopted in “Oslo” Repository ? How can we manage secured resource permissions in “Oslo” Repository?

Our main goal for the security model is the ability to provide a unified, flexible and yet secured approach for accessing repository contents from different type of database-enabled applications (e.g. SQL Reporting Services and Excel).

In "Oslo" Repository, we adopted the CardSpace security model (i.e. security claim-based). With claims based security, it is possible to emulate Access Control List (ACL) and Role-Based Access Control (RBAC) based security and therefore supporting the security models used in a wide range of products.

The claims-based security model, essentially, consists of the following concepts:

Concepts	Description	Examples
Claims	Claim type-value pairs	(Name = ‘Anthony’)
Claim types	Used for partitioning claims into related sets	Windows SID claim type
Issuers	Parties that issue a claim. This is identified by a certificate.	Contoso Corporation
Resources	Resource type-value pairs that are secured by the system	(Folder = ‘.Net Framework 2.0’)
Resource types	Used for partition resources into related sets.	A folder resource type
Operations	Secured operations that can be performed on secured resources.	A read operation
Permissions	The authorization model that allows principals with a valid issued claim to perform an operation on a particular resource.	A principal with the domain group ‘MyDomain\IT Admin’ may read the folder ‘.Net Framework 2.0’

The diagram below provides the overall pictures of the security model in "Oslo" Repository. In practice, resources, claims and operations are referred to by their GUIDs.

In terms of users and database roles, there are 2 distinguished users for security purposes:

· RepositoryOwner: A loginless user that owns all repository objects.

· RepositoryService: A loginless user that repository service runs as.

Below is a list of repository user roles defined in "Oslo" Repository.

· RepositoryAdministrator: A role that allows users to perform administrative operations such as granting resource access.

· RepositoryReaderWriter: A role that allows users to read and update the repository content.

· RepositoryReader: A role that allows users to read the repository content.

· RepositoryUser: A role that allows users basic access to the repository.

· RepositoryChangeTrackingReader: A role for users authorized to read SQL Server Change Tracking records (not CDC records).

· RepositoryChangeDataCaptureReader: a role for users authorized to read CDC records (not Change Tracking records).

There is a role used by repository infrastructure:

· RepositoryChangeDataCaptureService: An internal role that is used to access CDC records.

The security model comes with a number of helper functions for managing resource permissions in a repository. Below are some examples to give you some idea of how to play around with the security model in "Oslo" Repository. Note that the wrapper functions for the code below are already provided for Folder resources (see my previous blog for more details). For a complete set of examples, click here.

-- Returns the detailed summary of resource permissions of the current session

select CP.[ResourceKind],

CP.[ResourceKindDescription],

CP.[Resource],

CP.[Operation],

CP.[OperationDescription],

CP.[MayGrantOrRevoke]

from [Repository.Item].[CurrentPermissions] as CP

-- Returns a summary of resource permissions of the curresnt session

select SP.[Resource],

SP.[ResourceKind],

SP.[Operation],

SP.[MayGrantOrRevoke]

from [Repository.Item].[SessionsPermissions] as SP

-- Obtain the security claims associated to the current user session.

select TSSC.[Claim], DP.[Name] as [PrincipalName]

from [Repository.Item].[TheSessionsSecurityClaims] as TSSC

inner join [Repository.Item].[SecurityClaims] as SC on TSSC.[Claim] = SC.[Id]

inner join sys.database_principals as DP on SC.[Claim] = DP.[sid]

order by DP.[Name]

-- Give the new principal the read access to the folder '/Repository/New Folder'

declare @folderId int = [Repository.Item].[PathsFolder]('/Repository/New Folder');

declare @readOperation [Repository.Item].[SecuredOperationId];

declare @folderResource [Repository.Item].[SecuredResourceKindId];

select @readOperation = SO.[Id]

from [Repository.Item].[SecuredOperations] as SO

where SO.[Name] = 'http://schemas.microsoft.com/Repository/2007/10/Operations/Read';

select @folderResource = SRK.[Id]

from [Repository.Item].[SecuredResourceKinds] as SRK

where SRK.[Name] = 'http://schemas.microsoft.com/Repository/2007/10/Identity/Resources/Folder';

execute [Repository.Item].[GrantPrincipalPermission]

@principal = 'MyDomain\Bob',

@resourceKind=@folderResource,

@resource= @folderId,

@operation=@readOperation,

@mayGrantOrRevoke=0;

-- View a list of readable and updatable folders

declare @folderResource [Repository.Item].[SecuredResourceKindId];

select @folderResource = SRK.[Id]

from [Repository.Item].[SecuredResourceKinds] as SRK

where SRK.[Name] = 'http://schemas.microsoft.com/Repository/2007/10/Identity/Resources/Folder';

select * from [Repository.Item].[ReadableResources](@folderResource);

select * from [Repository.Item].[UpdatableResources](@folderResource);

Playing with “Folders”: The Magic behind “Oslo” Repository

2009-03-13T06:03:00Z

One of the highlight features in “Oslo” Repository is row labeling. Imagine that you can label instances in your database tables just like how you store files into different folders on your computer. That’s right!, including controlling security - read, write permissions, that we are talking about here. All the magic behind this functionality is captured under the concept of ‘Folder’.

In “Oslo” Repository, RepositoryUsers can see a list of folders that they have access to through the view [Repository.Item].[Folders] which is linked to the table [Repository.Item].[FoldersTable]. This table consists of four columns – [Id], [Name] ,[DisplayName] and [Folder]. The column [DisplayName] is designed to be used for application localization purpose. It is linked to [System.Globalization].[StringsTable] which consists of all localized strings in repository. The column [Folder] is basically there to enable the hierarchical folder structure.

Just like other databases, only users with db_owner and sysadmin roles can view all folders from the folder table. By default, RepositoryAdministrators have read and write permissions to all root folders (Note that they are not automatically assigned with a sysadmin role). Users with other roles can view all the folders that they have at least the read access to from the security view [Repository.Item].[Folders]. They also have the same permission to all sub folders of the root folders that they have access to.

“Oslo” Repository comes with a number of helper functions around Folders. Below are examples that demonstrate the usage of some helper functions and how Folders can be used to perform table horizontal partitioning. Note that these examples are provided based on the NET. Framework CLR instances. For more examples around Folders, see the SQL script (FolderExample.sql) attached.

Insert, update and delete a folder

insert into [Repository.Item].[Folders] ([Name], [Folder])

values (N'4.0', [Repository.Item].[PathsFolder](N'/Frameworks/.Net Framework'));

update [Repository.Item].[Folders]

set [Name] = N'4.0b'

where [Id] = [Repository.Item].[PathsFolder](N'/Frameworks/.Net Framework/4.0');

delete from [Repository.Item].[Folders]

where [Id] = [Repository.Item].[PathsFolder](N'/Frameworks/.Net Framework/4.0b');

Folder path

-- Given a folder id, return its full path.

select [Repository.Item].[FoldersPath](104); -- returns /Frameworks/.Net Framework/2.0

-- Given a folder path, return its folder id.

select [Repository.Item].[PathsFolder](N'/Frameworks/.Net Framework/2.0'); -- returns 104

Descendent folders

-- Given a folder id, list all its descendent folders

select [Repository.Item].[FoldersPath](S.[Id])

from [Repository.Item].[Subfolders](101) as S;

-- Given a folder id, list all its descendent folders including itself.

select [Repository.Item].[FoldersPath](S.[Id])

from [Repository.Item].[SubfoldersAndFolder](101) as S;

Folder contents

-- List all direct contents of a folder (/Frameworks/.Net Framework/2.0') in a view.

select *

from [Clr].[Assemblies] as A

where A.[Folder] = [Repository.Item].[PathsFolder](N'/Frameworks/.Net Framework/2.0');

-- List all contents of the folder with the id 101 and its subfolders in a set of views.

select A.*

from [Clr].[Assemblies] as A

inner join [Repository.Item].[SubfoldersAndFolder](101) as S on S.[Id] = A.[Folder];

Folder permissions

-- Grant a principal access to a folder.

execute [Repository.Item].[GrantPrincipalFolderAccess]

@principal = N'MyDomain\Principal',

@folderPath = N'/Frameworks/.Net Framework/2.0',

@mayRead = 1,

@mayUpdate = 0,

@mayGrantOrRevoke = 0;

-- Revoke a principal's access to a folder.

execute [Repository.Item].[RevokePrincipalFolderAccess]

@principal = N'MyDomain\Principal',

@folderPath = N'/Frameworks/.Net Framework/2.0',

@revokeReads = 1,

@revokeUpdates = 1;

To have a quick play with Folders, follow these links to download the Oslo SDK and then populate your repository with .NET Framework 2.0 & 3.0 CLR instances.

Storing CLR metadata in an "Oslo" Repository

2009-02-25T09:38:00Z

Populating over 30 assemblies, 16,600 types and 162,000 methods from .NET Framework 2.0 assemblies into your repository !!
[Prep time]: 5 minutes [Ingredients]: Oslo SDK, CLR instances (.NET 2.0), SQL Server Reporting Services

Another interesting application that we have developed on top of “Oslo” Repository is ‘BuildLoader’.

BuildLoader is the application which has been developed for the purpose of facilitating the process of analyzing the metadata within and across assemblies. It shreds the metadata in an assembly and then loads them into a repository.

Currently, in our team, we are using BuildLoader to shred some main assemblies that are produced from the nightly build process. For each build, we shred the test assemblies and product assemblies into different folders in our repository. This is so that we can perform the analysis across between the test and product assemblies easier.

BuildLoader consists of the 3 main components – CLR Schema, CLR Loader and CLR Reports.

CLR Schema: The Repository schema which contains a set of tables and views for storing the assembly metadata. This schema also contains a set of helper functions that are useful for text formatting (e.g. [Clr].[MethodToText] returns the full-qualified method name). Click here for more details on the Repository schemas.
CLR Loader: This component is responsible for shredding the assemblies and then loading the assembly metadata into a repository. The dependent assemblies are also determined during the shredding process and are also loaded into the repository for the assembly referencing purpose.
CLR Reports: A set of SQL Server Reporting Services (SSRS) reports which provide user-friendly interfaces for a user to browse through the assembly metadata in the repository. CLR Reports that we have developed consists of the following six reports.

FolderContentReport: This report basically displays a list of the assemblies that are in the selected folder.
AssemblyReport: The report which provides the high-level details of the assembly content. For example, the namespaces that are used within the selected assembly and the types in the assembly.
SearchByNamespaceOrType: This report allows a user to search for a type using a type or namespace or keyword.
CSharpTypeReport: This report provides the details of the selected type using the C# style.
MethodReport: This report displays the methods and the number of times that they call or receive a call from the selected method.
CalledMethodStatisticsReport: This report provides the bar graphs that display the number of calls made to and made by each method in the selected type.

BuildLoader has brought a number of benefits into our team. Having BuildLoader shredding and loading real data into a repository has helped us find and fix problems that customers will face. It also helped us discover holes in our documentation and guidance. Moreover, the test and product assembly metadata available allows us to perform the analysis such as the build content comparison. The called method statistics available from CLR Loader can be used to determine the areas in the product code that are still lack of testing. We use CLR Reports to get an overview of the build and the items within each assembly. For example, the CalledMethodStatisticsReport can provide us a rough estimation of the dependencies on the selected type in the selected assembly. This information is useful when there’s a need to re-design/ or refactor a component. The MethodReport and CSharpTypeReport are also particularly useful for debugging purpose. Using these two reports, a user can drill down to the method that is at the beginning and the end of a chain.

To get a similar experience to BuildLoader, install Oslo SDK and then install CLR instances (.NET 2.0) on top of “Oslo” Repository (comes with Oslo SDK). Then take a look at the CLR Reports under C:\Program Files\Microsoft Oslo SDK 1.0\Samples\Reports\. To give you some idea, this will populate over 30 assemblies, 16,600 types and 162,000 methods into your repository. This should be enough to keep you busy for a while :) Note that CLR schema is already comes with the default installation of an “Oslo” Repository.

Experiencing "Oslo" Through Event Pattern Mesh

2009-02-11T09:54:00Z

"Oslo" and Real-time Apps? Or maybe Complex Event Processing (CEP) in "Oslo"? Experiment through Event Pattern Mesh!

If you are unfamiliar with "Oslo", why not take a quick look here or learn more at ”Oslo” Developer Center. The first CTP version of “Oslo” was shipped at Microsoft Professional Developers Conference (PDC) 2008. The 5 “Oslo” video sessions from PDC are available from here (search for Oslo). I am actually very thrilled to hear that we made it into eWeek’s Top 10 Application Development Products for 2008, despite the fact that we have just released the first pre-beta CTP version last October.

Having recently completed a PhD in the event processing area, I could not wait to see how this concept could be implemented using the technologies within "Oslo". And that's basically the main motivation for Thomas Delrue and I to come up with our little application called 'Event Pattern Mesh’.

Event Pattern Mesh is an end-to-end application that has been designed for detecting patterns of occurrences from real-time data feeds. It also a good illustration of how the tools within "Oslo" can be used together in real-life scenarios. Event Pattern Mesh consists of

1. Model definitions. These include Event, Event Pattern and Event Pattern Mesh models defined using “MSchema”. Just to give you some idea, here is our Event model.

2. Event Pattern Modeling Language (EPML). This is a Domain Specific Language (DSL) that we have developed using MGrammar. The purpose of this language is to provide an intuitive way for a user create event patterns. For example, instead of creating each event pattern manually using “MSchema”, a user can simply use EPML to define an event pattern mesh as below.

3. Instance Generator: A tool which converts an event pattern mesh defined in EPML (described in 2.) into M instances.

Event Pattern Mesh makes use of the “M” Compiler to compile “M” instances into “M” images. From here, we can then use the MX tool to push the “M” images which contain event pattern mesh information into the “Oslo” Repository. Having event pattern details stored in the repository allows us to manage the information through a sleek design GUI in “Quadrant”.

One interesting design in Event Pattern Mesh is having the implementation of event and event pattern (binary files) as a part of the model as well as storing them in the repository. The nice thing about this is that now you can make use of the repository to label event pattern meshes and then assign each mesh, including its implementation, with different permissions. For example, if there are 2 traders trading stocks for 2 different clients, I am sure that they would not want to share event patterns that contain their trading strategies with each other. In this case, we store their event pattern mesh into different folders in the repository. Click here for a quick overview of the Event Pattern Mesh design.

So, after Event Pattern Mesh, what do I love about “Oslo”? Many things!

· MSchema: The concept of having a single data model works for me. I do not have to worry about having to define a model from the OO and ER perspectives. This means less development time as well as no overheads and errors from switching between these two concepts which is great!

· MGrammar: This concept goes well with MSchema because you can create several DSLs that are tied to a single model. For example, you can create different DSLs where each allows different types of user interaction to the model, depending on their technical skills, security levels or user groups.

· Repository: The row labelling mechanism in the Repository makes life much simpler when it comes to implementing and managing security. You can just assign different labels to different group of users. Another highlight of the Repository is security views which allow model modification (versioning) to be performed smoothly and without interruption.

· M Toolchain: This tool makes our life a lot simpler. It generates Repository-SQL from the MSchema models and also deploys them into/exports them from the Repository. The tool also provides the same options for a normal database.

On the downside, at this stage, some of the steps still have to be done manually. We used Intellipad to create our models (MSchema) and EPML (MGrammar). However, we had to deploy our models and generate M instances from EPML through the command line. Therefore, one thing would be nice to have is a tool that can take care all the steps from end to end? Perhaps Ipad will take care of these steps in the future release?

To learn more about Event Pattern Mesh, take a look at our video presentation or download the source code from here. If you are interested to find out more about the motivation behind this application, here are some references.

1. P. Mangkorntong and F. A. Rabhi, A Domain-Driven Approach for Detecting Event Patterns in E-Markets. World Wide Web: Internet and Web Information Systems. Springer Netherlands. ISSN 1386-145X (Print) 1573-1413 (Online), 13 December 2008.

2. P. Mangkorntong. A Domain-Driven Approach for Detecting Event Patterns in E-Markets: A Case Study in Financial Market Surveillance, PhD Thesis, School of Computer Science and Engineering, The University of New South Wales, April 2008.

3. P. Mangkorntong and F. A. Rabhi, A Domain-Driven Approach for Detecting Event Patterns in E-Markets: A Case Study in Financial Market Surveillance, In Proceedings of the 8th International Conference on Web Information Systems Engineering (WISE 2007), B. Benatallah et al. (Eds.), Lecture Notes in Computer Science, Springer-Verlag, Berlin Heidelberg. Vol. 4831 pp. 147-158, Nancy, France, 3-7 December 2007.

4. P. Mangkorntong and F. A. Rabhi, A High-Level Approach for Defining & Composing Event Patterns and Its Application to E-Markets. Proceedings of the Second International Workshop on Event-Driven Architecture, Processing and Systems (EDA-PS‘07) at the 33rd Very Large Data Bases (VLDB‘07), Vienna, Austria, 23 - 27 September 2007.