Bill Gibson's WebLog

Whitehorse Architecture TechNotes Published on MSDN

Phew!  A set of Visual Studio Team System Tech Notes have now been published on MSDN, including some fifteen new articles I've written on topics related to the Whitehorse architecture tools.  These should all be easy to read and are inter-linked, to make them an easy to browse.  Hopefully they will provide some useful background and some new insights into the Distributed Systen Designers. 

Below are links to each of my notes, ordered in what is probably a good sequence in which to read them. 

TN_1104: Understanding the System Definition Model

TN_1114: The Four Layers of Systems in the System Definition Model

TN_1100: Understanding Applications and the Application Diagram

TN_1105: Why Class Libraries are not shown on an Application Diagram

TN_1109: Understanding Systems and the System Designer

TN_1101: Understanding and Using the Default System

TN_1110: Using Systems to Represent Services in a Service Oriented Architecture

TN_1112: System Portfolio Management

TN_1102: Designing Substitutable Web Services

TN_1113: Representing Connections to Manual, Physical and Other Kinds of System

TN_1111: Top-down System Design

TN_1108: Connecting Applications to Web Services via Class Libraries

TN_1106: Copying Endpoints vs Creating Endpoints from WSDL

TN_1103: Web Service Endpoint Name Propagation

TN_1107: Understanding, Using and Creating Toolbox Prototypes

Why do ASP.NET WebService and ASP.NET WebApplication look alike on the Application Designer? 

On shadow applications and agile development

Randy Miller has published a paper on agile development that discusses the use of 'shadow' applications in an agile modeling and development context, in which he talks in somewhat veiled terms about the use of the Whitehorse design tools.  The term shadow application emerged from discussions Randy and I had at Tech Ed earlier this year as we developed the top-down design approach described in my earlier article.  In the article I show how you can use a shadow application as a placeholder for the behavior of a system and thus design systems top-down. In his paper, Randy take this idea further and suggests the shadow application is actually written or generated, consistent with the code-oriented agile development philosophy.

Logical vs. Physical Architectural Modeling Concerns

We've been having hallway discussions about some of the dimensions of modeling application architectures.  SDM adopts a particularly physical perspective, as it focuses on the deployment packaging stack of concerns.  Its focus is on resources (things like assemblies, config, XML files etc.) that are composed into independently deployable applications, that are composed into systems, which are then composed into still higher level systems.  SDM, does allow you focus on tangible software entities and express their behavior - very CBD-like.  But of course there are other perspectives, and particularly troublesome to some is the absence in the current designers of a logical perspective, in which you can define what an application does, independently of describing how it does it or how that implementation is packaged.  The discussion turned to the role of namespace in that logical perspective, specifically to the question, are namespaces the primary logical organizing perspective. I wrote some comments in an email on the subject and have elaborated on them here and invite comment.

While namespaces are clearly not a packaging concept as they can cut across assemblies, they are not necessarily the best or only way to organize the elements in a logical architecture.  Instead the notion of a cross-cutting and entirely logical organization of function is proposed.  The term "module" was suggested some months ago by a colleagure for this notion.  Thus a module is an arbitrary collection of function.  A module could be a collection of methods, one class, many classes; may be coincident with an assembly, cross assemblies or contain assemblies.  Modules can also describe data. Early on in your design, you might think about modules but go no further than giving a name and describing the role.  A classic block structure architecture diagram might be used to show the way modules interact, either depicting a specific instance of an architecture or a pattern before any details exist of the behavior of each module. 

You might assign namespaces to modules – although the naming of types/classes, while important, is not intriniscally linked to the logical structuring of software.  Namespaces might reflect the logical structure and you might be tempted to conflate the two, such that you use namespaces when really you arethinking about modules.  At some point in your design process as you refine your module designs and start to define classes, creating namespaces will become important.  And at some point in your design packaging considerations will become important, when you will specify applications, frameworks, and systems.  With your packaging decisions you harden the partitioning of the functions in your modules and make communication and binding decisions.  

Thus there are module, namespace and packaging dimensions, These represent orthogonal but related concerns which inform each other.  Different design strategies will have you tackle these aspects: physical structure (systems, applications, frameworks), logical structure (modules) and implementation (classes with namespaces) in different amounts at different times – being able to reason about all three as and when you choose will be the most flexible and should be what we aim for in tools.  Associating a namespace with a module or even a package may make good sense when the purpose of the module/package and the goals of the namespace align but the two are not intrinsically linked. 

ARCast on model-driven development

An interesting initial round on this model-driven development podcast, including comments from Jack Greenfield from our team.  Will be fun to watch this one develop...  As someone who was involved in what we believed to be one the big successes of the 80's and 90's so called CASE era, I always find it a little irritating to hear about all the failures, so it was interesting to hear some positive comments about the lessons to learn from that period.  From my experience, what we were doing at Texas Instruments was successful for a long time, in large part because the problem domain we tackled was limited and the patterns well recognized-- we were targeting large scale integrated database transaction processing systems. In effect, while we never described it in these terms, what we developed was a domain specific language - or at least a family of languages.  And as one of the contributors mentioned, the approach relied on the assumption that you never modified a line of generated code - there were lots of user exits but you never touched the code.  We even implemented a feature that deleted the source code after it had been compiled to save space :-)  The approach we used then would be much harder to make successful now, when frameworks are so much richer and the problem domains and application model so much more varied. 

Kick the VSTS tires...

Now you can try out Team System in a hosted environment without the hassle of downloading and installing.  Check out Rob Caron's post for details.

Going dark...

I've not lost enthusiasm for this blogging thing, its just that I've been asked to convert some of my blog entries into MSDN TechNotes and write others afresh to accompany the Visual Studio launch, so I've been busy with notes/articles in work on the following topics among others:

Understanding the SDM Type System
The Four Layer Model of Systems in SDM
Understanding Applications in Application Designer
When and how to use the Default System
The Role of System Designer
Using Systems to Define Services
Designing Substitutable Services
Understanding Creating and Using Toolbox Prototypes
Top-Down Design (published here already)
Why no Class Libraries on the Application Designer (blogged here already)

I will post the finished articles here and/or links to them on MSDN. 

Top-down System Design Article posted

I have posted an article on top-down system design using the System Designer.  This describes and illustrates a little-known technique for defining systems and applications top-down. 

Developer? Architect? All hail the Devarchitect!

No doubt you know that Microsoft is very persona driven.  A persona is a fictitous person that embodies the characteristics of a class of user - useful to us when working through product usage scenarios.  Mort is probably the most famous of Microsoft's development personas.  Whitehorse has its personas too - you'll hear us chatting about Andrew, the Enterprise Architect and Vernon the Application Architect as if we had a drink with them in a bar the night before. 

However, I think we're missing a persona which is a variation of the application architect - I don't have a name for him/her yet - but I think this persona is a Devarchitect.  Devarchitects think of themselves as architects (and may have that job title) - they are intimately involved in defining the function, structure, and design of systems, including performance, scalability, reliability, interoperability, etc - but they are also deeply involved in developing aspects of those systems, perhaps writing and testing early prototypes or core elements, often in the early days of a project so that the technical architecture and its execution characteristics can be proven, as well as digging in to deliver some of the more complex components as the project ramps up.  A devarchitect rolls up his sleeves and gets his hands very dirty.  Many high-end developers are devarchitects or aspire to be.  Devarchitects often don't want to be anything else - they don't want to get out of touch with development and the details of the technology.  And in some cases their organizations can't afford the luxury of full-time architects. 

At TechEd during a focus group I recently met several people who fitted this description well.  The profile of one of them confirmed this - he identified his primary job responsiblity as architect, and profiled his work with tasks associated with an architect role occupying 20% of his time and those associated with a (lead) developer occupying 70%.

Does anyone want to suggest a name for our devarchitect?

Understanding SDM: Systems and the Four Layer Model

The Whitehorse Distributed System Designers are based on SDM - the System Definition Model.  SDM offers a simple model for representing computer systems that can help in many parts of the design, deployment and management space.  I'll try over an occasional series of posts to introduce some of the key SDM concepts and how they play together.  I'll also show how we can extend these to do some neat things in the application architecture modeling space.

At the heart of SDM is the notion of a 'system'.  Let's look at a definition of system that I find useful.

A system is a configurable collection of resources that can be independently deployed .  If the resources in a system need to access resources of another system they do so by communicating via endpoints on the systems. 

A system can be atomic or composite; a composite system is composed of other systems. I'll explore system composition, resources and endpoints further in other posts, this post will focus on the notion of system hosting.

One system is considered to host another system if it provides infrastructural services to the hosted system.  Services can be thought of as infrastructural if the services are provided to many systems of the same type.  Introducing the hosting relationship allows the model to be layered, so that different concerns can be addressed in different layers.  For example, the requirement of an ASP.NET application on the services provided by IIS is an infrastructural hosting relationship.  Similarly IIS is hosted on Windows which in turn is hosted on an appropriate physical system.  It wouldn't be useful to have to include models of IIS and Windows and all the other systems that exist in those layers every time you wanted to model an application so we model those separately and then host a model at one level on the model of another. This reflects the reality that one tends to think of configuring standard system 'platforms' on which other systems are installed.  Most organizations use a small number of standard configurations of hardware systems on which they install standard configurations of OS and networking systems on which they run standard configurations of application hosting systems on which they run standard configurations of their business applications.

We typically think about there being four layers of systems: application systems, application hosting systems (such as IIS, SQL Server), operating and network systems (such as Windows), and hardware systems (such as a Dell or HP server, Cisco router etc.).  In reality there are no hard boundaries and there may well be more than four layers of interest, but it is often useful to think about these four general categories of systems, and it certainly helps people 'get' the overall model.

Different roles/people at different stages are interested in defining, deploying and managing each of the layers.  Information about one layer can be provided to those involved in defining another layer to assist in validating that systems at one level will successfully deploy on a specific configuration of systems at the next level.  The Distributed System Designers, for example, support the definition of models of the application layer and the application hosting layer and enable deployment validation - where the configuration of an application is verified against a configuration of application hosting software with the goal of increasing the likelihood of successfully deploying the application.   

Does Application Designer support operations defined on interfaces?

Another question that keeps cropping up is whether we support viewing and editing Web service operations when these are defined on interfaces.  It usually comes up in the form of the question, I've built a Web service and defined the operations using attributes on methods on an interface, but the operations don't show up in the Web Service Details window when I click on the endpoint.  What's wrong? 

Application designer doesn't support display of the operations where the WebMethod attributes are defined on methods on an interface.  The Web method attributes must be placed on the concrete class referenced by the asmx file for them to appear as operations in the Web Service Details window.  The endpoint does appear on the diagram and you can navigate from it to the class in code, but we don't walk up the inheritance chain to look for attributes on the interface, so we don't see the operations.  Asmx platform support for interfaces was only introduced into the VS 2005 release late in cycle, too late for us to respond with effective tool support.  We would have needed to support both viewing and editing operations and handle several complex use cases around mixed definitions.  It simply wasn't possible to address this in the time available and ensure the quality level we wanted so we had to cut the feature.

We recognize the value of using interfaces which are a cornerstone of the Windows Communication Foundation (Indigo) service model.  So, expect to see tool support for WCF fully incorporate interface support as part of a contract-driven service development approach.  We'll be able to talk more about this as our plans crystallize.

And on this topic, one observation is that if using a contract-driven approach in which a WSDL contract is viewed as the master, then it is not so critical that a CLR interface exists.  Using the Application Designer's support for creating web service endpoints from WSDL you can construct the web service implementation class from the WSDL contract directly.  While using interfaces undoubtedly helps some aspects of the engineering process it is not as critical when a WSDL doc is considered the master definition.  And note also that you can use interfaces in your design it is just that the Web service-related attributes must be defined on the concrete class for Appplication Designer to recognize them, and any edits in the designer are applied to the methods on the class and not the interface. 

Application Designer: copying endpoints vs. creating them from WSDL

There are two ways you can copy the specification of a Web service endpoint using Application Designer, which vary in subtle ways that you can leverage. You can simply use copy/paste - see my earlier post, or, if the source application has been implemented, you can use Create Web Service Endpoint from WSDL (right click on the target application and then reference the URL of the source application's endpoint).  In both cases you get what we think of as a shallow copy - that is, no implementation code is copied.  The differences are in how each approach handles complex types referenced by the operations.  In the case of copy/paste, the operation signatures only are copied and no new types are created in the target application.  If you create the endpoint from the WSDL, then the WSDL and any referenced XSDs are processed and new CLR types are generated for any complex types referenced in the message definitions.  If the target application is unimplemented, you cannot see or reference these new types until code is generated, but rest assured they are there! 

Copy/paste of an endpoint takes the view that the implementer of the target application will provide the definition of the parameter and return types out of band.  You can define the types from scratch, copy the type definitions from the source or more interestingly, reference the actual type definitions used in the source application.  This is likely not a good idea if the types are defined in the source Web service project itself, but may be very useful if you define the types in a separate project/assembly, which is arguably best practice.

The other approach, creating the endpoint from WSDL works rather differently.  This works the same way as WSDL.exe /server and will create a 'fresh set' of types in the target application based on the types in the WSDL and imported XSDs.  This approach echoes the SOA tenet of sharing schema and not types - although obviously that is directed primarily at avoiding implementation dependencies between consumers and providers of a service - not between providers. 

It can be argued that we could have gone further with copy/paste and created references in the target application to the projects/assemblies that contain the referenced types.  There proved to be several technical obstacles to doing this, not least that the referencing mechanisms in Visual Studio were only designed for creating references from projects and were not accessible to us - in many cases the target application will not have been implemented and there is no project.  Given that the workaround is to simply add the reference yourself to the project once implemented we thought this was a reasonable course.  And this approach recognizes that you may not want to create the tie between the service implementations anyway.

In any case, with the two techniques available you have a choice.  And choice is a good thing :-)

One other point to note is that you can create endpoints from the WSDL of External Web Service endpoints, which don't otherwise support copy/paste.

Why the name, Whitehorse?

A little bit of trivia.  Whitehorse is the capital of the Yukon Territory in Canada.  That made sense as a name back in the days when the group's mission was largely SQL Server data tools.  Times changed and our mission changed but the name stuck. 

Whitehorse is a great place to live... 

We later adopted the Uffington white horse as our 'logo' - this is very cool, it's amazing to see modern art that dates back to 1000 BC!

Yes, you can use copy and paste on the Application Designer

This came up in a discussion with some MVPs the other day - they didn't realize you can use copy and paste on the design surface.  It's actually pretty darn useful and worth a few comments.  If you copy and paste an application then we copy the entire application, endpoints and all.  If your copied application has consumer endpoints that are connected, then we assume you want the same configuration of those endpoints on the copy, so the pasted application will also be connected to the same application(s) on the diagram.  It's easy to reconnect them to something else later if that isn't what you wanted.  If the application has Web service provider endpoints defined with operations then the operation definitions are copied also.  This is a shallow copy, so you'll get the operation signatures including the names of the argument types.  This applies whether the application is implemented or not.  You are never copying the implementation (method bodies).

Copying an application with Web service endpoints is a great way to make substitutable services - the contracts will be the same, so subject to your implementations, the services can be used by the same clients.  This can be useful for testing - copying an application is great way to quickly create a test stub application for a service or a test client application.  And you needn't be concerned that these test applications will show up in your deployment definition, because you will create explicit system definitions that will exclude these test applications, (I'll talk more about the System Designer and using explicit system definitions in other posts). 

You can also copy endpoints alone - the endpoints must all be on the same application and you can only paste them onto an application that supports that kind of endpoint.  Again, the interesting case is copying Web service endpoints - useful for creating substitutable services or just getting a jump start on defining a new Web service.  In another post I'll compare using copy and paste for creating endpoints with the Create Web Service Endpoint from WSDL feature.  

You can also copy/paste operations inside the Web service details window - again this is a shallow copy which doesn't copy the underlying types.

And in all cases above you can copy and paste between IDE instances so you can copy from one application diagram to another.

Finally, copy/paste preserves all the properties and settings/constraints (where it makes sense).  For example, if you have defined the WSDL service and binding names and namespaces on your endpoints then these are preserved - that's way better than re-entering the attributes in code.  Although do remember to change these if the service you create isn't intended to be substitutable. 

 

Where are all the class libraries?

I’m often asked why class libraries aren’t supported on an application diagram.  At first glance the diagram seems like it should support visualizing all aspects of application structure.  In Whitehorse we use the term application loosely to talk about a collection of systems that serve some purpose and then more precisely in a slightly different context to refer to a discretely configurable and deployable entity – typically deployed into a single app domain on a single server.  These are the applications that show up on the application diagram.  While an application in this latter sense is atomic from a deployment perspective, it is still an abstraction – normally an application comprises a root entity (often a root project) and a set of resources deployed with it, which are considered to be part of the application and not part of the hosting environment or the infrastructure.  These resources may include class libraries, config files and other resource files.

A key part of the Whitehorse model is the notion of composing systems from one or more applications to describe what is actually required for a deployment. When a class library is referenced by an application the configuration of the class library is determined by the application that uses it – thus the value of a web reference URL or a database connection string required by a class library is provided by the root app.config or web.config, not the app.config defined in the class library.  Think about this situation - a class library that makes a database connection is referenced by two applications but in each case is configured differently to connect to two different databases.  If we added the class library to the diagram how would it be connected?  How would you indicate that the connection was determined by the context in which it was called?

The solution is to not think of class libraries and other resources as peers of an application but as part of its implementation; instead of seeing them connected alongside the application on the AD or SD one should drill-into the application to see them in context.  So in the same manner that a system can delegate its behavior to its members (other systems or applications), an application delegates its behavior to the projects, class libraries, dlls, etc within it. We’re thinking therefore of enabling a drill-in view which would expose the structure of an application and the delegation of its endpoints.  You'd be able to define where in the structure a specific web or database reference exposed on the application as a consumer endpoint is implemented, or where the services provided by the application are implemented.  

It’s likely that this drill-down view would be in a separate designer/diagram – think double-clicking on an app on the AD or SD to open the drill-down view, where you might see the root project and a nested hierarchy of class libraries/dlls contained within an outer application shape.  Each endpoint on the outer application shape would be delegated to either the root project or one of its referenced modules.   While this drill-down view could be shown expanded in situ on the AD/SD we’d more likely use a separate diagram to visualize each application – the use of a tree structure for the drill -in view wouldn’t blend well with the graph-style routing used on the AD/SD.  This designer would also address the problem of copying the config entries from a class library into the config file of the root project.

So while I know there are tricks people have used to make class libraries show up on the AD I hope you can see why we don’t think this is a good idea in the long run.  Doing this will not be consistent with anything we do in future.

Out of the gate...

Hi - I'm Bill Gibson, a senior program manager on the Whitehorse team.  I want to use this blog to share some insights and observations from the world of Whitehorse.  In case you missed it, Whitehorse is the internal project name for the Distributed System Designers in Visual Studio 2005, available in the Visual Studio Team Edition for Software Architects package.  Now there's a mouthful !  For those on the team we'll always be Whitehorse...

As one of the Whitehorse architects, I have been focused particularly on application and system design.  There's much we've achieved in the first release of these tools and still much to do.  There are also lots of misconceptions about what the tools are for and how they should be used and can be extended.  We obviously have not done a great job of explaining what an 'application' is or explaining why class libraries don't show up on the design surface, and I've yet to meet anyone who really gets the System Designer!   I'll try and talk to these kinds of issues as well as exploring ideas we have for the future - at least those I can share publicly.

And just in case you're interested in my background... I like others on the team have a long history in model-based development tools.  For anyone familiar with the Texas Instruments integrated CASE products -- known variously as IEF, Composer and more recently, COOL:Gen -- I was on the design team from 1985 to 1997 (!) working on both the modeling tools and the methodology that underpinned it.  Much of the thinking from those days, particularly the work we did on component based development, still permeates our thinking today.