Sandcastle : Architecture

Sandcastle September 2007 Release: VersionBuilder

aram — Wed, 03 Oct 2007 03:23:00 GMT

Sandcastle September 2007 release includes the VersionBuilder tool under \ProductionTools folder to provide support for documenting multiple versions of a project. The VersionBuilder tool generates a single reflection data file from multiple version-specific reflection files. The output reflection file can then be used in the Sandcastle build to generate reference documentation in which each API's topic contains information about the versions that support the API. For example, if you have three versions of the same DLL, with some APIs present in all three versions and other APIs present in only one or two versions, the topic for each API can include a Version Information section that lists the versions that include the API.

For Orcas RTM, the Version Information section of the documentation is as follows:

A member that’s been in all versions of .NET will say:

Supported in: 3.5, 3.0 SP1, 3.0, 2.0 SP1, 2.0, 1.1, 1.0

A member introduced in 2.0 SP1 will say:

Supported in: 3.5, 3.0 SP1, 2.0 SP1

A member introduced in 3.0 SP1 will say:

Supported in: 3.5, 3.0 SP1

To fully understand the version information, it’s important to understand how .NET 3.5 is built. Unlike versions 1.0, 1.1, and 2.0., .NET 3.5 (and 3.0) use dlls from the previous versions of .NET and adds its own as well. For example, there is no 3.5 version of mscorlib.dll. Instead, .NET 3.5 uses the same version that .NET 2.0 uses. But installing 3.5 on a computer updates the 2.0 version of mscorlib.dll. In fact, you can get the updated dll without installing 3.5; you can get the new dll by installing .NET 2.0 SP1. That means that any APIs added to mscorlib.dll after .NET 2.0 RTM are part of 2.0 SP1.

We think of Orcas as using .NET 3.5, but in reality, Orcas allows you to target .NET 2.0, 3.0, or 3.5. You’re not targeting the RTM versions of 2.0 or 3.0, though, because Orcas installs .NET 3.5, which includes the updated dlls for 2.0 and 3.0. As a result, when you target 2.0 or 3.0, intellisense shows the APIs that have been added since either version RTM’ed. But the recommendation to our customers is to only use the new APIs if you target .NET 3.5 because it’s difficult to tell whether a user has the RTM version of .NET 2.0 or .NET 2.0 SP1. Please see http://www.danielmoth.com/Blog/labels/Orcas.html and the video at http://channel9.msdn.com/Showpost.aspx?postid=333940 to understand the complexities of the Framework versions with the Orcas release.

Sequence of processing in VersionBuilder:

· Run MRefBuilder on each version's DLL set. Note that the DLL set for each version of a project includes the complete set of DLLs, not just the DLLs that are new or changed.

· Run the MergeDuplicates.xsl transform on each of the version-specific reflection files.

· Run VersionBuilder with the version-specific reflection files as input and the combined reflection file as output

· Proceed with the rest of the Sandcastle build using the combined reflection file

VersionBuilder command line:

The VersionBuilder command line has two switches.

/config:versionCatalog

versionCatalog is the path of a configuration file that specifies the reflection files for each version

/out:outputFile

outputFile is the path of the file where VersionBuilder writes the combined reflection file. If not specified, VersionBuilder writes output to the console. VersionBuilder throws an exception if the outputFile specifies a directory that does not exist.

For example:

VersionBuilder.exe /config:%projectPath%\config\versionBuilder.config /out:%projectPath%\reflection_allversions.xml

VersionBuilder configuration file:

The VersionBuilder configuration file specifies the input reflection files that MrefBuilder generated for each version. It also specifies a name for the project and names for each version. The names correspond to the ids of shared content items for the project and version names that appear in the Version Information section of an API's documentation.

Here is the content of a typical VersionBuilder configuration file:

<versions>

<versions name="ourproject">

<version name="ourproject30" file="%versionReflectionFolder%\reflection_ourproject30.xml" />

<version name="ourproject20" file="%versionReflectionFolder%\reflection_ourproject20.xml" />

<version name="ourproject10" file="%versionReflectionFolder%\reflection_ourproject10.xml" />

</versions>

Additional shared content items for version names

In a reflection file generated by VersionBuilder, each <api> node has a <versions> child node that contains information about the versions that include the API. When the <versions> node is present, the build transforms insert a Version Information section in each API's topic to list the versions that support the API. For example:

Version Information

Our Project

Supported in: 3.0, 2.0

The project name ("Our Project") and the version number strings ("3.0" and "2.0") come from a shared content items that you must define. For example, you can create a file named ourSharedContent.xml that has the following contents:

<content xml:space="preserve">

<item id="ourproject">Our Project</item>

<item id="ourproject30">3.0</item>

<item id="ourproject20">2.0</item>

<item id="ourproject10">1.0</item>

</content>

Then you need to update the BuildAssembler configuration file to include a reference to the ourSharedContent.xml file.

<component type="Microsoft.Ddue.Tools.SharedContentComponent" assembly="%DXROOT%\ProductionTools\BuildComponents.dll">

<content file="%DXROOT%\Presentation\vs2005\content\shared_content.xml" />

<content file="%DXROOT%\Presentation\vs2005\content\reference_content.xml" />

<content file="%DXROOT%\Presentation\shared\content\syntax_content.xml" />

<content file="%DXROOT%\Presentation\vs2005\content\feedback_content.xml" />

<content file="c:\ourSharedContent.xml" />

</component>

Hope this helps. Cheers.

Anand..

September 2007 Sandcastle release: Generating Framework Reflection data for resolving links

aram — Tue, 02 Oct 2007 09:25:00 GMT

Sandcastle September 2007 release was shipped without the Framework reflection data. This allowed us to reduce the Sandcastle installer size to 1.5 MB. In our previous releases we shipped the reflection files under the \Data\Reflection folder. For the September release we expect the users to generate this information.

Why do we need this reflection data?

The reflection files are used to resolve any reference links to the Framework. Please see http://www.dnrtv.com/default.aspx?showNum=84 for additional details.

Why did we leave the reflection files from our Install?

We could have generated the reflection files and shipped it with our installer. In our previous releases we shipped the reflection files for Framework 2.0 under Data\Reflection folder. However the reflection files we shipped were specific to a document model.

VS2005 and Hana presentation styles follow a document model (ApplyVSDocModel.xsl) and Prototype style follows another document model (ApplyPrototypeDocModel.xsl). We shipped the reflection files with VS2005 document model and users using Prototype style ended up using it. With Orcas RTM the users targeting 3.5 will need access to various versions of Framework reflection files. Please see http://www.danielmoth.com/Blog/labels/Orcas.html and the video at http://channel9.msdn.com/Showpost.aspx?postid=333940 to understand the complexities of the Framework versions with the Orcas release.

With Framework versions 2.0, 2.0 SP1, 3.0 and 3.5 things are getting complicated and it became very clear to us that shipping all the reflection files for all the document models will increase the Sandcastle installer size by 20 plus MB.

How do the users generate reflection data?

If you are using VS2005 or Hana presentation styles do the following:

1. Run MrefBuilder on all the Framework dlls

2. Apply the ApplyVSDocModel transforms.

If you are using Prototype presentation style do the following:

1. Run MrefBuilder on all the Framework dlls

2. Apply the ApplyPrototypeDocModel transforms.

Alternatively you can generate them by executing one of the the following under /Example/Sandcastle directory:

msbuild fxReflection.proj /Property:NetfxVer=2.0 /Property:PresentationStyle=VS2005

msbuild fxReflection.proj /Property:NetfxVer=2.0 /Property:PresentationStyle=hana

msbuild fxReflection.proj /Property:NetfxVer=2.0 /Property:PresentationStyle=prototype

Note: If you are have 3.0 or 3.5 installed please change the NetFxVer accordingly.

Generating Reflection data as a part of test project:

We have shipped a sample command line batch script and msBuild project with the /Example/Sandcastle directory. Users can run the following to build the test project CHM or HxS and also to generate the reflection data information.

1. Using the command line option for the batch file under Examples/Sandcastle folder to build VS2005 or Hana or Prototype style CHM or HxS file

1. Build_sandcatle.bat vs2005 test

2. Build_sandcatle.bat prototype test

3. Build_sandcatle.bat hana test

Note: The reflection files should be generated for the first time only. Please comment the following section in the Build_Sandcastle.bat files after generating them for the first time.

msbuild fxReflection.proj /Property:NetfxVer=2.0 /Property:PresentationStyle=%1

2. Using msbuild to generate build for VS2005 or Hana or Prototype, CHM or HxS file

1. Msbuild build.proj /property:PresentationStyle=vs2005

2. Msbuild build.proj /property:PresentationStyle=hana

3. Msbuild build.proj /property:PresentationStyle=prototype

Note: The reflection files should be generated for the first time only. Please comment the following section in the build.proj files after generating them for the first time.

<Exec Command="$(Framework)\msbuild fxReflection.proj /property:PresentationStyle=$(PresentationStyle)" />

Hope this helps. Cheers.

Anand..

Componentization - Building Assembly level HxS using Sandcastle

aram — Wed, 25 Jul 2007 09:21:00 GMT

The primary goal of componentization is to reduce the size of a run-time image. In addition to helping meet space requirements, componentization has the following benefits:

Performance - Reducing disk and memory use can significantly improve performance of both the platform and your software.
Reliability - As a general rule, reliability increases with fewer executable files running and interacting with each other, devices, and the user.
Customization - Componentization allows for granular customization that is not possible using traditional setup programs.

Componentization using Sandcastle:

Sandcastle now allows us to build API documentation on a per-assembly basis. That is, we can build an HxS that contains documentation only for the types in a given DLL and then combine the HxS into a cohesive document set (e.g., create a unified TOC). We will be shipping "Orcas Beta 2" FxRef documentation as componentized HxS to improve our F1 performance within the VS IDE. This feature is available with Sandcastle June CTP.

How does this help with performance, reliability and customization?

When we shipped Windows SDK in November 2006 (http://blogs.msdn.com/windowssdk/archive/2006/11/08/windows-sdk-for-windows-vista-is-live.aspx) the Framework documentation was a 400 MB monolithic Hxs. This created performance and reliability issues for customer downloads and also impacted the F1 calls in VS IDE. Also customization was an issue when we wanted to ship documentation for certain assembly with VS Express SKUs.

Here are some of the issues we investigated before designing this feature in Sandcastle:

1. Size of HxS - HXS size could theoretically be a problem with an assembly-level HXS, but it won’t practically become a problem any faster than it would under our current practice (used in VS 2005 documentation) of namespace-level HXS. Our largest HXS under the current practice is for the System.Windows.Forms namespace. Those APIs pretty much represent the contents of the System.Windows.Forms.dll assembly, so under the new practice the largest HXS would remain about the same. Sandcastle provides infrastructure which will make it easy to build assembly-level HXS.

2. FX Issues - Building documentation for the APIs in an assembly does require information from other assemblies, not just for inheritance hierarchies, but also for parameter and return types, interface implementations, etc. In order to deal with this requirement, Sandcastle expects to have reflection data files for such other assemblies available while it is building documentation for one assembly.

3. Distributed build ownership – If you are a big corporation (like Microsoft) with different build teams then individual teams can maintain the ownership of building their assembly.

Here is an overview of assembly level build process using Sandcastle. We use MRefBuilder on each assembly to generate reflection data file. SegregateByAssembly generates reflection information for each assembly and BuildAssembler creats documentation (HxS) from the reflection data files. We use AggregateByNamespace utility, which creates a namespace reflection data files from a collection of assembly reflection data files. See a conceptual sketch below that describes this process:

Attached is a componentized builds sample for Sandcastle users. The steps below provide information on using this feature. Please install Sandcastle June CTP before trying the following:

1. Copy the attached file to “c:\program files\sandcastle\examples”;

2. Unzip it to folder “C:\Program Files\Sandcastle\Examples\componentized”;

3. Run “msbuild comp_build.proj”.

4. When done, assembly level Hxs files can be found under output\hxs and its subfolders.

5. You can copy your dlls to “C:\Program Files\Sandcastle\Examples\componentized\dll” and comments files to “C:\Program Files\Sandcastle\Examples\componentized\comments”.

6. The properties in comp_build.proj can be modified per you needs.

<PropertyGroup>

<Name>Sample</Name>

<PresentationStyle>vs2005</PresentationStyle>

<Framework>$(WINDIR)\Microsoft.NET\Framework\v2.0.50727</Framework>

<BuildDir>$(DxRoot)\examples\componentized</BuildDir>

<OutputDir>$(BuildDir)\Output</OutputDir>

<ProductionTools>$(DxRoot)\ProductionTools</ProductionTools>

<ProductionTransforms>$(DxRoot)\ProductionTransforms</ProductionTransforms>

<Presentation>$(DxRoot)\Presentation\$(PresentationStyle)</Presentation>

<Dlls>$(BuildDir)\dll\*.dll</Dlls>

<Comments> $(BuildDir)\comments\*.xml</Comments>

</PropertyGroup>

Let me know your feedback and I hope you like this feature. Cheers.

Anand..

Introducing a new document model design code named "Hana"

aram — Sun, 17 Jun 2007 19:35:00 GMT

With Sandcastle June CTP we are releasing a new documentation design code named “Hana”. We expect the release to be available on June 18, 2007. This presentation layer will be released as a part of Sandcastle and will not be used for Orcas documentation shipping in MSDN anytime soon as we have to make changes to the online infrastructure. We will continue to use VS 2005 presentation for our Orcas release with additional design improvements. I will blog about this soon.

This new documentation presentation layer targets developer audience who need to find information quickly and easily in our documentation that grows significantly with every Visual Studio/.NET Framework release. As new .NET Framework features are released, we must update the documentation design to accommodate the associated new concepts. As a result, we need improvements to the way that we navigate, filter, and display the documentation. These new features allow developers to locate, filter, and sort information in ways that are necessary to perform their job. When this new documentation presentation layer is implemented, user must be able to take advantage of the following features:

Persistent language filter to let a user choose a preferred default programming language that automatically shows the associated syntax blocks, code examples, and language keywords that are associated with that language
Filter at the member level based on a type’s supported frameworks
Filter at the member level based on a type’s supported framework versions
Filter at the member level based on a type’s supported platforms
Filter on member types
Filter on member types’ accessibility modifiers
Restrict the documentation disk footprint
See more information in less screen space
non-English users to easily compare English with non-English documentation pages
Syntax Blocks will occupy less space on the screen

Improved filtering to accommodate new technology concepts

New technologies such as Extension Methods and XAML introduce entirely new categories of members and syntax blocks that the filtering mechanism must accommodate. No one view meets all user needs, thus we allow users to manipulate the view to create the display that meets their requirements. For example, a user wants to show only the XAML syntax blocks when they view class pages.

Multiple dimenstions of information

As the number of frameworks increase, we must display multiple dimensions of information, such as the matrix of various framework versions, platforms, types, and members. This requires a sophisticated filter that accommodates all the dimensions, yet does not overwhelm the user with choices. For example, a user wants to only show types that are supported on the Compact Framework.

Filtering Persistence

When a user sets a particular filter, since there are multiple levels of filtering possible, they should not have to reset the filter every time they navigate to a new page. Therefore, the filters must be persistent, meaning that the filter maintains the user’s choices as they navigate the documentation. Specific filters such as member filters do not persist to support searching members using Ctrl-F.

The programming language filter is focused on a user’s role, meaning whether the user is a C# programmer, or a VB programmer. Therefore, filter persistence applies only to the programming language filter. Other filters, such as member accessibility and the type filter on namespace pages should reset to the “All Types” view as a user navigates across the pages. These do not map to a user’s role specifically and instead relate to a sub-category of information they are looking for within their role.

Screen real estate and vertical scrolling

Even at high resolutions, screen real estate is limited given the growing amount of information we must present in the documentation. Previously we introduced collapsible section headers, which allowed a user to hide a section of content that they were not interested in, which reduced the overall vertical topic length and thus reduced the need to scroll. However, as the content grows over time and we add to the number of headers, most reference topics no longer fit on one page. Reducing the amount of time users spend scrolling and making best use of the available screen real estate is very important. For users that spend a lot of time in the documentation, saving a few seconds adds up over time and allows them to be more productive overall. Additionally, comparing elements on a page is easier when there is less scrolling involved.

Link between localized versions of topics

Users have provided consistent feedback saying that users in non-English locales like to compare localized content with the English version. Currently, switching between versions of topics is extremely difficult and requires a user to maintain a mapping of English URLs to localized URLs. Users need quick and reliable access to multiple language versions of a given documentation page. This is a feature for online side of MSDN and will not impact Sandcastle users.

Page Count

Applying a new presentation layer allows the opportunity to reduce the number of pages of documentation that a user must navigate. For example, when a class overview page also contains the member list (not separate pages), this reduces the amount of TOC navigation needed.

Documentation Disk Footprint

Sharing reference pages across multiple frameworks and versions allows a smaller documentation disk footprint.

I will blog more details with screen shots soon..

Anand..

The "Prototype" document model design overview

aram — Sun, 17 Jun 2007 19:16:00 GMT

When we released Sandcastle last July we released it with the “Prototype” document model design. The ‘Prototype” document model was designed with the following ideas:

1. Don’t segregate members by visibility

2. Don’t segregate – Filter

3. Don’t Separate a Type from its Members

4. Better (Chicken Feet) Navigation

5. Tabbed Language Filtering

6. True Language Filtering

7. No Intermediate Overload Pages

8. No Overload Pages At All

The syntax blocks, code examples, and member tables should be filterable by language. Users should pick a single procedural code language in one filter that determines their primary/default view for all of these. Code-behind examples should appear as appropriate for the user's procedural language choice and be clearly labeled and located to indicate their code-behind relationship. The filtering paradigm should translate consistently across online or offline deliverables and the user's filtering choices should persist across pages and sessions.

Because markup languages such as XAML or ASP.NET are not relevant to many reference topics, the language filter should omit them as choices for the user's primary/default view. In the tabbed filtering proposal, a tab for every available language will show as an option. If the user does not select a primary language, the primary/default language that appears should be based on the target audience and product priorities.

Member tables should provide views of members by all members (default view), member category (property, method, etc.), inheritance (inherited, declared), access level (public, protected), and other relevant modifiers (static, abstract, etc.). Member tables should include separate entries for each overload rather than a single entry that points to an intermediate overload grouping page. Member tables should appear on the type pages rather than on separate pages; the Members section should appear after the Syntax section (and its constituent sections). Type tables should appear on the namespace pages and offer views by all types (default view) and by category (class, structure, etc.). These tables should include descriptions for each item and use icons consistent with Visual Studio. Consideration should be given to consolidating filter mechanisms so that users don't have to choose one filter choice from the chrome (inheritance) and other choices directly in the member table (access, modifiers). We believe that using filters for this is more efficient presentation than the current segregation approach used for many of these views.

I will blog more details about the “Prototype” document model soon.

BuildAssembler Configuration

aram — Mon, 16 Apr 2007 08:40:00 GMT

In my previous blogs I have discussed the architecture of BuildAssembler, a configurable build component stack of Sandcastle. As each topic is build, it is represented as an in-memory XML document. The document is passed through a stack of build components, each of which can act on the document in any way. The component stack is specified in Sandcastle.config file, and the set of topics to be generated is specified in a manifest. A simple component stack that might be used to generate API reference documentation is shown below:

Load reflection data
Load authored comments
Execute an XSL transform
Save the document

We shipped Orcas Beta 1 Library (scheduled to be live at MSDN on APril 18th) using Sandcastle. We currently maintain four configuration files for each presentation layer:

· Conceptual.config: for conceptual projects. We do not ship this config with Sandcastle CTP.

· Sandcastle.config: for external managed reference projects. We ship this config with Sandcastle CTP.

· Reference.config: for internal managed reference projects that inherit APIs from other projects. We do not ship this config with Sandcastle CTP.

· Reference-core.config: for self-contained internal managed reference projects (e.g. fxref). We do not ship this config with Sandcastle CTP.

This table below describes the steps specified by each configuration file and the external data sources on which each step draws.

The steps used in each scenario are (reference and reference-core are the same in this table):

Step	Description	Conceptual	Sandcastle	Reference
Copy skeleton	Create skeleton XML document.	X	X	X
Topic API data	Copy in API data for the topic API.		X	X
Container API data	Copy in API data for the topic API’s container (e.g. namespace data for type APIs).		X	X
Event handler API data	Copy in API data for the handler API associated with an event.		X	X
Parameter API data	Copy in API data for the topic API’s parameter types (e.g. argument types of a method).			X
Generate syntax	Call syntax generators to create syntax blocks.		X	X
Comments	Copy in comments for the topic API.	X	X	X
Snippet resolution	Resolve code snippets.	X		X
Members	Copy in API data and comments for members (e.g. type data for namespace APIs).		X	X
Token resolution	Resolve tokens.	X		X
Metadata	Copy in project & topic metadata	X		X
Intellisense	Write out intellisense for the topic API.			X
Transform	Apply XSLT to produce HTML.	X	X	X
Art link resolution	Resolve art links.	X		X
Shared content resolution	Resolve shared content.	X	X	X
Conceptual link resolution	Resolve conceptual links.	X		X
Reference link resolution	Resolve reference links.	X	X	X
Save	Save the resulting file.	X	X	X

The associated data sources are (reference and reference-core are not the same in this table):

Source	File(s)	Conceptual	Sandcastle	Reference	Reference-core
Local comments (/// format)	Comments.xml		X
External comments (/// format)	%SystemRoot%\Microsoft.NET\Framework\\.xml		X
Local comments (DdueXml format)	DdueXml	X		X	X
External comments (DdueXml format)	%DXROOT%\Data\DdueXml\*.xml			X
Local topic metadata (XmlComp format)	XmlComp\*.xml	X
External topic metadata (XmlComp format)	%DXROOT%\Data\XmlComp\*.xml	X		X	X
Local API data	Reflection.xml		X	X	X
External API data	%DXROOT%\Data\Reflection\*.xml		X	X
Snippets	%DXROOT%\Data\CodeSnippet.xml	X		X	X
Tokens	%DXROOT%\Data\Tokens.xml	X		X	X
Art	%DXROOT%\Data\Art	X		X	X
Shard content	%DXROOT%\Presentation\\Content\.xml	X	X	X	X
Conceptual project metadata	ExtractedFiles\*.ContentMetadata.xml	X
Reference project metadata	SupportFiles\*Metadata.xml			X	X

Hope this helps users to understand the Sandcastle.config file shipped with Sandcastl CTPs. Please inform me of any discrepancies you notice.

Anand..

Video of Sandcastle session from MVP Global summit

aram — Fri, 23 Mar 2007 02:02:00 GMT

Microsoft Most Valuable Professionals (MVPs) are exceptional technical community leaders from around the world who are awarded for voluntarily sharing their high quality, real world expertise in offline and online technical communities. Microsoft MVPs are a highly select group of experts that represents the technical community's best and brightest, and they share a deep commitment to community and a willingness to help others. Please see the MVP site at http://mvp.support.microsoft.com/

Last week I had the pleasure of meeting with several MVPs as a part of the Microsoft Global MVP summit. My team held sessions around MSDN Wiki and Sandcastle. The session for Sandcastle was very interesting and most of the Windows Help MVPs attended the session.

Rob Chandler video taped the session on Sandcastle and posted it at http://picasaweb.google.com.au/robert.w.chandler/SandcastleVideo?authkey=sUo23QwaU-c. Please see the attached Powerpoint slides used at this session. Thanks Rob for attending and doing this.

Robert is author of must-have shareware HTML Help and MS Help 2 authoring tool FAR HTML - http://www.helpware.net/FAR/

Check out Robs site of all things HTML & Help - http://www.helpware.net/

Rob also maintains MSHelpWiki with the other help MVPs - http://MSHelpWiki.com

Sandcastle Performance improvements in February CTP

aram — Wed, 28 Feb 2007 09:41:00 GMT

David Smith made some great observations regarding Sandcastle perf. issues on large assemblies at http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1125897&SiteID=1. We have been testing our code on Phoenix (http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1125897&SiteID=1) assemblies and here is what we found:

	Prototype Style	VS2005 Style
Topic number	22289	32414
One big reflection file	6mins	16mins
One reflection file per assembly	5mins	13mins
One reflection file per assembly and not using MSDNWebService	4mins	8mins

Remarks:

1. Using ApplyVSDocModel for vs2005 transforms and AddOverloads for Prototype transform;

2. Memory peak value: one big file (600MB), one file per assembly (200MB).

3. The project tested was Phoenix (http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1125897&SiteID=1) assemblies.

As far as the difference between ApplyVSDocModel and AddOverloads, ApplyVSDocModel is doing what AddOverloads does plus a lot more, so it's to be expected that it will take significantly longer. We looked at the ApplyVSDocModel code and didn't see any obvious places to make perf improvements.

However when building phoenix documents, we found a performance issue during the CHM build process.

To create CHM contents file, we used the following command.

XslTransform /xsl:..\..\ProductionTransforms\ReflectionToChmContents.xsl reflection.xml /arg:html=Output\html /out:Output\test.hhc

In ReflectionToChmContents.xsl, xsl:document() is used to get topic title.

<xsl:value-of select="document(concat($html,'/', file/@name, '.htm'),.)/html/head/title"/>

<xsl:value-of select="document(concat($html,'/', key('index',$overloadId)/file/@name, '.htm'),.)/html/head/title"/>

When applying this xsl, the xsltransform consumed all system memory. For building Phoenix documentation, the process took an hour to create this hhc file because the cache mechanism of xsl:document(), all html files are loaded into memory and not released.

In February CTP we have addressed this issue by creating a user defined function to replace xsl:document().

<xsl:stylesheet version="1.0"

xmlns:xsl="http://www.w3.org/1999/XSL/Transform"

xmlns:msxsl="urn:schemas-microsoft-com:xslt"

xmlns:ddue="http://ddue.schemas.microsoft.com/authoring/2003/5">

<msxsl:script language="C#" implements-prefix="ddue">
  <msxsl:using namespace="System.Xml" />
  <msxsl:using namespace="System.Xml.XPath" />
  <![CDATA[
   public static string getTitle(string fileName) {
    XPathDocument doc = new XPathDocument(fileName);
    XPathNavigator node = doc.CreateNavigator().SelectSingleNode("/html/head/title");
    if (node != null)
     return node.Value;
    else
     return String.Empty;
   }
  ]]>
</msxsl:script>

This function is used at:

<xsl:value-of select="ddue:getTitle(concat($html,'/', @file, '.htm'))"/>

The perf improvement is lot better with February CTP. For phoenix project, it takes about 4 minutes to generate hhc file compared to 1 hour before.

A (Microsoft) code name a day: Sandcastle

aram — Wed, 06 Dec 2006 09:02:00 GMT

I was thrilled to see Mary Jo Foley, from ZDNet, blog about Sandcastle at her “Microsoft Codenames” page. I met Mary at TechEd 2006 in Boston when my team was presenting a chalk talk on MSDN Wiki. I remember asking her after our talk to blog on MSDN Wiki.

Mary had the following about Sandcastle in her blog on code name:

Meaning/context of the code name: No idea. (Anyone out there know why Microsoft chose this particular code name?)

Sure Mary and I will be happy to explain. When Microsoft shipped VS 2005 (code named “Whidbey”) we internally used a "beach theme" code base called “Coronado” to build and ship our conceptual and managed reference content. As a part of this beach theme, the managed reference reflection and build engine portion of the Coronado code base was called Sandcastle. Internally Sandcastle was used by developers across teams to document their /// comments. You can read more about the evolution of Sandcastle here - http://blogs.msdn.com/sandcastle/archive/2006/08/06/689946.aspx. Please see below a schematic of the internal documentation process using Coronado.

I will provide more details about our internal documentation process in a separate blog.

Anand..

Supporting Custom tags in Sandcastle

aram — Wed, 22 Nov 2006 04:28:00 GMT

Sandcastle supports C# (and VB) xml tags for developer documentation comments. The list of c# tags are available at http://msdn2.microsoft.com/en-us/library/5ast78ax.aspx. We added the following new custom nDoc tags in November CTP (http://www.microsoft.com/downloads/details.aspx?FamilyId=E82EA71D-DA89-42EE-A715-696E3A4873B2&displaylang=en):

1. <overloads>

2. <preliminary>

3. <threadsafety>

4. <note>

These custom tags are supported in nDoc and you can see the tags under tag reference section at http://ndoc.sourceforge.net/usersguide.html.

This blog provides details on how to define your custom tags. To support custom tags the user can modify the presentation transforms (XSLT) that are shipped with Sandcastle. The following steps provides details:

1) Add the custom tag to the code in C# file [test.cs]:

/// <customTag>This is a custom tag.</customTag>

public class StoredNumber

{

}

2) Compile the C# file and extract the /// comments to, say comments.xml.

csc /t:library /doc:comments.xml test.cs

The custom tag will be included in the XML.

3) Update main_sandcastle.xsl with the following:

<xsl:template name="body">

<xsl:apply-templates select="/document/comments/customTag" />

</xsl:template>

<xsl:template match="customTag">

<xsl:apply-templates />

</div>

</xsl:template>

The resulting page for the topic would look like the following with custom tag at the top of the page:

Syntax Generation

aram — Tue, 15 Aug 2006 08:22:00 GMT

Syntax in Sandcastle documentation is generated by SyntaxBuilder. SyntaxBuilder uses the Common Compiler Infrastructure (CCI) library to reflect over a set of managed assemblies and produce syntax blocks that represent each API in various languages. SyntaxBuilder was developed during VS2005 ship cycle.

In principle, the information required to generate syntax blocks is contained in the reflection data produced by MRefBuilder, so a separate reflection step is not strictly necessary. However, two strong arguments have convinced us to nevertheless keep this step. First, the SytnaxBuilder code base exists and does its job well; using XML data as input instead of CCI objects would require a near-complete re-write. Second, the logic involved in generating syntax blocks can be quite complex and require referencing obscure pieces of reflection data; using CCI objects guarantees that all the information in an assembly, up to an including individual IL instructions, is available to generate syntax.

SyntaxBuilder is extensible in two ways: users can define SyntaxWriters, which define an output format, or SyntaxGenerators, which define a syntax block. We will provide three SyntaxWriters:

UscSyntaxWriter (generates a shared content file for use by pDoc in the previous version of Sandcastle)
HtmlSyntaxWriter (generates HTML for direct viewing)
XmlSyntaxWriter (generates an XML file for use by the new toolset)

Sandcastle will provide seven SyntaxGenerators:

C# Declaration Syntax
Visual Basic Declaration Syntax
Managed C++ Declaration Syntax
J# Declaration Syntax
Jscript Declaration Syntax
Visual Basic Usage Syntax
XAML Usage Syntax

The abstract SyntaxWriter class is defined as follows:

public abstract class SyntaxWriter : IDisposable {

protected SyntaxWriter (TextWriter writer) {}

public abstract void WriteStartDocument ();

public abstract void WriteStartApi (string id);

public abstract void WriteStartBlock (string name);

public abstract void WriteString (string text);

public abstract void WriteStringWithStyle (string text, string style);

public virtual void WriteIdentifier (string identifier) {}

public virtual void WriteKeyword (string keyword) {}

public virtual void WriteParameter (string parameter) {}

public virtual void WriteMessage (string message) {}

public abstract void WriteReference (string reference);

public abstract void WriteReference (string reference, string display);

public abstract void WriteEndBlock ();

public abstract void WriteEndApi ();

public abstract void WriteEndDocument ();

public virtual void Dispose ();

}

The abstract SyntaxGenerator class is defined as follows:

public abstract class SyntaxGenerator {

public abstract void WriteSyntax (Member api, SyntaxWriter writer);

protected ApiFilter ApiFilter { get; }

protected AttributeFilter AttributeFilter { get; }

}

Sandcastle.config file has the following structure for syntax generation:

<component type="Microsoft.Ddue.Tools.IfThenComponent" assembly="..\..\ProductionTools\BuildComponents\BuildComponents.dll">

<if condition="not(starts-with($key,'Overload:') or starts-with($key,'R:'))" />

<then>

<component type="Microsoft.Ddue.Tools.SyntaxComponent" assembly="..\..\ProductionTools\BuildComponents\BuildComponents.dll">

<syntax input="/document/reference" output="/document/syntax" />

<generators>

<generator type="Microsoft.Ddue.Tools.CSharpDeclarationSyntaxGenerator" assembly="..\..\ProductionTools\BuildComponents\SyntaxGenerators.dll" />

<generator type="Microsoft.Ddue.Tools.VisualBasicDeclarationSyntaxGenerator" assembly="..\..\ProductionTools\BuildComponents\SyntaxGenerators.dll" />

<generator type="Microsoft.Ddue.Tools.CPlusPlusDeclarationSyntaxGenerator" assembly="..\..\ProductionTools\BuildComponents\SyntaxGenerators.dll" />

</generators>

</component>

</then>

</component>

Sandcastle users can provide custom syntax generators at the <generator /> section.

BuildAssembler (EXE)

aram — Mon, 14 Aug 2006 09:10:00 GMT

I my previous blogs I discussed MrefBuilder and Sandcastle.config in detail. Sandcastle process uses three executables:

MRefBuilder: Reflects over managed assemblies to produce an XML-formatted reflection information file.
BuildAssembler: Collates information from comment files and reflection information files into a set of HTML pages.
XslTransform: Applies XSL transforms to XML files. This executable is used to generate and manipulate auxiliary data files.

Sandcastle architecture uses a configurable build component stack. As each topic is build, it is represented as an in-memory XML document. The document is passed through a stack of build components, each of which can act on the document in any way. The component stack is specified in Sandcastle.config file, and the set of topics to be generated is specified in a manifest. A simple component stack that might be used to generate API reference documentation is shown below:

Load reflection data
Load authored comments
Execute an XSL transform
Save the document

BuildAssembler (EXE)

BuildAssembler controls the component stack at the center of Sandcastle. The command-line inputs for BuildAssembler are simple:

BuildAssembler manifest /config:config

The manifest is a list of topics to build, and the Sandcastle.config file defines the BuildAssembler component stack.

The manifest has a simple XML structure that defines a list of topics.

<topics>

<topic id=”string” />

</topics>

A manifest may contain any number of topic elements; each topic element must have an id attribute. Although BuildAssembler does not require it, we expect that GUIDs will be used for conceptual topic IDs and CERs will be used for reference topic IDs.

Sandcastle.config file is structured as follows:

<configuration>

<dduetools>

<builder>

<components>

<component type=”type” assembly=”path”>

</component>

</components>

</builder>

</dduetools>

</configuration>

Any number of component elements is allowed. Each component element must specify an assembly from which to load the component and a fully qualified type name corresponding to a BuildComponent class. A component element may have an arbitrary descendent structure that specifies configuration information for the component. Note that this allows different instances of the same component to be used for different purposes, even within the same component stack, by providing different configuration data.

BuildAssembler instantiates each component by pointing its constructor to its section of the configuration file. Once the component stack has been instantiated, BuildAssembler invokes the component stack for each topic specified in the manifest.

Note that what occurs during a component stack invocation is entirely up to the individual components in the stack. Depending on the components and their configuration, nothing may occur, or the XML or HTML document is saved corresponding to a topic.

As it reads the configuration file and manifest file, BuildAssembler may report any of the following error conditions:

Condition	Description
IO	The configuration (or manifest) file was not found, or is not accessible.
XmlException	The configuration (or manifest) file is not well-formed.
XmlSchema	The configuration (or manifest) file does not fulfill the required schema.

We refer to these error conditions as the “file reading errors”; other tools that read XML files may also produce them.

During the component instantiation phase, BuildAssembler may report any of the following error conditions:

Condition	Description
IO	The specified component assembly was not found, or is not accessible.
BadImageFormat	The specified component assembly is not a managed DLL.
TypeLoad	The specified component assembly does not contain the specified component type.
MissingMethod	The specified component type does not define the appropriate constructor.
TargetInvocation	An exception occurred while the specified component was being instantiated.
InvalidCast	The specified component is not a BuildComponent.

When it reports these errors, BuildAssembler also reports which component generated the error. We refer to these errors as the “dynamic loading errors”; other tools that dynamically load type from external assemblies may also produce them.

During the component stack invocation phase, BuildAssembler may report any error that occurs within a build component. When it does so, it also reports which component generated the error.

Within the BuildAssembler code, build components are represented by the following abstract class:

public abstract class BuildComponent {

protected BuildComponent (XPathNavigator configuration) {}

public abstract void Apply (XmlDocument topic, string id);

protected void WriteMessage (LogLevel level, string message) {}

}

To instantiate each build component, the constructor is called with the configuration parameter set to the build component’s section of the configuration file. For each topic, the Apply method of each build component is called in turn with topic parameter set to the XML document as modified by previous components and the id string equal to the id of the topic being generated. Build components can write messages to the log by calling the WriteMessage method.

To create a new build component, a developer just creates a new class that inherits from BuildComponent, and compiles his/her source with a reference to the BuildAssembler executable, which defines the abstract BuildComponent class:

csc /t:library MyComponent.cs /r:BuildAssembler.exe

He/she can then use the new component in the BuildAssembler configuration file without having to re-compile the BuildAssembler application.

BuildComponents (DLL)

The build components we ship fall into three categories: action components that affect the document, flow control components that affect the order of invocation of other build components, and diagnostic components that can be used to supply information about the state of the XML document that represent the topic at any point in the component stack.

Action

Flow Control

Diagnostic

CopyFromFile

CopyFromDirectory

CodeExample

Transform

SharedContent

ResolveLinks

Save

ForEach

IfThen

Switch

Branch

Display

Validate

Anders Ljusberg explains in detail how Sandcastle works and has provided some details about these components. I will write a separate blog with details about these components.

MRefBuilder (EXE)

aram — Sun, 13 Aug 2006 21:31:00 GMT

I my previous blog I discussed Sandcastle.config in detail. Sandcastle process uses three executables:

MRefBuilder: Reflects over managed assemblies to produce an XML-formatted reflection information file.
BuildAssembler: Collates information from comment files and reflection information files into a set of HTML pages.
XslTransform: Applies XSL transforms to XML files. This executable is used to generate and manipulate auxiliary data files.

The MRefBuilder application reflects over a set of managed assemblies and creates an XML output file containing information about the APIs found in those assemblies. APIs are namespaces, types, and members defined in the assembly. Reflection data includes, but is not limited to: visibility, return types, parameter names and types, inheritance data, whether a class is abstract or sealed, etc. The output of MRefBuilder is consumed by the reference build component stack when creating API reference documentation.

The invocation syntax for MRefBuilder is as follows:

MRefBuilder.exe dll /dep:dep /out:out /namespaces- /internal-

Here dll is the path to an assembly to reflect over. More than one dll path may be specified, and each dll path may be a wildcard path corresponding to multiple assemblies.

The /dep option specifies a dependency assembly to load. (A dependency assembly is an assembly upon which one of the reflected assemblies depends; i.e. the dll assembly was compiled with a reference to the dep assembly.) Multiple /dep options are allowed, and each option may specify a wildcard path corresponding to multiple assemblies.

The /out option species a path to an output file; if no /out option is specified, output is sent to the console.

The /namespaces (not available in July CTP) option specifies whether namespaces are included in the output; by default they are. The Boolean /namespaces option enables the componentized build scenario. In that scenario, MRefBuilder would be run with /namespaces- on each assembly. The resulting reflection data would be used to generate separate HxS documentation, without namespace topics, for each assembly. Then a separate tool would be run over all the assembly reflection data files together to produce a namespace reflection data file, listing all the namespaces and their corresponding members in all assemblies. This namespace reflection data file would then be used as input to generate one more CHM/HxS containing only namespace topics.

The /internal (not available in July CTP) option specifies whether APIs visible only inside the assembly are included in the output; by default they are not.

If a required dependency assembly is missing, MRefBuilder will report the name and version information of the missing assembly and of the assembly that depended on it. This error condition is fatal.

MRefBuilder uses the Common Compiler Infrastructure (CCI) reflection library to do its work.

The schema of MRefBuilder output is complex. Fundamentally, the structure is just a list of APIs:

<apis>

<api id=”cer” />

<!-- many child elements containing

reflection data on cer -->

</api>

</apis>

There are, however, many possible descendents of each api element, each providing different reflection data. Just which descendents may appear depends on the kind of API described.

The format for reflection data produced by MRefBuilder is very important and the reflection data is consumed by the following:

A BuildComponent in the component stack for reference builds.
A transform that produces the BuildAssembler manifest for reference builds.
A transform that produces the HxT table of contents for reference builds.

We are currently looking at the naming convention and will probably rename MRefBuilder to Mreflector for Sandcastle RTW. We also expect the Sandcastle reflection data format to be targeted not only by MReflector, but also by unmanaged reflection tools (for example, a UReflector tool that parses IDL or header files) and SReflector (for generating schema documentation). In order to make it easier for other to target the reflection data format, we plan to supply an XML schema file that defines the format.

Sandcastle Configuration (sandcastle.config)

aram — Mon, 07 Aug 2006 08:23:00 GMT

Overview

Sandcastle builds documentation for any managed class libraries. Sandcastle requires the following inputs:

A set of managed assemblies (.dlls or .exes) to document
The set of assemblies on which the operation of those assemblies depend (aka dependencies)
XML comment files like those generated from /// comments using the /doc option. XML documentation comments can be added to your source files (.cs or .vb or .h/.cpp). In Visual Studio 2005, the C# (csc.exe), VB (vbc.exe), and C++ (cl.exe) compilers all support XML documentation comments using the /doc:assembly.xml switch. In previous versions of the .NET Framework only C# supported XML documentation comments. For more information on authoring documentation comments, click here.

The output of Sandcastle consists of HTML pages. These pages can be packaged in a CHM file using the HTML Help Workshop tools.

The Sandcastle process uses three executables:

MRefBuilder: Reflects over managed assemblies to produce an XML-formatted reflection information file.
BuildAssembler: Collates information from comment files and reflection information files into a set of HTML pages.
XslTransform: Applies XSL transforms to XML files. This executable is used to generate and manipulate auxiliary data files.

In addition to these executables, a number of useful XSL transform files and auxiliary data files are provided.

Configuration

The sandcastle.config file specifies the steps that BuildAssembler takes to build each topic and the data sources used in those steps. Whether you do something as simple as change the name or location of an input file, or something as complex as adding a new step of your own, you will need to develop a basic familiarity with this file.

The steps are specified as <component> elements under the <components> root element. The configuration data used by each component is represented as XML data under the <component> tag.

The components used in the Sandcastle stack are:

Create a skeleton XmlDocument to store topic data.
Load reflection data for the topic from the reflection.xml file.
Copy in reflection data for namespace (in type and member topics) and types (in a member topic) that contain the API.
Generate syntax blocks for the API.
Copy in authored comments for the topic from the comments file.
Copy in reflection data and authored summaries for members (in a type topic) or types (in a namespace topic).
Apply a transform to the topic to generate an HTML page.
Resolve any references to localizable shared content.
Resolve any references that link to other topics.
Save the result to a file.

Many of these components read data from external files. You may want to change the files that are read, or add new ones.

If you are a developer, you can create and use your own component. A component is an implementation of the abstract BuildComponent class, defined in BuildAssembler.exe. The only members required are a constructor that accepts an XPathNavigator, from which the component should read its configuration data, and an Apply method that accepts an XmlDocument representing the topic and a string representing the topic ID. Add your component to the stack by adding a <component> tag with assembly and type attributes that point to your component. Using the configuration data loaded in its constructor, your component can do anything it wants to the topic XmlDocument.

Notice that the reflection data step reads the reflection.xml file that you generated, and also reflection data files in the cpref_reflection directory to get reflection data for .NET Framework assemblies. (You could generate these files yourself by running MRefBuilder on the assemblies in your .NET Framework installation directory.) Notice that you can specify multiple <data> elements to read data from multiple sources, and use wildcards in the files attribute to specify multiple source files using each <data> element. If the assembly you document depends on types defined in other assemblies, you may want to add reflection data files for those assemblies here.

Notice that the copy comments step reads the comments.xml file that you generated, and also comment files in the .NET Framework directory that contain authored summaries of the APIs in the .NET Framework assemblies. (These files are installed by the .NET Framework SDK, so if you want these summaries to appear for the APIs that your types inherit from .NET Framework types, you will need to install the .NET Framework SDK, or at least copy those files from a .NET Framework SDK installation into a directory that you reference here.)

Notice that the transform step loads its topic transform from a file in the Presentation\transforms directory. If you want to change the look-and-feel of a topic, you will need to make changes to that transform.

Notice that the resolve shared content step reads shared content elements from files in the Presentation\content directory. If you want to change the names of sections, headings, etc. that appears in topics (for example, in order to localize them into a non-English language) you will need to make changes to those files.

Notice that the resolve reference links step requires topic data files in order to correctly render links. If you add a topic data file to the reflection step configuration, you should add it here also.

Finally, notice that the path to which a topic file is saved is constructed via an XPath expression in this component’s configuration. If you want to save topic files to a different location, make that change here.

I will post new blogs about BuildAssembler and MrefBuilder.

Evolution of Sandcastle

aram — Sun, 06 Aug 2006 07:57:00 GMT

Thanks for all your comments and feedback about the July CTP. I was away last week at a management training and will get back to all your comments. I also received several emails about Sandcastle and have started getting back them. This forum post had some interesting comments and made me write this blog about the evolution of Sandcastle.

Sandcastle builds documentation for any managed class libraries. The only required source files are your assemblies (.dlls or .exes) themselves. For a more complete set of documentation, Sandcastle also supports XML documentation comment files which allow you to provide summaries, remarks, examples, and other content to enhance the documentation. XML documentation comments can be added to your source files (.cs or .vb or .h/.cpp). In Visual Studio 2005, the C# (csc.exe), VB (vbc.exe), and C++ (cl.exe) compilers all support XML documentation comments using the /doc:assembly.xml switch. In previous versions of the .NET Framework only C# supported XML documentation comments. For more information on authoring documentation comments, click here.

During VS 2005 ship cycle we wrote Sandcastle to ship .NET Framework documentation for MSDN online and offline libraries. This version of Sandcastle had 2 components cDoc and pDoc.

cDoc

cDoc (comments doc) compiles assemblies (and optionally your comment files) into a cDocML file. The cDocML file contains all of assemblies' metadata and all of XML documentation comments (if any) in an XML format.

pDoc

pDoc (presentation doc) compiles the cDocML file into individual topics organized by namespace, type, and member. pDoc also creates table of contents files, index files, and project files necessary for the HTML Help compilers that will follow.

HHC (HTML Help 1.x Compiler) compiles the pDoc target files into a CHM file and HxComp (Microsoft Help 2.0 Compiler) compiles the pDoc target files into an HxS/HxI pair of files.

The attached diagram illustrates an overview of Sandcastle used for shipping VS 2005 documentation. This version is also used by several teams inside Microsoft for their reference documentation.

After shipping VS 2005, we decided to re-write Sandcastle to improve the perf, scalability and reduce our .NET Framework build time. We started our planning for the new version of Sandcastle (code named “manifold”) in April 2006 with a goal to make this version available externally. The July CTP version of Sandcastle helped us reduce .NET Framework build time from 10 plus hours to 30 minutes. Details on the architecture for the July CTP are available here. We plan to release couple CTPs of this new version before RTW. Also we plan to ship .NET Framework documentation in Windows SDK using this new version.

Anand..