Vesa "vesku" Juvonen : WSS 3.0

How do we do SharePoint development, including TFS integration?

sonofthesun — Sun, 25 Oct 2009 15:32:18 GMT

There’s lot of information available concerning development processes, automated builds and continuous integration in SharePoint development using Team Foundation Server, but unfortunately most of the information is in relatively high level or just concepts, but details are missing. It’s also fairly common misconception that setting for example automated build with TFS is difficult – it’s not.

Following chapters defines the practices we use in MCS Finland during development and for setting up the automated daily builds from TFS for enterprise projects. We will concentrate on the development process and setting up automated build process, but I’ll give some additional context from the project management point of view as well.

This content applies both SharePoint 2007 and 2010 development. There are minor changes and improvements with SharePoint 2010 development, which are pointed out.

In this blog post, we cover following topics

Development process
Visual Studio solution configuration
TFS side configurations
Automated testing environment setup

Development process

Following picture defines in high level the process we used for our enterprise projects. Actual details depends of course on the project size and project team. Actual process or methodology depends as well on the project, but process is always based on the agile approach with numerous iterations. Usually methodology used can be considered as adaptation from the SCRUM.

#	Step	Description
1	Requirements and tasks	Project requirements and tasks are collected from project owners
2	Collecting tasks to TFS	All tasks to be accomplished (also documentation) are entered as tasks to the team foundation server. Tasks are divided to iterations based on the initial plan of the project.
3	Virtualized development environments	All development environments are virtualized for easy management. Standardized base image has been created, which can be setup to server in matter of minutes. We commonly used centralized box to host our virtualized environment, but this depends also on project. Some consultants have 8GB on their laptop, making it as more appropriate and flexible hosting environment using Hyper-V.
4	Integration and testing servers	Separate testing servers are part of the project setup, which are used to verify the delivery before installing that to customer environment. Automated builds are used and depending on the project, automated testing is being done, using process defined later in this blog post.
5	Evaluation of status	Daily builds are being validated and based on the status and iteration schedule new deployments are being installed to the customer’s quality assurance environment.
6	Quality assurance testing	Customers can follow up the progress of the projects and give instant feedback, if there’s something to change. Release cycle to customer environment depends based on project, but is approximately two weeks to ensure that there’s constant dialog on the functionalities being developed. Quality assurance environment is also used as the acceptance testing environment, when production release is planned to be done.
7	Change requests and feedback	Feedback and change requests from the quality assurance environment are tracked to team site and entered to TFS system for developers to see the possible changes and new tasks. All incoming change requests are prioritized and scheduled to iterations based on discussions with project owners.
8	Production installation	When first release can be done, QA environment is used for acceptance testing and decision to move forward is done.
9	End user feedback	Since quite rarely projects end on the first production release, feedback and tasks are obviously collected from the end users to be processed to change requests and actual development tasks.

Iteration and production release count obviously depends on the project.

Visual Studio Solution configuration

There are few things to do in the Visual Studio Solution level to enable automated builds, if you use Visual Studio 2008 in development. By default Visual Studio 2008 does not understand the solution packaging concept, so we need to enable this by modifying VS project file. Also by modifying the project settings, we can improve our development experience.

In our case the standard VS2008 project structure usually looks something like following. There’s few things to notice from the solution structure.

Code is divided to multiple projects based on their usage models
- ApplicationLogic – contains business logic of the project, including all actual code manipulating data of the lists etc. If patterns and practices models are used, services, repositories etc. are placed here as well.
- Data – this is project for the data objects, which are used to transfer information between different solution layers. For easy manipulation in the UI layer, all code is transferred to data objects, rather then transferring actual SharePoint objects. This also helps in unit testing.
- FeatureReceivers – this project has the custom feature receiver code. Code is separated, since feature receivers have to be placed to GAC in every deployment scenario.
- Resources – this project contains the 12 hive files
- Solution.Assemblies – project to create the *.Assemblies.wsp solution package
- Solution.Resources – project to create the *.Resources.wsp solution package
- Web.UI – Project containing all the code for web parts, web controls, http modules and code behinds for application pages
As you see, there’s actually two projects, which are used to generate solution packages. One for the resources (stuff to goes to 12 hive) and one for the actual assemblies. This is controversy approach also in our team, but if the assemblies are separate from the resources, we can more easily and risk free, just update the actual code, without touching the xml files on disk.

For the projects, which are outputting solution packages (wsp files), we’ve configured new .targets file, which contains separate settings for Debug and Release configurations. The Target WSSSolution has been configured as the default target to these project by modifying project file.

Notice that output folder of the assemblies has been configured as “..\..\..\Binaries\Release\” in the Release configuration and that wsp package is also placed to the same folder using the above xml configuration. This is same relative path to folder what’s used by the TFS automated build process to output assemblies, before the whole folder is copied to drop location. Since the wsp package is also placed to this folder by default, it’s automatically copied to the drop location without any additional configurations. So we use the same relative path already in the development environments, so that we don’t need to change anything as part of the automated build process.

How to debug efficiently in VS2008? – This is quite common question, since in VS2008, there’s no native way debug the SharePoint customization efficiently, unless you’re using the VS extensions for WSS 3.0. Since the extensions were introduced quite late and they didn’t first support TFS integration fully, we haven’t got used to utilize them in our projects. This does not however mean that you shouldn't use them.

In our case, customers usually accept deployment of our assemblies to GAC, so don’t have to create CAS files for the deployments, which makes also the debugging little bit easier, since during development time, we can deploy our assemblies to the bin folder of the IIS application and start debugging without IIS recycles etc.

This means that we’ve setup the output folder of the Visual Studio projects to bin folder of the IIS application and configured the web application to fully trust the assemblies (<trust level="Full" originUrl="" /> in the web.config – not good practice if CAS files are used). To avoid unnecessary application pool recycles, we’ve also added additional AllowPartiallyTrustedCallers attribute to assembly, so that assemblies are loaded to IIS process memory runtime.

After these settings have been done, only thing left to do, is to attach to IIS process (w3wp.exe). To make this as efficient as possible, we use VS macro and short cut keys to establish the debug process. Basically this means that only thing we need to do to get the latest versions to development portal is to compile the code and press the shortcut in the Visual Studio. After you request the page from you browser, debugger get’s automatically attached and you can start debugging.

What kind of third party tools we use? – Actually none. Managing solution manifest and the ddf file is something which we do manually as part of the development process. Each person involved in the project has responsibility to keep the files in sync. Generally we aim to create stub versions of each control, web part and feature file in the first iterations of the project. Meaning that each feature file and template is already included in the package, even though it doesn’t do anything. Also each web part or control only write the purpose of the particular functionality – “This web part will provide the article comment functionality in later Iteration”. This way each iteration will just increase the functionalities, which are working as designed in technical architecture for customization document.

There’s lot of great third party tools available for the build process in the VS2008, but we have not decided to use them, since in the end manual updates do not take that long and if the process is automated, there’s no way to control what actually get’s packaged from VS structure. Meaning that if some features are not finalized yet for the solution package, we don’t need to add them to manifest and ddf.

How about SharePoint 2010 environment with VS2010? – When you develop SharePoint 2010 projects using Visual Studio 2010, solution package is by default compiled to the same output directories as the assemblies, so there’s no reasons for these additional configurations. Visual studio solution structure also looks quite different, since in VS2010, you’d place the code and 14 hive elements to same Visual Studio projects.

In SharePoint 2010 development, the common way to divide the files between the Visual Studio projects is based on the functionalities, not based on the file types. This means that for example the MySite functionalities are compiled to single solution package and portal and collaboration customizations to other packages. This model enables you a way to test and version the functionalities independently.

Debugging your customizations is extremely easy, since for all of the SharePoint customizations, this happens just by starting the debugging session directly from the Visual Studio IDE. Similar ways as for the 2007 environment, debugging happens in the same computer, where you develop your customizations. Target url for debugging is defined during the VS project creation or from the project properties.

How about SharePoint 2010 environment with VS2008? – This is definitely valid option and quite easy to establish (it’s only references). If you want to migrate your customizations as fast as possible to SharePoint 2010, this is most likely the temporary option you’ll use. In this case, you’ll have to modify the VS2008 project types as declared above. Since VS2010 however improve the productivity of the developers, we strongly suggest to migrate your code to VS2010.

VS2010 supports importing existing solution packages to your Visual Studio solution. This way you can relatively easily migrate your VS2008 package to VS2010. After importing, you’ll need to reconfigure the packaging model, which will require refactoring, since the Visual Studio 2010 solution structure is not total imitation from the 14 hive.

TFS side configurations

In the TFS side, all we need to do is create the build agent and the build definition for the project. This is relatively trivial process, but basically we are enabling the automated build using standard TFS side configurations. Since we’ve configured the solution package to be dropped to the same locations as the compiled assemblies (dll files either in Debug or in Release folder), we actually don’t need to do any additional steps or configure any additional msbuild settings of the build project.

TFS 2008: A basic guide to Team Build 2008

Some things to remember when you setup the build server

Ensure that you have the proper assemblies installed and/or available on the build server. Setup is dependent on the decencies on your code.
Ensure that the build service is running using domain account, which has access to the team project, so that the build service can get the latest version from the source safe
Setup share to the build server or any staging server to where the compiled builds are stored, including the solution package
Remember to setup proper cleaning process for the automated builds, since each build will consume space – this can be controlled using the retention policy of the build definition file

Depending on your code and unit testing practices, you can also add unit testing as part of your code compilation. Since some of the SharePoint dependent code can be difficult to mock, this might be difficult or require third party tools.

Automated testing environment setup

TFS automated build only verifies that the code itself compiles, but there’s still additional tasks to be done to ensure that code can be deployed and that the actual use cases work as expected. Especially for the larger projects, we’ve created additional steps, which are used to imitate the customer installation of the customizations. This way after each automated build, we have similar upgraded version of the portal, that the customer would have. To automate this, we’ve added few scripts to be executed.

The following model also decreases the inefficient working time of the tester to setup the actual testing environment, before the actual functionalities can be tested. We can certainly test some of the functionalities using automated testing, but for larger projects, it has been fairly common to have dedicated tester.

In one of our centralized hyper-v servers, we’ve setup a scheduled batch file, which is executed after the automated build process.

set Folder=\\mcstfs\contosoautobuild
set FileMask=autowspbuild_*
set VMName=contosotest02
set SnapshotName="18.10.2009 latest win updates"

set LatestFile=
for /f "delims=" %%a in ('dir /ad /b /o:d "%Folder%\%FileMask%"') do set LatestFile=%%a
if "%LatestFile%"=="" goto :eof
echo %LatestFile%

powershell -command .\startsnapshot.ps1 %VMName% %SnapshotName%
echo Sleep for 3 minutes...
ping -n 240 localhost >nul
applybuild %LatestFile% %VMName% contoso\moss_admin pwd

As you can see from the batch file content, we first solve the latest solution file, continue with snapshot modifications and as final step installs the build to the particular virtualized environment.

We use powershell to restore the status of the testing server to same status as in the production environment. This is done by using following powershell script, which restores the snapshot given as parameter to the script.

$VMname = $args[0]
$SnapshotName = $args[1]

$VMManagementService = Get-WmiObject -Namespace root\virtualization -Class Msvm_VirtualSystemManagementService
$SourceVm = Get-WmiObject -Namespace root\virtualization -Query "Select * From Msvm_ComputerSystem Where ElementName='$VMname'"

$Snapshots = Get-WmiObject -Namespace root\virtualization -Query "Associators Of {$SourceVm} Where AssocClass=Msvm_ElementSettingData ResultClass=Msvm_VirtualSystemSettingData"

#Write $Snapshots.Length

foreach($Snapshot in $Snapshots)
{
#    $Snapshot.ElementName
    if($Snapshot.ElementName -match $SnapshotName)
    {
        #Write $Snapshot.ElementName
        $TheSnapshot = $Snapshot
    }
}

Write "Applying the snapshot..."
$result = $VMManagementService.ApplyVirtualSystemSnapshot($SourceVm, $TheSnapshot)
ProcessWMIJob($result)

Write "Starting the VM..."
# Start the VM
$result = $SourceVM.RequestStateChange(2)

When this script has been executed, the virtualized environment is back online and the defined snapshot has been restored. Following step is to apply the actual build to the restored VM. For this the following script is used.

rem parameters: buildname, vmmachinename, username, password

xcopy /y /s \\mcstfs\autobuild\%1 \\contosotest02\drop\%1\

c:\pstools\psexec.exe \\%2 -u %3 -p %4 -w c:\drop\%1\release c:\windows\system32\cmd.exe /c c:\drop\%1\release\upgrade.bat http://%2 dev V12 V5

As you can see the script, we are copying the solution package from the TFS autobuild server to share located in the testing server (which was just started). After that we can execute the upgrade script on the testing server, which contains the project dependent script to upgrade the existing portal with the new version of the customizations.

By using these relatively simple scripts we now have clean and updated environment for testing every single day with all the latest customizations. Model obviously decreases the overall project costs, since we don’t need to spend time setting up the environments manually.

How to further improve the process?

After the testing environment has been setup, we can also improve even further the process. Common improvements would be to add web testing and load testing to this scenario. Due the extensive support for scripting, we can automate these to be included to the daily process. This would mean that after the testing environment has been setup, we start the automated web tests for use case validation and performance testing to ensure that new code has not dramatically decreased the performance.

Web testing is again relatively trivial task to do with the Visual Studio and it provides quick way to ensure that basic functionalities are still working as expected.

Performance testing uses the recorded web testing scripts and imitates larger load. Even though the development testing environment is not using same hardware as the production, we can do estimates and assumptions based on baseline tests.

Baseline testing means that we have made tests in the customer environment and in testing environment using same customizations. If the new version is decreasing test environment performance for 20%, we can assume that the result would be most likely the same in the production.

Improvements in SharePoint 2010 for the process

As mentioned already earlier in this post, when you develop customizations for the SharePoint 2010 with Visual Studio 2010, there’s native support to compile solution packages to the output folder of the Visual Studio project, so there’s no need for any msbuild script changes.

Other huge improvement is the native support for upgrade and versioning in the feature framework side. This will help to upgrade already existing portals and manage the versioning of the customizations. Traditionally in SharePoint 2007, this has been done with custom feature receivers, which modify the existing content. In SharePoint 2010, we have much more powerful tools to manage the portal lifecycle and to update the already existing content. Also the logging mechanism have been dramatically improved to help with customization troubleshooting, if deployments fails or something unexpected will happen.

We’ll release lot of new information concerning the SharePoint 2010 development in upcoming months, so stay tuned.

Summary

Setting up the standardized development process might seem like a large task, but when the environments and processes have been defined, it saves lot of resources to actual efficient project work. The processes we use for our enterprise projects (30-500-xxxx man day projects), is constantly evolving process, which is improved and changed based on the experiences from the projects.

Following people have also contributed to our standardized process: Jaakko Haakana, Juhani Lith, Janne Mattila, Tom Wik and Jukka Ylimutka.

Hopefully this was useful.

Project planning and application lifecycle management

sonofthesun — Tue, 16 Sep 2008 21:00:01 GMT

One of the most important and often overlooked thing to cover in planning phase of the projects is the application lifecycle management (ALM) for upcoming portal. In this context I don't mean just the lifecycle management for the customizations, rather for the full process and environments. Every single project done with SharePoint technologies (WSS/MOSS), should define the clear rules and practices to manage the process as early as possible.

By setting the ground rules immediately in the planning phase of the project, you can take them into account during the technical planning of the deployment and of course also in the operational planning. Following image and table defines one example process, which follows the continuous integration model for SharePoint development (Continuous integration in MOSS development using TFS).

Following table defines the steps and phases one-by-one.

#	Phase / Element
1	Developers develop individual features and functionalities based on of the technical specification using their independent virtualized environments, which have access to the TFS server for work items, source control etc. Virtualized environment have to be in-sync with the production environment concerning the licensing and patching. Customizations developed with enterprise license; don't necessarily work in standard environment. Patches and service packs should be also keep in sync. Ensure that it's somebody’s responsibility to keep the virtualized environment up to date.
2	TFS Server used to store source code and other project related information. Developers can also sync their environment using the artifacts stored in TFS. It's obvious, that developers have to have ensured connection to the source control repository.
3	SQL Server instance of the TFS, used for actual storage of the different artifacts and document. Ensure that this database is fully managed, monitored and operational; otherwise the development cannot be conducted.
4	Development integration server, which is used to verify the builds from the TFS, preferable using automated build process. Server should be kept in sync with the production environment. Integration server is used for integration and deployment testing. Also the initial functional testing for full package should be conducted in this environment.
5	Project members (for example project manager, testers and even customers representatives in some cases) can follow the progress of the project and give feedback based on the builds deployed. Define the ground rules for accepting the deployments to quality assurance environment from the integration environment. If the development happens in ISV premises, this server is most likely located also there. It's not however good practice to have similar server in these cases also at the customer premises, so that it can be used to deploy pilot or draft versions of the solution. This way we can establish communication channels concerning the upcoming features as soon as there's something to deploy. This approach minimizes surprises at the end of the project and the customer representatives can use the environment for training purposes as early as possible There has to be clear rules to follow for accepting the deployment for following phases.
6	Quality assurance environment used for functionality testing and acceptance testing. In ideal world this environment is identical as the production environment, so that it can also be used for load testing. Quite often though, the environment is virtualized for more convenient maintenance. Load testing can be nevertheless conducted also in virtualized environment, if project performs the initial load testing (base line testing) during the first version of the portal. This way the future load testing results can be compared to these base testing values. In sense of licensing and configuration, the environment should be completely identical to production. This ensures that if the deployment is successful in this environment, it will work also in production.
7	SQL Server of the quality assurance environment. Obviously the configuration should be identical as for the production. Operational setup should also follow the production, so that full portal behavior can be observed also in this environment.
8	MOSS environment used for production purposes. There should be clear responsibilities and operational guidance for this environment to ensure that possible issues caused for example customizations can be solved in timely fashion. One of the key things to document and to follow is the version handling model of the customizations (how things are updated). As written earlier, there has to be crystal clear model for deployment of new versions and guidance to follow in case of any issues encountered.
9	SQL Server for the production, which is fully managed and monitored 24/7. In case of any issues, there should be clear guidance on how to proceed. Backup and restore process should be verified in quality assurance environment and if possible also in production environment, before going live with the portal.

Considerations

Following chapters defines few points to consider when the full process is planned. I also want to raise few pointers, which I have personally run into in multiple partners and customers.

Are the code and SharePoint artifacts in safe place?

It should be clear that the all of the customizations developed for SharePoint portal are stored in some source control system, like Team Foundation Server. You don't want be in situation where the customization are gone, due failure in single laptop or desktop. Also consider and ensure that the source control system is in safe hands. If there's critical issues in the production, which requires instant code level fix to be deployed, you need to ensure that the source control is up and running.

Basically this means that the source control system should be high available or at least there's a backup plan to access the source code in timely fashion, so for example possible SLA's can be achieved.

Is the process high available?

We need to go through the steps in the process to ensure that there's no single point of failure in the process. Technical failures can be fairly easily identified, but we also need to consider the steps, which require human intervention. You don't want to be dependent from single persons, which could compromise the environment for example during summer vacation. There needs to be a back up person for each critical responsibility, with sufficient knowledge concerning the possible tasks to be done.

Is there clear model to update the customizations?

Setting up the initial version of the portal is quite straight forward, but the possible updated versions have to be carefully designed. It's quite common that after the initial version, there will be additional functionalities included in the portal. At some customers, there are quarterly releases, which enhance the functionalities provided for the end users by introducing new possibilities and options. If the upgrading model is not clear before the initial customizations are deployed, life can get complicated. Can we use the in-place upgrade of the solution packages? Does the end users use SharePoint designer in production? Every decision has it's effect and there for we need to ensure the model already in the planning phase.

Personally I've seen way too many SharePoint deployments, which have been build up nicely, but when we need to update any of the customizations, the life get's over complicated.

Is the customization deployment model scalable?

The usage of the solution package in deployment of the customizations can be considered as the de-facto way of deploying ANY customizations to the portal. By using solution packages we can ensure that in case of increased load for the MOSS farm, we can scale the farm out, by adding additional servers. If the deployments are done manually, there's too huge risk for human errors and the servers won't be in-sync. Manual deployment would require also huge amounts of additional actions to be conducted when the new server is introduced to the farm.

Summary

I listed only few pointers to give you an idea of the things to cover in you project. Of course model and required processes is fully dependent on the scale of the deployment. If you only use closely out of the box SharePoint installations and the modifications are done using SharePoint Designer, you don't necessarily have to think through these kinds of things. For large scale projects, the models and processes have to be planned carefully. It's not rocket science and you should not over think the processes, but by planning a head, you can ensure the long running service, which will be definitely worth of investment.

Continuous integration in MOSS development using TFS

sonofthesun — Tue, 29 Jul 2008 19:58:09 GMT

I've been delivering quite a few technical training's during past year and one of the most discussed thing is the setup of the development environments for large scale projects. Especially large ISV's are really interested on the the practicalities of utilizing the TFS as the continuous integration (CI) and/or application lifecycle management (ALM) platform. For standard .net projects this has been the way to manage large projects and it's obvious that the investment and practices are wanted to be utilized also for the SharePoint based development.

Since SharePoint development differs quite a lot from the standard asp.net development, this has not been that straight forward. Following scenario is example done using Visual Studio and TFS, but the principles and practices can be easily adapted also for other continuous integration solutions, like the CruiseControl.NET (CCNet).

Setting up the Visual Studio solution for the TFS

Before the continuous integration can be setup in the TFS side, we need to configure the Visual Studio project correctly, so that when ever build is initialized, newly compiled solution package (.wsp) is created. There are numerous blog entries available from the Internet including the detailed steps for this.

MSDN - Automating Solution Package Creation for Windows SharePoint Services by Using MSBuild

Basically the idea is to configure the Visual Studio solution such away that each assembly is first compiled and then the solution package is compiled using the MakeCab.exe. For the VS solution where you have multiple projects, make sure that you have defined the project dependencies such away that that the actual solution package project (the one which output is the wsp file) is dependent on the assembly projects (outputs dll's). This ensures that the assembly projects are compiled, before the wsp package is generated.

Creating the auto build project for TFS

When the auto build process of the TFS has finalized by compiling the Visual Studio solution, we have received fully package solution package(s), which are ready to be deployed to any SharePoint server. Since the TFS is not aware of these kind of file types, it does not by default copy the wsp package to the drop location. This is not an issue, since we can modify little bit the build project to be able to initiate the portal recreation. By opening the build project file (by default TFSBuild.proj located in the TeamBuildType/[build name]/ folder in TFS source control) and adding following xml elements, we make sure that the wsp package is also copied to the drop location and additional batch file (this case the rebuild.bat) is executed.


<Target Name="AfterCompile">
  <Copy SourceFiles="$(SolutionRoot)\[TFSProjectName]\[ProjectName]
                    \SolutionFiles\Package\[SolutionPackageName].wsp" 
        DestinationFolder="$(DropLocation)" />
  <Exec Command="C:\TFS\rebuild.bat" 
        WorkingDirectory="$(DropLocation)" />
</Target>

Note. Above example of using rebuild.bat is dependent on the fact that the SharePoint is located in the same server as where the build happens, which in most of the time, is not the case. Alternative solution is declared in the following chapter.

Really nice feature concerning the auto build is also the fact that operations and actions logged by the MakeCab are automatically included in the TFS auto build report, which is generated for each executed auto build. If there's anything wrong with your solution files (manifest, ddf etc.), the errors will be automatically logged here. Each executed build has it's own detailed information, from where you can access the build log as we can see from the following image.

Build log (BuildLog.txt) has huge amounts of details concerning the actions taken in particular build. All the MakeCab logged information is also included in the log for detailed analyses on the SharePoint solution package compilation.

Adding rebuild of the portal to scenario

When the wsp package has been created, it has to be of course deployed to the portal first, before it can be tested. This can accomplished manually, but it can also be automated, so that the portal is recreated automatically as part of the auto build process.

Personally I have done this few different ways. Initially I created a console application, which was executed as a scheduled task by the Windows OS. More convenient way to do the same would be to create few new extensions to the stsadm, which are responsible of setting up the environment, so that project members can access the latest version without any manual intervention. If the build server and SharePoint server are different servers (most likely the case), you can schedule the execution of the stsadm commands to batch file located in the server of the drop location.

Following defines one approach used. The tasks are dependent on the type of development and can be customized based on your requirements.

Objectives

Redeploy the new solution package to the farm - remove any previous versions, if exists
Recreate the portal hierarchy using portal site definitions
Define access to the newly created hierarchy for the project managers and testers

For these objectives, I created following stsadm extensions, which are sequentially processed. These commands access the farm using object model. By default the stsadm provides already similar functionalities, but by creating my own commands, I can easily improve and/or add any actions to be deployed as part of the auto build process.

Command	Description
deploysolutionadv	Responsible of deploying the new solution package to the farm. Retracts and removes any previous versions from the farm, if exits. Command is used to redeploy the solution package as part of the daily builds.
recreatesitecollection	Command recreates site collection using specific template defined as parameter. If site collection already exists in the farm, it's deleted. Command is used to recreate the site collection for the daily builds. Portal site definitions are great way of providing immediately full hierarchies for the newly created site collection.
assignuserstogroup	Grant access to defined site collection for the users defined as parameter. Command is used to define access to the newly created site for the persons responsible of verification tasks.

Full scenario for continuous integration

Following image defines the key steps for the continuous integration within the SharePoint development. This model can be considered as the development time process for the project.

Following table defines the steps and phases one-by-one.

#	Phase / Element
1	Developers develop individual features and functionalities based on module plan (part of the technical specification) using their independent virtualized environments, which have access to the TFS server for work items, source control etc.
2	TFS Server used to store source code and other project related information. TFS is scheduled to build the integrated version of the package using build automation functionalities. Developers can also sync their environment using the artifacts stored in TFS.
3	Development integration server, which is used to setup the outputs from the TFS. If required, this server environment can be utilized by multiple projects as long as they have separate application on which the solution is automatically deployed (often the case in ISV environments).
4	Project members (for example project manager, testers and even customers representatives in some cases) can follow the progress of the project and give feedback based on the builds deployed. By providing instance access to daily builds, project will get instant feedback for the developed functionalities Daily builds provide flexibility to follow the progress and to discover any required changes in the design as early as possible

Possibilities to test and verify the provided functionalities in the development integration server depend heavily on the type of the solution to build. In case declared above, complete custom site definitions with initial configuration of custom web parts are included and there for when the portal site definition is used to create the structures, new functionalities are directly visible in the portal.

On the other hand, it's quite common that you are developing functionalities, which are associated to the out-of-the-box site definitions using site stapling techniques. In these kind of projects, the new functionalities would be available, as long as you create the definitions, on which the stapling has been added.

Even though you would be developing only few custom web parts, by utilizing the deployment model as declared above, you could verify the deployment packages for your project and test the web parts in the environment. If you are only adding few new web parts to out-of-the-box portal, you might want to consider automated activation of your custom features, which deploys the .webpart files to the portal. This way the tester(s) could verify the functionalities by adding new web parts to the portal using standard web part picker.

SharePoint artifact development

One of the thing to consider when setting up the project is the storage location of the SharePoint artifacts. Even though SharePoint provides version control for artifacts it stores, it cannot be considered as a actual source control system. Especially if the development is done by ISV, which is the most common case, it's good to have the source code including the artifacts in sync in the source code system (like TFS) to be able to label the actual releases of the developed features. Consider the practicalities to update your customizations from version 1.0 to 2.0 (I'll write later practices for version handling of your SharePoint project).

Artifact development on the ISV side can of course utilize the standard SharePoint tools, like SharePoint Designer, which increase the productivity during the initial creation of the functionalities. There's however no easily way to sync the artifacts from SharePoint to the Visual Studio package responsible of the encapsulating the solution packages. Whenever the development for particular artifact is finalized, it can be however copied to the package manually. This way for example the master page developer, can first finalize and verify the functionalities in her/his own virtualized environment and provide versions to the official solution package when appropriate.

Real life experiences

I initially created this process and the necessary configurations for one enterprise project, which started July 2007, where I was acting as the technical lead for the infrastructure architecture and for the customized development (at the time project started developers from ISV didn't have that much experience on the customization models). Overall amount of developers in the project was up to seven persons and since the development happened at customer premises, the daily builds provided easy way for the customer representatives to follow the progress and give instant feedback whenever required.

Similar setup would be however extremely useful for also any ISV, which does SharePoint development. Since the recommended deployment method for any customizations in the SharePoint landscape is to use solution packages, this process would be useful to any development project, no matter the amount of the customizations (from one web part to enterprise projects with tens of developers).

One additional advantages from automation came as a surprise during the project - or initially it was not foreseen. One of the guidelines we kept in the project was to utilize immediately fresh copy of the virtualized environments, if unexplained errors were encountered during development, that could not be solved in timely fashion. By utilizing the same process as for the auto build, we could recreate the full portal for the development environment from scratch (huge WCM portal) just by running the predefined batch files. This decreased the time required for setting up the development environments with latest build and there for saved project resources for actual activities to be completed.

Summary & more information

Utilization of continuous integration practices within the SharePoint development projects provides fairly easy way to increase the quality of the delivered functionalities. The process might first seem difficult, but when the initial configurations and actions have been completed, process can be reproduced easily to any number of projects.

Links to the concepts defined in this blog post

I'll write more guidelines concerning the ALM (Application Lifecycle Management) and other project practices for SharePoint development in upcoming posts.