Using Jenkins + Testswarm + QUnit for javascript test on Windows environment

  • Article

Recently I set up an environment for testing our javascript projects using Jenkins + Testswarm +QUnit on Windows. Here's how I did it:

Background:

We have a bunch of tests written in Javascript using QUnit and checked in to Github. We'd like to set up a continuous integration system that pull the code from Github, and run the tests on the browsers/devices that we are interested in. After some research, we decided to start with the following:

  • Jenkins for its integration of Git plug-ins.
  • TestSwarm 
    • Nice integration with Qunit and be able to integrate into Jenkins as a build step. 
    • Natual support for multiple browsers and devices
  • We tried to stay on Windows Server as much as possible.

Experience:

TestSwarm

TestSwarm is not designed to be installed on traditional Windows servers natually.  However, since it simply uses php + MySql + javascript and some Apache RewriteRules, it's pretty easy to run it on Windows Server as well.  Here's what I did:

  • Install Apache for Windows, MySql and PHP.
  • Follow the instruction of TestSwarm's wiki on Github: https://github.com/jquery/testswarm , including set up mysql database, setting config.ini and .htaccess.  I just ignored item #5 regarding cronjob.
  • Go to https://<serverName>/<swarmPath> (such as https://localhost/testswarm/ ) and register a new user.  Go to the MySql database and find token value of the user just created in the "Users" table, which is needed later to add new jobs
  • To test if it works, open the testswarm page, log in, and open this page: https://localhost/testswarm/?state=addjob; *** NOTE The trailing ";" is very important.  Otherwise, you'll get a "Incorrect user name or Auth Token".  I was able to repro this on testswarm's website.  It seems this manual page is rarely used.  It's more common to add jobs programmatically.  Add a job and suite to point to some test page, and go to https://localhost/testswarm/user/<username>  to see the new job added or open https://localhost/testswarm/run/<userName> to run the actual test

 

Jenkins

Installation

Configuration

  1. Install plug-ins
    • Open https://localhost:8080 from browser. Click "Manage Jenkins", then "Manage Plugins".
    • Add "Jenkins Git plugin", "Github Plugin", "Jenkins MS build Plugin" from Available Plugins
    • I aslo need the Jenkins-TestSwarm plugin, which cannot be found in the available plugins list of Jenkins. The Jenkins-Testswarm project is on Github https://github.com/appendto/jenkins-testswarm . It contains a testswarmbuilder.hpi, which can be installed to Jenkins plugins (advance). Ideally, everything should just work now. It's true if you will not use any variables in your Testswarm settings. It's not true in our case. We'd like to deploy each commit to a new VDir, something like https://myserver/myproject/%Build_Number%/mytestpage1.html . We also wanted to customize the testswarm job title to show the Git commit number, which is available in Jenkins Git plugin as an environment variable: %GIT_COMMIT% However, there's a bug in Jenkins that does not replace the variables in settings with actualy environment variables. I have filed an issue https://issues.jenkins-ci.org/browse/JENKINS-11451 to track the issue. My fix is at https://github.com/xusun/jenkins/commit/b8dee2c710637e718290f93c31d3854099d2d157#comments To workaround it, I did not recompile the entire Jenkins project. Instead, I downloaded the testswarmbuilder plugin and replaced the reproMacro calls with my fixed version. Rebuild the hpi and reinstall it to Jenkins. That fixed the variable issue.
  2. Config system ("Manage Jenkins" => "Configure System")
    • Under Git section, set up "Path to Git Executable". This points to the full path to Git.exe
    • Under Git Plugin, input your names on Git and email.
    • Email Notification: it seems Jenkins has problems on authentications to exchange smtp servers. see Jenkins' issue 3983.https://issues.jenkins-ci.org/browse/JENKINS-3983. There are two workaround that I tried sucessfully:
      • use smtp.gmail.com as smtp server
      • config your local IIS to act as a smtp relay server.  see this msdn for details.  In the local SMTP server, you can set up the authentication credentials used for relay.  In this way, you don't need authentication with Jenkins. Note that even you have IIS7.0, this smtp server is managed in IIS6 management portal.

Add Jobs

Our workflow is like this: [Sombody checkin on Github] - > [Jenkins pull from Github ] -> [msbuild our solution] -> [msdeploy the websites to our IIS/Asp.net server] -> [Add test jobs to TestSwarm] -> [TestSwarm executes and return results] -> [Jenkins build Success/Fail] -> [Email groups]

The following might need to be mentioned:

  1. Config the Git repository: In the help of Jenkins, you can set up the git url as https://... or git://... or ssh://.   Since we have a private repo, the easiest way is to use ssh with a no-password key in the CI machine.
  2. Build triggers.  It's hard for us to define a Git callback proxy.  So, we just simply use Poll SCM per minute: "* * * * *".
  3. Deploy to IIS server.
    1. We would like to keep some build history for reference and possible issue investigations.  But we also wanted to keep the IIS server as clean as possible.  So, before deployment of a new build to IIS, we simply query the server for existing deployed vDirs and delete those that is older than n builds.  Since we use the jenkins build number as the vDir name, it's easy to simply write a batch file to call msdeploy to do this.  Here's the code, (although I have to admit that I'm not good at batch programming).  Here we used Jenkins environment variables %BUILD_NUMBER%
 @echo off
 setlocal ENABLEDELAYEDEXPANSION
 
 IF /I '%1' == '' GOTO usage
 
 SET TARGET=%1
 
 SET KEEP=10
 ECHO CLEANING APP %KEEP% LESS THAN %BUILD_NUMBER%
 set cmd="msdeploy -verb:dump -source:iisapp="Default Web Site\MyProject",computername=%TARGET%"
 
 
 FOR /f "tokens=1-5 delims=\\" %%G IN ('%cmd%') DO (
 IF "%%J"=="" ( 
 IF "%%H"=="upshot" ( 
 IF NOT "%%I"=="" ( 
 set /a test=%%I
 
 IF !test! GTR 0 (
 set /a commit=%BUILD_NUMBER% - !test!
 IF !commit! GTR %KEEP% (
 echo msdeploy -verb:delete -dest:iisapp="Default Web Site\MyProject\!test!",computername=%TARGET%
 msdeploy -verb:delete -dest:iisapp="Default Web Site\MyProject\!test!",computername=%TARGET%
 )
 ) 
 )
 )
 )
 )
 
 GOTO end
 
 :usage
 ECHO.
 ECHO CleanOldVDirs [TARGET_COMPUTER_NAME]
 GOTO end
 
 :end
 EXIT

  b. deploy script is even simpler:

 @echo off
 setlocal ENABLEDELAYEDEXPANSION
 
 IF /I '%1' == '' GOTO usage
 
 SET TARGET=%1
 
 CALL msdeploy -verb:sync -source:iisapp="%WORKSPACE%\test\MyProj.Test" -dest:iisapp="Default Web Site\MyProject\%BUILD_NUMBER%",computername=%TARGET%
 REM ...
 REM OTHER DIRS TO DEPLOY 
 GOTO end
 
 :usage
 ECHO.
 ECHO CreateNewVDir [TARGET_COMPUTER_NAME]
 GOTO end
 
 :end
 EXIT

4.TestSwarm integration:

 

That's it. Final tip would be that the Testswarm machine may need some config update on the php.config to make the server on production state. Otherwise, the php error log may heavily slow down the server and cause timeout on TestSwarm jobs.