CodeLens Team Indicators - Processing Pipeline

CodeLens Team Indicators - Processing Pipeline

  • Comments 10

Introduction

CodeLens is a heads-up display in your Visual Studio Editor where you can find information about your code in-context. See this MSDN article for an introduction to CodeLens - Get history and other info about your code.

You can use the CodeLens Indicators to quickly identify the changes that have happened on the code and the authors of these changes. You can also see the code reviews, work-items and bugs associated with the changes. In addition, CodeLens will also show you incoming changes from other branches. These CodeLens Indicators (Changes, Authors, Code Reviews, WorkItems, Bugs & Incoming Changes - collectively called Team Indicators) are computed on the Team Foundation Server. [This feature requires at least Team Foundation Server 2013 version control. If you are using Git version control, please see CodeLens for Git] When you open a file in Visual Studio 2013 Ultimate, it contacts Team Foundation Server, retrieves this data and displays them as CodeLens Indicators.

NOTE: References and Test CodeLens Indicators are computed locally and do not depend on data from Team Foundation Server.

In this blog post, I will describe the processing done to compute CodeLens data and discuss some tips to manage the CodeLens jobs and data.

 

CodeLens Jobs

Step 1 - Keepup

When the user checks in a file into Team Foundation Server Version Control, it triggers a notification which is consumed by CodeLens. CodeLens initiates a "Keepup" job to process this changeset. This job will pick each file in the changeset and using Roslyn identify the code elements which have changed in this checkin. It will also identify the workitems & bugs which have been associated with this checkin and associate them to the changed code elements. This information is serialized into a JSON format.

Step 2 - Aggregator

Another job "Aggregator" is then initiated which picks up the information created by the Keepup job, summarizes it, merges it with past data and creates the final CodeLens data file. This file contains all the information required to show the CodeLens Team Indicators for the corresponding code file. Thus there is a matching CodeLens file for each code file stored in Team Foundation Server Version Control.

Step 3 - Cleanup

Yet another job "Cleanup" runs at periodic intervals and cleans up the temporary data created by Keepup and Aggregator jobs. In order to improve performance, this job does not actually remove the files but marks them for deletion. Another Team Foundation Server job comes in and deletes the file after a specific retention period.

Step 0 - Catchup

Since CodeLens was introduced only in Visual Studio Ultimate 2013, there may be lots of changesets already existing in the system. There is another job "Catchup" which processes all changesets that are existing in the system before upgrading to Team Foundation Server 2013. This will pick each changeset in descending order and process it exactly like Keepup. This job will be disabled after all historical processing is complete.

 

Job Run Frequency

Job

Run Frequency

KeepUp

Every Checkin/Every hour

CatchUp

Every hour

Aggregator

On Demand/Every hour

Cleanup

Once a day

 

Job Characteristics

  • Keepup & Catchup jobs pick each file and tries to process it as quickly as possible.
  • An instance of each of these jobs is created for each Team Project Collection.
  • Each job processes one changeset at a time.
  • CodeLens jobs uses the existing Team Foundation Server job agent infrastructure.

 

Administration

The administration of CodeLens jobs is done using the TFSConfig command line utility. The capabilities are described in this MSDN article - CodeIndex Command.

Using this tool, it is possible to

  • Understand the current status of all jobs.
  • Turn off indexing for specific team project collections
  • Ignore a set of files or folders in CodeLens processing
  • Reindex all the CodeLens data
  • Destroy CodeLens data

 

Team Foundation Server Upgrades

When Team Foundation Server is upgraded to a new version, in some cases, CodeLens will re-initiate Catchup. This generally happens when there is a change in the CodeLens data schema. We attempt to minimize such cases, but were required to make a schema change when we introduced the Incoming Changes Indicator in Team Foundation Server 2013 Update 2 RC.

Thus you will see that catchup job is restarted after an upgrade to a Team Foundation Server 2013 Update 2 RC from an earlier version. Since catchup job is resource intensive, you may see a spike in CPU usage on the Team Foundation Server Job Agent(s) when it is running. You can use TFSConfig CodeIndex /IndexingStatus to understand progress of the catchup job.

Another side-effect of the catchup job restart during the upgrade is the buildup of temporary data. Since the catchup job is processing a large number of changesets (potentially), an equivalently high volume of temporary data is created. As I described above, the cleanup job marks all temporary data for deletion. But the data is deleted only after a retention period, which by default is 7 days. During this period of 7 days, the database size will be significantly larger than normal. If you want the temporary data to be deleted sooner, you can set the retention period to a lower value. (See Tip#2 below).

 

Tips

1. We have seen some cases where CodeLens files grow very large. The examples we have seen are all for code files generated by some tool (with a large number of code elements) and updated frequently. Since these code files and their corresponding CodeLens files have to be serialized/de-serialized, CPU utilization will be high when processing them. If you have such files, you can ignore them from CodeLens processing using TFSConfig CodeIndex /IgnoreList:add

2. The temporary data marked for deletion is deleted after a retention period which is by default 7 days. The following PowerShell snippet sets the retention period to 1 day.    [Caveat - This is intended to be run by experienced TFS administrators]

#Load Team Foundation Server Client library

[Reflection.Assembly]::Load("Microsoft.TeamFoundation.Client,Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a")


#Connect to Team Foundation Server - Replace with your Team Project Collection URI

$tpc = [Microsoft.TeamFoundation.Client.TfsTeamProjectCollectionFactory]::GetTeamProjectCollection(http://localhost:8080/tfs)


#Get the Team Foundation Registry Service

$registry = $tpc.GetService([Microsoft.TeamFoundation.Framework.Client.ITeamFoundationRegistry])


#Set the Retention Period to 1day

$registry.SetValue("/Service/FileService/SqlRetentionPeriod", 1)

NOTE: Setting the value of retention period to 1 (as shown in the script above) does not have any side effects and it is safe to do so.

 

Finally

In this blog post, I described the various jobs which compute CodeLens Team Indicators data. I also described how these CodeLens jobs can be managed. In addition, I also explained the behavior of CodeLens jobs during a Team Foundation Server upgrade. As always, looking forward to your feedback and questions on this topic and CodeLens in general.

Leave a Comment
  • Please add 5 and 8 and type the answer here:
  • Post
  • Please consider adding CodeLens to the VS Premium features.

  • Thanks for this, I'll keep it handy when I have to upgrade our TFS to 2013.

    It looks like the powershell at the end is incomplete, you might need to add another line :)

  • @Chad

    Please see the formal response at visualstudio.uservoice.com/.../4150271-make-codelens-available-outside-of-ultimate

    @Dave Shaw

    Thanks for the heads up. I have reformatted the page. You should now be able to see the full content.

  • It looks nice, but certainly not worth the huge price difference.

    I don't understand the bragging about CodeLens when it is in the ivory tower of Ultimate.  I would love to have CodeLens, but the fact that I (and most real developers) cannot, makes me less confident in Microsoft's thought process, then if it was never released.

  • شكرا

  • A;VH

  • Thanks for this post Mathew. As you know we hit the High CPU spike during a recent upgrade and the /IgnoreList is pretty useful on large files, like Linq-to-CRM.datacontext.cs files of about 8MB and huge typed datasets hidden away on old branches.

    Another very useful addition to this, is the ability to use wildcards, so if you have an Attic or Archive folder in source control, you might want to do:

    TfsConfig codeIndex /IgnoreList:add $/Project/Archive/*

    It can reduce the duration of the upgrade process quite a bit :).

    And if none of your developers are using Ultimate, or of your TFS server is sized very conservatively, then you might consider turning off indexing completely using:

    TfsConfig codeIndex /setIndexing:off

  • Thanks Jesse. Agree with your observations.

  • Thanks for the info!

    I am wondering why the catchup job is set to Disabled? On the TFS Monitoring pages this results in job executions being listed together with the failed jobs.

    Shouldn't they be set into something like HostDormant?

  • @2re

    There is an item in the product backlog to significantly improve the TFS monitoring pages. As part of that work, the team will review how disabled and failed jobs are displayed.

Page 1 of 1 (10 items)