vincem's WebLog : Build Topics

The Build Master book has shipped!!

vincem — Sat, 08 Oct 2005 11:13:00 GMT

Hello, it’s been a long while.

The two major reasons for my absence is the amount of time needed to get The Build Master book completed and the birth of my son! I am happy to say that both events have gone well and now I hope to have more time to blog and spend time with my baby boy and his older sister!

Since the book has officially shipped, I would like to say that it was an honor and privilege to work with an amazingly great and truly professional group of people at Addison-Wesley. The delay in getting this book published was entirely my fault. Sorry if I kept some of you waiting – it was necessary to get everything right (and all of the appropriate permissions). If anyone reading this blog is considering writing a book, I could not give a higher recommendation than going with A-W.

(to get more info or order click here)

versioning part 2 (breaking down a windows file version)

vincem — Fri, 01 Apr 2005 07:02:00 GMT

I have had a couple of good questions from my previous blog entry on versioning of which I will address the first one here and the other one in another entry.

1) What should be used for a build number?

Well I think as a rule-of-thumb a codeline in a source tree should have it's own build number and that number should start at zero when the branch is first created. If it is a build off of the main trunk then the build number should go into the <revision> field. You should keep your mainline incrementing with each build as long as it is still the tip of your source tree. I just right-clicked on kernel32.dll on my machine and went to properties, then clicked on the version tab and found 5.1.2600.2180 listed as the file version.

Let’s break this down in more detail –

5.1 is the internal version number of Windows XP which is always used in our bugtracker, if this was a Win 2k3 kernel this number would be 5.2. If it was Longhorn it would be 6.0.

2600 is the RTM(Released To Manufacturing) build number of Windows XP

2180 is the RTM build number of Windows XP SP2 – SO how could this build be earlier than the original XP RTM (2600)?

The reason why XP SP2 has a lower build number is because the SP2 codeline is a branch off of the XP mainline (the 2600 codeline) and when the branch was made the build number was reset to 0. The large number of builds is due to the fact that SP2 contains all of the SP1(I think the RTM build number was 1086 for SP1 – same branch) changes.

Confused yet? It is if you do not try to visualize what our source trees look like.

This version format is consistent with my previous post of the 4 part file version number.

For the .NET folks, remember, do not confuse or try to use this number for your assembly versions. Keep the two properties separate but make sure each assembly has both an assembly version and a file version when you ‘right-click’ on it. I got a few questions about that.

Furthermore, I do not like to see dates mixed into a version number simply because there are so many date formats out there that it can be difficult to standardize on one. Also, what if you have more than one build on a given date? How do you handle that scenario?

This leads into how to label your source tree. For the short, simple answer, use the file version that is described above. Use more than just a build number for a label. Of course this is very dependent on the source control tool you use but I have seen at several customer sites (and MS) people just using the build number when you get more value using the complete version string. This will really help you when you need to do a ‘one-off’ hotfix or a SP yourself.

I guess there is also one more point, the build number should be set before the build is fired off not when it is done. I’ll let you think about why.

some Versioning ideas

vincem — Sun, 27 Feb 2005 19:33:00 GMT

When people talk about versioning this can be a very broad subject so let’s narrow this down a little bit…

There really is only one version number that anyone should care about and that is the ‘file version’ of the files that you ship. I think this topic gets easily confused and convoluted since there are so many different ways of how this file version number (usually a four part number – explained in detail later) gets generated. Furthermore, with the introduction of the Microsoft .Net Framework and .Net assembly versions, this topic gets very confusing as people move from unmanaged to managed code.

Now if you are doing some .Net programming you will have to worry about .Net assembly versions but you should Not confuse this with the actual file version.

The two versions should not be linked together. You can use the same format described below for file versioning as for your assembly version (as this is what is recommended with the last field being a little different). Assembly versions are meant for binding purposes only. They are not meant to keep track of different daily versions – use the file version for that instead. It is recommended to keep the assembly version the same from build to build and only change it after each external release. For more details on how .Net works with these versions I will refer you to the link below:

For Assembly versioning info on MSDN look here.

For general win32 versioning info on MSDN look here

Why worry about file versioning

I think there are several reasons why having a good version scheme for your software is very important. Of all of the good reasons I think the top five are (in random order):

1) To be able to track product binaries back to the original source files.

2) To recreate a past build.

3) To avoid DLL hell – multiple versions of the same file (library in this case) on a machine.

4) To help your setup program handle upgrades and service packs.

5) So your Product Support and Q/A teams can easily identify the bits they are working with.

So how do we keep track of files in a product and link them back to the owner? How can you tell that you are testing or using the latest version of a released file?

This is a recommendation (taken from “The Build Master” book) of how to best setup the way you apply versioning to your software. There are a lot of different schemes out there and you should feel free to create your own versioning method. What is described here is what I have found to be the most effective and easiest way to do this to satisfy the top five reasons above.

Ultimately, like most of the other topics in the book, the versioning responsibility of the files in a build tends to fall into hands of the build team. The reason why may be because it is usually the build team that has to deal with the headaches that come from having a poor versioning scheme. Therefore it is in their best interest to publish, promote and enforce a good versioning scheme. If the build team does not own this then someone who does not understand the full implications of versioning will make the rules. Needless to say, this would not be very desirable for anybody involved with the product.

Microsoft Sidenote: “I’ve been slimed….”

In the old days (early nineties) in the Windows and Small Business Server group we used a very primitive source code control (SCC) tool called Source Library Manager (SLM, affectionately pronounced ‘slime’) and the labeling function was so unreliable (and poor when it did work) that we would not even bother labeling the sources after each build. But how did the developers, testers, and build team look up previous check-ins to previous builds? Well fortunately back then the projects were not as complicated or as large as they are today so this was all manageable through our release and build process. As mentioned in an earlier chapter we kept all of the sources and binaries of each build on a release server and we kept about two weeks worth of those releases. This allowed the developers, testers, and builders quick access to the previous builds without having to rely on the labeling function of the SCC tool. If someone needed files from a build other than what was on our release server we had custom tools that we wrote that would scan the sources on the source trees by date stamp and file version, copy the sources to a share, then we would rebuild those sources on that share. Since we had weekly tape backups of the release servers we could always restore past shares on the rare occasion we had reason to do this.

Bringing this forward to today (2005), Microsoft uses a very powerful SCC tool that was developed internally that handles labeling and branching extremely well – this was probably the most compelling reason to move to the new tool several years ago. If you want to adopt this tool take a look at the SCC tool in the Visual Studio Team System. This tool has all of the features of Microsoft’s in-house tool and more.

File versioning

Every file in a product should have a version number. A four part number separated by periods such as the one below seems to be the established best practice. There are a lot of variations of what each part represents so I will explain what I have found to be the best way of defining these parts.

<major version>.<minor version>.<build number>.<revision>

<major version> – usually assigned by the component owner, should be the internal version of the product, rarely changes during the development cycle of a product release

<minor version> – usually assigned by the component owner, this is normally used when an incremental release of the product is planned rather than a full feature upgrade, rarely changes during the development cycle of a product release

<build number> - usually assigned by the build team based on the build that the file was generated with, changes with every build of the code

<revision> - usually assigned by the build team and can have several meanings: bug number, build number of older file being replaced, or service pack number, rarely changes most of the time and mostly used when servicing the file for an external release

NOTE: These numbers range between 0 and 64k and can be enforced by a pre-build test tool or a build verification test.

For example

“5.1.2600.2180” is a typical file version for a Windows XP SP2 binary. Where 5.1 is the major.minor (5.0 was the version for Windows 2000) version of the product and 2600 is the build that this binary was built in and finally 2180 is the build number of the file it is replacing. Now if this was a one-off hotfix rather than a service pack release the <revision=2180> field may have the number of the bug that is being fixed such as 1000 (if this was the bug number).

Product Marketing numbers such as Windows NT 4.0.

WARNING: There seems to be a common misuse of product marketing version numbers for the major and minor numbers in the version string.

Do not use the product marketing version in the file version string since many components can be used by different products.

Furthermore, sometimes marketing numbers do not follow a sequential order or tend to be random. Now if on the occasional circumstance, the marketing people

want to use the internal version number of a product, e.g. NT 4.0 for their marketing version, then this would be ok just do not link the two numbers together in a permanent way.

Instead, for major and minor numbers use whatever numbers you use in your bug tracking database.

You should also try to avoid tools that inject version numbers into binaries (as a post-build step). Although the tools may seem reliable, they introduce an instability factor into your released binaries by hacking hexadecimal code. Most Q/A teams would have a cow if this was happening to the binaries they were testing – and justifiably so. The build number should be built into the binary or document files.

Comments, insults, or better ideas?

More details and info in “The Build Master” book that will be out in July by Addison-Wesley.

What bug tracking software does MS use?

vincem — Sat, 22 Jan 2005 22:45:00 GMT

I get asked this question a lot when I'm with customers so I wanted to add a link to ScottGu's explanation, and how customers will see this(or similarities to the in-house Microsoft tracker) in VSTS.

ok, notepad is the most popular editor but notepad2 is much cooler

vincem — Wed, 24 Nov 2004 19:24:00 GMT

I think most developers and testers will admit that when they need a 'quick-edit' tool they use notepad. If you want to use a notepad that is on steroids try this one:

Tip of the day: If you save your file using "<filename> " notepad will NOT add the .txt extension to the end of the file.

http://www.flos-freeware.ch/notepad2.html

I've set this as my default notepad (as should all build teams or add it to your common tools) courtesy of Roland's directions...

http://weblogs.asp.net/rweigelt/archive/2004/08/12/213085.aspx

Some (but not all listed) features include....

- Customizable syntax highlighting:
    - HTML, XML, CSS, JavaScript, VBScript, ASP, PHP, CSS, Perl/CGI
    - C/C++, C#, Java, VB, Pascal, Assembler, SQL, Python, NSIS
    - INI, REG, INF, BAT, DIFF
- Drag & drop text editing inside and outside Notepad2
- Basic regular expression search and replace
- Useful word, line and block editing shortcuts
- Rectangular selection (Alt+Mouse)
- Brace matching, auto indent, long line marker, zoom functions
- Support for Unicode, UTF-8, Unix and Mac text files
- Open shell links
- Mostly adjustable

Stop the Nightly Build madness (but keep Daily Builds)

vincem — Fri, 19 Nov 2004 08:12:00 GMT

I think it is pretty much common knowledge in the software world today that ‘Daily Builds’ are a good and necessary practice. I do not think that daily builds was that common a practice five years ago. I love this movement to building your product every day.

What I do not like is that some people (including some Microsofties) start their daily builds at *night* after everyone has gone home. I plead to all Microsoft writers to stop recommending that builds be run at night and instead recommend that automated tests run overnight. Builds should be run during the day – hence the name “Daily Builds’ - The main reason being that build breaks will be fixed a lot faster when people are in the office and not at home asleep. And sorry to break the news to you but build breaks will happen. It is just a fact of life. No matter how good your software or build process is, you will have build breaks - except maybe in the Windows Main Lab builds but then that's another complex story. There are ways to minimize breaks and the impact but that'll have to be another blog entry.

Many years ago in the NT Build lab we tried the nightly or overnight builds thinking that we would save a lot of time utilizing the high-power machines at night. The reality was that someone (aka: hero) on the build team ended up coming in very early in the morning to fix a build break that happened at night. Our overnight build success rate was around 15% which is totally unacceptable in any group, especially the NT Group. What we ended up doing was spending the whole morning fixing the build break and then restarting the build. The whole overnight effort actually slowed us down rather than helped us because it promoted bad development behaviors (such as no one took the build serious since it failed most of the time).

When I was in the BackOffice team we went through the same mistake; just ask my wife, she loved it when I would get a phone call at 4 or 5 in the morning (from the Build Verification Team) that something was broke in the build and they needed help getting it fixed asap. To put an end to this madness we just switched to building during the day. Sometimes we would be at the office until 11pm getting that daily build out but at least we could sleep in (until 9am) the next day because everyone was able to pick up a solid build (from the day before).

Here is roughly how this would look on a Daily basis (and pretty much how many teams build at MS):

9-10am Bug triage meetings (talk about check-ins that will be built today)

11-2pm Check-in window open (only allow check-ins into main source line during this time)

2-5pm Build product (times may vary)

5-6pm Release product to test shares

6pm-9am Run automated tests at night

As part of the daily build process, a regular build schedule should be published. This should include the last check-in time, the time that the build will be completed, and the time that initial Build Verification Tests (BVT) tests will be completed. When any part of this process is broken, the person and module that broke the build should be published via build intranet page and/or e-mail.

If you need more details or examples, wait for my book <grin>

adding Korby's blog to my excellent list (for tons of VSTF info)

vincem — Wed, 17 Nov 2004 19:44:00 GMT

Visual Studio Team Foundation (VSTF and some VSTS info)

What build tools does Microsoft use?

vincem — Wed, 17 Nov 2004 05:25:00 GMT

This is the most common question I get when talking with customers.

I think it is important to point out that Microsoft does not have specific rules that every product development team has to follow. Instead we have a lot of best practices or guidelines on what or how to run a product group. This is flexible by design - some processes or tools may work for some groups but can be a real hindrance to others. My observation has been that the Windows team tends to be the de facto leader when it comes to the best build tools and processes. Since each product is rather unique, there are some information exchanges on what works in the Windows team and then the questioning product team decides on what they can/will adopt to improve their processes. I guess the assumption is when a process works for a group with over 2000 developers (Windows), it is safe to assume that it will scale down just as effectively.

So to answer the question, currently we use in-house developed tools with a movement of adopting the VSTS tools (NOTE: some of the VSTS tools that will ship are what we have been using internally for years such as prefast) that will be available when Whidbey (Visual Studio 2005) ships (see Soma’s blog). I do not think we have any statistics on what percentage of developers use Visual Studio for their code editor but it has to be in the very high 90’s (I mean how can you beat VS’s intellisense and debugging!). For the build in the various build labs, build.exe is very popular with MSBuild.exe gaining some ground. Some groups use nmake and makefiles and some have developed their own wrappers for devenv (command-line tool for Visual Studio builds).

What should you be using?

I always tell my customers to adopt MSBuild asap. It is the future of builds at MS and a very powerful tool (see Alex’s blog).

If you insist on using devenv then you should use Andy Reeves wrapper. He is keeping the xml tags compatible with MSBuild’s so this is a nice transition tool if you are hesitant in using MSBuild. I have got a lot of good feedback on this tool.

The Build Master Book - Microsoft's SCM Best Practices

vincem — Fri, 12 Nov 2004 22:27:00 GMT

Well I know I've been lame about getting this blog started but I have been very pre-occupied with writing a build book -

"The Build Master" will be complete at the end of January 2005 and be available from Addison-Wesley in March 2005.

(IN MICROSOFT FASHION THE RELEASE HAS SLIPPED TO JULY - edited post Feb 27, sorry)

Contents at a Glance:

Source Code Control – The “Golden” Rule.
- Since it seems that the build team on a project tends to live in the source code trees and are usually the administrators of the trees, I spend a chapter talking about the best way to configure your sources.
The Build Process – The Mission Critical Assembly Line.
- The heart and soul of this book. Nine chapters dedicated to covering, in detail, how to build your product. For a more in depth overview please read the book’s introduction.
Deployment/Release – Ship It!
- This is another area that tends to spillover to the build team’s responsibilities that is covered in three chapters.
Sustained Engineering – The only sure things in life: “Death, Taxes, and Bugs.”
- If you are struggling with your build process, this usually tends to be the first area that symptoms of a failing project start to show up. Most notably everyone on the project team is in ‘reactive mode’ instead of working on new features.
The Future – How to Get There from Here
- I give some suggestions on how to influence upper management to make the correct tough decisions in your organization that help you do your job easier.
- If you are interested in the new tools that Microsoft will be releasing with the future release of Visual Studio, I touch on how to utilize those tools using the processes described in this book.

This book will be based on the 15 years I've been at Microsoft and the various Software Configuration Management roles I've been in.

Although some of the examples in the book are with Microsoft tools and technologies, I have tried to write this book in a tool agnostic way. In other words, irregardless of the tools or platforms that you develop on, you will still be able to use these processes to ship your software very effectively.

stay tuned for more...