What To Do When The Source Control Server Is Down

What To Do When The Source Control Server Is Down

Rate This
  • Comments 34

I have not forgotten about my series on method type inference; rather, the contrary. I have been thinking hard about how to change method type inference to be more accurate in a hypothetical world with covariant and contravariant interfaces, and this has led me to dig in even deeper to the method type inference specification and implementation. I've got acres of notes on this now; getting them into bloggable form will take more time.

Until then, some of the wit and wisdom of the managed languages team.

 We "dogfood" our source control system here; that is, we now use the same source control system that we sell to customers to control our own sources. Even better, we use recent builds of the source control system, so that we find the flaws in new versions of it before customers do. We feel the pain so that you don't have to. But this means that occasionally the Software Development Company Abstraction breaks down for a few hours here and there.

Some wags have started writing suggestions for what to do when the source control server is down on the whiteboard outside my office. So far, the list consists of:

0) Teach the VB team what index lists begin at.
1) Send email to [the source control server team].
2) Complain.
3) Update this whiteboard.
4) Learn helplessness.
5) Go to Tosche Station. Pick up some power converters.
6) Consider the management track.
7) Write a new source control system -- yes, you have enough time.

What do you do when your source control / bug database / network / email / whatever crucial system is temporarily unavailable?



  • Do you know, there is an option in Visual Source Safe to enable a shadow folder (technical details are here http://msdn.microsoft.com/en-us/library/byx8tbka(VS.80).aspx)

    In short, this folder should be on a different server (that preferably gets backed up), so in the event of the VSS server crash developers can simply obtain the latest code from the alternative location, given of course, the team makes daily/hourly check-ins.

    Since I used to be a VSS admin I am aware of this option which saved a lot of time for my team back 7 years ago when we had our VSS database server's Hard Disk failure.

    So for us any issue with the VSS availability is not a drama.

  • >> What do you do...

    I go mountain biking :-)

  • @Michael, I don't think you appreciate the scope of development at the level of Eric's group. Let's just say that the number of companies doing development at that level is very small (few dozens at most) and though Subversion may be robust under certain circumstances, it is wholly unsuitable to use in the situation of gigabytes of code being modified by literally thousands of developers and testers and migrated around through several hundred different branches on a near continuous basis.

  • No one has mentioned just letting VS go into disconnected mode for a while.   I work from home against VSS hosted at the office.  Works fine for me but then I don't get many collisions with others on the team.

    Still it makes a good excuse to read blogs like this one ;)

  • The VS IDE is tied to VSS all of the time.  Two large shops I worked at did this and forever had rookie and even experienced developers committing code that had been accidentally modified (extra blank at end of line) or had debug lines in it.  This is because the IDE (VS) will prompt you with an ALL TO EASY TO CLICK YES on the 'your code is modifeid, do you want to chedk it in?' dialog.

    The always connected forces developers to have one and only one working directory for source code (a bad bad idea if you have to do both production support on the current version and new development on a new version).  It's too easy to get the source code mixed up (I know branching may fix this but the lack of real parallel development on the trunk is a problem).

    A new developer should be able to a) pull down the source code to any directory and b) build it in VS 2008 without hardcoded paths.  That same developer should be able to a) pull down a second copy of the project to a different directory and b) build it in Vs 2008 without having to un-do the path/dirctory setup for the first copy pulled down.

    The other aspect of IDE tied to source control all of the time is that upgrading one or the other usually breaks the source control link.

    The last IDE integration with source control issue is that non-source code files are difficult to manage for business analysts, documentation people, QA people, and anyone not a develoepr that would normally commit significant files to a source code tree.

    Lastly, IDE, Source Control when combined with integrated trouble ticket/task system fall down in that a) everyone has to have a license to the IDE, b) the users entering tasks/bugs are usually not devlopers and do not want to use the IDE, c) There is no capability for the ticketing system to integrate help desk, customer support, sales support, internal users, and developers all entering and working on tickets for the same produce (i.e., the ticketing system should be easily usable and low cost to roll out to the enterprise to have decent value.  Ticketing systems limited to developers only are low productivity, low value, and force the developer to waste considerable time entering/restating/clarifiying tickets and being a first level support phone answering machine).

    I'm sorry to be hard on integrated tools like TFS in the ticketing and source control aspects but being forced to use them for source control/ticketing wasted much of my time.  

    A decent combination of an excellent IDE (VS 2008), an excellent CRM (ticketing) system, and an excellent version control system (Subversion) works much much better.

    This is  based on using RCS, CVS, Subversion on (unix and/or windows), using PVCS, Microsoft Delta, VSS (on windows), plus an ancient primitave version control system on VAX/VMS over a 20+ year timespan.

  • A two hour lunch sounds right.

  • I agree wholeheartedly with Ted.

    SCC integration in VS is nice when it works, but is a total nightmare when it does not. Performance with SourceSafe was a joke, even with SourceGear. With TFS, speed is better, but still unuseable in slow network conditions (think VPN from another continent).

    Also, having SCC info embedded in project/solution files is just wrong: it makes everything complicated, especially branching. On many occasions, I had to resolve issues by editing project files by hand.

    All in all, we got tired of dealing with integration issues and we are now manually checking out files. It makes it easier to forget to add or check-in a file, but compared to the previously lost time, it is a small price to pay.

  • I'd just like to say thanks for dogfooding source control, because this can only make it better. It would be great for MS to release an easy to use/install product taking the best concepts out there. Only by dogfooding it can you expect the market (us developers) to use it.

    My only 2 cents is don't requite it to have it's own dedicated beefy server. This has been the deal breaker in TFS. Impossible setup, requires sql server and a beefy machine. Hard to justify when VSS is free (included) and just requires a file share.

    If you build it, they will come.

  • Basic RCS with a few script is a simple but very efficient way to do version control.

  • Basic developer tasks like source code control, versioning, patch deployment, etc. seem to be downlpayed and thus quite problemmatic in many shops I've worked in.  I do not know if this is just because the majority of developers in those shops have under 5 years experience or if no one stays around long enough to dogfood their own code thorugh multiple production releases.

  • what's a source control server?

  • Dogfood your own project for awhile to appreciate following standard best practice with source code control.  

    Too often, consultant and rookie coders treat the entire project and all of the add-on parts (version control, build process, etc.) as a semester project that does not live after the developer leaves the project/company.

    Meshing with and contributing to that is the business analyst and project manager downplaying version control, documented source code, and other best practices as wasted time usually since they have never worked in a production software development shop.

    Both of these greatly inflate the overall cost of IT.

  • >Robin Goodfellow said:

    >Let's just say that the number of companies doing development at that level is very small (few dozens at >most) and though Subversion may be robust under certain circumstances, it is wholly unsuitable to use in >the situation of gigabytes of code being modified by literally thousands of developers and testers and >migrated around through several hundred different branches on a near continuous basis.

    And VSS/TFS is more suitable than subversion in this environment exactly how? Neither are suitable, if you ask me.

  • If the source control server is down I go and ask the build/source control guy why (it's a small company). Since I'm his manager I should know because somebody is going to ask me!

  • HI all,

    Sincerely, I don't have a clue about what to do because it never happened to me.  Even if my main server was down for a catastrophic crash, only thing I need to do is attach the main repo FSFS folder to another LAMP server and continue serving SVN in a matter of a few minutes.  Even if the main copy got lost, we do live backups every day so we would only loose MAX 1 day of work in theory.  In practice, it is close to NIL.  That’s what our team planned on.  There are other ways to achieve way higher up time percentages than this but we did not feel the need for it.

    This points out to an important problem.  SCM is the most single important thing in a software company as small or big it is.  There are ways to reach 99.9999% up time on any good SCM for any number of teams or employees.  It is just not something that people put in priority because the event of a real crash is so remote.  Unfortunately, it is not something achievable at a reasonable cost with TFS.  Of course, it is very achievable with SVN.

    If you or your company did not ask yourself yet what type of down time you can sustain, what type of redundancy is needed and you do not have a disaster recovery plan in place for your development team, you’re doomed...

    To Robin Goodfellow:  Please check www.sourceforge.net for a proof that SVN can support large deployments.   I guess you do not know WANdisco either or SVN hooks to create Master-Slave replication?

Page 2 of 3 (34 items) 123