Being Cellfish

Stuff I wished I've found in some blog (and sometimes did)

Change of Address
This blog has moved to
  • Being Cellfish

    Applying agile: where to start


    In the last year or so, Scrum has become very popular and people are trying to apply it left and right. I don't know if it is Scrum but the number of projects that are trying to use agile methodology have exploded in the last one or two years. And there is a problem with this. Many projects that "start being agile" are not agile at all.

    For example: imagine you're working on a project where you used to develop for 4 months and then test for 2 months and then prepare a release for one month. Your boss have heard that agile increases productivity and you start having 3 week iterations; 2 weeks of development, one week of test and prepare a release for one week. Instantly you see productivity drop 50%. And the reason is that you have just chopped up your regular work in smaller chunks and the productivity feels like it drops because you now do 3 weeks of work in 5 weeks.

    The problem with agile is that you need to understand it fully before you can implement it correctly. And this applies to the large picture (Scrum, XP) and the small picture (TDD, Daily stand-ups) - if you do not understand the purpose, you will probably do it wrong. Agile methods like Scrum will act as an catalyst helping you understand what all the building blocks are but you will still need to understand the purpose in order to be successful.

    Then we have another problem; Agile methods, especially when applied as a bundled concept like Scrum may not fit every organization. For example I heard that one company I've previously helped start using Scrum (and I think I was successful since more and more of their development teams are using Scrum even without my help) are now trying to apply Scrum to the marketing team. I'm sure their marketing team will have benefit from some of the parts suggested in Scrum but if they try to force all their work into a Scrum-like cycle I think they will need a really, really good coach to get it right.

    Yet another problem with applying agile is that it will probably be a big change in how you work and I know many developers who don't like change. These developers will try to resist and see all the problems with the new methodology and you will be working upstream in order to get them aboard. And I think there is an easier way to do it.

    So where do I start?

    There are several books you can read in order to prepare your self and try to not fall into the same pits as everyone else before you. Think that will be a future post... I have come to the conclusion that applying agile in small steps are probably the easiest way to do it. If you're going to apply agile in tiny steps I think this is the way to do it:

    1. Start having daily stand up meetings!
    I think this is the most important thing you should do, especially when you consider the cost against the positive effects of daily stand-up meetings. Communication is the key to success regardless of if you're developing games or building cars.
    2. Respect the retrospects!
    The retrospect is the time after each iteration where the team sits down and talks about what went well and what went less well in the last iteration. Things that can be improved are discussed and a plan to improve is put into place. If you have many problems (as many teams think they have in the beginning of their agile journey) you should concentrate on the top two or three problems the team perceives. Also remember to respect the outcome of these meetings! If the team perceives something as a top problem - management must let the team try to remove that problem.

    Thats it... If you do those two things correctly you will successfully implement agile into your team. If you think my list is to short you can always look at this Scrum checklist. But what about iterations? I mentioned that you should have retrospects after each iteration but that is not on the list... Well iterations are not needed to have retrospects. Just have your retrospects at a regular interval and I suggest no more than four weeks apart in the beginning. The longer the period between the retrospects the more problems will the team see and the slower change will be because after each retrospect the process used by the team will improve. If you want really quick changes in the process you should have a retrospect each week. Also iterations gives you the power of closure which should not be forgotten but if I have to choose between iterations and daily stand-ups or retrospects - iterations come in last. Also there is no point in making the list any longer because the last item, respect retrospects will over time have you implement all other good things than are considered agile.

    Finally I think there is one more thing you should do in order to quickly become successful with applying agile. Get a coach! Maybe that should be the only item on my list because everything else will come over time with help of a good coach. I did not put it on my two item list because I know many teams will not have the opportunity to hire a coach. But if your retrospective shows the team wants a coach - you should get one. If your organization does not allow you to hire an agile coach there are two other things you can do; Ask around on agile forums or get someone from outside the team come and listen to your daily stand-ups and retrospects. This person will look at the process from the outside and may in many cases come up with a few new ideas on how to improve that you have not yet thought about. This person from the outside may also say things that the team is afraid to say. This will also bring problems to the surface.

    Preemptive comment: How does Microsoft do Agile? Well Microsoft Research have one report you can read. And I will also come back to this subject in later posts on this blog.

  • Being Cellfish

    CCR vs Task Parallel Library


    I was reading this about profiling gotchas and couldn't resist the urge to download the samples from the book and implement a few of them using CCR to compare performance against TPL. I played around mostly with the parallel quick-sort. The first observation is that CCR takes a little more setup than using TPL but most of that would be something you do once in your application anyway so I would consider them similar in readability. Both implementations looked very similar and straight forward. I guess that is what I like with both TPL and CCR is that your code looks almost like regular single threaded code, but with the benefits of concurrent execution.

    While executing these two variants and tweaking thresholds I noticed that CCR almost always ended up executing 10-20% slower than the equivalentTPL implementation. Looking a little closer at what was happening I think it all made sense. First of all this was on a twin core machine so there are really only two threads running. CCR is built around message passing and the way I implemented the sort algorithm was to post a message when a part of the array was sorted signaling when all parts had been sorted. Since quick-sort uses a pivot number putting it in the right place it also means that a lot of these partial sorts were completed by just putting a single number in place. That is a lot of messages being sent. The handler also must be a run once handler that then puts itself back in the queue so that I don't update the remaining unsorted counterconcurrently. Hence a lot of time is relatively spent in the CCR dispatcher just to decide when everything is done rather than sorting. Did that make sense without a code listing?

    Anyway... My gut feeling is that TPL is more well suited for parallel computations than CCR. The code looks a little cleaner and performance is a little better. However CCR is more well suited for event based implementations where TPL is not that big of an advantage. As usual it is a matter of choosing the right tool for the right problem.

    Preemptive comment:I think I could have made my CCR implementation a little more efficient and I suspect the differences would have been smaller with more cores but I didn't test that. I'll let you do that!

  • Being Cellfish



    I'm currently using MSTest in my day to day work and I'm not always writing true unit tests but occasionally I'm also writing what I call integration tests. That is tests that uses a real database and/or web service. As such these integration tests are fragile and might fail because of the configuration on the machine running the tests or just because a service is temporarily down. Getting a test failure because of this is annoying and you really want to make sure that whoever sees the failure knows an occasional failure may be "OK". So in order to signal that the test didn't really fail but was unable to run as expected and intended I use Assert.Inconclusive. Basically if I'm relying on a web service I ping it if there is a failure. if the ping also fails I'll use Assert.Inconclusive to signal that the test can be ignored in the test run.

  • Being Cellfish

    @"Tired of Scrum"


    I got a comment here that deserved more than just a comment response. It is an interesting comment and I wish I could sit down with whoever wrote that and discuss that comment. But I guess that will never happen so here it goes. First of all I feel sorry for whoever made that comment and everybody else that have had a bad experience with Scrum. Scrum is by far the most abused and misused agile practice there is out there I think. Way too many organizations have tried to implement Scrum "because it makes all projects successful" without knowing what they're getting themselves into. Scrum is a tool that may kick-start a team into being more agile. But the team has to be open to embrace agile rather than scrum. Teams can be very agile without scrum. A very good team can probably be agile with a waterfall type of process too. Scrum has a tendency to focus on sprints and planning while the most important concept is hidden in the retrospect; the ability (and will) to improve the team's productivity. A team that is "happy with how things are" will in my opinion sooner or later experience an utter failure or just be unhappy. A team that continuously looks for ways to improve will more likely avoid big failures but more importantly; everybody on the team will be much more happy.

    The second part of the comment intrigues me. I agree that most of the bad attempts at implementing Scrum have been initiated by managers. But I think that is just bad copy cats... And I've seen Scrum being used for an excuse to micro manage. That does not make it right nor does it make Scrum beign a nice experience for anybody involved. All good implementations I've seen however have typically started with a team that decides they want to improve their situation and then supported by management since good management will see (or already knows) that happy teams are productive teams and micromanagement doesn't make anybody happy.

    One thing that I'm still wondering is; if you're a person who have been forced to try Scrum and it didn't work out well; what would you like to do? What does your perfect way of working look like?

    So while this is merely more than a long comment I would like to recommend some reading:

  • Being Cellfish

    CCR tips and tricks - summary


    So it turned out to be an advent calendar in the middle of the summer... Here is an overview:

    1, 2, 24, 25 - writing unit tests for CCR
    3, 4, 21 - causalities
    5 - SuccessFailure ports
    6 - CCR APIs
    7 - 4 ways to receive
    8 - scatter gather
    9, 10 - yielding on the right thing
    11 - result ports
    12 - problems when reusing ports
    13 - Receive with no handler
    14 - calling asynchronous APIs from CCR
    15, 16 - calling synchronous APIs from CCR
    17 - abort port
    18 - exclusive handlers
    19, 20 - throttling events
    22, 23 - adding timeouts to receiver tasks

    I'm sure there will be more tips and tricks for CCR in the future.

  • Being Cellfish

    TPL Dataflow and async/await vs CCR - part 2


    Dealing with asynchronous APIs will also be much easier than with CCR. First of all you can expect most (if not all) classes in the .Net framework to have another method added to them that is declared async. For example the Stream object used to have a synchronous Read method and then BeginRead and EndRead for asynchronous processing. In .Net 4.5 the Stream object also have ReadAsync which can be used to process asynchronously. If you however need to work with an API that have not been updated with an async method you can either apply the same pattern as with CCR by using TPL data-flow or you can just spawn and await two tasks like this:

     1: var asyncResult = default(IAsyncResult);
     2: var result = default(int);
     3: var mre = new ManualResetEvent(false);
     4: await Task.Run(() => 
     5:     {
     6:         stream.BeginRead(buffer, offset, count, 
     7:             r => { asyncResult = r; mre.Set(); }, null);
     8:         mre.WaitOne();
     9:     });
     10: await Task.Run(() => result = stream.EndRead(asyncResult));

    You could naturally wrap this in a helper method to deal with this generically exactly the same way as with CCR.

    UPDATE: First of all I was too quick to post this so there was an error in the original code. The use of BeginRead must wait on completion before completing. Second (I guess that is the price for using the latest stuff) I missed to point out that there already is a helper for making Begin/End into a task; Task.Factory.FromAsync. Last (but not least) I admit the code above is kind of stupid and yes it would block a thread pool thread. I was stuck in a pattern commonly used in CCR. Stephen Toub's first comment below show's how this should really be done using a TaskCompletionSource object. So to really learn something useful, look at that comment and the use of TaskCompletionSource.

  • Being Cellfish

    CCR tips and tricks - part 15


    In part 14 I showed you how to work with an asynchronous API from CCR. Today we'll handle synchronous code form CCR. Since one of the most common ways to work with CCR is to use a dispatcher with one thread per core you do not want to block one thread by waiting on some synchronous, long running operation. Let's create a blocking operation first:

    private void BlockingMethod(int seconds)

    Calling this from a CCR thread would in most cases be devastating for performance of your code. If this method is called multiple times from different CCR tasks you could easily have all your CCR threads blocked in this method and that is probably not what you want. But we can fix this by using a feature of the DispatcherQueue. If you create a DispatcherQueue with no arguments it will not use a default dispatcher (with one thread per core), it will use the CLR thread pool! A good pattern to use if you do not know which dispatcher your code will be used with (this might be true if you have a library of functions for others to use) is demonstrated in this code:

      1: private IEnumerator<ITask> MethodThatNeedToCallBlockingMethodFromCcr(
      2:     Port<EmptyValue> donePort)
      3: {
      4:     if (!Thread.CurrentThread.IsThreadPoolThread)
      5:     {
      6:         using (var clrTaskQueue = new DispatcherQueue())
      7:         {
      8:             yield return
      9:                 Arbiter.ExecuteToCompletion(
     10:                     clrTaskQueue,
     11:                     new IterativeTask<Port<EmptyValue>>(
     12:                         donePort,
     13:                         MethodThatNeedToCallBlockingMethodFromCcr));
     14:             yield break;
     15:         }
     16:     }
     18:     BlockingMethod(42);
     19:     donePort.Post(EmptyValue.SharedInstance);
     20: }

    There are two reasons for the guard clause in the beginning that reschedules the task on a CLR thread;

    • You do not want to reschedule if already on a CLR thread since every time you schedule a task there is a small overhead.
    • You do not want to use the CLR thread pool for all of your CCR code since using the CLR thread pool dispatcher queue has a greater overhead for each task executed than using a standard CCR dispatcher queue with a more limited dispatcher. CCR has very small overhead when running with one thread per core.
  • Being Cellfish

    Should a professional developer always use TDD?


    The answer is no - a professional developer should always use BDD.

    No seriously. I read this today and didn't hesitate a second to steal the topic since I there are two discussions around TDD I love; the discussion with people who struggle to write good code and the discussion with people who write really good code. I guess this is more about the latter. When I get into a discussion with the great developer it is often claimed that they think about testability when they write the code but then never find the time to actually write the tests. Yet there is always something in the code that could have been better if some tests were actually written; a few bugs that would probably have been found with tests or a design that is hard to test.

    So the next argument is that writing the tests slows this great developer down. Well I would never do something that slowed my team down. Even if one person can write code that is so good it is not worth the time to write tests if that person is the only one ever working on the code - I would agree that it is a bad idea to write those tests since it is a waste of time. The problem is that you do not know when that is the case. And you need to consider the full lifetime of the code (which is typically much longer than you expected) and the time of the team to work on the code. Time to first release is typically not what is interesting but what most people think about.

    So far I've been ranting about writing tests, not TDD, so what is the deal with TDD? Well I couldn't agree more with the conclusion from the fishbowl referenced in link above; a professional developer should not always use TDD but I would expect them to have tried it, mastered it and use it when appropriately. To me, that's a no-brainer.

  • Being Cellfish

    20 tips to write a good stored procedure (is really just 12)


    A few days ago there was an article with 20 tips to write a good stored procedure (requires free registration to read). The problem is that there are really only 12 good tips (and 4 bad and 4 neither good or bad). So let me go over the tips one by one and comment on them:

    1. Capital letters for keywords and proper indentation. With todays code editors with syntax high lighting I don't see why you want to high light keywords with capital letters. The code editor will do that for you. And suggesting proper indentation is not really a tip to write a good stored procedure. It's common (coding) sense! So I don't think this one counts... Score (good-not really-bad advice IMHO): 0-1-0.
    2. Use SQL-92 syntax for joins. If MS SQL server drops support for the old syntax this is good advice. Score: 1-1-0
    3. Use as few variables as possible. The article mentions cache utilization as an argument. Sounds like premature optimization to me. I'd say use as many variables as makes sense to make the code most readable. If that turns out to be a problem, then you optimize. So in general I found this advice to be bad. Score: 1-1-1
    4. Minimize usage of dynamic queries. Kudos to the article to pointing out how to minimize the bad of dynamic queries and I guess technically minimizing could mean zero but that is really the only good advice; don't use dynamic queries. So once again a bad, or at least misleading advice IMHO. Score: 1-1-2
    5. Use fully qualified names. If you don't do this you might end up with some weird behavior so this is a good advice. Score: 2-1-2
    6. Set NOCOUNT on. Good advice: Score: 3-1-2
    7. Don't use sp_ prefix. Score: 4-1-2
    8. KEEPFIXED PLAN. Learn from this article and use it correctly. Hard to argue with "learn something and use it right". Score: 5-1-2
    9. Use select instead of set. Once again performance is mentioned as a motivator. However the potential bad side effects of using select rather than set are more important in my opinion. The problem with select is that the variable might not be set if a query returns no rows and the set gives you an error if the select returns more than one row. Read more about it here. I'd definitely prefer set over select. Score: 5-1-3
    10. Compare the right thing in the where clause. This advice just confuses me. The article talks about what operators are the fastest and then refers to this page talking about preference. Even though the article is confusing on this point the basic idea is correct. For example using IN is generally faster than NOT IN. So I'll call this one a draw. Score: 5-2-3
    11. Avoid OR in WHERE clause. This is good advise for good performance. Score: 6-2-3
    12. Use CAST over Convert. CAST is SQL92 standard. Convert is not. Score: 7-2-3
    13. Avoid distinct and order by. Once again this is common sense. Don't do things you don't need... Score: 7-3-3
    14. Avoid cursors. This falls into the same category as dynamic queries to me. The only good advice is don't use cursors. Score: 7-3-4
    15. Select only the columns you need. Common sense! Score: 7-4-4
    16. Sub queries vs joins. Article lists a few good rule of thumbs. I think you should use whatever is most readable. Score: 8-4-4
    17. Create table vs select into. Article points out important differences. Score: 9-4-4
    18. Use variables instead of temporary tables. Score: 10-4-4
    19. Use proper indexes. Score: 11-4-4
    20. Use profiler. Many tips in the article suggest you do things to improve performance. But I think doing so before you know you have a problem is a waste of time and resources. So this advice is actually one of the best advices in the article. Score: 12-4-4
  • Being Cellfish

    Object Calisthenics: Rule 3: Wrap all primitives and strings


    This rule states that no parameters should be a of a primitive type (e.g. int, double) nor a string. Instead new objects should be created to represent what the parameters are. Is the integer really representing money or a coordinate. This small classes also creates great places to put methods that manipulate whatever the primitive type represents.

    When I first tried the object calisthenics this was one of the harder rules actually. Not by itself but because of the ninth rule; don't use getters or setters. Together it really makes you have to think about how to represent things in your code.

    By itself I think this rule is one rule that easily translates to your real code. Making sure you're passing objects to your methods which actually are what they represent rather than a primitive type that represent something is generally a good thing if you some day have to change the representation. E.g. money should be a double rather than an integer.

Page 2 of 49 (482 items) 12345»