James Whittaker is one of the most distinguished testers at Microsoft. He's written a couple of books and is a well known in the industry for his work in testing – particularly in the Security field (see this interview). James is now the Architect for Visual Studio Team System – Test Edition. And he's started blogging in the past month – check out JW on Test!
One of his posts, measuring testers, caught my eye. One of the things that I like to think about when considering performance is how much better a tester has made the product. A tester can make the product better in a variety of ways. Finding bugs in code before we ship is an obvious way and one that is easily measurable. Another thing that is fairly measurable is the quantity and quality of the regression automation that a tester writes to help us avoid bugs in existing code. Those are good ways to make the product better, but even better is the impact a tester has in the functional spec and design phases of a feature. Helping to build the right feature in the right way the first time has a much bigger impact on the product, but is also much more difficult to measure.
James' post throws another dimension to this evaluation. James' advice on evaluating testers is to 'measure how much better a tester has made the developers on the team'. What I like about this is that it is more sustainable than any of the things that I mention in the previous paragraph. Making lasting improvements through improved development practices (individual and team wide) on the team is a great thing to consider. How a tester is impacting the product in the short term is also obviously important, and I don't think a tester should come to work with an attitude of "I'm going to make my developer peer better today whether he wants to or not!" But improved developers and practices is a logical side effect to great testing.
An even better way to improve developers is for them to be a tester! Some of the best developers that I've worked with are also some of the best testers that I've worked with – whether or not they have formally been in a test role or not. I especially like a developer career path that includes a tour in the test group.
James has gotten off to a fast start on his blog – check it out!
As I mentioned in my previous post, Dynamics AX has an integrated development environment and a specialized language called X++. Inspired by the usefulness of 'xUnit' based frameworks like JUnit and NUnit, the developers for Dynamics AX 4.0 (released in 2006) implemented a unit testing framework called SysTest in the AX development environment. With this framework, AX developers can easily write unit tests for their X++ implementations – even following a Test Driven Development (TDD) approach if desired.
The purpose of this post is to bring together some resources for learning more about SysTest. Here they are:
More on SysTest later…
As I mentioned in an earlier post, I am now working on the Microsoft Dynamics Ax product team. I plan to start posting on topics closely related to the product and felt an upfront post with a brief product overview and some context would be helpful.
Like many Enterprise Resource Planning (ERP) products, Dynamics Ax has an integrated development environment for rapid application development. The environment is metadata driven and contains a specialized language called X++. Nowadays, we would call X++ and supporting metadata a Domain Specific Language, or DSL.
Microsoft delivers base business application functionality and a large network of Independent Software Vendors (ISVs) and Implementation Partners customize the application for specific customer needs. Since my previous job as a Development Lead was focused on customization, I have a great deal of interest in helping to make Ax ISVs and Partners successful.
One thing that has come through loud and clear for me in my current role as a test manager is that ERP systems are very challenging to test. First of all, their functionality is very broad. Second, there is a variety of deployment configurations that need to be considered. Third, non functional requirements like security, performance, and extensibility are paramount. Fourth, upgrade is very important since businesses can't just start over when installing a new system.
The four items listed in the previous paragraph is a starter list for what we need to worry about for the base application that we ship from Microsoft. On top of this, there is the likely possibility of ISV add-ons followed by implementation customization that tailors the product for a particular company's business needs.
One way to integrate my interest in the partner ecosystem with my current role in software test is to focus on how Microsoft can enable partners to deliver quality solutions to our mutual customers. This is something that I am focused on in addition to my internally focused responsibilities. Much of what I plan to post in the coming weeks will be related to enabling quality in the Ax ecosystem. Look for more soon…
Check out the MSDN Tester Center for helpful information for software testers and the software testing discipline! The material includes whiteboard videos, blogs, technical articles, book reviews, and a variety of other resources.
One of the great debates in the Software Test Engineering world involves the optimal mix of automated testing and manual testing. Like all good debates in software engineering, there are the religious fanatics that position themselves at either end of the argument. In pretty much all cases that I've seen, the truth lies somewhere in the middle. This debate is another one where that is the case.
In his recent article in Better Software, Jonathon Kohl does a nice job of showing how automated and manual testing can be very complementary. Check out his blog post on the topic. The post contains a link to the article.
I moved into this Test Manager gig a little over a year ago. I also stopped posting consistently to this blog about that time. Coincidence? I think not…
Another event that happened in the past year is that my team was re-org'd into the Microsoft Dynamics Ax product group. This happened last spring and we joined in the midst of the 5.0 release cycle. We immediately picked up some critical features for the release. Needless to say, it's been rather hectic.
I've learned a lot about both the testing discipline and Ax in the past few months. Those two topics, especially where they overlap, is what I want to start blogging about in the next few months. As before, I also intend to sprinkle in some Fargo related topics.
The challenge is finding the time to do it, but this post is the first step!
Since joining Microsoft almost six years ago, I have enjoyed the columns written by I.M. Wright. I.M. is the alter ego of our Director of Development Excellence, Eric Brechner. His columns always bring a smile to my face while drilling home a key point in development process.
Eric has recently written a book (I. M. Wright's Hard Code (Best Practices)) and I just realized that he is now posting his columns outside the firewall at http://blogs.msdn.com/eric_brechner/default.aspx. Take a look…
A co-worker pointed out this article from the MSN Careers site. Fargo was rated #1 in a list of "15 Great Cities for Job Seekers".
When are you going to check it out? We have openings on our team for Software Developers…
Conway's Law states that "Any piece of software reflects the organizational structure that produced it. " Further, Conway's Law is identified as a key pattern in Organizational Patterns of Agile Software Development, stating "Make sure that the organization is compatible with the product architecture". Finally, the relationship between architecture and organization is the key component of the Architecture Business Cycle described in Software Architecture in Practice.
I first came across these ideas several years ago when I was working in another organization. The idea that architecture would influence organization was pretty intuitive to me. The corollary that organization would influence the architecture was not as intuitive. As a matter of fact, it seemed flat out wrong to me. The way that a company is organized shouldn't influence a technical matter as core as a product's architecture.
But then I thought about how we made decisions on the product that I was working on. The product was built on a 'platform' built by another team. The platform released about once a year. The product that I was working on was delivered as a standard package and was used for custom applications, often releasing more than once a year. How we met feature needs for both standard and custom deliveries depended on what we could get from the platform team and when we could get it. If the feature was important and we couldn't get it into the platform due to either priority or timing considerations, then we figured out how to build it within my team's code base. Our organization was driving the technical decisions!
Does Conway's Law still apply when using agile practices? I believe it does - especially for larger product teams like we see at Microsoft. But there is definitely an added degree of coordination in order to meet the agile goal of building a system incrementally where the focus is on end to end scenarios that provide customer value. It can be very difficult to accomplish a scenario within a single iteration if it involves significant development efforts in multiple teams. But enabling the scenario in as short a timeframe as possible is still the goal.
An alternative to Conway's Law would be to iteratively organize around the end to end scenarios. The danger here is that the architecture grows in an ad hoc manner as many different teams and individuals will be mucking in different areas of the system. For example, how would you grow the data access layer functional in a way that architectural integrity would be preserved? I also think that release planning would be difficult from a resource perspective.
So Conway's Law still applies as a best practice for me, even for agile projects. But I would be very interested to hear about its application on agile projects as well as alternatives for organizing large projects that want to be As Agile As Possible.
As a follow on to my As Agile As Possible post, here's a recent blog entry from David Anderson on Large Scale Agile Development. He references another article from Scott Ambler on scaling agile, this time focused on how architecture enables scaling of agile projects.
Leveraging 'just enough' up front architecture work to enable teams to work in an agile manner makes a lot of sense to me. It's great to see more good information becoming available for this approach.
Plan Driven vs. Agile, ScrumBut, Good Agile, Bad Agile – there's a constant dialogue going on regarding what is agile, if you are truly being agile, if agile is good or evil, if you are following or violating the rules, etc. This debate especially comes into play when you move past the optimal team size of less than 10 people.
I just read an article by Scott Ambler in the March, 2006 issue of Software Development called SuperSize Me (may need subscription for access). Scott summarizes the article about applying agile processes to a larger team in a sidebar that I've quoted directly from the article:
The following strategies enable large teams to remain agile.
- Organize your large team into several small, colocated subteams.
- Model your high-level requirements and architecture early in the project.
- Deliver working software regularly.
- Coordinate via daily stand-up meetings, not status reports.
- Collaborate, collaborate, collaborate.
- Hire good people.
- Adopt common philosophies and guidance.
Put another way, be as agile as possible. Take the principles of the Agile Manifesto to heart, but determine what practices that you need to be successful given your situation. If something works, keep doing it. If it doesn't, reflect and make it better in the next iteration.
Bottom line, let common sense be your guide.
Trust is like earning interest in a bank. It can take forever to earn it, but you can lose it in a moment of spending.
I can't recall where I picked up the paraphrased quote in above – it was in somebody's blog that I read and it was in the last year. I really like the metaphor and I've found myself using it more and more lately.
I'm out in Redmond this week meeting with other Test Managers in Microsoft Business Solutions. While we did get some other work done, the biggest thing that we did was get to know each other and get a start on trusting each other. Trust is absolutely essential with multi-site development.
A couple of months ago, I was involved in a very real example of how quickly trust can be spent. A team from my site and a team from another site were working closely together on a challenging project. We had brought the teams together to build relationships and work together on the problem. Things were going fine and 'trust interest' was being slowly earned. Unfortunately, much of this trust interest was spent in a late night e-mail exchange where frustrations over a problem came out in bold, shouted, print. Just like that, the teams took several steps back and had to re-start earning interest.
Brad Appleton said it well with his first blog post, The first thing to build is TRUST! That's some very good advice.
My new role as our team's Test Manager marks the fourth team that I've managed in the past 13 years of leading software teams. The transitions have always been a time when I'm forced to step back to think about what I want to accomplish as a team leader and team member of my new team. This transition is no exception to that process.
I want any team that I am a member of to be an outstanding team. How can I help my new team get recognized as an excellent test team – one of the top test teams in our division and the company? How do we challenge ourselves to be the best that we can be?
With this context, I've been thinking about the recipe for team excellence. Here are the four key ingredients that I've been thinking about:
Great People – Any high performing team starts with a great mix of outstanding individuals. You need a team with a variety of skills, experiences, and core competencies. As the 'plucky' Minnesota Twins have shown us again this year, it's not always the team with the superstars that wins. It's often the team with the best complementary set of team members that can work well together that ultimately wins out (and not those 3rd place White Sox J). You need both talented individuals and the right mix to build a great team.
Clear Mission – A team with a clearly articulated, motivating, mission can do amazing things. JFK's challenge to put a man on the moon was a mission that NASA and the whole country could back. Team members should be able to articulate the team's purpose and why it's important to the company and its customers.
Capable Tools – Tools, including processes, that enable a team to focus on creating a great product for its customers are an important success factor. As I've learned at Microsoft, this is especially true for large scale development.
Feedback – Fundamentally, you need feedback to get better. One of the things that I like best about Scrum is the quick turnaround of Sprints. You plan, execute, review, and reflect all within 30 days. Fast, thorough feedback is especially a key factor for test teams. The more real time that information on the product is communicated, the more efficient the organization will be.
I should note that I could have stopped with the first two ingredients. A team made up of great people with a clear mission will figure out what it needs to do to be successful. But the last two ingredients are certainly worthy additions to the list.
So that's my short list of key ingredients and what I plan to focus on in my new role. Is it the right list? What's missing?
Steve McConnell has a new book out, Software Estimation – Demystifying the Black Art. While I’ve only read the first chapter, I found some interesting content.
Say you’ve estimated and planned a project to complete in 20 weeks. Twenty weeks later, your team successfully delivers the project – woo hoo! Sounds like you and your team have done a great job of estimating – but have you?
In a typical 20 week software project, how much will change? A lot in my experience – requirements change as you learn more, some staff leaves and new staff comes, priorities change and re-focus your team, and more. So how many of the estimate assumptions and inputs are still valid at the end of a 20 week project? How can a team possibly say that it accurately predicted how much it would take to complete the project?
This scenario leads to McConnell’s main point of the first chapter of his book. I quote, “The primary purpose of software estimation is not to predict the project’s outcome; it is to determine whether a project’s targets are realistic enough to allow the project to be controlled to meet them”.
Back to the 20 week project scenario, the estimate must have been a good one even though many of the assumptions and inputs have changed. Why? Because the project was able to be controlled to a successful conclusion. McConnell’s definition of a “Good Estimate” supports this –
“A good estimate is an estimate that provides a clear enough view of the project reality to allow the project leadership to make good decisions about how to control the project to hit its targets.”
Thinking about estimation like this is a subtle, yet very useful, mind shift. Think about what you are doing with your estimates. Is the accuracy of your estimate good enough to plan against? Are you planning with the appropriate amount of detail given the accuracy of your estimate? Are the estimate inputs (requirements, analysis, etc) good enough to create a ‘good estimate’ to control your project?
I would recommend this book based on the ideas of the first chapter alone, but the rest of the book also looks good. Check it out!
The weather is cooling, school has started, and the long, warm days of summer are starting to fade from my mind. It’s hard to believe that fall is upon us, but it is. It also means that my seemingly annual hiatus from blogging must end and I must get back to posting.
But before we move on, I suppose I should give a brief summer retrospective…
As usual, my family took a couple of vacations this summer. Around the 4
th of July, we went to Ely, MN. Ely is a gateway to the Boundary Waters Canoe Area and the home of
the International Wolf Center. My 10 year old daughter has been very interested in wolves and, being dog lovers, the Behind the Scenes program at the center was pretty cool for all of us.
Summer in Minnesota was warm and dry this year, leading to record low number of mosquitoes and some great lake weather. We spent a week and a few weekends enjoying the family lake cabin.
On the Fantasy baseball front, my team has been a disappointment. My
MBF powered RotoDrafter application helped me buy a team which had the best projected season in the league coming out of the auction, earning the team the name “The Paper Champions”. Unfortunately, analysis is only as good as the input data and it turns out the projections for my players haven’t been all that accurate. If they haven’t underperformed or been injured, they underperformed AND been injured. After spending most of the season near the bottom of the league, the Paper Champions ‘surged’ to the middle of the pack a few weeks ago. But I’ve been slowly sinking back down instead of making the next step up into the payout money. Oh well, there’s always next year.
Thankfully, my Minnesota Twins have continued to make the baseball season interesting. Can they hold on for a playoff spot? It’s too close to call at this point.
I started using my bike for my 23 mile round trip commute some this summer. In doing so, I managed to justify the purchase of a new bike to myself in August. I haven’t gotten to use it as much as I would like, but my
Redline Conquest cyclocross bike is a fun ride.
Oh yeah, we did manage to get some work done this summer. We continue to flesh out our Customization story and checked in some cool functionality in the milestone that we are just wrapping up. I got to demo our bits to Steve Ballmer in August – a definite summer highlight!
Finally, there’s also the new job thing that I referenced in the title of the post. I’m just now transitioning to a new role as the Test Manager for our Dynamics Tools team in Fargo. I’ve done a fair amount of testing related activities in my pre-Microsoft days, so I’m pretty excited about this opportunity. It will bring lots of new challenges as it’s a bigger team with a different set of problems.
I expect my blogging will take on a decidedly more test oriented spin in the coming months as I come up to speed on testing a large scale project. What does it take to build a great test team? Can testing be done in an agile manner? I certainly hope so as I won’t be happy if it can’t! I’ll keep you posted as I go.
