Adventures in Software Engineering : ALM

Keep your SOX clean

clemmend — Wed, 19 Dec 2007 00:16:25 GMT

I have been to a few customers who have implemented or are implementing Sarbanes-Oxley (SarbOx or SOX) compliance in their development processes using VSTS. Andrew Delin from the VSTS Process team is creating a whitepaper on how to do that with VSTS. In the meantime, here are some reflections based on my personal work with this topic so far.

[The next is a PPT-like intro to the topic. For those who know what SOX, you can skip it].

What is SOX?

Federal legislation signed into law in July 2002
It requires higher accounting standards, improved trustworthiness in corporate reporting, and greater financial transparency
Two key sections of the law that have drawn the most attention
- Section 302: Requires executives to personally certify the validity of financial statements
- Section 404: Requires complete documentation of financial controls and auditor attestation to management's evaluation

Section 404

Requires “an internal control report, which shall

1) State the responsibility of management for establishing and maintaining an adequate internal control structure and procedures for financial reporting;
and
2) Contain an assessment, as of the end of the most recent fiscal year of the issuer, of the effectiveness of the internal control structure and procedures of the issuer for financial reporting.”

[end of introduction]

Ok, given this very brief summary, I can now tell you that the best general guide I have found so far to understand how to implement SOX in an IT environment is "IT Control Objectives for Sarbanes-Oxley, 2nd Edition".

This book explains the rationale for establishing the controls needed from the IT perspective, starting with SEC's own recommendation:

"Historically, assertions on control by an organization have been mostly voluntary and based on a wide variety of internal control frameworks. To improve consistency and quality, the SEC mandated the use of a recognized internal control framework established by a body or group that has followed due-process procedures, including the broad distribution of the framework for public comment. Specifically, the SEC referred to COSO".

and

"For Sarbanes-Oxley compliance efforts, it is important to demonstrate how IT controls support the COSO framework. An organization should have IT control competency in all five of the components COSO identifies as essential for effective internal control. They are:
• Control environment
• Risk assessment
• Control activities
• Information and communication
• Monitoring"

How does that relate to the normal IT framework controls that we are used to, such as ITIL/MOF, and SDLCs such as MSF for CMMI Process Improvement?

Here is a short summary plot:

SOX recommends COSO per SEC
COSO maps to COBIT (Control Objectives for Information and related Technology) standard
portions of COBIT map to relevant parts of CMMI
other parts of COBIT map to ITIL and other IT management standards

Said in this way it would seem that by implementing ITIL/MOF, and by using MSF CMMI as the standard SDLC, we would be covered in SOX compliance. This seems like a lot of overhead. However, you don't need all that, as we will see next.

SOX is about financial reporting

This was very eloquently mentioned by Dave Erickson:

“Sarbox is about assessing risk. While risk assessment is an element of ITIL, it isn’t the framework’s primary focus. Furthermore, CIOs who put ITIL or any other IT framework in place solely to comply with Sarbox will have gone overboard, says Erickson. The Sarbanes-Oxley Act requires only that companies establish controls over the systems relating directly to financial reporting. ITIL, Cobit and other frameworks for IT help companies put in place general controls for IT—a good thing to have, but much broader than the narrow scope required by law.”

So one of the first things that needs to be established from an IT perspective is a control that identifies the application being developed as impacting financial reporting. These type of applications will need to follow SOX constraints. Other types of application do not need all the overhead, especially if you are doing Agile development.

Usually SOX compliance teams will keep their own database of such applications. In VSTS it is possible to create a work item to identify those for reporting purposes. That would be the first of several work items that might be needed for SOX compliance.

So given that part of what is needed in already in the MSF CMMI template, it is clear that a few items need improvement. Remember that this just a sample of what might be needed, not a comprehensive list:

Strategic planning alignment
- One way to implement this might be to have new work items explicitly owned by upper management and/or CIO, and mapped to strategic objectives (such as in Microsoft's Development Division with Value Propositions/Experiences/Features)
Risk management process
- We need to add risk reports per project and across portfolio (slice risk management by financial management applications)
Traceability
- We need to implement reports that show traceability of work items that impact financial reporting. This will be easier to do with
  - Adding new fields to work items (such as a task work item with a tag “SOX regulation” )
  - Adding work items that have have more workflow steps to deal with regulations
SCM (as part of change management)
- Add work items that correspond to checkpoints for branching (see article by John Jacob et alii on branching guidance)
Audit trails
- Have additional reportable fields, pivoted with the SOX attribute, and provide more reports for auditors
Security
- Existing process guidance already handles part of this, but it is not enacted in tooling
- We need to implement Secure Development Lifecycle with work items as checkpoints, and corresponding work products and reports

As mentioned above, another big part of SOX compliance is covered by ITIL/MOF. I won't go into the infrastructure topics per se (see the book above for that), but there is one clear common implementation point with VSTS/TFS/MSF CMMI in security groups. Segregation of duties is mandated by SOX. However the currently default process template security groups are loosely defined, allowing persons without the proper authority to review/modify documents.

The full implementation of security model described in MSDN documentation is a solution.
Reporting needs enhancement to provide evidence of compliance showing that groups are separated in their duties.

Finally, part of SOX compliance is covered by IT Portfolio Management. Therefore, reporting needs enhancement to provide evidence of compliance using, for instance, a portfolio view of a dashboard showing compliance status. This view could used departments as pivots.

So as I mentioned above, these are just initial thoughts in a very complex topic. Andrew Delin and the VSTS Process team are working on getting more comprehensive guidance on how to tackle this subject.

Presentation on ALM foundational concepts

clemmend — Tue, 18 Dec 2007 21:46:00 GMT

I did a presentation for the VSTS Inner Circle in September 11th, and I am still getting requests for the video link and slides. Here they go:

Fundamentals of ALM

Abstract: What you should know to elevate an enterprise to an intermediate or higher level of maturity regarding SDLC and ALM. Includes discussion of the features of VSTS that enable integrated ALM, and an overview of what is coming in the next couple versions of VSTS (Orcas and Rosario).
View Recording
Recording Details
    Subject: Fundamentals of ALM
    Recording URL: https://www112.livemeeting.com/cc/microsoft/view
    Recording ID: K7K7ZZ
    Attendee Key: PFSN5?2$m

This presentation has a five minute delay to start (recording started too soon). I have asked the organizers to edit those minutes out, and I will post the link to the edited version when it is available.

I want to thank Sam Guckenheimer who co-authored an earlier version of this deck which was co-presented at TechReady 4 (an internal Microsoft conference).

Guidelines to choose your ALM pilot project and pitfalls to avoid

clemmend — Tue, 18 Dec 2007 00:17:31 GMT

Some Agile and/or ALM adoption efforts are canceled midstream because of lack of understanding of the basics of finding a suitable candidate development project. I have seen in more than a single situation that the chosen project is cutting edge in all three aspects of technology, process and people:

The technology is brand new, or new to the team, sometimes in even more than a single tier (for instance, new database software coupled with new UI development tools and a new programming language)
The development process is being changed (say from waterfall to Agile)
New people are being added to the team just after receiving their first training in the new technology

But the biggest mistake with Pilot efforts is to to use a strategic, high profile brand new project as the proofing ground for all these aspects, all at the same time. This is more common than expected. It starts as something like this:

Business has some urgent need for strategic functionality that allows them to challenge the existing technical architecture
However, the effort still has to abide by the usual existing waterfall processes that dictate that all must be done in a single pass
So the project is approved, but no cycle is allowed to try out the new tools and processes in a smaller context , and multiple changes to the environment are bundled together in an insurmountable ticking bomb that will later explode as a "death march" project.

To add insult to injury, sometimes on top of all this no proof-of-concept is ever tried with the new technology and processes (Proof-of-concept differs from pilot in that it is done in a lab environment, with no impact on business). Pilot projects do have business justification, but usually one chooses a minor project instead of betting the "jewels of the crown" on risk upon risk.

The mistake on all these lies usually in the governance management tier(PMO, office of the CIO, etc). It is usually associated with just enforcing the status quo, and it takes some brand new business need to act as a catalyst to challenge it. This governance tier should be the one to understand how to evolve their environment through carefully taken steps, and to know how to spread the risk underlying the business need into preparatory small projects (using proof of concepts and pilots) that will pave the ground for more ambitious ones.

If a governance tier is not active in doing this, the new project decays into a rogue that just reinforces the "didn't tell you so" attitude of those who see governance only as keeping IT madness in straightjackets.

Allowing this to happen is like building on moving sand: the construction might be impeccable but will collapse upon itself if it doesn't have firm ground to support the pressure of adding new layers.

The best practices for selection of a Pilot project are widely known, and for quite a long time. Here is an excerpt from a Microsoft Official Curriculum course from 1993. It is part of Course 124 - Managing the Migration to Client-Server Architectures. I modified the text to fit ALM adoption (the text in brackets [] replaces "client-server" and updates the context of other points):

"Start small - with a Pilot Project

We suggest you start your exploration of [new ALM processes and tools] with a pilot project.

Maintain excitement:
- Sponsors will lose enthusiasm
- Team members will lose enthusiasm
- Reduce risk of turnover
Need strategic answers quickly to be of value.
Avoid management problems of large projects:
- Large projects require more layers of management
- Coordination of client developers and server developers is critical
- Coordination will be much easier in a small group that talks to each other

Selection Criteria of Pilot Projects

Well defined data requirements
- Don't want to get bogged down in data analysis
- Could be existing application
- Could be part of a new application, where data analysis has been completed
Benchmark available
- If don't have, need to build in-house benchmarking capability
- Performance criteria
- Quantify savings and benefits
- Define ball-park expectation
- Use to validate tool selection
- Use for quality control in future projects
Decision support application [Business Intelligence in today's jargon] as opposed to data entry application
- More showy, if that's what's needed
- Safe place to start - it won't disrupt business operations
- Usually a simpler system
- Deliverable flexibility - keep concentration on testing the [ALM processes and tools]
Committed and supportive users
- Might be #1 critical success factor [that includes not only end users of the application in the role of product managers, but also developers, project managers and upper management]
- Willing to work with the team
- Willing to allocate the time required for the project
- Could use internal IT system so "end users" are IT
Low Cost
- Use equipment you already have [for instance, VPCs]
- Look for idle equipment [for instance, a PC with Windows XP can be a build server for a small project]
Low Risk
- It's better if this might be considered a throw-away project
  - Objective is to evaluate [new ALM processes and tools], not build an application. Concentrate on tools and platform rather than application development"
- If you need to choose a project that is mission critical, at least let it not be time-critical

As you can see, those best practices are nothing more than codified common sense, and I highly recommend you have those in mind when scoping your next ALM project.

Technorati Tags: ALM