J.D. Meier's Blog : Competitive Studies

Security Innovation Security Engineering Study

J.D. Meier — Sun, 02 Apr 2006 05:41:00 GMT

The Security Innovation Security Engineering study, Comparing Security in the Application Lifecycle - Microsoft and IBM Development Platforms Compared, is timely, given the emerging industry emphasis on integrating security in the life cycle.

My favorite quote in the study is "The patterns & practices security guidance covers the key security engineering activities better than any other resource we’ve found." I think this reflects the fact we have more than 2,500 pages of security guidance (see Security Guidance, Security Engineering, Threat Modeling, and Improving Web Application Security) , and we've integrated our guidance into MSF/VS 2005 (see MSF/VS 2005 and p&p Integration.)

The study was available from the MSDN Security DevCenter for a while but seems to have fallen off. I've summarized the study here for quick reference:

Overview
Security Innovation evaluated the guidance and tools of Microsoft's and IBM's development platforms. The study compared the support available to a development team via security guidance, documentation and security focused features in the life-cycle tool suites. Gartner reviewed the approach.

Evaluation Criteria

Coverage. How well do the provided tools and guidance cover the key set of security areas?
Quality. How effective and accurate are the tools and guidance?
Visibility. How easy is it to find the tools and guidance and then apply it to your security needs?
Usability. Are the tools and guidance precise, comprehensive and easy to use?

Ratings

Outstanding: 81-100%
Good: 61-80%
Average: 41-60%
Below Average: 21-40%
Poor: 0-20%

Scorecard Categories

Basic Platform Security. When used in accordance with its documentation, a platform should be inherently secure.
Platform Security Services. A mature platform should include services that make it easier for developers to implement security features in their applications.
Platform Security Guidance. A secure platform is much less useful if it lacks proper guidance.
Software Security Engineering Guidance. It is not possible to develop a secure application unless security is a focus during every phase of the development lifecycle.
Security Tools. A secure platform should include tools that make it easier to define, design, implement, test, and deploy a secure application.

Results of the Study

First, here's a couple key points, then the summaries are below:

Microsoft beat IBM in every category around guidance.
Microsoft beat IBM in three out of four categories around tools.

IBM

Platform Overall
1. Overall: 36%
2. Coverage: 62%
3. Quality: 70%
4. Visibility: 17%
5. Usability: 72%
Platform Security Guidance
1. Overall: 50%
2. Coverage: 81%
3. Quality: 85%
4. Visibility: 17%
5. Usability: 84%
Security Engineering Guidance
1. Overall: 25%
2. Coverage: 50%
3. Quality: 64%
4. Visibility: 17%
5. Usability: 69%
Security Tools
1. Overall: 32%
2. Coverage: 55%
3. Quality: 59%
4. Visibility: 56%
5. Usability: 63%

Microsoft

Platform Overall
1. Overall: 67%
2. Coverage: 88%
3. Quality: 85%
4. Visibility: 61%
5. Usability: 80%
Platform Security Guidance
1. Overall: 76%
2. Coverage: 93%
3. Quality: 85%
4. Visibility: 67%
5. Usability: 91%
Security Engineering Guidance
1. Overall: 78%
2. Coverage: 100%
3. Quality: 89%
4. Visibility: 67%
5. Usability: 79%
Security Tools
1. Overall: 47%
2. Coverage: 71%
3. Quality: 78%
4. Visibility: 50%
5. Usability: 68%

Quotes from the Study

Microsoft’s overall rating of 67% reflects the impressive level of focus Microsoft has applied to application security in the past several years.
IBM’s overall score of 36% is the result of a more disjointed approach to security. Security guidance is spread throughout the IBM web site and is difficult to discover.
The patterns & practices security guidance covers the key security engineering activities better than any other resource we’ve found.

More Information
For more information, see Comparing Security in the Application Lifecycle -
Microsoft and IBM Development Platforms Compared at Security Innovation's site. They created four documents that take you through the details and results: Executive Summary, Research Overview, Full Detailed Reports and Results, and Methodology.

OpenHack 4 (eWeek Labs): Web Application Security

J.D. Meier — Sun, 02 Apr 2006 04:31:00 GMT

Whenever I bring up the OpenHack 4 competition, most aren't ware of it. It was an interesting study because it was effectively an open "hack me with your best shot" competition.

I happened to know the folks on the MS side, like Erik Olson and Girish Chander, that helped secure the application, so it had some of the best available security engineering. In fact, customers commented that it's great that Microsoft can secure its applications ... but what about its customers? That comment was inspiration for our Improving Web Application Security:Threats and Countermeasures guide.

I've summarize OpenHack 4 here, so it's easier for me to reference.

Overview of OpenHack 4
In October 2002, eWeek Labs launched its fourth annual OpenHack online security contest. It was designed to test enterprise security by exposing systems to the real-world rigors of the Web. Microsoft and Oracle were given a sample Web application by eWeek and were asked to redevelop the application using their respective technologies. Individuals were then invited to attempt to compromise the security of the resulting sites. Acceptable breaches included of cross-site scripting attacks, dynamic Web page source code disclosure, Web page defacement, posting malicious SQL commands to the databases, and theft of credit card data from the databases used.

Outcome of the Competition
The Web site built by Microsoft engineers using the Microsoft .NET Framework, Microsoft Windows 2000 Advanced Server, Internet Information Services 5.0, and Microsoft SQL Server 2000 successfully withstood over 82,500 attempted attacks to emerge from the eWeek OpenHack 4 competition unscathed.

More Information

For more information on implementation details of the Microsoft Web application and configuration used for the OpenHack competition, see "Building and Configuring More Secure Web Sites: Security Best Practices for Windows 2000 Advanced Server, Internet Information Services 5.0, SQL Server 2000, and the .NET Framework"

@Stake Security Study: .NET 1.1 vs. WebSphere 5.0

J.D. Meier — Sun, 02 Apr 2006 03:15:00 GMT

I like competitive studies. I'm usually more interested in the methodology than the outcome. The methodology acts as a blueprint for what's important in a particular problem space.

One of my favorite studies was the original @Stake study comparing .NET 1.1 vs. IBM's WebSphere security, not just because our body of guidance made a direct and substantial difference in the outcome, but because @Stake used a comprehensive set categories and an evaluation criteria matrix that demonstrated a lot of depth.

Because the information from the original report can be difficult to find and distill, I'm summarizing it below:

Overview of Report
In June 2003, @Stake, Inc., an independent security consulting firm, released results of a Microsoft-commissioned study that found Microsoft's .Net platform to be superior to IBM's WebSphere for secure application development and deployment. @stake performed an extensive analysis comparing security in the .NET Framework 1.1, running on Windows Server 2003, to IBM WebSphere 5.0, running on both Red Hat Linux Advanced Server 2.1 and a leading commercial distribution of Unix..

Findings
Overall, @stake found that:

Both platforms provide infrastructure and effective tools for creating and deploying secure applications
The .NET Framework 1.1 running on Windows Server 2003 scored slightly better with respect to conformance to security best practices
The Microsoft solution scored even higher with respect to the ease with which developers and administrators can implement secure solutions

Approach
@stake evaluated the level of effort required for developers and system administrators to create and deploy solutions that implement security best practices, and to reduce or eliminate most common attack surfaces.

Evaluation Criteria

Best practice compliance. For a given analysis topic, to what degree did the platform permit implementation of best practices?
Implementation complexity. How difficult was it for the developer to implement the desired feature?
Documentation and examples. How appropriate was the documentation?
Implementor competence. How skilled did the developer need to be in order to implement the security feature?
Time to implement. How long did it take to implement the desired security feature or behavior?

Ratings for the Evaluation Criteria

Best Practice Compliance Ratings
1. Not possible
2. Developer implement
3. Developer extend
4. Wizard
5. Transparent
Implementation Complexity Ratings
1. Large amount of code
2. Medium amount of code
3. Small amount of code
4. Wizard +
5. Wizard
Quality of Documentation and Sample Code Ratings
1. Incorrect or Insecure
2. Vague or Incomplete
3. Adequate
4. Suitable
5. Best Practice Documentation
Developer/Administrator Competence Ratings
1. Expert (5+ years of experience
2. Expert/intermediate (3-5 years of experience)
3. Intermediate
4. Intermediate/novice
5. Novice (0-1 years of experience)
Time to Implement
1. High (More than 4 hours)
2. Medium to High (1 to 4 hours)
3. Medium (16-60 minutes)
4. Low to Medium (6-15 minutes )
5. Low (5 minutes or less )

Scorecard Categories
The scorecard was organized by application, Web server and platform categories. Each category was divided into smaller categories to test the evaluation criteria (best practice compliance, implementation complexity, quality of documentation, developer competence, and time to implement).

Application Server Categories

Application Logging Services
1. Exception Management
2. Logging Privileges
3. Log Management
Authentication and Access Control
1. Login Management
2. Role Based Access Control
3. Web Server Integration
Communications
1. Communication Security
2. Network Accessible Services
Cryptography
1. Cryptographic Hashing
2. Encryption Algorithms
3. Key Generation
4. Random Number Generation
5. Secrets Storage
6. XML Cryptography
Database Access
1. Database Pool Connection Encryption
2. Data Query Safety
Data Validation
1. Common Validators
2. Data Sanitization
3. Negative Data Validation
4. Output Filtering
5. Positive Data Validation
6. Type Checking
Information Disclosure
1. Error Handling
2. Stack Traces and Debugging
Runtime Container Security
1. Code Security
2. Runtime Account Privileges
Web Services
1. Credentials Mapping
2. SOAP Router Data Validation

Host and Operating System Categories

IP Stack Hardening
1. Protocol Settings
Service Minimization
1. Installed Packages
2. Network Services

Web Server Categories

Architecture
1. Security Partitioning
Authentication
1. Authentication Input Validation
2. Authentication Methods
3. Credential Handling
4. Digital Certificates
5. External Authentication
6. Platform Integrated Authentication
Communication Security
1. Session Encryption
Information Disclosure
1. Error Messages and Exception Handling
2. Logging
3. URL Content Protection
Session Management
1. Cookie Handling
2. Session Identifier
3. Session Lifetime

More Information
For more information on the original @stake report, see the eWeek.com article, .Net, WebSphere Security Tested.