edjez's WebLog : /etc/other

Invoking the ghost

edjez — Fri, 28 Apr 2006 23:37:00 GMT

Let’s say you are talking about a program with someone. Imagine stakes are somewhat high. Suddenly one of these two phrases gets thrown out:

1. "It could be faster"

In any but the healthiest of teams, these are great words to put folks on defensive and stop rational conversations about code. Suddenly the neurons spin (?) and we wonder- is there something we missed? Is my code wrong? Did I screw it up all over the place? Does this guy/gal have some privileged piece of information? When someone tells you "It could be faster" - what's your typical reaction…?

2. "It's for security reasons"

That's another one. And it's funny I even dare to bring this up, being from Microsoft and all that. Who would be brave enough enter a debate about code being written for such an honorable purpose? Who would be enough of a security expert to dare question the existence of code that is there to protect us from evil, shame and regret? Do you want your grandchildren to remember you for being the dude/dudette that left that gaping security hole open? That was exploited by that worm that took civilization back to the 19th century in one fell swoop? I know I don't want to be associated with an accidental return to hunter/gatherer lifestyle ...so we block the issue in our heads and move on.

What's going on here?

These are just two examples of what I call 'invoking ghosts' in discussions about software. We bring up a topic that is vague, crosscutting, obscure, to 1) provide an excuse to do work or not do work, independently of how sensible it is to do it or not 2) win a debate against The Other Idea 3) raise FUD levels and gain control of the situation or reduce others' credibility 4) basically steer the conversation away from the code and into people manipulation.

Now, don't get me wrong. Perf and security are important (Doh! I sound like an idiot....I guess you can quote me on that...No, the first phrase) Things can be faster and you could be doing things for real security measures - but how to tell? What do you do if in a conversation someone invokes these topics as ghosts, and steer it back into a productive place?

Fortunately, I think there are good patterns and tools to communicate around these issues. A huge one is to be transparent about the goals and unknowns. This basically puts on the table 1. What is it that you are trying to achieve, and 2. An acknowledgment that the understanding of the problem and the shape of the solution are still evolving.

What has helped me is to get to a shared (implicit or explicit) commitment to an approach:

1) Establish the needs & the priorities

2) Analyze current situation and set a goal for an improvement

3) Work on improving the system, checkpoint against goals

4) do it all again

This actually sounds quite obvious, but it has helped me bring sanity to discussions about quality attributes such as performance & scalability, security, manageability, maintainability, and other areas where 'ghosts' tend to live. Quality attributes tend to be particularly good 'haunted houses' because of their crosscutting, distributed nature, and because we tend to have less than ideal tools to think about tests of success.

If you look at the 4 steps above the first one is about establishing needs - figuring out your tests, if you will. There has been a lot of work for many years around helping people figure out their tests/needs for quality attributes.

For security you have Threat Modeling, for manageability you have Health Modeling, for performance you have Perf Modeling, and I hope good frames come up for other quality attributes as well. But what I particularly like about these models is that they aren't models about the solution, but models for tests (and steps 2, 3 & 4 are just process guidance to help us make them pass…the “green and refactor” of the “red, green, refactor” cycle).

In my opinion the focus on tests first for these complex areas makes these approaches and tools intrinsically suitable for expressing partial, incomplete or overlapping goals (which tends to happen whenever humans are involved..), and helping people iteratively collaborate around defining, verifying and refining them.

By starting with tests, we support an inductive approach to building the solution, akin to what you do with TDD. Now maybe we still don't have good tools to automatically test our systems based on all these testing models, but I hope that's just a matter of time (base DSL for tests that can be specialized for specific domains, anyone?). Eventually, these tools could become part of the suite of tests that drive the creation of the solution, treated as acceptance tests tied to business needs, and maybe even verifiable against systems at runtime.

…In the meantime it takes toil, tinkering, and a dabbling with mental experiments. But most of all I think it takes clear communication between people. Here’s some resources around these topics that I’ve used to help me scare the ghosts away from the conversations, and keep them productive, collaborative, and smart:

What makes a good threat model – by J.D. Meier

Performance & Scalability Checkpoint to improve your software engineering – by JD too

MSDN page having lots of things threat model (the generic portal)

Rico Mariani’s blog – Entry with the foreword of the p&p perf & scale guide with a quote of Donald Knuth quoting Tony Hoare saying ‘

Health Modeling - whitepaper on the Windows Server. They’ve done us all the favor of dumping the health model of a printer in XML. Whee! (But can you write a test runner for it?)

(Whither my simplicity model paper? Related to maintainability of code? I digress)

Kudos: It’s clear that J.D. has done a lot of work in this area. He’s the best I know at distilling these areas in significant, actionable, and insightful ways.

(OK, My cereal is going something like "Ed - W-T-F-is-this-fluffy-post? Why don't you tell us about dependency injection, aspects, executable models, Object Builder, using XAML to define types dynamically, gps, CAB running in mobile devices, workflow and UI or concurrency or whatever?" A: "I believe writing software is a social activity. I believe that if we learn to communicate, improve how we reason, become more self-aware, and question assumptions in more effective ways we will plain simply write much better software and enjoy it more along the way. Plus - it's my blog. And my doctor told me I shouldn't talk to cereal.)

Release of CAB (Composite UI Application Block) !!

edjez — Wed, 07 Dec 2005 02:38:00 GMT

And so it releases. This has been a great project and team to work with. I feel privileged to have been able to work with all the folks who made it possible, and to have made such a step forward implementing a close customer connection throughout the project.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag2/html/cab.asp

So what is CAB? It's an application block that provides common infrastructure components and a programming model for smart client applications that are composed out of views and business logic that might come from different teams or need to evolve independently. It has been architected considering many a convergence of many patterns observed in large successful customer applications and whre the platform and tools are going in the future in this space.

It contains:

WorkItems: a programming abstraction to simplify encapsulating use cases into 'contexts' that have shared state and orchestrating logic and nested recursively
Plug-in infrastructure: providers for enumerating available modules and loading them into the environment, and orchestration of the application bootstrap
Shared shell abstractions: a set of interfaces that allow logic to 'share a shell' and to facilitate separation of concerns between UI-intensive shell development and business logic development
- Workspaces - a set of interfaces that specify how to show controls in an a given area or style - such as portal, tabbed, MDI windows, etc
- UI Extension sites - named 'slots' in a shell where you want to add controls such as menus or status bar panes
- Commands - a common way of hooking up multiple UI events to a specific callback inside your application
Composition infrastructure that helps objects find each other and communicate - such as the ability to share state, auto-wire-up of pub-sub of events
A service locator/lifetime container/dependency injection++ foundation
- Built on ObjectBuilder - which allows you to extend the architecture specifying what it means to 'contextualize' an object.
A reflective architecture that you can explore to see the current state of the application, and a visualization architecture that allows architects and troubleshooters have views that exploit this reflective nature and can show you the internals of the application structure and how it's running while it's live (see the sample visualization in a separate download).

This is a 'large' app block - you will probably want to do the hands-on-labs and see the webcasts to understand the key concepts in depth. But you can get going with little effort by looking at the quicktstarts and included how-to documentation.

When the time comes around, we'll be working on showing how WinFX technologies such as Windows Workflow and WPF (Avalon) can be used in CAB applications. As always, the goal of these p&p assets is to help you build your applications in that make the best use of the existing platform and sets you up for simplified adoption of the next generation.

I hope you enjoy it. To all the customers who participated in our expert advisory board, on-site visits, workshops, webcasts and gotDotNet community - THANK YOU.

//7-12 Updated to add link to Visualization download

Of EntLib 2.0, a local AOP Workshop, and the infamous TDD article

edjez — Tue, 06 Dec 2005 07:19:00 GMT

Quickie blog post, just to get jamming:

EntLib internals webcast. Yes, EntLib and CAB ship with ObjectBuilder, our common factory & dependency injection infrastructure ++. Answering the FAQs: Yes, we have one. Yes, it's this one. Yes, you can build something like Spring with it, if you wanted to. Yes, we want your feedback and you to do cool stuff with it. And Yes, I am hoping that folks implement a creation strategy that do a transparent proxying, see my next post why. Interested in ObjectBuilder and how we externalized the config interpretation and wiring to instrumentation out of each app block? See this:
So what's this Configuration Source thing meant to be? Why do you have so many kinds of factories? How can I reuse the instrumentation capabilities in my own app? What's different in the new Configuration Designtime?
Get the answers to these and many more questions at the Enterprise Library for .NET Framework 2.0: Core Architecture webcast. It's on this coming Thursday, December 8th at 10:00am US Pacific Time / 18:00 UTC. Register now to view the webcast live (this is best if you want to ask us questions), or as always it will be available on demand from the next day

By the way, shipping EntLib on ObjectBuilder was not an easy feat, congrats to the team for pulling it off and consolidating the EntLib and CAB factory and dependency injection approach...at one point in time I started wondering..."what have I done! Include a new component as part of the foundation of something that needs to ship ASAP for what some think is a hazy fluffy ivory-tower architectural benefit! I'll be lynched!". But our p&p devs + testers rock. (Guys - if you are reading this: If you don't pull these feats, nobody can. However, if I ever have to ship a full OS, I'll keep the learning in mind).
Aspect Oriented Programming Workshop We recently held a workshop around AOP here at MS. I was glad I finally got to meet Prof. Mehmet Aksit in person (from Twente, of Composition Filter fame). I hoped I would also see Christa Schwanninger from Siemens but she's having a baby (congrats Christa & Michael!). My conclusion? To better influence Microsoft the AOSD community should 'consolidate efforts' and express the commonalities that exist in approaches. I mean, there ain't that many e.g. in runtime you'll find 3 main flavors - assembly rewriting, using dynamic languages, or using transparent proxies. Only so many pointcut expressions cover a large % of advice types, and so on. I think it would be good to have a pattern language around AOSD. The AOP alliance has most of the material needed but sometimes I feel the discussions around problem space and solution space get mixed; or that the academic efforts focus on the uniqueness of each solution (which they should) instead of the commonality (which is a lousy strategy for a researcher - to write about what everyone understands!). Mehmet was going to try to get a AO on .NET wiki started somewhere to get this dialogue around commons going, when I find out that it materializes, I'll post.
- In the meantime, if anyone from the AO community wants to define a factory for proxying, please consider doing it via an ObjectBuilder strategy. It will help others use your solution, on stuff we've built e.g. CAB. And oh, if only I got a dollar for every time I meet a customer who could have benefited from richer interception on .NET. If you want it too ping a comment here, I'll print it nail it to doors on bldg 42.
- No, I don't have MS's official stand on AOSD, so please don't ask me- there's an old obscure BillG 2nd hand quote from an Australian interview where he allegedly said MS would be embracing it. Definitively the platform is embracing more and more requirements of separation of concerns throughout, which is good. Indigo looks great in this regard. I'd like to see more of a unified approach one day. But a lot of questions are unresolved yet (when you think about trust models around assemblies, for example, who is allowed to advice what?). So I don't know what the end solution would look like. Especially when I think about crosscutting concerns and concurrency, hm. We'll see.
The infamous TDD article gets pulled Unless you have been living in a tupperware, you know there was a fallout over some article about TDD deep in the tree of MSDN VSTS content. Yes, it got removed. Here's the link, anyways. It described TDD in terms of some steps that in my opinion was just..plain..wrong...and it irked folks in the TDD community. Anyways, here are my key points late in the game:
- Sorry Jimmy! I'm so late to this flame party it's not even funny... Jimmy Nilsson pings me for comment, I start writing a post in this web form and my machine runs out of battery...there goes my IE with all my text. It would have been a good post. To your question...no, what the article described is not what we do when we say we do TDD in patterns & practices. I respected Randy Miller's reaction a lot, even though he didn't get to blog about it. And I truly beleive in TDD as a design technique.
- To MS folks...how about some internal review cycles...there's agile and TDD lists... one... email... away.
- I'm glad the community responded so vigorously. Kudos to all who by nature or effort were able to articulate a response that did not evoke a Turret syndrome diagnosis.
- I would also like to see those passionate about the subject take advantage of the situation and engage with MS in a proactive way. I mean, it's good agile approaches will be getting more visibility. As the word gets spread, it may get mangled. That's just 2nd law of thermodynamics. MS and its tools channel is a huge megaphone for the agile message. I know, many people ask, "who needs a megaphone in this era of blogs & social web 2.0 long-tail software"? Well, not everyone is in-the-know of what Michael Feathers or Brian Button are saying, or has their RSS pointing at the community gurus..so influencing an ISV like MS to help spread the word in a way the respected leaders think it's right not only will help the overall industry, but also creates a market for those engaging in it. Consider this personal estimate: probably more Microsoft customer-devs relate OOP advice to Deborah Kurata than the GoF. So what about ganging up on MS and saying "we will publish books, training, etc and furnish you at a fee with agile-community-approved guidance that the masses can take a shot at without requiring going on a 4-month spiritual walkabout with Kent Beck on some rainy hill in Oregon". I think it could be a win.

What's the news? CAB shipping, EntLib shipping, and we are off to a great start with patterns & practices in 2006 and starting to plan the wave of stuff for WinFX+ timeframe technologies.... Avalon...Indigo...Workflow...LinQ This is my 5th year in this group. Any one year is never similar to the previous one. Hope you are finding the existence of p&p as helpful as I enjoy working here.

Namasté! no, wait, that's Soma, not me. Thank you!

@ Seattle Code Camp this Weekend

edjez — Sun, 23 Oct 2005 05:11:00 GMT

Today my brother & I headed towards Seattle Code Camp. Chris Kinsman is the director & the DeVry University provides the place. The agenda is decided upon by volunteer speakers (it is a bit more structured than Foo, but I still like the un-conference feeling to it). It's nice to see the local community congregate and have such an event, and the fact that many speakers try out their skills at being up front for the same time. I was able to see many friends, present and past. I got to see Andy Dunn although I missed his talk on Shaders ("I think I scared them with math again..." he mused right after). Brad Wilson (of p&p fame) gave an intro to Ruby & Rails, and John Luif, Olaf and Kzu were around as well.

Kzu, Brad and Peter Provost & TShak have talks scheduled for tomorrow about stuff like Monad, dependency injection containers, CAB, GAT, and LINQ. Cool people, cool topics. See you around. I'm the one with the badge that says "Hello my name is Ed"

One Bad Unit Test..How many angels can dance on the head of a pin?

edjez — Sun, 07 Aug 2005 09:01:00 GMT

I was coding some stuff this last Friday, trying to clarify to myself some of the feedback we've been getting for CAB, the Dependency Injection work we've done there and we are doing in EntLib, partial specifications, and related topics. I was playing on a prototype, which inclines me strongly to write unit tests carefully and with rigor. I do this because a good set of tests will make you think hard of the semantic requirements, scopes of collaboration, and other important issues of your program. I find it that choosing carefully which test to write next is orders of magnitude more significant than making code that will pass it. With good unit tests I consider this second step almost a trivial, inductive activity. It must be weird, if you're not used to seeing TDD, to watch me staring at empty space for a while, then after 10 minutes saying "aha!" and writing a 10 liner unit test... that fails.. and then writing 3 lines to make it pass. I guess that's my quirky way of being in the zone when I am by myself (pairing around spikes and prototypes is an order of magnitude better.. I cringe when I think the hours people -including me- spend coding on their own at weird times of day, when Mon-Fri 8-5 they congregate in buildings to do...email. But I digress.)

At one point during the day there was a couple of hours when my rigor kind of went down. That is something that pairing will prevent, but I didn't have anyone to turn to (not in the war room). I reverted inch by inch to the 'old way', making some notes on paper, hacking a bunch of classes 'that you just know what they have to do' with that high-speed caffeine/sugar/videogame high feeling..that you know will you pay for later. But I was typing typing typing typing pretending all was OK and exhilarated at the apparent speed. At one point in time I wrote a quick test (failed) and changed the code to make it pass and..failed. duh! I know what the issue is..went fix the code - remove-this-here, change-this-to-here, delete-this and run test and ....fails. Huh?. I looked again, and I had changed a different class than the intended one. My change made sense anyways, it was an improvement of the semantics (not just a refactoring) but then it hit me:

I changed a different class than I intended. The tests around that class still passed.

I panicked. Not only I had coded a new subsystem with appalling test coverage, but I had left a trail of vestigial cruft out of half-thought though requirements, interfaces with just one implementation, and the one trivial unit test that was NOT breaking was waving all my bad practices in my face. So I immediately reverted to the point where I hadn't made that change and try to analyze what test would have broken by that change. Finding the test that would have failed because of my change actually was easy, but once I coded it, it was clearly not good: it was an horrible statement of intent, a disgust of a specification, a shame of OOP, an insult to Liskov, a mooning to Demeter, and all those things that make your code brittle and plain evil. I did not want my code passing that test at all!

So I decided to stop and revert and analyze what had gotten me here. As a matter of fact making the ugly test meaningless (and thinking of the meaningful test) sent me on a quest though my code to revert, delete, and clean. Which made me happy since deleting code is my favorite programming activity. It was amazing how much semantic trash had crept in through a couple of missing and bad tests. (Of course if the architecture you are working in favors good modularization and separation of concerns it helps act as a 'containment wall' against such ugliness...but I was working at the core of a very generic component without these benefits. And, to reinforce, paring would have been even better). All that infectious spreading darkness, through one little fixture done wrong. The semantic spaces that we represent in code - especially OO code - are twisty and rich, and through one apparently small mis-understanding (by error, or in my case, by omission) can cause large, unpredictable and almost emergent side-effects elsewhere.

Which reminded me of the 18th? 17th? century discussions of space and spirit- how many angels can dance on the head of a pin? The question is strange because it seems to discuss dimensions totally orthogonal to each other - matter and spiritual force, exagerated by the use of a trivial pin and respect-inducing angels. So how much meaningful insights can leak through one bad or one missing unit test? I guess... just as many.

...godspeed to the sts-114 Discovery astronauts.

p&p Community Process 2.0 is out!

edjez — Sat, 09 Jul 2005 05:57:00 GMT

We have relased the p&p Community Process to 2.0, incorporating feedback we got from customers, partners and field on the 1.0 version.

This one includes:

- A more realistic relationship between the internal process and externally visible process. In the 2.0 version now we open up the information about the project to the broad community as early as possible but still making sure we will actually have a project to ship :)
- More explanation around the options available for 'deeper' engagement - how to become an Expert Advisor and week-at-p&p, for example. This was done in an ad-hoc way before

Here's the URL. Please send us feedback if you have ideas on how we can work in a way that suits you better, or if you think having more visibility into specific aspects of our business would be valuable for you

http://practices.gotdotnet.com/releases/checkfordownload.aspx?id=60dd1bb9-0d1e-45e0-975a-a7f398697344&ReleaseId=cc9f9070-7133-4729-93dd-b088bd9a2e30

5 years of guidance groups at Microsoft! - What's your story?

edjez — Fri, 01 Jul 2005 08:02:00 GMT

5 years ago Microsoft executives approved the existence of a group focused on guidance - understanding enterprise customer scenarios, creating a dedicated engineering team to producing guidance in the form of code,patterns, frameworks and guides, and consolidating the authoritative content that enterprises would use under an identity. This team, called PAG internally, eventually became patterns & practices.

Some of us in the team put together some wiki pages with the motivations and approaches of our group, sprinkled with candid and colorful comments about how we got here, what it's like to be outside the normal product group hierarchy at Microsoft,and other interesting facts.

This is not just about explaining p&p - It's about having a 2-way dialogue with YOU -a conversation about why we exist, and areas for growth. We put it in a wiki so you can go and tell us your perspective and your own anecdotes.

The pages include things like:
- why did p&p get created, where does it live in the MS organization, and what's it like to be there?
- How do we explain internally and externally the difference between product stuff and p&p stuff?
- What are some of the things currently going on inside of our group? What tools and methodologies do we use?
- What are the observed trends and directions of our group?

http://channel9.msdn.com/wiki/default.aspx/Channel9.PatternsAndPracticesInsideOut

(We didn't have the time to make it to short but it's light reading)

I'm thrilled to work in this group and this paper is part of our commitment to be transparent and more effective at learning how to be your -our customers'- ambassador.

Most people go hiking to lakes...architects prefer hiking around:

edjez — Thu, 12 May 2005 10:04:00 GMT

Meta Lake!

Sorry. Couldn't resist.

Darth vader's Blog

edjez — Sat, 30 Apr 2005 00:23:00 GMT

Missed it when it hit /. a couple of days ago. It's hilarious. http://darthside.blogspot.com/

You know it's going to be good when Darth Vader's post title is "I Am Surrounded by Idiots"