Riding Herd : WinFS

Another mini-guide update

jmazner — Tue, 13 Sep 2005 07:07:00 GMT

We heard from some attendees that they were having a hard time choosing between DAT209 (WinFS Future Directions: An Overview) and TLN306 (The .NET Language Integreate Query Framework: An Overview) in the Wednesday 1:45pm timeslot. To make it a little easier on folks, we scheduled a repeat of DAT209 for 5:15PM on Thursday. Now those who wanted to catch both sessions have a way to make it work.

Stunning PDC details revealed!

jmazner — Mon, 21 Mar 2005 01:45:00 GMT

Okay, maybe not so stunning, but in response to several requests that have come in from commenters and private email, I’m happy to be able to share a few key details that might help those of you trying to figure out your conference plans and budget for the rest of the calendar year.

Content:

As I noted previously, PDC is focused on our forward looking platform roadmap. If you’re looking for education on shipping or soon-to-be shipping Microsoft software, TechEd might be a better choice. Likewise, if you’re an IT administrator, TechEd will be a better fit for you. PDC is really geared towards developers who will be making platform technology decisions and need to understand Microsoft’s future roadmap.

So, what counts as “future roadmap technology” and thus meets the bar for presence at PDC? There are some obvious suspects that come to mind – Longhorn, Avalon, Indigo and WinFS are all technologies we covered in 2003, and we’ll be revisiting them all to show how we’re progressing against the vision and goals we talked about in 2003. We’ll likely also include discussion around some newer technologies that haven’t been discussed publicly yet.

Whidbey (Visual Studio 2005) and Yukon (SQL Server 2005), which are about to become our current generation platform technologies, likely will not have a lot of broad content. These technologies are so close to shipping that they aren’t really part of the future roadmap, they’re part of the here-and-now story. That’s why they will have tons of broad educational/training content at TechEd. What we will likely do for Whidbey and Yukon at PDC is focus on some really deep architectural and performance content, 400-level stuff that will help you understand how we intended the technologies to be used and how to get the maximum benefit from them. We'll also have some content in our pre-conference sessions.

Price:

The conference fee will be the same as last year. Full price will be $1995, but we’ll offer some discounts, including a 15% early bird discount. We expect to open registration on the main site by the end of May, and then you’ll be eligible for the early bird discount for around a month or two after that. The pre-conference sessions will have an additional fee associated with them, I'll have to check on whether we're ready to share the price for that yet.

Hope that helps with your budget and conference planning!

PDC planning and your input

jmazner — Wed, 09 Mar 2005 20:21:00 GMT

Back in January, my manager (Steve Cellini) mentioned that he’d kindly volunteered me to be Content Owner for PDC. I haven’t blogged about that yet, as I’ve been spending the past 6 weeks trying to get my head wrapped around what exactly it is that I’m supposed to be doing. I’m finally starting to get clarity – which means not that I have all the answers, but that I’m starting to get a sense of what questions to ask, at least.

Here’s the first one that comes to mind. Why do you attend to PDC? What do you hope to get out of the experience, and what makes it worth the cost of attending? We know from our attendee survey in 2003 that around a quarter of our guests said they just wanted to keep up to date on existing technology, and another two thirds said they wanted to learn about new technology.

But what does it really mean to want to learn about a technology? I can think of a few different intentions:

I want to see Microsoft’s big picture vision and demos of future apps, to inspire me in planning the next version of my own app/features.
I want to understand what the big new pieces of platform technology provide, so that I understand what new end-user capabilities I can add to my app and how to architect my app to support them
I want to deeply understand the internal architecture of these new platform technologies, so that I can evaluate whether they are robust enough to meet my app’s needs
I want to learn about how Microsoft expects its platform to be used, so that I can be sure I’m implementing best practice architecture and coding in my own apps that leverage MS technology.
I want to learn how to write code against the new platform technology, so that I can start hacking on this stuff on the plane ride home and see if I can make it do anything useful
I want to talk with my peers, architects and developers at MS and other companies, to understand how they’re approaching technology and make sure I’m not missing the next big thing.

And there are probably more beyond that. I expect most people don’t have just one answer, but it would be interesting to see how you balance these goals. Do you want to spend 10% of your time on #1, and 90% of your time on #6? An even split of 16% of your time across all 6 areas?

We’re early on in the content planning process, and this is the time for me to guide what kind of breakout sessions, labs, pre-conference sessions, etc, we’ll be providing. So, what do you want to see? If you’ve attended PDC in the past, what did you like/dislike? If you haven’t attended PDC, in favor of other industry conferences, why? If you don’t think conferences are a valuable use of your time and money, why not?

Post a comment, or better yet, blog your thoughts and send me a trackback so I can read them. I've plenty more questions to ask in the next several months if we can get a good dialogue going...

Recent statements on WinFS, "Project Green"

jmazner — Tue, 08 Mar 2005 18:51:00 GMT

Here are a couple pointers to the latest info we've made public on some ongoing development projects. I don't have any more insight to add, just wanted to make sure the links were out there in the world.

WinFS:

Comments from Product Manager Tom Rizzo in Microsoft Watch (and the requisite rampant speculation on Slashdot)

"Project Green" update from Convergence:

"Project Green," the code name for next-generation Microsoft Business Solutions' development efforts, will be delivered over the course of two waves. The first wave will occur between 2005 and 2007, and will include the release of a shared user interface based around 50 common configurable roles that people have within a company, all seamlessly integrated with Microsoft Office. Microsoft's business applications also will interoperate with service-oriented applications and include a common configurable reporting environment based on SQL Server (TM) Reporting Services and a common security-enhanced intranet and extranet environment based on Microsoft Office SharePoint® Portal Server to enable new levels of collaboration within and across companies.

The second release wave, which will begin shipping in 2008, will build on the first wave's innovation and apply a model-driven approach to business processes. Innovations released during the second wave will draw on the power of WinFX (TM) and Visual Studio® .NET.

Evangelism and credibility

jmazner — Wed, 02 Feb 2005 06:02:00 GMT

I’ve been meaning to write about this for a while, but finding Rosyna’s post (thanks, Feedster!) finally provided the motivation I needed. In reference to my WinFS post from last April, she writes “I can't help by [sic] chortle demonically as I read this”. There was another link too that caught my attention (from a comment on Scoble’s blog, can’t find it now) that basically said my credibility was shot because of that April post.

What I wrote back in April was:

“I, from my vantage as a Longhorn Evangelist, haven’t seen any changes that significantly impact anything we’ve said at PDC, or since, about the WinFX platform”

Clearly I was wrong about any changes coming ;) Did that post destroy my credibility for all time? I’d like to think it didn’t, and let me explain why. When I wrote it, I was following rules 1, 2, 15 and 16 of Scoble’s Corporate Weblogger Manifesto. Which is to say, I told the truth, I didn’t hide anything, and I did it fast. At the time when I wrote that post, it was as accurate and honest as I knew how to make it. Clearly the information has since become severely outdated, and reading it now, it looks like I was just blowing sunshine. The truth is, though, I posted the most accurate information I had available to me at the time.

When the Longhorn and WinFS plan changed, I blogged about that, too. Again I provided the best information I could in the shortest possible timeframe. And I still feel good about what I wrote in that post.

What’s become clear to me, however, is that I didn’t follow Scoble’s rules 8 and 9

8) If you screw up, acknowledge it. Fast. And give us a plan for how you'll unscrew things. Then deliver on your promises.

9) Underpromise and over deliver. If you're going to ship on March 1, say you won't ship until March 15. Folks will start to trust you if you behave this way.

I over promised, which was a screw up, and then I never acknowledged it. This post is my belated acknowledgement. And also recognition that I have since taken rule #9 to heart and been more careful about how and when I talk about what work we’re doing at Microsoft.

I’ve been a big fan of WinFS from the start. The WinFS team set out to enable a series of scenarios that are near and dear to my heart, and I was incredibly excited to be working with them. I wanted the world to know about WinFS, I wanted the world to share my excitement about WinFS, so I wrote about it with enthusiasm and passion. And, I can see now, optimism ;) I was lucky enough to work closely with the core product team, sitting in on their meetings, feature reviews, etc, and in some ways I drank the kool-aid. I caught their excitement, and didn’t temper it with the healthy dose of skepticism that should be part of any realistic discussion of a new software project. And there was plenty of skepticism around (even from other Microsofties) that could have, and should have, made me pause.

I’d think I learned from the experience. When I see something new and promising now, I approach it with a more skeptical eye. I ask myself whether I’m ready to suggest that our developer community invest time in learning about this new thing, or whether it makes sense to stay quiet until the technology (the features or the schedule) is more baked.

Given how little I’ve blogged lately, maybe I’ve erred too far on the side of caution now. I’ll keep working on finding the right balance.

What happened to WinFS?

jmazner — Mon, 30 Aug 2004 03:55:00 GMT

Friday’s announcement has left a lot of people asking what happened to WinFS? The short answer is: nothing. But a lot happened to Windows.

As I explained in my previous post, the customer and developer feedback we’ve been getting was consistent on two points: We want to start using WinFX as soon as possible, and we don’t want to require Longhorn to run WinFX applications. To respond to this feedback, the product teams had to ask themselves what they could do to tighten their schedules, and figure out how well their platform would behave on XP.

The WinFS team spent a solid couple weeks going through this evaluation. There are of course plenty of things you could do to increase the confidence level on a project the size of WinFS, since it has so many features, including:

Built-in schemas for calendar, contacts, documents, media, etc
Etensibility for adding custom schema or business logic
File system integration, like promotion/demotion and valid win32 file paths
A synchronization runtime for keeping content up to date
Rich API support for app scenarios like grouping and filtering
A self-tuning management service to keep the system running well
Tools for deploying schema, data and applications

If you cut one of these, or reduced its functionality, you could probably shorten the schedule. But I think the team concluded that the real sweet spot of WinFS is all these features delivered together, in an integrated package. The feedback I’ve heard from ISVs, certainly, is that if you take any one of these things away, you significantly diminish the value of WinFS overall.

In Bill’s interview with CNET, he talks about even adding additional features that ISVs have been asking for, such as “adding the tabular stuff and figuring out a server plan”. In his words, “The WinFS team, in terms of its progress and performance, is doing very, very good work, but it couldn't take the additional features and make an '06 schedule.”

The team also looked into what it would take to get WinFS working on XP. The biggest sticking point, as I understand, is around the file system integration. There have been several changes to NTFS in Longhorn which enable WinFS to guarantee consistency between an NTFS file and the WinFS Item which represents it. If you run WinFS on XP, you lose that guarantee, which could affect the reliability of the system (unless someone ports NTFS changes back to the XP code base, which makes for even more work to be done.)

So what happened to WinFS? Nothing. Others Windows teams concluded they could make some changes in order to deliver more quickly, and so they are accelerating and aiming to deliver to WinFX functionality on XP and Server 2003. The WinFS team concluded that neither of these was viable, so their plans are unchanged.

Actually, “unchanged” is misleading. They are updating a lot of their plans. The API went through one of Steven Clarke’s usability studies, and the API team has really listened to that feedback and come up with some new API patterns and a revised data model. I have seen the proposed changes and they are a huge improvement. The schema team has been busy taking feedback on the default, in-the-box schemas, and those are getting refined as well. The performance team has achieved tenfold gains in some areas, and they’re really just getting started with the profiling and tuning process. The file system integration team has been working closely with ISVs to tune the promotion/demotion model. The core data model team has been working with MBF to come up with a single model that will support information worker apps, PIM apps, and line of business apps. There’s a team hard at work figuring out how WinFS feature manifest themselves in the next release of Windows Server, as well as the next release of SQL Server.

The work the entire WinFS team is doing is really amazing, and I am looking forward to the day that we can share some of it back out to the community.

Do I wish we could have found a way to include WinFS in the 2006 releases announced yesterday? Yes. But am I glad that the team stayed focus on building the right thing for ISVs, and accepted the trade off is shipping in 2007? Emphatically yes.

Jon Udell's blogs turn into a cover story

jmazner — Tue, 20 Jul 2004 16:49:00 GMT

Jon Udell has turned his series of blog entries on Longhorn into an InfoWorld cover story . His editor makes the point that the blogging drove some great discussion and contributions to the final article. It’s cool to see that discussions I took part in with Jon, like here and here, had an impact on the final article. I’m also glad that after talking to Quentin Clark, Jon concluded that “If you're investing today in XML document formats, you should expect WinFS to do a good job running XPath or XQuery searches over them”. There’s a lot more from Quentin on Jon’s blog.

I think the final article came out well. I particularly like the order in which Jon lays out the WinFX pillars, since this is exactly how I do it when I give a WinFX overview:

First you need to get your data, no matter what device, server or service it lives on. You want to get to it in a secure, reliable way. That’s Indigo
Now you need a way to store it locally, so that you can find the relationships between data from all these disparate sources and turn it into useful information. That’s WinFS.
Finally, you want a way to see that information, a compelling visualization that makes sense of it all. That’s Avalon.

Jon also made the point about WinFX betting heavily on .NET as a foundation: “One thing that's not in question, however, is Longhorn's deep commitment to .Net. […] That's great news for the long-term health of Windows, the productivity of its developers, and the security of its users.” I’d agree. When ISVs ask what they can do to get ready for Longhorn, the first step is to get familiar with .NET and managed code. If you have an existing Win32 app, look into the enhanced C++/CLI support in Visual Studio 2005. If you’re starting on a new app, write it in managed code from the start.

Only one sentence in the article struck me as not quite right: “Microsoft is doing nothing to improve Internet Explorer's support for DOM, CSS, SVG, or other standard ways to enrich the browser.” It might be accurate to say that the Avalon team aren’t investing here, but read the comments of IE’s Group Program Manager, Tony Chor, over at Channel9, or read IE Program Manager Dave Massy’s blog, or take a look at the community discussion on the IE wiki. I’m pretty sure these guys are doing more than nothing to enrich the browser.

Thanks to Scoble for pointing out that the article had hit the web.

Did I misunderstand Udell's argument against WinFS?

jmazner — Tue, 15 Jun 2004 05:37:00 GMT

Several responses to my last post on Jon Udell’s entries on WinFS suggest that I didn’t get the Jon's point at all. Dare wrote “You missed his point almost completely.” And David (no blog link provided) said “Next time, try reading the article before you respond to it. Where does he contend that "full text search over XML files is good enough"?” I bumped into Alex Hopmann (another active MS blogger, I see ;) at the gym, and he said I missed the point too.

Well, David, I was just going by what Jon said his point was:

Here's the point of this installment. To the extent that our personal information stores contain information represented in XML, we have standard ways to search them.

Note also that the one example he gives (“There's no need to wait until 2007 to see what this would be like”) is all about searching over XML tags,

The same XML data will be open to the more powerful kinds of search available in the newer XML technologies now coming online: XPath 2.0, XQuery. Meanwhile, a growing number of databases are gearing up to do this kind of search efficiently, often in combination with both relational and free-text querying.

And here’s how Jon himself summed up that original entry, as a prelude to his second entry on WinFS:

I concluded that the compelling benefit of WinFS must lie in the realm of "organizing stuff" rather than just "finding stuff" -- else why not just leverage existing and well-understood relational, free-text, and XML search methods?

So let my try again. Here’s how I read Jon’s argument across his two WinFS entries:

Everybody agrees that users need help finding and organizing their data.
There are two basic ways to approach this problem

Simply full-text index all your content, to allow quick full-text search
Allow for semantic relationships between data to enable a richer search/organization experience

The first option, full-text, is pretty darn appealing: “The power of pervasive free-text search, by the way, is something that Microsoft seems consistently to underestimate”, “brute-force free-text search routinely trumps navigation and structured search”
There is added value to the relationship approach: “it's easy to state the practical benefit. If my personal information store contains items of types Person, Organization, Project, and Document, and if it knows about relationship types like Employment and Authorship, then I can easily answer questions like "Which Project X documents were written by Doug?"”
But, we’re years away from a world like (4), with consistently rich relationship data: “in practice, I wonder if anybody […] can mandate such an approach given the chaotic messiness of reality”, and quoting Joshua Allen, “real-world information is chaotic“
On the other hand, the trends do support a world where full-text is a reality, thanks to two trends: “the growing use of open XML file formats, and the steady advance of databases that can index and search XML content.”
These trends in (6) foster the following philosophy: “Let's get schematized information out into the open, where any XML-aware tool can see it and touch it and work with it”
WinFS, by contrast, “envisions a canonical set of schemas woven tightly into Longhorn”, and embraces the philosophy “Let's put schematized information into Windows, where any CLR-aware Windows application can see it and touch it and work with it”
In conclusion, “Personal information management, in Longhorn, will be a walled garden with its own notion of schema, and its own query language. To give users the benefit of finding stuff, Longhorn-style, developers will have to implement the Longhorn model. And then they'll have to find ways to unify that approach with the XML-oriented model prevailing in the world at large -- and indeed, even on pre-Longhorn Windows systems.”

I see that Dare restated Jon’s argument in far fewer words than I just did:

"If the software industry and significant parts of Microsoft such as Office and Indigo have decided on XML as the data interchange format, why is the next generation file system for Windows basically an object oriented database instead of an XML-centric database?"

But I don’t see that as what Jon was asking. He didn’t argue about a relational model vs. hierarchical model, or XPATH vs. T-SQL vs. ADO.NET or what have you. (What is an “XML-centric database” by the way? Is Yukon object oriented, relational or XML-centric?)

Jon’s point as I read it really boils down to “why are you taking this very complicated, rigid structured approach, when we can get along just fine with simple search over transparent XML files?” You see it in his second post, where he's not arguing XML vs. relational, he's arguing the value of “RDF/SemWeb“ vs. just plain simple XML files (and using Dare's words to do it.)

Let me stop here with this entry. Did I get the point? If not, help me out by providing your own breakdown of Jon’s premises and conclusions. Once I'm sure I have the argument, I'll provide some commentary on each of the presmises.

Jon Udell questions the value and direction of WinFS

jmazner — Tue, 08 Jun 2004 06:06:00 GMT

Jon Udell at InfoWorld is doing a series of blog entries on Longhorn. Feedster just discovered his first one, from Wednesday on the justifcation for WinFS defining a new way to manage metadata.

It's a well written entry, and deserves a well thought out response. I did want to get out at least one quick response into the blogsphere, though, because I think there's a misleading statement towards the end:

two powerful trends point to a brighter future for this scenario: the growing use of open XML file formats, and the steady advance of databases that can index and search XML content. WinFS embraces neither trend, and that looks to me like a looming headache. Personal information management, in Longhorn, will be a walled garden with its own notion of schema, and its own query language. To give users the benefit of finding stuff, Longhorn-style, developers will have to implement the Longhorn model.

Jon seems to have missed a few key entries in MSDN about WinFS's support for XML APIs, as well as the support for metadata handlers that copy metadata between WinFS and the filestream, precisely so that there is no walled garden -- if you're using Longhorn, you see WinFS properties, if you take the file somewhere else, you see EXIF headers or whatever other metadata format your file type supports.

XML formats with well-defined, licensed schemas, are certainly a great step towards a world of open data interchange. But XML files alone don't make it easier for users to find, relate and act on their information.

Jon's contention is that full text search over XML files is good enough, but is it really? I did a series of blog entries on WinFS scenarios back in February, and I don't think's Jon full text search approach would really enable these things. Take the simple media scenario, where I want to add background music to a movie by browsing through my media library. As it happens, I've recently started using iTunes to manage my music, and iTunes stores its metadata in “iTunes Music Library.xml“ on my hard drive. So let's say I wanted to search for jazz music to add. Here's a little snippet of what iTunes's XML format looks like for one of my jazz CDs:

- <dict>

<key>Composer</key>

<string>Lee Morgan</string>

<key>Album</key>

<string>Cornbread</string>

<key>Genre</key>

<string>MPEG audio file</string>

<key>Location</key>

<string>file://localhost/D:/files/Music/Lee%20Morgan/Cornbread/01%20Cornbread.mp3/</string>

</dict>

So what would full text search do for me over this file? If I searched for “jazz“, it would certainly show me a result for “iTunes Music Library.xml“, since that file contains many instances of the string “jazz“. It would also probably return other documents on my system that mention jazz, like emails, or papers I may have written in school. How exactly does this help me find the right piece of music to add to my movie?

To help at all, the programmer who built the movie editing program would have to add in a bunch of smarts about how to index this particular iTunes file, understanding the key/string pairs, and also recognizing that Location has a particularly special meaning. To make a really performant system, you'd probably need to make a smart indexer as well, so that if you change the genre of only song in you collection of 4000, the indexer doesn't have to recrawl the entire XML file to update itself.

Perhaps Jon's point is that the file format of the music itself should support XML, and we should replace .mp3 with a pretend new XML-media file format, .xm3. Well, now you've still got to worry about the schema definition of .xm3, and you've got to also worry about what to do if your user happens to prefer media encoded with Windows Media Player or Ogg or whatever. What you want is a common storage engine, and you want a shared schema with strongly typed metadata. That's WinFS.

I could go on more here, but wow, this was supposed to be the simple scenario! I also wrote up a more complicated event planner scenario. The key value of an event planner app is that it relates together content that would otherwise be completely unrelated. If I want to find the presention on Longhorn that I gave to Infoworld, full text search doesn't help at all, because “Longhorn“ and “Infoworld“ likely never appear together in any document. Perhaps they appear together in a calendar entry, and full text search might help me find that one entry. But then how would I find the agenda for that Infoworld meeting, or the notes I took from that meeting, or the presentation I gave?

Anyways, like I said at the start, Jon's well-written entry deserves a well-written response, but this is what I came up with off the top of my head. If we're doing a bad job about explaining the end-user benefits of WinFS, keep in mind that so far we've tried to really focus the message on developers, since it will be a while still before a home user needs to think about Longhorn. If you're a developer and you're interested, there's plenty of WinFS info up on MSDN today. Take a read through, and see if you reach the same conclusion Jon did -- “There's no question that Longhorn aims for lock-in ”.

What? They cut WinFS??

jmazner — Wed, 14 Apr 2004 05:07:00 GMT

There has been a great deal of hubub about what Jay Greene’s BusinessWeek article means for Longhorn, and WinFS in particular. I’ve been surprised by the general silence on MS blogs on this topic – which is perhaps an indication that I ought to be careful myself ;) But let me throw caution to the wind and get to the punch line: I, from my vantage as a Longhorn Evangelist, haven’t seen any changes that significantly impact anything we’ve said at PDC, or since, about the WinFX platform. You won't see me editing my slides, changing my talking points, or cutting back my demos, because the message about WinFX's capabilities hasn't changed.

The line that got everyone thinking was Jay’s comment that. “The current plan calls for the file system to work on PCs but not extend to files shared over a corporate network.” Note that although he writes this immediately following a quote from an internal mail, this sentence is not in fact a direct quote – it’s Jay’s interpretation. MaryJo Foley picked up the thread, and added her own interpretation that Microsoft is “holding off on allowing WinFS to work over a corporate network”.

Scoble weighed in here and here. But I haven’t seen any other MS comments on this (and Technorati is no help). The article is clearly causing a lot of confusion, as evidenced by the customers who have been writing to me over the past few days asking for an explanation. Part of me wishes we’d just publish the entire text of Joe Peterson’s original email that Jay drew from, as well as the latest data on what’s changed in the platform. But someone above my pay grade will have to make that decision.

Instead, let me tell you about what I learned about changes to the platform recently. My blog has been quiet the past few weeks because I was working on assembling a presentation for Jim Allchin, Senior VP of Windows. As it so happens, the subject of the review was an update on WinFX platform capabilities, with a specific eye towards whether anything we discussed at PDC might have been cut since then.

Understand, of course, that it wouldn’t be a big surprise if things we showed at PDC in October ’03 weren’t in the final product when it eventually ships in…whenever. Anyone who’s worked on a software project knows that the only thing you can say about a pre-Alpha release like the one at PDC is that it’s guaranteed to change before RTM. And it wouldn’t be the worst thing in the world to take a serious step back, realistically evaluate what could be built and shipped in the next two years, and just focus on that. That’s how you ship software.

But the good news is: I found that the vast majority of the WinFX technology we showed at PDC is still on track. I had conversations with product team leaders around the company to try to ferret out any recent cuts. We also just had a week of quarterly Longhorn Reviews, where each team presented their latest plans, including risk items, to the leadership team. And having gone through all that data, my conclusion is that the platform we showed at PDC is intact.

WinFS hasn’t been cut. WinFS hasn’t even really been scoped back. Hillel showed an amazing demo at PDC of using WinFS to build a new document experience in the shell – real time word-wheel filtering, dynamic queries and organization (aka Stacks), the CaseBuilder demo showing off relationships between items, and more. These things are all on track. Take a walk through the WinFS talks from PDC. Are we still building an extensible synch architecture for bringing data from remote servers into WinFS? Yup. Will we have a default synch adapter to help one WinFS machine talk to another? Yup.

Two questions might come to mind.

First is: what were Jay and MaryJo talking about when they said WinFS wouldn’t work over a network? I’m not entirely sure. We’re talking about one sentence from a sidebar, written by the same guy who was no doubt focusing his energy on the cover story he wrote for the same issue of the magazine. Perhaps that line got mangled, or he meant to say something else, or he just misinterpreted some snippet he saw in an email somewhere. I went back and read the Joe Peterson mail of March 19 that Jay quotes right before his WinFS corporate network quote, and the only instance of the string “WinFS” is in the paragraph where Joe talks about the core pillars of the release remaining the same.

It’s possible that Jay had some confusing information about Longhorn Server – a product whose feature set, particularly around WinFS, has not been the subject of any public comments that I know of. We recognize that for WinFS to be really interesting in the enterprise, it needs to be able to scale up to a server environment. And we know that the Longhorn client version of WinFS is not optimized for that kind of scale – it’s tuned for client scenarios instead, which makes sense. At PDC, I think we were consistent in saying that there are set of interesting enterprise scenarios which won’t be enabled until Longhorn Server comes out, and we’ll talk about the specifics of that product when we have a firm plan in place. I suppose you could hear that statement, and end up deciding that WinFS won’t work over a corporate network, but it’s the wrong conclusion to draw.

OK, so the second question that might come to mind is “well, is there anything that you did cut?” The answer is yes – features get cut all the time in a software product. We’ve had two milestones since the PDC build, which means two opportunities to find features that weren’t working as expected, or couldn’t get the right performance level, or otherwise proved to be untenable. But the scope of these cuts is so small that it’s barely worth mentioning. I really wish we’d just publish the whole darn list to stop the speculation, but since I can’t do that, let me be bold and mention at least a few by name. I’ll stick with WinFS, because it’s the area I know best (and the team I know best, which means perhaps they’ll be less likely to tell my boss to fire me ;)

One feature the WinFS team cut was around how granular you can make schema change units. Change units are a nice trick to avoid conflicts during item synchronization -- by default, the entire item is treated atomically and any change to any property on the item marks the whole item as changed. But the schema designer can break the item down into sub-units that are likely to change independently, so that only part of the item gets marked dirty. For example, the designer of the contact schema might make Business Phone and Home Phone distinct change units, since it’s likely that you only edit one number at a time. By marking them as distinct change units, you can change the Business Phone on machine A, and the Home Phone on machine B, and you won’t get a conflict the next time the two synch.

Nice feature – and it’s still shipping. What’s gone is the ability to divide complex properties (aka nested collections) into their own sub-units. By way of (an oversimplified and not accurate with our actual schemas) example, while Age might be a simple Integer property, Name is likely a complex property that actually includes Name.FirstName, Name.LastName, Name.MiddleName, etc. The cut is that you can’t say Person.Name.FirstName is in a separate change unit from Person.Name.LastName. You have to put all of Person.Name in the same change unit.

Raise your hand if you knew this was a potential feature. If your hand is up, put it down unless you planned on using it. Still up? Put it down unless the absence of this feature impacts your ability to use WinFS. If your hand is still up, let me know and explain your scenario.

Another “feature” that’s been cut: it looks like one of the internal schemas, for natural language user interface, has been dropped. I gotta say, I’m not even sure what this thing was used for in the first place, nor how a 3^rd party developer might have benefited from it, and it’s something that has come up exactly zero times in my discussions with developers over the past six months. Again, if you were counting on this, let me know why.

I hope that gives you an idea of how things are changing -- and staying the same -- in WinFX. The teams are looking at trimming out the features that might effect performance, or significantly increase the test burden, or other factors that might unreasonably delay the release of the product. But everyone is committed to keeping a viable platform with compelling new functionality for developers, and I’m feeling really confident that they’re making exactly the right choices.

A WinFS scenario from someone other than me...

jmazner — Mon, 15 Mar 2004 22:22:00 GMT

...and he's a real developer, even! Dan Crevier, an architect on the People & Groups team on Longhorn, has posted a write up of how WinFS would have helped him on his last software project.

WinFS Scenario #3: focusing your dev time on what really matters

jmazner — Mon, 08 Mar 2004 06:53:00 GMT

Wikis seem to be catching on some at Microsoft. Several product teams are using Wikis to brainstorm features and product design, or even more mundane things like recording meeting notes. I love the simple collaborative nature of a Wiki, the openness that just invites every ready to contribute and respond to one another.

One thing I don’t like about Wikis is how they work when I’m on offline -- which is that they don’t work at all. All the goodness of that collaborative web of data is trapped up on the server, and unless I’ve got a connection back to the corporate network, I can’t see any of it. Even when I am online, if I’m want to edit several entries, it slows me down to have each click launch a round trip, and wait for the server to return with the next page.

Others have observed the same shortcomings, and have started Wiki client projects here and there. JHorman seems to have wanted the disconnected Wiki experience so much that he built a personal Wiki-like notepad app. These are all neat client apps, and I’m glad to see them on Windows today, but I suspect they required a lot of code to handle storing the data on the local client.

So for my third not-bad WinFS scenario, I figured I’d taking an existing scenario – the local Wiki client – and talk some about how the WinFS platform would simplify the development process.

At the core of FwSynch and WikidPad must be some sort of data engine. You’ve got a collection of distinct Wiki entities, and a web of keyword relationships between them. How do you persist them on the client? How do you track changes on the client and synchronize them with changes on the server? Will your storage architecture be performant -- how long will it take to boot the app and display an entry? Will the performance scale as the number of entries end edits reaches the hundreds, or thousands? Is your storage format able to recover if a small portion of data becomes corrupted?

There are plenty of options for client storage today, of course. The simplest solution might be to store all the data in a big XML file. Maybe you're more comfortable with relational DBs than with XML's hierarchical storage concepts, so instead you redistribute a DB engine like MSDE or SleepyCat’s Berkeley DB. Maybe you've got the right skill set to build your own custom storage engine. Each of these has its own tradeoffs across development time, feature set, and performance characteristics.

But, no matter what you choose, you will end up spending development time on something that is, in all likelihood, not a competitive differentiator. If you were building a competitor to WikiPad, where would you focus your energy on creating a different and more valuable product – would it be in tuning the performance of the storage engine, or would it be in improving the UI, or editing experience, or adding new features? If you were building a competitor in the music player market, going up against iTunes, MusicMatch and WinAmp, etc, would your distinguishing features be around visualizations, skinning, and play list management, or would they be around storage optimizations?

I’d argue that for most desktop applications, performance of the storage engine is generally not a competitive advantage – you’re not going to sell users on switching to your software just because it’s 10% faster than the other guy. Performance can be a disadvantage, if you’re the slowest app in the pack, but the goal in that case is just to catch up to everyone else so that you can get the same check mark in the performance column as all your competitors.

There are, of course, some apps whose main value-add really is performance – if you’re building a client search app like X1, then it’s probably worth spending a lot of dev cycles to eke out a 10% perf gain over your competitors. But for apps that manage music, photos, or Wiki entries, I think the real-value add is in the features and user experience, not the perf. I’m looking forward to reading everyone’s opinions in the comments, but really, have you heard of anyone switching from Quicken to MS Money (or vice-versa) because checkbook register look-ups happen a few milliseconds faster?

So how does WinFS help out? It provides an integrated storage engine that’s built into the platform, and which has been tuned for client performance by a dedicated team of storage experts. It's has a schema and data model that's built specifically to store relationships between items. As a core part of the WinFX platform, you can expect there to be a good set of tools from Microsoft and others to help tune the performance to suit your particular application. And if your app has to synchronize with data from a server, WinFS is built from the schema on up to support the notion of efficient synchronization, and has an extensible provider model for communicating with whatever your server back-end may be.

All in all, less time spent on developing storage infrastructure, and more time spent on building the real value-add features of your next great application. This isn’t quite as exciting as the last two scenarios, since they really focused on adding those killer new features, but I’m hoping that a lot of people will indeed breathe a sigh of relief when there’s a decent storage engine in the Windows platform, and they can turn their focus to more important matters.

WinFS designers on .NET show -- what would you ask?

jmazner — Tue, 24 Feb 2004 06:51:00 GMT

Robert Hess tells me the next episode of The .NET Show on MSDN will cover WinFS. He's interested in feedback on what topics you'd like to see covered.

His guests will be: Mike Deem, Quentin Clark, and Anil Nori.

What would you ask these guys about -- the philosophy behind WinFS? How they design the schemas, data model or API? Questions around performance or security? Architectural guidelines on how to best leverage WinFS in your apps? What you can do to prepare your apps today to use WinFS?

WinFS Scenario #2: event planning

jmazner — Fri, 20 Feb 2004 07:18:00 GMT

For my second not-bad WinFS scenario, I’m going to again avoid anything that requires a bunch of new work entering or editing meta-data. We’ll get to those eventually, but I want to emphasize that WinFS can be useful without requiring the average user to suddenly become a meta-data freak.

In my job as evangelist, I’m responsible for a lot of events. Sometimes they’re bigger, like getting 150 people together for a design review. Sometimes they’re smaller, like the panels at PDC. Or maybe it’s something as relatively simple as getting a few program managers together to go brief a customer on Longhorn.

There are also event organization tasks that come up outside of work. Two years ago, I became the CTO for the Mazner-Emmerman wedding, which seemed like a full-time job some days – invite lists, thank-you cards, seating charts, the works. Over the summer, my wife and I are planning a vacation with a few of her high school friends – that means coordinating calendars, picking a house to rent, lining up some activities while we’re there, etc. We’re also doing some work on the front porch of our house – emails with the architect, emails from the contractor, floor plans, bills and more.

What all these events have in common is that I am responsible for keeping track of them. The relevant information I have to track includes:

People involved in the event
Date, time, duration and location of the event
Agenda
Presentations
Tasks
Discussions about all of the above

Where do I track all this information today:

People: I might look up people in the corporate CRM database if I need to invite customers, in the Microsoft directory if I’m inviting employees, or in my Outlook or Hotmail contacts if I’m inviting friends. If there are a lot of people and I need to track RSVPs, I end up with a spreadsheet of names and status.
Date/time: I store this in Outlook for my own tracking. If not too many other people are involved, I’ll send out a schedule request from Outlook so that they can have it in their calendars as well, plus I can track responses. If there are more than 25 people involved, I post the event to a calendar on the Microsoft intranet, or for customers, on an extranet or public web page (and then everyone else has to copy that info to their own calendar).
Agenda: Usually a text document or spreadsheet, which I might save on a file share, or maybe post to a web site
Presentations: Powerpoint, of course. A bigger event will have multiple presentations authored by multiple people, so we create a dedicated file share or Sharepoint library for them
Tasks: Maybe in Outlook, maybe in Sharepoint, or maybe in a Project file if the set of tasks is complex enough
Discussions: email. Lots and lots of email.

So as you can see, the information for any given event is spread all over the place, which means I can’t really keep track of it all. If I want to see the invite list and RSVP status, I’m either in Outlook (if I’m lucky) or off to some file share to find the spreadsheet. If I want to see some information about an invitee (their phone number, or who their account manager is), it’s off to the directory or CRM system to look it up. If I want to know what presentations have been approved for use, I crawl through email to find the one message from my general manager where he says the presentation is ready. If I want to see the actual presentation, it’s back to a file share or Sharepoint.

What I really want is a way to corral all these related items together, stick them in a big bucket with a label that says what event they’re for. I want a simple UI where I can see what events are coming up, then browse through all the related material for each one, and maybe be able to answer some simple questions: how many presentations for this event are in the Approved state? How many attendees have declined the invitation?

I’ll assert that this is really, really hard to do today. Outlook wizards would probably argue that you could do this with some series of catagories, public folders, and shared calendars. SharePoint gurus would say this is exactly what a Meeting Workspace is for. Old school event planners might claim you could track this all in a nice big spreadsheet with multiple pages and links out to file locations.

Yeah, with enough work, you could meet my needs with one of these models, but it would just be so unnatural. When I get email, I want to manage and store it in my email application, Outlook. I don’t want to have to copy every email up to a Sharepoint server and figure out which doc library it should live in. When I’m working with files, I want them somewhere that’s easy to access from Word’s File Open dialog, not off in some crazy Exchange Public Folder. I like keeping track of my contacts in Outlook, not in a bunch of columns in a spreadsheet. Maybe you’d argue that you could leverage the Outlook, or CDO (or even MAPI, if you’re a masochist) API to extract a lot of this information into a new app. Okay, fine. But now make it work if I’m using Lotus Notes as my PIM. Or Hotmail. Or AOL Communicator.

The solution I want will let me continue to edit each type of content in the application that’s best suited to the job, and with which I’m familiar. I’m okay with a separate app that I use to manage the overall project, but when I find a presentation, I want it to open in PowerPoint, where I should be able to edit and save without having to do anything special.

What does WinFS provide that will help?

A common storage engine, one unified namespace for storage of any application data on your machine. Whether I use Outlook, AOL Communicator, or Notes, my emails can all be stored in WinFS. (Yes, I understand that new versions of these apps will have to built…encouraging that is my job as evangelist.)
A set of common schemas, so that an email is an email is an email, no matter what app created it, and it always has a To:, From: and Subject: that I can access through the same API.
A data model that supports relationships, so that my event management app can specify “this email from Bill is related to the Longhorn Design Review event, as is this calendar appointment for next month”
A data model that supports extensions and meta-data on relationships, so that I not only say “this contact Jon is associated with this Design Review event”, but also “Jon is a speaker at this event” and “Jon is the author of this deck that he’ll present” and “Jon has not yet confirmed attendance at the party afterwards.”
Win32 file system access, so that even though files are stored in WinFS, applications can still get to their streams via a Win32 path

My dream app would have a left pane with a list of events. When I clicked on an event, the right pane would open up to show me all the documents, people, events, tasks, email, etc, associated with that event. When I had new content to add, I could just drag and drop it onto the event – it wouldn’t copy it, but simply create a new relationship to it.

If you saw the CaseBuilder demo in the PDC keynotes, you have some idea of what I’m getting at here. If you didn’t see CaseBuilder, I have a video of it that I’m going to try to get posted somewhere publicly soon.

So, not the simplest of scenarios to explain, but I hope I did an okay job, and that you agree it would be useful – without requiring people to add a ton of new meta-data. The kinds of meta-data I need to know – “contact Jon is invited to event Design Review” is information I already have to enter and track today, whether in an Outlook meeting request or in some spreadsheet column.

Feedback on scenario #1 and the meta-crap challenge

jmazner — Fri, 20 Feb 2004 06:31:00 GMT

Before going on to my next scenario, I wanted to address feedback I’ve gotten so far.

On the general usefulness of WinFS, Stephane Rodriquez commented that “It's not about information, it's about MESSAGE.” You can quibble with my language, but we both mean the same thing – you need a simple way to get at the stuff that’s important to you, when you need it. We showed a demo at PDC of trying to find documents written by Bill about Longhorn. Using the Longhorn document library, which exposes a simple query interface on top of data stored in WinFS (yes Michael, I agree that a great user experience is what really makes WinFS shine,) it takes two clicks (one to stack by Author, a second to stack by Project) to go from 1100 documents down to just the 40 or so where Author=Bill and Project=Longhorn. Or to stick closer to my first scenario, two clicks to find music where Genre=Jazz and Mood=Mellow or what have you.

Both Mario and Harold (in comments and in his blog, respectively) want to see an easy way to get meta-data out of WinFS for use by other systems. Right on – that’s why the system includes an extensible demotion engine to write meta-data back into the filestream as appropriate. Of course you can also query for meta-data using SQL, if you want, and even request the results in XML format.

Regarding my first scenario, I’ve got Dare agreeing that “scenario is compelling”, although he suggests “that there doesn't even need to be explicit inter-application sharing of data” to make the scenario work. Along the same lines, Simon comments that “I can already do this by selecting details view and letting explorer show me all of the meta-data stored *inside* my mp3 files.” It’s certainly true that for this scenario, you don’t need WinFS to get access to the meta-data for an individual file. But you do need WinFS to get to it in a consistent way across multiple kinds of items, stored across multiple folders, and to do it fast. Mario addresses some of this in his comments, but I’ll be more explicit.

Let’s say you use iTunes to purchase AC3 files, you’ve got some old MP3’s lying around, a collection of WMAs, and maybe some Oggs as well. And they’re spread across a hierarchy that’s 10 folders wide and 10 deep. Now you’re in Movie Maker, and you want to find that mellow jazz tune to add. What you need is a queryable index across all those files. You need a storage engine that extracts the genre information from the files, stores it in a common schema, and indexes it, so that you’re not grinding through the hard drive every time you want to find a piece of music. That’s where WinFS adds value in this scenario.

A few more comments worth pointing out: Asbjørn suggests this kind of common store with a schema that allows for relationships would be useful for organizing his inbox such that: “My e-mail is basically in a pool altogether, and is placed with pointers (shortcuts) in different views (compare them to database views, as there is no physical moving of files going on) that I can create.” David wants to get the same sort of quick and easy sorting/filtering experience I mentioned above for the PDF files he downloads, and Mario points out that the promotion engine in WinFS is designed for exactly this case.

Okay, moving on to the next scenario...