What's in Store

Update to the Update

2006-06-27T06:26:00Z

Hi. Wow there has been a lot of, let’s say, interest in the posting Friday. I knew there would be interest, and I knew that the news that we would not ship WinFS as a separate thing would make news, but didn’t expect quite the thread lengths we are seeing! Whew.

There are obviously a lot of questions on people’s minds. So I post again today trying to answer some of them. To those who think I am not a real person but rather a name in front of PR machinery – that’s just not true. I am flesh and blood – with a job, a team, and a passion for what we have been pursuing in WinFS. And even a life outside of Microsoft Building 35 with a wife, kids and other interests. Certainly seems like I might have been too careful in wording last week – was not my intention to offend bloggers everywhere, really.

OK, here are the questions/answers.

Is WinFS dead?
Yes and No. Yes, we are not going to ship WinFS as a separate, monolithic software component. But the answer is also No - the vision remains alive and we are moving the technology forward. A lot of the technology really was database stuff – and we’re putting that into SQL and ADO. But some of the technology, especially the end user value points, are not ready, and we’re going to continue to work on that in incubation. Some or all of these technologies may be used by other Microsoft products going forward.

Does your plan for WinFS have any impact on Windows Vista?
There is no impact on Windows Vista. We announced back in August 2004 that WinFS would not be in Windows Vista.

Will the "Relational Filesystem" ever be in Windows?
Hey – we are very busy finishing Vista, and just aren’t ready to talk about what comes next. The vision for a richer storage in Windows is very much alive. With the new tools for searching and organizing information in Windows Vista, we are taking a good step towards that vision.

Why are parts of WinFS going into SQL Server?
We have a vision around data that guides us we call the "Data Platform Vision". We’ve been talking with customers about this for some time, and we have heard consistent positive feedback. It was clear that the integrated storage and automation features of WinFS will help SQL Server deliver on the "Beyond Relational" and "Continuous Availability and Automation" promises of that vision. We decided to focus resources on delivering these technologies to our customers as part of the Data Platform Vision in the near term.

What's the upside to developers?
We believe that including some of the WinFS work in SQL will broaden which developers will benefit from that database, and further we believe the ADO.NET for Orcas innovations will make using a database a lot easier and more productive for developers. Our Data Platform Vision talks about Your Data, Any Place, Any Time. It’s a compelling vision, and we will continue to invest in the desktop versions of SQL (SQL Express and now SQL Everywhere) as well as the Server.

What is a ship vehicle? Why does it matter that WinFS is not a "separate ship vehicle"?
A ship vehicle is the method in which we bring a technology to market. This could be a separate product release, a service pack to an existing product, or an integrated technology in a larger product platform. We announced the removal of WinFS from Longhorn two years ago, and talked about WinFS being a separate ship vehicle. But we are no longer are planning to release a separate WinFS delivery vehicle.

Was WinFS "killed" because of its design?
No. In fact, the Beta was coming together really well. People have speculated on "redesigns." The original goals of WinFS have never changed, but the technology we are building isn’t easy – so we did take a number of internal design changes and re-writes. And I am not going to apologize for that. Getting the relational engine to behave and perform like the Windows filesystem isn’t a matter of a few lines of code – it has to be done very carefully and architected right. The bars on performance, compatibility, etc. are all super high.

Why did Microsoft announce this now after talking about WinFS at TechEd so recently?
When we were at TechEd, we had not made the decision. Sure, it was under discussion, but we did not have all the information we needed and we had not made the call yet. We did share the news as soon as we had the final word. We could have waited longer to disclose the information and made the change in plans less of a contrast, but we chose to notify people as soon as we could. This is why we used the blog and didn’t fire-up the big MS PR machinery – that takes time.

Author: Quentin Clark

WinFS Update

2006-06-23T23:00:00Z

It's been nearly a year since I wrote my entry about WinFS Beta1, but rest assured, we have been working furiously since then. Today I have an update about how we are delivering some of the WinFS technologies. It represents a change to our original delivery strategy, but it's a change that we think that you'll like based on the feedback that we've received.

As most people who read this blog know, WinFS has always been about many things – a new model to enrich how users manage information, rich storage technology, and sometimes also a packaging of technology. The real change I am addressing today is in the packaging strategy.

There are many great technical innovations the WinFS project has created – innovations that go beyond just the WinFS vision but are part of a broader Data Platform Vision the company is pursuing. The most visible example of this today is the work we are now doing in the next version of ADO.NET for Orcas. The Entities features we are now building in ADO.NET started as things we were building for the WinFS API. We got far enough along and were pushed on the general applicability of the work that we made the choice to not have it be just about WinFS but make it more general purpose (as an aside – this stuff is really coming together – super cool).

Other technical work in the WinFS project is at a similar point – specifically the integration of unstructured data into the relational database, and automation innovations that make the database "just work" with no DBAs – "richer store" work. It's these storage innovations that have matured to the point where we are ready to start working on including them in our broader database product. We are choosing now to take the unstructured data support and auto-admin work and deliver it in the next release of MS SQL Server, codenamed Katmai. This really is a big deal – productizing these innovations into the mainline data products makes a big contribution toward the Data Platform Vision we have been talking about. Doing this also gives us the right data platform for further innovations.

These changes do mean that we are not pursuing a separate delivery of WinFS, including the previously planned Beta 2 release. With most of our effort now working towards productizing mature aspects of the WinFS project into SQL and ADO.NET, we do not need to deliver a separate WinFS offering.

Be encouraged that we are able to get the underlying feature work into Orcas and Katmai. It's great technology and we are super-excited to be productizing this way. And most importantly, it's what people have been asking for – as we work with customers, we're constantly hearing that they want many of the technologies to be more broadly available in the data platform products. That feedback was taken seriously.

Of course, there are other aspects of the WinFS vision that we are continuing to incubate – areas not quite as mature as the work we are now targeting for Katmai and ADO.NET. Since WinFS is no longer being delivered as a standalone software component, people will wonder what that means with respect to the Windows platform. Just as Vista pushed forward on many aspects of the search and organize themes of the Longhorn WinFS effort, Windows will continue to adopt work as it's ready. We will continue working the innovations, and as things mature they will find their way into the right product experiences – Windows and otherwise. Having so much ready for SQL Server and ADO.NET is a big impact on the platform, and more will come.

That's all for now. I know people won't be shy with questions and comments.
- Quentin

Author: Quentin Clark

Showing off the next release of WinFS at Tech Ed 2006, in Boston! Join us!

2006-05-22T23:12:00Z

Hello! Let me briefly introduce myself. My name is Shan Sinha. I am a program manager on the WinFS team, having joined the team approximately 8 months ago now. It has been a fast ride from the day I got started and it is only accelerating. I have the good fortune to spend my time working with partners, customers and developers to build out our early adopter ecosystem for Microsoft. It’s one of most fun jobs anyone can have! First off, I get to work with people like you, investigating new opportunities for innovative applications on WinFS and dealing with the challenges of delivering those ideas. Second, I’m part of a team at Microsoft building THE most innovative technology our company is currently working on (or at least we think so :-)).

Imagine a world where data storage and retrieval just work the way they should- no need to create clumsy mappings between objects, relational tables, and byte streams being stored in files. We finally can realize a world that simplifies the persistence, manipulation and retrieval of data, giving us an opportunity to create unique new applications based on those new capabilities. If you are reading this, it would appear that you believe in this vision too. It could not be more exciting!

Let me take a few minutes to tell you about Tech Ed, which is occurring the week of June 11th, this year. Since we announced the availability of Beta 1 last September, we have been busy incorporating all of the feedback you gave to us and preparing Beta 2. Well at Tech Ed, we will be demonstrating why we continue to be so excited about WinFS, showing off features from our Beta 2 release, which will be available later this year. If you have the opportunity to attend Tech Ed, be sure to join us- it will be an exciting set of sessions! You will want to join us for 4 sessions:

WinFS & Integrated Storage Overview [DAT202]
Monday, 6/12 9:00AM – 10:00AM
Speakers: Quentin Clark, Product Unit Manager, and Shan Sinha, Program Manager
Target Audience : General, 200 level

Learn about Microsoft's vision for Integrated Data – a single platform for storing and accessing structured, semi-structured, and unstructured data. See how WinFS, Microsoft's new relational filesystem for Windows, delivers on this promise. Get a glimpse of how the next generation of your applications will benefit from a relational file system.

Deploying WinFS : Bringing a Relational Store to Windows [DAT211]
Tuesday, 6/13 8:30AM – 9:45AM
Speaker: Sethu Kalavakur, Lead Program Manager
Target Audience : IT Managers and Administrators, 200 level

WinFS is the new relational file store for Windows. This session will describe WinFS in the windows ecosystem – supporting backward compatibility for existing applications, performance tradeoffs, easier management through features like Automated Recovery from Data Corruption, security semantics and more. You will walk away comfortably understanding how to deploy, use and maintain WinFS on Windows.

Building on WinFS : Developing a Great WinFS App from Scratch [DAT416]
Tuesday, 6/13 2:45PM – 4:00PM
Speaker: Roger Lueder, Software Development Engineer
Target Audience : Developers, 400 level

A deep dive into building WinFS applications. This session will walk through the capabilities offered by the WinFS platform. We will create a rich data oriented application from scratch, showing you some great examples of new scenarios you can enable in your applications. You will leave the session with the knowledge you need to begin building your own data-oriented applications on WinFS.

Next Generation Data-Access in .NET Applications with ADO.NET vNext [DAT304]
Monday, 6/12 10:45AM – 12:00PM. Repeated: Thursday 6/15 2:45PM – 4:00PM
Speaker: Pablo Castro,
Target Audience : Developers, 300 level

ADO.NET 1.0 presented a break-through in data-access technologies with explicit support for disconnected scenarios and a lightweight, high-performance provider model. In ADO.NET 2.0, the API was extended to enable more scenarios, to perform faster and scale better. In this session we're going to discuss what comes next. The future of ADO.NET has both evolutionary aspects and serious innovation in it; from language-integrated query to object services to mapping, ADO.NET will bring simplification and great expression power to the Microsoft data platform, enabling the construction of sophisticated business applications and tools with less effort and more functionality. The WinFS API takes advantage of the capabilities we are building into ADO.Net vNext.

Have no doubt- WinFS will be the future of Windows data storage! Those who choose to explore WinFS early will be in the best position to know how to utilize it to improve their development experiences and enhance their users’ lives. I look forward to seeing you at the conference and meeting in person.

In case you will not be able to attend Tech Ed this year, we will be posting content from the conference on our site at http://msdn.microsoft.com/data/winfs. Keep checking for updates!

Tech Ed 2006 Home : http://www.microsoft.com/events/teched2006/default.mspx

Author: Shan Sinha

The killer app for getting users organized

2006-05-16T21:34:00Z

Hopefully by now, most of you have already seen some of the very cool WinFS demos (PDC 2005 Real Estate Demo, WinFS iWish Video) that we've built. Microsoft recently made a decision to productize some of these ideas into a new application - currently codenamed Project Orange. Unfortunately, I can't say much publicly about exactly what Project Orange is yet, but I can use this opportunity for a shameless recruiting plug. If you're excited about the opportunity to develop a new application for people to organize their information - entirely build on the new storage platform (WinFS) and new presentation platform (WPF, AKA Avalon) - check out job postings here: UI PM, Product Designer, Schema PM, SDET, Schema SDET, SDE, SDE, SDE Lead, Schema SDE Lead, or the many other jobs in WinFS-land!

Just to tease your curiosity, I've pasted in a couple of the descriptions here:

UI Program Manager

Do you want to help users finally get organized? Are you tired of working on standard usability challenges and excited by the chance to work on a whole new user paradigm for information management? Think that you have an eye for the “killer app” for a new platform? Did you ever want to work in a fast paced, start-up environment but still be involved with a project with huge impact? Do you want your work to ship to hundreds of millions of Windows users? Do you think Windows Presentation Foundation (AKA Avalon) and WinFS offer a truly new world for applications?

If so, Project Orange may be the right fit for you! Project Orange is a brand new team tasked with building a next-generation Information Explorer based on WinFS and WPF (AKA Avalon) to help users finally get organized. This is a soup-to-nuts project focused on defining a breakthrough user experience for users to unify, organize, and explore their data in meaningful new ways. WinFS and WPF offer dramatic new opportunities for information management by merging the traditional world of relational databases with end user data and offering new opportunities for interaction & visualization.

We are looking for a unique person that can blend skills from usability, design, and program management to spearhead definition of this new application. Successful candidates will be creative and be drawn to solving end-user interaction design problems in radical new ways.

Candidate should be experienced in the process and execution of user-centered interface design and presentation. Since this will be a small, fast-moving team, candidates should be strong multi-taskers who are able to balance work on multiple areas simultaneously. Experience working on UI design, data visualization, V1 projects / startups, and end-user desktop applications is a big plus. Candidate should either have a degree or significant work experience in Design (interaction design, industrial design, visual communications, or related field), HCI, Human Factors or something similar. Please be prepared to provide a portfolio for review. A BS/MS degree in Computer Science or related technical field would be bonus but is not required.

Software Development Engineer

Have you ever wanted to work in a start-up? Want to work on a fast-paced project with huge impact?

We're looking for talented and motivated developers who: have an eye for the "killer apps" for a new platform, want to help overcome the hurdle of information management for end-users, and are excited by the idea of finally delivering a relational filesystem in Windows. If so, we may be the team for you!

The WinFS team is starting up a new effort to build a killer application for WinFS. This is a soup-to-nuts project focused on defining a breakthrough user experience for users to unify, organize, and explore their data in meaningful new ways. Successful candidates will be creative and be drawn to solving end-user problems in radical new ways. Candidates should relish the opportunity to work on bleeding-edge technologies such as WinFX, Avalon and WinFS, and stay up to speed on a rapidly changing technology base. Since this will be a small, fast-moving team, candidates should have strong multitasking skills, be able to balance work on multiple projects simultaneously, and have the ability to take on planning and strategic tasks.

There will be ample opportunity for personal career development, with everyone on the team exposed to new challenges as part of the WinFS integrated storage project. A successful candidate will have excellent technical and communication skills, work well with other teams, will have successfully shipped product(s), and will have a strong customer focus.

Experience working on UI design, data visualization, Windows Shell extensions, V1 projects / startups, and end-user applications is a big plus. Two years of development experience; strong background in C#, C and C++; demonstrated experience shipping high quality products. Database and systems development experience is a plus, but not absolutely required. A BS/MS degree in Computer Science or related technical field preferred.

If you’re interested, you can use the career site or hit the email link on the right side of this page.

Author: Shishir Mehrotra

My tryst with Destiny, err… Integrated Storage

2006-02-25T02:16:00Z

Hi, my name is Sanjay Anand. I run the Program Management team for WinFS. I’ve been at Microsoft for almost 12 years now in which time I have been involved with networking, security, messaging, file systems, and of course integrated storage :-)

Prologue:

A very very long time ago, my friend and I joined Microsoft to work on “next generation” Operating Systems – I was on Windows NT, he was on Cairo. Cairo was this sexy, glitzy, very futuristic attempt at building an integrated store - an object file system. This was the thing that would replace Windows NT the moment it shipped. I plodded along on the not-so-new NT, knowing that Cairo would come along and replace most of what we were working on. Then one day, Cairo disappeared and the machinery moved on to salvage parts of Cairo to deliver on the premium OS – Windows NT. At that time I didn’t think much of the whole thing except that it was perhaps far ahead of it’s time…

Webstore circa 1999:

A half-decade later, I encountered “Integrated Storage” for the second time while working on the Exchange Webstore. The premise was to build a single enterprise store that would have integrated manageability, multi-tier deployment, supported multiple protocols and data access stacks and a platform for app development. It was a file server, a web server, a corporate email server, internet email server, collaboration server, all rolled into one. The file server in particular was the piece I worked on. We actually shipped this product in Exchange 2000, but it was still-born. The technological underpinnings weren’t quite there – it lacked real transactions, had issues with scalability, was too mail-oriented. The win32 support was too hard – we mastered the entire file system in a single file, literally re-implementing NTFS within a file, with our own block management strategy, our own locking implementation, complex interactions with the cache manager – oh! It was a nightmare, but we got it limping along. A brave effort nonetheless and something we were to learn from eventually!

The client manifestation of this webstore was called the LIS (local information store) – a valiant attempt at building a client platform to match the Webstore on the server. Office Designer was the dev environment on the client and Outlook was the marquee application on the client to use that store. The beginnings of the Outlook offline store took root here. The challenges were two-fold: take the server store and “shrink” it to the client and provide a MAPI shim over an HTTP wire protocol for the sync. The performance never really matched the requirements of Outlook, the store was too general purpose... this went by the way side as well.

Fastforward to 2003:

Another half decade later, we launched with renewed energy towards the 5th attempt (and my second) at Integrated Storage – WinFS. A lot of factors were in our favor – the need for a storage platform was more acute than ever with digitally born data exploding every day, a lot of technological underpinnings were there for the taking: transactions in file systems, mature databases, sophisticated programming languages, but most importantly, a set of very motivated individuals who had deep experience in the various elements required to bring together this complex technology. We had folks with deep database experience from QO/QP to transactions, aces in programming languages, folks with emmense experience in file systems, distributed systems, web services, hard core synchronization wonks, O/R mapping gurus, but above all, a fearless leader who was ready to take a real shot at this thing once and for all and who had the maturity and discipline to control the engineering of this innovation – Peter Spiro.

The project started with much fanfare - a who’s-who of very senior people passionately oversaw the vision, tons of internal teams hooked into the common schema effort - identity services, state management, windows help, natural languages, Windows shell, Pix, Media, etc. At one point we also had over 80 external ISV’s who were at various levels of commitment. These were heady times; we knew these were also unsustainable for any single platform to serve, but we also knew these engagements were essential to distilling the core nuggets of innovation to target for a v1. The Vista change occurred and while negative on the surface, from a project perspective it actually helped us take pause, distill our core value props and run really fast at hitting these. There are many ways where WinFS adds value, however I want to touch upon one that really appeals to me: a storage platform for application development.

Storage platform:

First off, we are about building a storage platform that enables applications to model their data in a much richer fashion, while still benefiting from common services one would expect from a file system: administration, sync, sharing, security, backup, drag-n-drop, move/copy, delete and yes, search. File systems don't allow applications to model their data in anything richer than byte streams when it comes to interacting with other applications, Windows shell, etc. Databases allow you to model your data in richer ways by leveraging relational capabilities, however none of these automatically enable that data to be shared with the Windows Shell much as a file is. So, there you have it - we are the best of file systems (Integration and services) and databases (rich data modeling capabilities) rolled into one - hence Integrated Storage.

Now, lets move further and see how you can do much more if you actually share that rich data definition across applications. Let me take an example: reusable organization is an extremely powerful concept – something that application-specific tagging cannot even come close to achieving. Consider this scenario: at each of my daughter’s b’day celebrations at her school, we take in a poster of her pictures from previous years. I search for the pictures in one app (Digital Image Suite), auto-fix in another (Photoshop), print using HP’s custom album printing app. Each app has some notion of query-based searches, but no two share it. So, whenever I transition between applications, I end up duplicating my photos into folders since that’s the common denominator across applications. Then, I go through and delete these duplicates which I have collected over time, including copies of edited ones, ones I staved off to put on ophoto.com, post on spaces, email to my sister, the list goes on and on. Let me admit that one of the most difficult item types to delete is a photo – you can buy music again, re-write a document, but not a photo. So, I end up not deleting the photos many times… Yes storage is cheap, but this is ridiculous – and I am a reasonably tech savvy person... I recognize that there are several answers to this scenario: build a better single app to do the entire photo workflow; standardize on a common tag file format, etc. However, the same scenarios apply to music – my organization in Media Player is lost in iTunes. I use both: the first to stream the second for my iPod. In fact they apply to all forms of digital data.

The point is that the file system today is too primitive a substrate for these applications to work over and WinFS attempts to close this gap by raising the level of involvement in the application’s data: it’s a trade – the more apps tell the system about their data, the more the platform does for their types in terms of allowing an ecosystem of applications to interact with their content. The user organizes her content in one app - sets up a query to filter down the photos – “all 5-star photos of my daughter that I haven’t used earlier, grouped by year taken, ordered by resolution” and another app uses these queries to edit the set, etc. This is achievable with a rich underlying storage platform, just not today.

Now and beyond:

At this point, we have perhaps the best shot at this than ever before – we have robust file system support – perhaps the most optimal implementation one could hope to build. We have gone through a number of iterations on our data model so have higher confidence that we are closer to the eventual answer ;-). A great platform needs a great API and we have that with our alignment with ADO.Net vnext, we have solutions to hard questions around business logic, cross-store versioning, security, a strong synchronization story, the list goes on and on. There is a lot more to solve and gell, but this for me is by far closest to hanging together.

I am also a huge believer in learning from shipping and real world exposure. So, right now, it’s on to shipping this baby and getting it in your hands. As with any platform, the real proof lies in the universe of applications and how they build on that platform. Non-Microsoft applications, btw ;-)

The dream continues…

Author: Sanjay Anand

Synchronizing Data between WinFS Stores

2006-01-26T03:12:00Z

Hi, my name is Mark Scurrell and I’m a Program Manager on the WinFS Sync team. I’d like to give you an overview of the functionality we provide to allow applications to synchronize data between WinFS stores.

If you haven’t already done so, I would recommend you first read a previous post where Neil Padgett provided an overview of synchronization and described the scenario where data is synchronized between a WinFS store and a non-WinFS application store.

As well as providing a set of services to synchronize with non-WinFS stores, as Neil described, we also want to allow developers to build peer-to-peer applications. So, as part of the WinFS Sync platform we provide services that will allow applications to synchronize their data between multiple WinFS stores without application developers having to design and code the sophisticated synchronization algorithms that are required for peer-to-peer applications.

A good example of a peer-to-peer application that was built using WinFS Sync is Microsoft Rave, a sample application included in the WinFS Beta 1 SDK. Rave allows WinFS data to be shared between users without the need for a server – each user can directly synchronize folders on their computer with other user's peer-to-peer. The developer of Rave discussed his experience producing this application in a previous post.

Let’s start with the simplest use of WinFS-to-WinFS synchronization which is to synchronize two WinFS stores.

So, what happens when synchronization is initiated? WinFS Sync will do the following:

Determine the changes made to each store since the last sync (change enumeration)
Communicate those changes to the other store and update the other store (change application)
When applying the changes, determine if there is any conflicting data – the same data has been changed on both stores since the last synchronization (conflict detection)
Either log the conflicting data for later resolution or have it resolved immediately (conflict resolution).

WinFS stores will contain many different types of data such as files, custom item types, links between items, and so on. WinFS Sync will synchronize all data stored in WinFS; no custom code is ever required, for example, even if a new item type is defined.

Synchronization between two stores is useful; however more interesting scenarios involve multiple stores. With multiple stores there are different ways the stores can be configured to communicate – the sync topology.

The Microsoft Rave sample application, for example, allows multiple users to synchronize, with each user being able to synchronize with every other user in a full mesh topology. Another application, for example, may require a topology with all changes being communicated through a central “hub”. The point that I want to emphasize here is that WinFS Sync is very flexible and has been designed to cater for any topology; there does not need to be a “master” node and true peer-to-peer applications can be built.

I’ve described how WinFS Sync was built to handle peer-to-peer scenarios and different topologies, but how do we handle the fact that sync applications will need to operate in many different network configurations? Here are some possible scenarios for peer-to-peer sync applications:

There are multiple computers on a home workgroup network; a user could synchronize their data between computers so it can be accessed and updated on any computer in the home.
Users are members of a domain on a corporate network and can configure folders for sharing and collaborating with other invited users.
Family members are located in different parts of the country or in different countries; they can share photos with other members of their family by having them synchronize their computers over the Internet.

WinFS Sync has no knowledge of the network configuration or the transport that will be available for communication of the changes. The application developer must provide a transport over which the sync protocol will operate. We define an interface for the transport and the sync application must supply an implementation of that interface to WinFS Sync. The developers of the transport do not have to concern themselves with the complex logic of synchronization; they implement simple methods such as ReadMessage and WriteMessage and WinFS Sync does the rest.

Those of you who have had a close look at our Beta 1 release may be confused as we provided a facility called Store Synchronizer that synchronized data between WinFS stores and included a transport suitable for a local network. For Beta 2 we have decided to focus on providing the peer-to-peer synchronization platform and will not provide any specific transport implementations. We have therefore modified Store Synchronizer so it requires a transport implementation. In the Beta 2 SDK we will of course provide sample code for a transport, guidance on how to build a transport, and reference material for the Store Synchronizer classes.

My main goal with this article is to raise awareness of the set of synchronization services that WinFS provides to support peer-to-peer sync application development and also to highlight the flexibility we allow in terms of diverse sync scenarios, topologies and network configurations.

I hope you found this post useful. There is further information available in our Beta 1 SDK documentation if you want to dig deeper. I would be interested to hear about any scenarios where you would utilize our WinFS-to-WinFS synchronization capabilities.

Author: Mark Scurrell

WinFS Mailbox II

2005-12-16T21:59:00Z

Hi Everyone! Before I disappear for the holidays, I thought I’d dig through our inbox, the blog, and newsgroup, and answer a few more questions.

Q: Chris asks, “While it's good to have a common silo that all apps can easily use, it's seems a bit risky. Wouldn't some rogue app/spyware have an easier time getting to financial Quicken/Money data?”
A: Even though all your data is in a single silo, all the data isn’t equally accessible. WinFS provides two data protection mechanisms. First, there is share-level security that controls access to your WinFS share. Second, there is item level security that supports NT compatible security descriptors. In order to access an item, the caller must be provisioned at the both share and the item. To manage this, WinFS will provide a rich API for security administration. Also in Vista there is the concept of “integrity level” for an application (Mandatory Integrity Control). Data can be configured to prevent lower integrity apps from accessing it. Simon (our security PM) will post more detailed description about this soon.

It’s important to note that an application shouldn’t be using file format complexity as a primary security measure. For important information like bank account and credit card numbers, you could continue to store them in an encrypted and password-protected file. But, there’s no reason that this file couldn’t live in WinFS.

Applications may have some data they definitely want to lock away. WinFS, however, gives the application developer the opportunity to share the data he chooses to share. Today there is a good amount of information in a money file and not all of it necessarily needs to be private. For example, Money could have “payees” (i.e. someone you get a check from or write a check to) represented as WinFS contacts.

Q: Retla1 asks, “What’s the best way to get some data into WinFS?”
A: This is a great question. After all, once you install WinFS, there isn’t that much to look at except an empty relational filesystem. :-) Thankfully, there are number of good ways to get data into your store. The first set of ways is to copy it in. Open up our awesome WinFS Shell Namespace Extension (in the Windows Explorer, click through My Computer->WinFS Stores->DefaultStore). Now just drag and drop some files in.

Or, if you prefer, use copy or xcopy. Our Win32 support is robust, so please try it out (note the Win32 path is a UNC path: “\\<machinename>\defaultstore\foo.doc”). You can also redirect your “My Documents” folder to your WinFS store. This would move that data into WinFS. Any application that respects Shell APIs will work fine. (If it doesn’t work, let us know.)

If you prefer, you can sync data into your store. Beta 1 came with some sample sync adapters. Open up StoreSpy (one of our unsupported tools) and import some stuff from Outlook. If you install WinFS on two computers, be sure to use Rave to share that data.

Q: Lars asks, “Does WinFS look inside files (word, excel, pdf, etc.) like desktop search engines do?”
A: Yes. When a file is stored in WinFS, its metadata will be extracted by type specific property handlers. Part of our Beta 2 work is to integrate this with the Windows Desktop Search handlers. The end result is that a Windows Desktop Search query will return WinFS and non-WinFS files.

Q: Qearsa asked, “What about all the Shell improvements in Vista? How does WinFS fit into that?”
A: Vista provides a rich user experience using indexing technology. By integrating into that technology and providing a Shell Namespace Extension, WinFS will provide this same experience to users. This means your Vista searches will have results from many stores including WinFS, NTFS and Outlook. In addition, WinFS will provide the opportunity for the new and compelling user experiences we discussed in earlier posts.

Q: A few people asked, “Will Beta 1 run on 2003 or x64?”
A: Nope, we’re not planning to re-release the Beta 1 to support any other platform. We’re currently evaluating what platforms our Beta 2 will support.

Q: A few people also asked, “Are there any new features in the Beta 1 refresh?”
A: The re-release has the same functionality of the original, but runs on the RTM version of the .NET Framework 2.0.

Q: Timbu asks, “Desktop search applications already aggregate my data from different apps (like Outlook and Sharelook). Isn’t this ‘Unify’?”
A: Desktop Search represents the first step in a truly unified store. While you can use desktop search to see different types of data from different silos in a result view, the data itself still resides in separate silos. This means it is hard to operate across all that data. For example, I can search for all my files that have “Shell Namespace Extension” and get a huge list of emails, docs, and other files. At this point, there are some operational limitations. For example, I can’t copy all that data and put it in a single folder or USB drive, burn it to CD, or even share it out. Desktop search is a great end user application, but it’s not a platform. WinFS is a platform with great programmer access through managed APIs targeting the unified and extensible data types.

One last question, but I need the answer from you: “What should I write about next?” I’ll like to start getting into more details and away from the high level stuff. :-) Please post some comments with suggestions and ideas.

Happy Holidays!

Author: Vijay Bangaru

WinFS Beta 1 Refresh now available

2005-12-01T21:51:00Z

MSDN subscribers can now download our Beta 1 Refresh release. This release contains the same functionality of our original Beta 1 release and runs on the final release of the .NET Framework 2.0 . Before installing the refresh, make sure you uninstall your previous WinFS install along with any Beta releases of Microsoft Visual Studio and the .NET Framework 2.0.

There’s also a handy tool that uninstalls all the VS products for you.

Author: Vijay Bangaru

Unify, Organize, Explore, and Innovate. Oh my! (Part 4)

2005-11-19T03:32:00Z

…and then there was “Innovate”. We’ll close this series of posts by discussing how the WinFS relational platform provides powerful, next generation tools to the application developer.

WinFS is built on a true relational store that all applications can leverage. Today, tons of popular apps lock their data into proprietary stores. This includes photo apps like Photoshop album, music apps like WMP and iTunes, financial apps like Quicken and Money, email apps like Outlook, and so on. By using WinFS, developers can concentrate on building rich applications without worrying about creating a storage silo. After all, WinFS can give applications a place for all their structured, semi-structured, and unstructured data.

In addition, WinFS will provide services like synchronization, notifications, rules, and backup/restore. The services provided by WinFS lead to a zero-admin and extremely reliable experience.

All the data in the store is accessible as CLR objects, so developer can leverage all their Visual Studio and .NET knowledge while building apps. Developers can also utilize our full query engine over their data and transport their data with our robust serialization support. This is a significant improvement over today’s cumbersome interaction with multiple forms and multiple locations of data.

So we’ve covered Unify, Organize, Explore, and now Innovate. Hopefully you have a better understanding of where you can go once you combine a traditional file system and traditional database.

As always, please post any questions or comments.

Author: Vijay Bangaru

Getting Data Into WinFS with WinFS Synchronization

2005-11-18T04:45:00Z

One of the first topics developers ask about once they start learning about WinFS is “How do I get existing data in?” We tend to think of this in terms of a bigger problem: “How do I move a lot of data in and out of WinFS?” Our answer here is WinFS Synchronization.

My name is Neil Padgett and I’m a program manager working on the WinFS Synchronization APIs. The goal of the API set is to provide access to all of the services WinFS provides for developers building sync solutions. But, before we delve too deeply into that, let’s talk for a bit about what exactly synchronization is.

The simplest idea that pops into most people’s minds when they want to get data into WinFS is to write an importer. That, is, they plan to just write a simple application to pull data from some application store and then use the WinFS API to create WinFS entities to represent their data. This is a one way importer.

So, this seems great, right? You’ve got your data and it is moved into WinFS. And this works well assuming you aren’t going to use the non-WinFS application to update the data anymore. But what happens if you want to update the data in the non-WinFS application? Let’s make this a bit more specific, let’s assume we have a contact in the application store and we’ve imported it into WinFS. And then let’s assume that we’re going to go ahead and keep using the non-WinFS application to update the contact.

So this works fine? Right? We rerun the importer periodically and update the WinFS contact from the application store.

And, this will work. It works because we never update the data in one of the two places. This means that we can just overwrite the data in WinFS every time. However, WinFS is a shared data store – that contact is available in a well-known schematized format. And the user may choose to make it available to their other applications. So, others may update it. But, if we run our simple importer, we’re going to lose data. How can we solve this?

The answer lies in detecting (and later resolving) this conflict. But further to that, we want to try and merge together changes that happened on the different stores. This means we’ll need to be able to figure out what changed on each store so that we can try and apply those changed to the corresponding item in the other store.

So, let’s consider our contact again. And now, let’s be more specific about what we’re changing.

Let’s assume that we did some initial sync to ensure we had our contact in both stores. (We can talk about what this exactly means later, but for now we can think of it to be like running our importer.)

And let’s imagine that, after we sync we make some updates:

In the application store, we’ll update:
-Home address
-Telephone

And, in WinFS we’ll update:
-Home address
-Cell

So, now imagine we want to bring our two stores back into sync. So, how do we do that? We know we were in sync before, so we need to figure out what changed on each store, and then apply those changes to the other store. We can call these processes change enumeration and change application, respectively, and we want to do them in both directions. (In fact, WinFS does the hard work of figuring out what changed for us and of making sure that remote changes brought to WinFS are not echoed back to us later.)

Considering our example, we have some changes that are straightforward – the telephone numbers were each changed on the one store, but not on the other. We call these non-conflicting changes. For these non-conflicting changes, we can simply apply them each to the other store. The difficulty comes with the home address – we made changes on both stores – so-called conflicting changes. We’ve detected a conflict and we’ll need to resolve it, either by prompting the user or, more likely, according to some policy (for example, keeping the latest change.) Then, we can bring our two stores back into sync.

WinFS Synchronization is fundamentally about providing services in the store that do just these things: Change Enumeration, Change Application, Conflict Detection, Conflict Resolution either by deferring for manual resolution or via automatic resolution, and other things. We also generalize these services for the case of many stores with arbitrary topologies, and we provide specialized solutions for common cases like synchronizing files or synchronizing several WinFS stores. In upcoming posts I’ll talk about the services WinFS Synchronization provides and how they can be used to solve interesting data moving problems. We’ll also talk about some more interesting scenarios involving multiple synchronized stores with interesting topologies (such as in peer-to-peer) scenarios.

In the comments for this post, I’m interested to hear about how you think you might use WinFS Synchronization and what you’d like me to focus on first in the upcoming posts.

Author: Neil Padgett