Microsoft RSS Blog : RSS Platform

Still Gone? Ok – got it!

rss — Fri, 27 Apr 2007 09:28:00 GMT

Sam Ruby pointed out HTTP 410 GONE support in feed readers or rather the lack thereof. He links to the list of User-Agent strings that continue to request the feed that is gone. One of the entries points at the Windows RSS Platform as an "offender":

Windows RSS Platform/1.0 (MSIE 7.0; Windows NT 5.1)

It's listed with 282 hits. At first I was surprised to see the Windows RSS Platform in that list since we specifically added 410 GONE support. But then it dawned on me: That's not the Windows RSS Platform!

Well, it is, but it isn't. The above User-Agent string is the one from the Beta 2 Preview release (Jan 2006) of the Windows RSS Platform. The User-Agent string changed in Beta 2 (April 2006) to the final string:

Windows-RSS-Platform/1.0 (MSIE 7.0; Windows NT 5.1)

I described the string here a year ago. See the difference? The dashes instead of spaces! Why the change? Well it turns out that the product token of the User-Agent string may not include spaces, since spaces delimit product tokens and comments.

So it turns out that there are still people running the Beta 2 Preview version of the Windows RSS Platform, or some application is "faking" the User-Agent string.

Either way, I just verified that the RTM version of the Windows RSS Platform handles 410 GONE correctly. I used the following Powershell script:

$fm = new-object -comobject "Microsoft.FeedsManager"

$feed = $fm.rootfolder.CreateFeed("gone","http://www.intertwingly.net/blog/index.rss")

$feed.SyncSetting

$feed.Download()

$feed.SyncSetting

When you run it you will see that the SyncSetting property is changed from 0 to 2 after the Download() call. Note that the SyncSettings are defined as:

typedef enum {

FSS_DEFAULT = 0,

FSS_INTERVAL = 1,

FSS_MANUAL = 2

} FEEDS_SYNC_SETTING;

FSS_DEFAULT - Use the system-defined DefaultInterval value.

FSS_INTERVAL - Use the Interval value defined by the feed.

FSS_MANUAL - Do not automatically update the feed. Use Download to manually update the feed.

which means that the feed initially uses the default sync interval to get updated. Upon download, the setting is changed to Manual since a feed that is GONE should no longer be updated automatically.

- Walter vonKoch

Feeds Plus: An Intern Adventure

rss — Wed, 24 Jan 2007 07:41:00 GMT

Last summer, we had a couple of interns on the RSS team here in IE: Nate Furtwangler, a developer intern, and Chrix Finne, a Program Manager intern. Nate and Chrix both did an amazing job helping us to ship IE7 and Vista, and they also found the time to knock out a really cool project. Here's a big thanks from all of us to them.

I'll let Chrix describe the project (and their experience) in his own words. In case you're wondering, the "where's Sean's office" thing is a reference to Chrix's decision that it would be fun to relocate my office to the roof of the parking garage while I was on vacation. Good times.

- Sean

Hi! My name is Chrix Finne and this past summer I was an intern Program Manager on the IE RSS Team – also known as ‘Team RSS got new digs so the interns got a window office.’ I had the privilege of working on several cool projects this past summer, and had a blast while doing it. Shoutout to my officemate Nate Furtwangler, developer intern and my partner in crime. Though they consumed far fewer Swedish Fish than Nate and I, my mentor Jane “where’s Sean’s office?” Kim and my manager Sean “Romulus” Lyndersay were also awesome and kept me on my toes. Congrats to the whole team on IE7 and Vista RTM!

We are proud to announce that our intern project, Feeds Plus also shipped! It’s a free IE7 add-on that adds two features to the Windows Feeds experience: aggregation and notification. We hope that Feeds Plus will help users get more flexibility and engagement with feeds in Windows.

Feeds Plus, running in the background, can combine multiple feeds into a single, river of news-style feed. All the user has to do is turn on Feeds Plus’ aggregator, and every folder of feeds will sprout an aggregate feed at the top. This can be very useful – for instance, I can make a folder with all of my news feeds and then read all my news at once through the News’ aggregate feed. Read/unread information is synced between the aggregate feed and its children, and aggregate feeds are searchable just like any other feed. If multiple feeds use Simple List Extensions (SLE) then the corresponding aggregate feed will as well – this is cool because you can combine multiple different eBay feeds, and use SLE to sort by price across all of them.

A pop-up feed notification reminiscent of Outlook’s is the second feature in Feeds Plus. The Windows Feed Download Engine always runs in the background, so users have to check to see if new items have arrived. With Feeds Plus, the user can choose which feeds matter most to them and get a pop-up ‘toast’ as soon as those feeds have new items waiting. The pop-up is designed to be unobtrusive—it fades in and out and won’t go crazy and flash every half second—and it includes a handy link to the feeds that it’s announcing:

Feeds Plus is an unsupported IE7 add-on (meaning that it’s not supported by Microsoft technical support or by the IE development team). One important note is that it does not have accessibility support in this release. Don't use it if you're uncomfortable using unsupported software.

Download Feeds Plus here.

So, that’s Feeds Plus. I also had the chance to help design the Feed Headlines gadget on the Windows Vista Sidebar. Feed Headlines shows the user a scrolling list of headlines from one or many feeds with a nice little preview window and links to the browser. It’s a great way to keep content handy – I keep an instance pointed at Engadget that I use to procrastinate all the time.

My favorite thing about Feed technology in Windows is how much freedom it gives the user to choose how, when, and where to consume different types of web content – from news feeds to Craigslist searches. I think that it’s critical to provide different ways to read and consume the feed content that’s coming in, so I hope you enjoy these little feeds extras.

Yours,

Chrix Finne

PS – One shameless plug: Nate and I were lucky enough to get interviewed by Channel 9, so if you want to see those Swedish Fish for real, keep an eye on Channel 9 (hint: they have a feed). They’re on the middle shelf above the demo laptop.

Enclosure Download

rss — Wed, 06 Dec 2006 19:46:13 GMT

A while ago I posted details about the RSS Platform Download Engine. That post focused on downloading of feeds, but did not include additional details on enclosure downloads.

Enclosures are, as most readers know, files that are "attached" to items in an RSS feed. Typically, a publisher will include a reference to a binary file, which an RSS aggregator can optionally download when the feed content is downloaded. The most common example of enclosure use in RSS feeds is for podcasting, where the attached (or "enclosed") files are audio files.

As with feed download, we designed the enclosure download with server and client bandwidth in mind since feed as well as enclosure downloads also happen in the background. Their impact on foreground applications should be limited. Similarly, the impact of large enclosure downloads on servers should be limited.

Let me sketch how the enclosure download process works:

Every time the feed download engine runs it processes feeds that have the "Automatically Download Enclosures" setting set to true it. If it comes across a new item with an enclosure it adds the URL of the enclosure to a FIFO queue.
Before the enclosure is added to the queue, the URL is checked with Attachment Execution Service API (AES) to assure the enclosure file type is one of the permitted types. If it's not, the enclosure download is failed (IFeedEnclosure.LastDownloadError = FDE_DOWNLOAD_BLOCKED).
The first 4 enclosures in the queue are then handed off to the Background Intelligent Transfer Service (BITS). BITS is a background download service that ships in Windows and which enables downloading of files in the background while limiting its affects on network usage. In particular, BITS uses HTTP RANGE requests to download files in chunks. BITS also monitors whether foreground applications (like email or browser) are using the network, and if so, it throttles back its own network usage to limit its impact on those applications.
Once BITS completes downloading an enclosure, the Download Engine uses AES to save the enclosure to the folder corresponding to the feed. Saving via AES associates zone information with the file. The zone information is used when the file is launched at a later time.
If there are more enclosures waiting to be downloaded and there are less than 4 enclosure downloads active, the next enclosure is handed off to BITS as in step #3.
If, however, the server of the enclosure does not support HTTP RANGE requests, the Platform Download Engine falls back to downloading the enclosure via a regular HTTP GET request. If this attempt fails as well, then the enclosure download is fails and will not be attempted again automatically.

Note that the enclosure fall-back download (HTTP GET) is size limited to 15MB to limit the impact of denial of service (DoS) attacks against the RSS Platform Download Engine. Since the RSS Platform Download Engine runs in the background, a malicious server could consume all of the client's download bandwidth without the user having any idea. Enclosure download via BITS (HTTP RANGE requests) is less impacted by such an attack and is consequently not size limited.

In other words, if you are an enclosure publisher that wants to serve enclosures larger then 15MB to IE7 users, then you should use HTTP servers that support HTTP RANGE requests. Most popular web servers support HTTP RANGE requests.

It's also worth noting, that when a server does not support HTTP RANGE requests, the RSS Platform Download Engine will issue two requests for each file (the first testing for HTTP RANGE support, and the second to download the file without range support).

For more details on the security measures used to protect applications and users from potentially malicious enclosures, see Miladin's enclosure security post.

I hope that this description of the enclosure download process explains the "multiple-requests" that some publishers have seen, as well the security restrictions associated with enclosure downloads.

-Walter vonKoch

Program Manager

Windows Vista and Feeds

rss — Mon, 04 Dec 2006 20:01:00 GMT

As noted pretty much everywhere on the web, Windows Vista launched (for businesses) last week.

Windows Vista includes IE7 and the Windows RSS Platform, and is therefore the first Windows operating system to ship with built-in support for RSS (and the first OS of any kind to have RSS support built-in as a native platform component).

Windows Vista is, in fact, the fulfilment of a promise we made over a year ago at Gnomedex 5.0: Longhorn loves RSS.

In addition to the reading experience in IE7, and the platform features, Windows Vista also include the new Windows Sidebar, which ships with a Feed Headlines gadget.

The team that built the gadget have written up a great post on how the gadget was built, and how they leveraged the RSS platform to make development much easier for themselves.

Read their post here: Building the Feed Headlines Gadget.

In case you haven't seen the gadget in action, the screenshot below shows the gadget after the user has clicked on a headline (I've configured it to show the headlines from the MSNBC News feed).

Many thanks to the folks on the Sidebar team that developed such a great gadget, as well as to Chrix Finne, who interned on the IE RSS team as a PM this past summer, and helped out the Sidebar team with feature design for this gadget.

- Sean

Note: Apologies to readers who downloaded an earlier version of this post, which used a photograph taken by Niall Kennedy and posted on flickr.com. He did not appreciate the usage, and replaced it with a different image. I forgot to include an attribution, which I had fully intended to do, but for which I apologise to him.

RSS Platform MiniSDK

rss — Fri, 22 Sep 2006 22:20:00 GMT

Have you wanted to use the Windows RSS Platform from C++? Unlike managed code or script there is no simple way to create header files with the declaration of the IX.. interfaces which are designed for use from C++. Of course the msfeeds.h header file is included in the Windows SDK. If you are hardcore about Windows development you might already have it installed. However, not everyone wants to install the 1GB+ just to get the msfeeds.h header file.

Fear not, I've recently posted on my blog a MiniSDK which includes the required headers to use the RSS Platform from C++. I hope this will save you some time and effort.

-Walter vonKoch

Securing feed enclosures

rss — Thu, 21 Sep 2006 01:25:00 GMT

Greetings,

I am one of the developers on the RSS team, and to complement Sean’s and Walter’s recent postings on feed security, I would like to talk about one topic that didn’t get as much attention in recent discussions on feed security as perhaps it should have - feed enclosures. Enclosures are files “attached” to feed items, commonly used in podcasting and often automatically downloaded to user’s machine by aggregators.

In IE7 and the Windows RSS Platform, we have taken a number of precautions to protect users and developers against feeds which may attempt to use enclosures in malicious ways.

To begin with, when a user subscribes to a feed in IE7 enclosure downloads are turned off by default. Users can easily opt-in to enclosure downloads via the feed properties.

We also treat enclosures as inherently un-trusted files – in many ways similar to email attachments. We decided not to permit directly-executable (i.e. any file that would execute arbitrary code when double-clicked) or other dangerous files to be downloaded as feed enclosures (there are no common scenarios that require this today, and if it is absolutely necessary, it is possible to wrap an executable file in another format, so that it is no longer directly executable). For this we use the most flexible mechanism possible, the Attachment Execution Service (AES). In simple terms, the AES maintains a list of file extensions that are considered dangerous, including the directly-executable file types, which the RSS platform consults to decide whether or not to block a file.

Besides blocking the dangerous file types, AES also has a mechanism which allows security programs, such as anti-virus or anti-spyware, to integrate with it, allowing them to inspect files before we make them available to developers or users. Windows Defender has implemented this integration, so on Windows Vista (or if the user has installed Windows Defender on Windows XP), the user will gain that additional level of protection from the malicious files.

IE also has a mechanism to block file downloads on a per-zone basis, so before fetching the enclosure we also verify that downloads are allowed for the URL. You can find this per-zone setting in your Internet Options, under Security tab. The simplest way to prevent enclosure downloads from a site is to add it to the Restricted Zone, where downloads are disabled by default.

If an enclosure download does get blocked for security reasons, this is reported in the feed view as well as through the RSS platform’s LastDownloadError property.

Downloaded enclosures are stored in a subfolder of the Temporary Internet Files folder. The full path to the enclosures is different on every machine, preventing malicious feeds or other malicious code from using enclosure downloads as a vector to get known files on the system, as well as ensuring that other applications don’t unknowingly access enclosure files. If an application wants access to the downloaded enclosures it needs to obtain the path from the RSS platform.

To summarize: enclosures are treated as un-trusted files, and the following security mitigations are used:

Enclosure download is off by-default for all feeds.
Directly-executable files are blocked from being downloaded, using the Windows Attachment Execution Service (AES).
Anti-virus and Anti-spyware applications (like Windows Defender) can integrate with AES to dynamically block malicious files.
Files are stored in a variable location on each PC, ensuring that applications must opt-in to consuming the enclosures.

As before, we want to make sure all aggregator developers know that the tools we are using to make IE and the RSS platform more secure are available for their use as well:

AES can be utilized through the IAttachmentExecute interface.
To determine if file downloads are allowed, applications can invoke the ProcessUrlAction method to query for URLACTION_SHELL_FILE_DOWNLOAD.

Once again, we would like to reiterate our commitment to working with the community to improve feed security, and as always we are open for your feedback and questions.

Thank you,

Miladin

Update 9/25/2006: Added a summary paragraph for clarity

More on Feed Security

rss — Sat, 09 Sep 2006 03:31:00 GMT

Shortly after the SPI Dynamics presentation that sparked a renewed discussion on feed security in the community last month, James Snell developed a suite of tests (based on an earlier set by James Holderness), and generously made them available quietly to aggregator developers. He has now made the tests public.

I contacted James last month (via email as he requested) and he pointed me to the test suite, so we could test them against our own security mitigations. We have done full test passes using his test suite.

The result: IE7 passed all of the tests (which means that no script from the feeds executed successfully in IE, and that developers using the RSS platform would not have been vulnerable to the class of attacks in the tests). This confirms SPI Dynamic's findings that IE7 was not vulnerable to the attacks described in their paper.

I thought it might be useful to use this opportunity to talk about our commitment to security, the defense-in-depth strategy that we have taken, and how other aggregator developers might benefit from the work we have done.

Our commitment to security

To put it bluntly, we are keenly aware that IE is a target for security researchers and hackers. We know we cannot afford to be lax in how we approach security. It has therefore been our #1 guiding principle that we would aim for a secure experience first -- sacrificing functionality, if necessary, to achieve it.

Long-time readers may remember this post from last November, in which we announced that we would only support well-formed XML in feeds -- the post was the direct result of a long internal discussion about ways to securely handle malicious feeds. Refusing to handle malformed XML eliminates a large class of potential attacks.

Walter posted last month on the details of how IE7 and the Windows RSS Platform protect users and developers from script in feeds. To summarize what he wrote, IE7 employs a (roughly) two-level defense-in-depth strategy:

Sanitization: First, the Windows RSS Platform uses several techniques to strip out script (and several other variations of malicious HTML) before storing the feed content.
Restricted Feed View: Second, just in case the first step misses something, IE's feed view uses a variation on the Restricted Zone to show a feed, meaning that no script in a feed will run, even if made it through the previous step.

Each of the two defense-in-depth steps described above require a significant amount of code and investment, but security has been always important enough to us that they where the first major pieces of development that we did when we began implementing the RSS features. In fact, these security features have been in place since the first public release of the IE7 RSS platform features last February.

To give you a sense of what is involved -- at one point in development, the sanitization code accounted for fully one-third of all the code in the RSS platform. The code takes lessons from similar libraries used for years to clean the billions of messages that Hotmail receives, and used for a number of releases in various parts of Office. It includes a number of feed-specific additions (for example, if an element is supposed to only contain text, then we can remove all HTML, not just the script). We validate and sanitize every documented element in each format we support, as well as a set of common RSS extensions. This is all done before an item is ever stored on the system.

In IE itself, the "restricted feed view" was also a significant challenge because of the interactive nature of a feed view. We designed and developed a feed view that required no javascript for the various controls (subscribing, filtering, sorting, or searching) to work. In fact, the IE7 feed view implementation is effectively that described by Nick Bradbury in his recent post on feed security (using a script-less page, and manipulating the view from the hosting code).

The bottom line is that IE takes security very seriously. We have invested a great deal of time in hardening IE7 across the board, and nowhere more seriously than in our RSS features. It is an ongoing process, however, and we deeply appreciate the efforts of those in the community who have developed additional security tests and allowed us to use them.

Looking forward

We also look forward to continuing to work with the community to improve the security of all aggregators. To that end, we want to make a couple offers to developers of Windows aggregator developers:

First, you should feel free to contact us if you have questions that come up while implementing a fully restricted feed view using the techniques that Nick talked about in his post. If there is enough demand, we may write a blog post on how the IE feed view is built, so people can learn from what we’ve done.
Second, the Feeds API includes a utility function called Normalize(), which can be used to gain access to the platform's HTML sanitization code. Contact us if you'd like more information on how you could use this to supplement your own sanitization code.
Finally, I’ll make the obvious point that the entire platform is available for your use, including not just the security features described here, but storage and a suite of bandwidth management features. I understand, of course, that for many existing aggregator developers, switching storage and download engines may be too significant a change in their applications, but I do want to encourage developers of new applications to consider it.

Thanks for reading,
Sean

PS. Of course, there will be some readers who see this post as a challenge and start looking for exploits in IE's RSS features. If you do find any, please let us know! :) We know that no security is perfect, and that it is an on-going process.

Script in Feeds

rss — Mon, 07 Aug 2006 21:31:00 GMT

You might have read the c|net article "Blog feeds may carry security risk" which summarizes the presentation given by Robert Auger ~~and Caleb Sima~~ of SPI Dynamics. The presentation points to potential dangers of malicious script embedded in feeds. This has sparked some discussion in the community.

We think it's good for the RSS community and users that the potential dangers of malicious script in feeds are pointed out and thereby can be addressed by application developers before any attacks materialize.

In IE7 and the Windows RSS Platform we've implemented several mitigations that specifically address potentially malicious scripts in feeds:

Sanitization
When downloading feeds, the RSS Platform passes the feed through a sanitization process which among other things removes script from HTML fields like the description element. Also, text fields, like the title element, are treated as text and not as HTML, so HTML tags are entity encoded. These steps are performed before the feed content is accessible by application including IE7's Feed View. Further, the feed content is persisted in the Feed Store in the sanitized form, so that applications accessing the feed data benefit from the sanitization.

Feed View in Restricted zone
The IE7 Feed View displays feeds in the Restricted security zone, no matter where the feed originated, even if for example the feed came from a site in the Trusted Sites zone. By default script is disabled in the Restricted zone. In addition, the Feed View disallows URL Actions including script and active content.

We designed and implemented the RSS features using the principles of the Secure Development Lifecycle as embraced by Microsoft. One of the principles is defense in depth. The idea being, even if script somehow were to sneak by the first layer of defense, the impact that the script could have is restricted, if not entirely negated.

Hosting IE in Applications
The second mitigation above can be of interest to application developers who are hosting MSHTML inside their applications. When using MSHTML to render feeds, we recommend that the host application implements a custom security manager, which allows the application to control which URL Actions are permissible. In order to reduce the attack surface of the application it is advisable to limit the permissible URL Actions to the smallest number possible.

I hope this will spark even more discussion about security and RSS which will ultimately benefit users.

- Walter vonKoch

[Update 8/16] Peter Plamondon of SPI Dynamics provided the link to the paper itself in the comments.

[Update 8/17] As noted by Sean Kerner in the comments, the presentation was given by Bob Auger solo. I've correct the intro above. Thanks.

RSS Platform Beta 3 Changes

rss — Sat, 29 Jul 2006 00:41:00 GMT

Jane talked about reading feeds with ease in IE7 Beta 3. I want highlight what is new in the Windows RSS Platform in Beta 3.

With the release of Beta 3, the Windows RSS Platform is now API complete. This means that, barring any serious bug that we must fix, applications written against the Beta 3 API will run unmodified against the final RTM release of the platform.

Since the Beta 2 release we've made the following changes to the RSS Platform:

Added support for updating all feeds: FeedsManager.AsyncSyncAll()
Added support for adding and deleting enclosures from the store. This allows for improved enclosure management and alternate enclosure download outside of the RSS Platform's enclosure download: FeedEnclosure.SetFile() .RemoveFile() and properties .DownloadUrl, .DownloadMimeType
Changed signatures of FeedFolderEvents.FolderItemCountChanged and .FeedItemCountChanged as well as FeedEvents.FeedItemCountChanged.
They now indicate which item count property has changed, either the item count or the unread item count of a feed or folder
Added FeedItem.Modified to reflect xml element <atom:updated> or <dcterms:modified>
Added FeedItem.Guid to reflect xml element <atom:id> or <guid>
Split the RSS Platform internally-generated XML elements from the Simple List Extension (SLE) namespace "cf" (http://www.microsoft.com/schemas/rss/core/2005) into their own namespace "cfi" (http://www.microsoft.com/schemas/rss/core/2005/internal)
For example, <cf:read> is now <cfi:read>. This new namespace is reserved for the internal use of the RSS platform, so we remove any elements in the "cfi" namespace from incoming feeds before processing them.
Final API changes based on feedback, including
1. Proper capitalization of certain properties in the automation/dual interfaces which are used by .NET managed code interop
2. Additional FEEDS_DOWNLOAD_ERROR's
3. Renamed Feed.Id to Feed.LocalId
4. Renamed FeedItem.Id to FeedItem.LocalId

We want to thank the numerous people that provided feedback in blogs, comments, emails and newsgroups posts which resulted in the above changes.

Let me reiterate that we now consider the RSS Platform API complete. We are not planning to change the API. Applications written against the Beta 3 API will run against the final release of the RSS Platform.

Note: We are aware that the MSDN documentation does match the latest API changes. We will update it, but don't have an ETA at this time.

- Walter

Events in VB.NET

rss — Thu, 08 Jun 2006 08:16:00 GMT

We’ve been getting a few questions on how to use RSS Platform events in VB.NET. The problem faced by most inquiries was the fact that the interface returned by the GetWatcher() method call couldn’t be cast to the FeedFolderWatcher or FeedWatcher type. Instead the returned interface can be cast to IFeedFolderEvents_Event or IFeedEvents_Event interface. As done here:

Imports Microsoft.Feeds.Interop

Module Module1

Dim WithEvents watcher As IFeedFolderEvents_Event

Sub Main()

Dim fm As IFeedsManager = New FeedsManager()

Dim rootFolder As IFeedFolder = fm.RootFolder

watcher = rootFolder.GetWatcher(FEEDS_EVENTS_SCOPE.FES_ALL, FEEDS_EVENTS_MASK.FEM_FOLDEREVENTS)

AddHandler watcher.FeedDownloading, AddressOf watcher_Downloading

End Sub

Sub watcher_Downloading(ByVal path As String)

Console.WriteLine("Downloading {0}", path)

End Sub

Sub watcher_DownloadCompleted(ByVal path As String, ByVal err As FEEDS_DOWNLOAD_ERROR) Handles watcher.FeedDownloadCompleted

Console.WriteLine("Downloaded {0}", path)

End Sub

End Module

The key is the highlighted part.

Note: We are aware that the IFeedFolderEvents_Event and IFeedEvents_Event interfaces do not appear in the Object Browser or in IntelliSense. We are looking into it.

Keep the feedback coming!

-Walter vonKoch

Book talking about the RSS Platform

rss — Fri, 05 May 2006 23:54:00 GMT

Last month, I reviewed the chapter about the Windows RSS Platform in Dave Johnson’s upcoming book RSS and Atom in action. According to his blog post, it’s set for publication in mid-June after which you should be able to pick up a copy. We’re looking forward to it!

- Walter vonKoch

Niall Kennedy, Feed Platforms, and Services

rss — Wed, 12 Apr 2006 11:17:00 GMT

I, for one, am very excited to have Niall Kennedy join the Windows Live team and drive the development of an RSS platform.

For one thing, it's great to have someone with a voice that is as well-respected as his joining Microsoft in any capacity. Just last month, at Etech, Jane Kim joined Niall onstage for his session on Feeds as a platform, where she talked about the principles behind why we are building a shared central platform for feed syndication into Windows. It was clear then (and from his excellent writeups on the topic) that he deeply understood the value of feed platforms.

More generally, it is great to have Windows Live focusing on building an online RSS platform. I have long believed that integration between online services and clients enables far richer experiences than either one alone (in fact, a key goal of our own Windows RSS platform is to make it super-easy for a client developer to integrate online content and services into their application).

On a related topic, I will go out on a limb and say that I can be a bit of a NewsGator fanboy. I've been a NewsGator ~~Outlook Edition~~ Inbox user for years (though these days, I admit, I feel compelled to use IE7's feed reader :) The synchronization support between the RSS platform and NewsGator Online that Greg showed off at Mix06 is just awesome. Greg and the NewsGator team demonstrate exactly what I love about integration between clients and services combining to deliver a great service that makes them both that much better.

In short, there is no end to the cool things that services enable, both the Windows Live ones and the ones done by folks like Greg and his team.

Sean

Windows RSS Platform Download Engine

rss — Sat, 08 Apr 2006 13:09:00 GMT

I wanted to take a couple of minutes to describe how the RSS Platform's download engine works. The behavior can be of interest to feed publishers who might be concerned about scalability as well as to developers and individual users who want to understand how their feeds are being kept up to date.

Features that help publishers manage network usage

Some of the concerns of feed publishers may have:

Number of hits (a client will hitting the same server many times; update frequencies)
Traffic spikes (many clients hitting the same server at the same time)
Bandwidth (many clients getting large responses from same server; response sizes)

Let's look at ways in which the RSS Platform addresses these issues. First, number of hits from a given client:

Download Schedules

Each feed in the Common Feed List has its own update schedule (such as "every 4 hours," "Once a week" or "Once a day."). The RSS Platform download engine operates in the background while a user is logged in, and checks each feeds for new content on the appropriate schedules.

Default and Minimum Intervals

As popularity of RSS increases, feed publishers may be concerned about the increasing number of hits they get from aggregators checking for feed updates. The Windows RSS Platform takes a fairly conservative approach and sets the default interval for feeds to 24 hours, meaning that by default each feed will be checked no more than once in a 24 hour period. This frequency might not work for every feed type and the user is able to set a custom interval for each feed. Users can also change the default feed interval.

However, in order to avoid accidental overuse of the server bandwidth, the RSS Platform limits the feed interval to a 15 minute minimum, meaning that the RSS Platform download engine will not perform a scheduled background download more frequently then every 15min. It is possible for an application to request an update at any time (for example, when the a user clicks the Refresh button - or hits F5 - in IE7, IE will ask the RSS Platform to update the feed immediately and will then display the results). However, the RSS Platform background download engine will not automatically update more often than 15 minutes.

TTL

The 15 minute minimum interval might not be large enough for some feed publishers and might still result in too many hits. Or a feed publisher might know that there won't be any updates to their feed for a certain time period and they'd like to advise clients to not hit their server more frequently then a specified frequency.

The RSS Platform download engine respects RSS 2.0's ttl tag (and the Syndication extension for both RSS 1.0 and Atom) by limiting the background downloads to no more often than the publisher specifies. For example, if an RSS 2.0 feed has a ttl of 180 (minutes) specified, the download engine will not check for updates more frequently than every 3 hours, even if the user has set the feed interval to 1hr. Note: as with the case of minimum interval, the user is able to manually refresh a feed more frequently then the 3hour ttl defined by the publisher.

The second major concern that publishers have is with several clients hitting their servers at the same time. Let's look at how the RSS platform helps here.

Interval "Salting"

Suppose the RSS Platform download engine were to check for updates for feed A exactly on the hour every hour. Thanks to Internet time servers, client's clocks tend to be fairly well synchronized. Taken togther, this would make it likely that many clients would make requests to the feed A at exactly the same time. This would lead to traffic spikes which are expensive (at best) for servers to handle, since they would need to scale out to handle the peak traffic.

In order to minimize the likelihood of severe traffic spikes the RSS Platform introduces a certain amount of randomness to each feed interval (this is referred to as "salting" the interval). After each successful download, it sets the next download time of a feed to be the time of successful download plus the interval plus a random fraction of the interval. The effect is that the download time is, in aggregate, spread out over a period of time, so that requests made to the same server from many clients.

Error back-off interval

Assume that a download does not successly complete for a particular feed, let's say because of a temporary problem on the server. One approach would be for each client to retry every in a couple of seconds. However, this might make things worse for a server that is already having problems. Over time more and more clients would "join the party" -- constantly trying to get updates from the server making it hard for a server recover. Conversely, if the client were to simply mark the download as failed and wait until the next scheduled download time (hours or days later), it may miss updates if the error was a transient one.

The RSS Platform uses a progressive back-off algorithm when there are errors getting a feed. Instead of retrying every couple of seconds, it doubles the retry interval on each iteration. On successive failures the retry interval eventually becomes as large or larger then the normal feed interval, at which point the normal interval will used.

The final major concern of publishers is with bandwidth usage. The RSS platform implements several of the recommended features that will help reduce bandwith on servers (e.g. see Nick Bradbury's post, or Randy Charles Morin's HowTo).

Conditional GETs

As mentioned earlier, bandwidth for RSS feeds will be of increasing concern for publishers. In order to help reduce bandwidth the download engine supports Conditional GETs using ETag and If-Modified-Since HTTP headers. If the feed hasn't been updated at all since the last time the client checked, the server can respond with an HTTP 304 (Not Modified) response.

Delta Encoding

In addition to standard conditional GETs the RSS Platform download engine supports Delta Encoding (for details, see Bob Wyman's post "Using RFC 3229 with Feeds") which allows the server to respond with only the feed items that are new or have been updated, thereby possibly reducing the response significantly.

Compression (gzip encoding)

Another beneficial feature that the RSS Platform supports is compression. Specifically, the RSS Platform supports gzip encoding of server response bodies which can reduce the response size significantly especially for RSS/XML.

Automatic unsubscribing

Finally, the RSS Platform implements support for the HTTP response 410 (Gone). When this response is received, the platform will automatically change the feed's update schedule to "Never." So when a feed is shut down, the server can inform clients that the feed is gone, so that they stop polling.

That covers the features of the RSS Platform download engine that address feed publishers primary concerns, and provides options on how to best manage the scalability requirements of their servers.

Features to help manage client network usage

But wait, there's more! --- (isn't there always?) :)

If you're a developer or a user, you might be interested to read about some additional features that the Windows RSS Platform has implemented to help minimize bandwidth usage on the client.

By default, the RSS Platform background download feature is off for new installations. This means that applications can request manual updates, but otherwise, the content will never be updated. Applications that use the platform should ask the user whether they want background updating, when they first use feed-related features (or at another appropriate time).

Once enabled, the download engine runs in the background whenever the user is logged into Windows. It is important that the download engine does not adversely impact other applications that the user is running at the same time. Since the download engine runs in the background the user typically won't know when it's started up. It would be very frustrating for the user if, all of a sudden, normal browsing or email downloading became slow for "no apparent reason." To help reduce the likelihood of this, the RSS Platform download engine implements the following set of features:

Parallel Download

When the background download engine starts up, it creates a list of all feeds that are ready to be updated. To speed up the process, up to four feeds will be checked in parallel, but no more than that. Too many simultaneous outbound requests might impact foreground Internet usage severely.

Throttled requests

Once one of the four parallel checks finishes the next feed in the pending list is checked. However, this could lead requests being made in a tight loop which can impact foreground Internet applications. In order to reduce this impact, the download engine throttles the number of requests that will be made in a given time span.

The engine uses an algorithm that works by gives the engine a "token" once per second up to a maximum of 4 tokens that it can "store up". If it has a "stored up" token then it can make the next request. So assuming that requests finish quickly, then in "steady state" it will make a new request once per second but not more frequently. Obviously, if it has more then one token "stored up" (due to a feed taking a while to download) then it can "burst" and make multiple requests but only up to the max of 4.

I hope that this overview of some of the features of the Windows RSS Platform download engine provides some information for feed publishers as well as developers and curious users that are interested in how the download engine works, and what impact it may have on the overall network as well client performance.

Even more...

The inquisitive mind will rightly point out: "Hey Walter, you haven't talked about download of enclosures!" You are correct. I will cover the details of enclosure download by the RSS Platform in a future post. If you have any particular questions about enclosure download you'd like to see answered, let me know in the comments.

-Walter

RSS at Mix06

rss — Fri, 24 Feb 2006 10:06:00 GMT

A quick plug for Mix06 - it's coming up fast (Mar 20-22), so get registered quickly.

In case you missed it, Mix06 is a conference that’s focused on Internet technologies so, not surprisingly, RSS will be there in force, with a bunch of sessions that feature RSS fairly prominently.

We'll have a session on the Windows RSS Platform, and you’ll see aspects of the platform show up in several other sessions and keynotes.

The Channel9 team is doing a series of videocasts on the Mix sessions. Check them out.

Members of the RSS team will be there for the entire time, so if you’re planning on being there and you’d like chat, drop us a line. It’s in Las Vegas, so how can it not be fun?

- Sean

Windows RSS Platform ala carte

rss — Thu, 09 Feb 2006 05:43:00 GMT

We just shipped the Windows RSS Platform with the IE7 Beta 2 Preview and questions about the RSS Platform are sprouting up. I wanted to address one of the most common questions first: how should applications can take advantage of platform?

Within our team we think of 3 major ways of how applications can use the RSS Platform:

Bronze: Integrating with the Common Feed List
Silver: Integrating with the Common Feed List and leveraging the platform’s Download engine
Gold: Leveraging the entire platform: Common Feed List, Download Engine, and Feed Store (including enclosures)

1. Bronze: Integrating with the Common Feed List

Applications in this bucket are interested in the list of feeds that the user is subscribed to. For a number of reasons these applications are less interested in leveraging the RSS Platform for downloading feed content. Examples of applications of this type are:

Existing RSS aggregators:
Existing RSS aggregators (like FeedDemon or RSS Bandit) typically have a significant investment in their own download engine and feed store. The Common Feed List was designed to make it possible for aggregators to hook into IE’s feed discovery feature – so that whenever a user chooses to subscribe to a feed using the IE subscription features, it’ll show up in the whatever aggregator the user uses.
Browsers or other applications that perform feed discovery and allow users to subscribe to feeds:
Any application on the PC that has the ability to find feeds for the user can add feeds to the Common Feed List if the user chooses to subscribe to those. Other applications can then “find” those feeds via the Common Feed List.
Online feed reading services:
Online feed reading services can provide tools that keep the Common Feed List in sync with other services, enabling these online services to tap into the IE’s feed discovery feature, in the same way that local aggregators do.
Import/export tools (e.g. into OPML or some other format):
IE 7 provides an OPML import/export function for users, but it’s possible for developers use the RSS platform Common Feed List to export the user’s feed list into any format desired.

Applications in this category generally work by reading the Common Feed List, listen for folder and feed events, and adding or deleting feeds to keep a mirror of all or part of the feed list. Outlook 12’s upcoming support for RSS uses this level of integration with the RSS Platform and thereby IE7’s feed discovery.

2. Silver: Integrating with the Common Feed List and leveraging the platform’s Download Engine

An application might choose to become RSS enabled for which the Windows RSS Platform can be helpful. Typically, such an application already has a significant investment in a data store and a user experience built on top of that data store. Even though the application has not previously supported RSS, it would be a major undertaking to implement the same user experience on top of another data store.

These types of applications are interested not only in the feeds the user is subscribed to but they also want to access to the downloaded content of the subscribed feeds.

Just two examples are email programs and NNTP newsreaders which have their own stores for items (news items, emails, …) and whose user experience is designed to read and write to their stores. It is often difficult for such applications to re-implement their user experience on top of another data store.

The RSS platform implements a Download Engine that handles scheduled downloading in the background, parsing of multiple feed formats, and the merge logic of determining whether an item is new or updated, etc. An application can avoid having to implement these building blocks by simply using the RSS Platform APIs including events to copy the feed contents into its own store. New RSS content is then “automatically” available to the application’s users.

In this way, the application takes advantage of the RSS Platform’s Common Feed List and Download Engine, while maintaining its own data store and UI rendering code.

3. Gold: Leveraging the entire RSS platform: Common Feed List, Download Engine, and Feed Store

This category applies to applications that have little existing in vestments in their own item store or rendering code, or they simply wish to take full advantage of the functionally the RSS Platform offers. These applications want to become “RSS–enabled” without having to implement an RSS stack.

These applications implement all of the functionality described in the scenarios above, but do not maintain their own store. They display the feed content directly from the the Feed Store (including enclosures). Many of the new RSS applications that we’ve talked about fit in this bucket. Imagine an RSS screen saver which displays pictures of photo feeds.

Conclusion

The key point I want to convey is that there is not just ONE way to integrate with the RSS Platform, but there are several. We’ve designed the platform to support several different ways of integration. The integration that you (as a developer) choose depends on the requirements of your application, the existing code base and functionality of your application. Those factors determine if and how deeply you might want to integrate with the RSS Platform. There is no one-size-fits-all for any developer platform, and the RSS platform is no different. Choose Bronze, Silver or Gold, and let me know which choice works for you and why.

- Walter