Welcome to MSDN Blogs Sign in | Join | Help

Microsoft RSS Blog

All about RSS and feed technology at Microsoft and across the community
Enclosure Download

A while ago I posted details about the RSS Platform Download Engine. That post focused on downloading of feeds, but did not include additional details on enclosure downloads.

   

Enclosures are, as most readers know, files that are "attached" to items in an RSS feed. Typically, a publisher will include a reference to a binary file, which an RSS aggregator can optionally download when the feed content is downloaded. The most common example of enclosure use in RSS feeds is for podcasting, where the attached (or "enclosed") files are audio files.

   

As with feed download, we designed the enclosure download with server and client bandwidth in mind since feed as well as enclosure downloads also happen in the background. Their impact on foreground applications should be limited. Similarly, the impact of large enclosure downloads on servers should be limited.

   

Let me sketch how the enclosure download process works:

  1. Every time the feed download engine runs it processes feeds that have the "Automatically Download Enclosures" setting set to true it. If it comes across a new item with an enclosure it adds the URL of the enclosure to a FIFO queue.
  2. Before the enclosure is added to the queue, the URL is checked with Attachment Execution Service API (AES) to assure the enclosure file type is one of the permitted types. If it's not, the enclosure download is failed (IFeedEnclosure.LastDownloadError = FDE_DOWNLOAD_BLOCKED).
  3. The first 4 enclosures in the queue are then handed off to the Background Intelligent Transfer Service (BITS). BITS is a background download service that ships in Windows and which enables downloading of files in the background while limiting its affects on network usage. In particular, BITS uses HTTP RANGE requests to download files in chunks. BITS also monitors whether foreground applications (like email or browser) are using the network, and if so, it throttles back its own network usage to limit its impact on those applications.
  4. Once BITS completes downloading an enclosure, the Download Engine uses AES to save the enclosure to the folder corresponding to the feed. Saving via AES associates zone information with the file. The zone information is used when the file is launched at a later time.
  5. If there are more enclosures waiting to be downloaded and there are less than 4 enclosure downloads active, the next enclosure is handed off to BITS as in step #3.
  6. If, however, the server of the enclosure does not support HTTP RANGE requests, the Platform Download Engine falls back to downloading the enclosure via a regular HTTP GET request. If this attempt fails as well, then the enclosure download is fails and will not be attempted again automatically.

       

Note that the enclosure fall-back download (HTTP GET) is size limited to 15MB to limit the impact of denial of service (DoS) attacks against the RSS Platform Download Engine. Since the RSS Platform Download Engine runs in the background, a malicious server could consume all of the client's download bandwidth without the user having any idea. Enclosure download via BITS (HTTP RANGE requests) is less impacted by such an attack and is consequently not size limited.

   

In other words, if you are an enclosure publisher that wants to serve enclosures larger then 15MB to IE7 users, then you should use HTTP servers that support HTTP RANGE requests. Most popular web servers support HTTP RANGE requests.

   

It's also worth noting, that when a server does not support HTTP RANGE requests, the RSS Platform Download Engine will issue two requests for each file (the first testing for HTTP RANGE support, and the second to download the file without range support).

For more details on the security measures used to protect applications and users from potentially malicious enclosures, see Miladin's enclosure security post.

   

I hope that this description of the enclosure download process explains the "multiple-requests" that some publishers have seen, as well the security restrictions associated with enclosure downloads.

   

-Walter vonKoch

Program Manager

Posted: Wednesday, December 06, 2006 7:46 PM by rss
Filed under:

Comments

rickw said:

I'm using IE7 and the feed list to manage the podcast I listen to and the auto-downloads work well.  Except for trying to get WMP to autodiscover the new media.  Right now I'm using a manual process.

1) Setup the feed to auto-download

2) Click the "view files" button and copy the folder to the clipboard

3) Go to WMP and add a new folder to monitor.  Paste the path from the clipboard.

4) Setup a Auto-Playlist to include the new podcast

5) Setup the sync to my portable player (a Sprint PPC-6700)

Any suggestions for simplifying this process?  

Are there events that an app could respond to notice that new feed is downloading attachments?

# December 6, 2006 4:06 PM

Palomar College said:

Is there a way to define the directory for downloads? It would be so much better for findnig the files later.

# December 6, 2006 4:37 PM

Matt Hamilton said:

Hi guys,

Is there perhaps a registry hack to extend the 15MB limit? We have work-related podcasts we're trying to download that are just outside that range, and so we have to manually download them.

# December 6, 2006 6:10 PM

ptvGuy said:

That will place a serious limitation on the Internet Explorer browser for feeds that contain video since it doesn't take much video to pass 15MB.  This will adversly impact not only video bloggers, but all public media content producers.

Just as more and more public broadcasting stations are adopting RSS feeds for sharing their local content, you're capping the file size in way that prevents most video and nearly any full-length audio program.  There has to be a better alternative than HTTP-Range checking.

The point of RSS is to make this kind of content syndication simple.  RSS is an XML format and should be totally independant of the server.

# December 7, 2006 12:39 AM

Todd Cochrane said:

A number of companies including mine absolutley hate that you all decided to continue using BITS it completly destroys stats reporting of downloads and puts ridiculous loads on servers that are serving the data.

Here is why instead of having to deal with one file request and serving it at once thus one call to our databases servers that are managing file distribution for load purposes you all send 100's of request for one single file.

One service provider is so pissed at the way Bits is abusive to those of use serving the data that they have BLOCKED BITS releated transfers.

All of consider the protocol to not be friendly to us serving up data and you continue to not win friends in your utilization of bits. You should have talked to the people really serving data!

# December 8, 2006 12:26 AM

Ari Pernick said:

It's intresting what you did with the security model, but I think you should allow "infinte" sized simple HTTP GET downloads for enclosures that come from the same server who supplied the feed.

# December 9, 2006 12:11 PM

Rob Walch said:

So if I am reading all of this correctly.  IE7 is really not an aggragator in that it does not actually download all of the shows and then place them with your other Rich Media.  

The real reason podcasting took off - was not that you could add an enclosure and have some one subscribe and download to it.  Rather it went a crucial step further (or at least Adam Curry did) it was setup such that the files after being downloaded were placed with all of your other rich media files so the subscriber could easily find and consume the media.  

With out that crucial step - it is not podcasting.  It is just downloading via RSS.  At least if the file size is less than 15 Meg.  

Which by the way most podcasts are well over 15 Meg in size.

So when is MS coming out with a truly intergrated Podcasting solution for the subscriber?

Rob @ podCast411

# December 26, 2006 7:55 PM

Wilbur said:

How about a new year's resolution for the RSS team:

Allow users to choose the maximum size of enclosure to download and let them decide whether or not to trust the server.

If you are feeling really virtuous, how about another one: Allow users to set the download location for enclosures.

I know that Microsoft is always being lambasted for not catering to the lowest common denonminator but these two things are pretty vital to achieving the functionality needed to make IE7 achieve its objectives.

# January 4, 2007 9:14 PM

Wilbur said:

How clever to aviod DoS attacks the size of the enclosure is limited to 15MB. Nobody would ever have a podcast bigger than that surely?

Check out: http://msdn.microsoft.com/msdnmag/podcast/default.aspx

8 podcasts, 4 of which would not be automatically downloaded by IE7. What internet vandal would host such large files?

# January 22, 2007 7:49 PM

rss said:

@Wilbur,

You might have misunderstood the 15MB limit. It's a fallback limit which *only* applies if the webserver does *not* support HTTP RANGE requests. The enclosure server download.microsoft.com does support HTTP RANGE requests. So if you try it and turn on automatic enclosure download for MSDN podcast feed you will see all enclosures downloaded.

I hope this clears up the confusion.

- Walter [MSFT}

# January 26, 2007 12:52 PM

Tom Gleeson said:

Does the feed manager recognise links with the micro format rel=enclosure as enclosures or does it look for the <enclosure> tag? What about Atom enclosures?

# March 10, 2007 3:59 PM

James said:

Please make it possible to set the download location of enclosures!

# April 20, 2007 2:09 PM

Dann Veldkamp said:

One more request for the ability to specify a download folder for each feed. Without this I am left with iTunes or manually downloading my podcasts so I can burn them to CD for listening in the car.

Dann

# May 9, 2007 7:59 AM

ali said:

Walter  -

You're not being misunderstood. You need to make it so that this does not bother users by not allowing them to download podcasts. Here's a better way of preventing bandwidth wastage: Monitor the feeds bandwidth usage pattern and inform the user of any abnormalities that appear. This will actually turn a stupid restriction into a useful feature.

# May 31, 2007 8:46 AM

DavidWT said:

I agree with Dann, being able to specify (different) download folders for different auto downloaded  feeds would be a great advantage. Also feeds that are different auto downloaded go to Temporary Internet Files and so could easily be deleted forgetting that there are feeds downloaded.

# June 2, 2007 7:56 AM

Lanad said:

لوتسمح هل من الممكن توضع رابط تنزيل لا الشرح وحدة لا يكفي ولك مني جزيل الشكر .

# July 5, 2007 8:16 PM

LinkDir said:

It's intresting what you did with the security model, but I think you should allow "infinte" sized simple HTTP GET downloads for enclosures that come from the same server who supplied the feed.

# July 10, 2007 9:22 PM

RSS Podcasting said:

Thanks Walter, Good to know. 15mb limit would be useless for our guys. I see what you've done now, it's starting to make sense. Thanks again.

# September 5, 2007 11:49 PM

Tweak Vista said:

A number of companies including mine absolutley hate that you all decided to continue using BITS it completly destroys stats reporting of downloads and puts ridiculous loads on servers that are serving the data.

Here is why instead of having to deal with one file request and serving it at once thus one call to our databases servers that are managing file distribution for load purposes you all send 100's of request for one single file.

One service provider is so pissed at the way Bits is abusive to those of use serving the data that they have BLOCKED BITS releated transfers.

All of consider the protocol to not be friendly to us serving up data and you continue to not win friends in your utilization of bits. You should have talked to the people really serving data!

# January 19, 2008 7:08 AM

Veridom said:

How about a new year's resolution for the RSS team:

Allow users to choose the maximum size of enclosure to download and let them decide whether or not to trust the server.

If you are feeling really virtuous, how about another one: Allow users to set the download location for enclosures.

I know that Microsoft is always being lambasted for not catering to the lowest common denonminator but these two things are pretty vital to achieving the functionality needed to make IE7 achieve its objectives.

# January 19, 2008 7:08 AM

TheCableGuy said:

That will place a serious limitation on the Internet Explorer browser for feeds that contain video since it doesn't take much video to pass 15MB.  This will adversly impact not only video bloggers, but all public media content producers.

Just as more and more public broadcasting stations are adopting RSS feeds for sharing their local content, you're capping the file size in way that prevents most video and nearly any full-length audio program.  There has to be a better alternative than HTTP-Range checking.

The point of RSS is to make this kind of content syndication simple.  RSS is an XML format and should be totally independant of the server.

# January 19, 2008 7:09 AM

LFERC said:

I'm using IE7 and the feed list to manage the podcast I listen to and the auto-downloads work well.  Except for trying to get WMP to autodiscover the new media.  Right now I'm using a manual process.

1) Setup the feed to auto-download

2) Click the "view files" button and copy the folder to the clipboard

3) Go to WMP and add a new folder to monitor.  Paste the path from the clipboard.

4) Setup a Auto-Playlist to include the new podcast

5) Setup the sync to my portable player (a Sprint PPC-6700)

Any suggestions for simplifying this process?  

Are there events that an app could respond to notice that new feed is downloading attachments?

# January 19, 2008 7:09 AM

moderowany katalog said:

How do I change the default directory for downloads?

# March 20, 2008 3:23 PM

enciclopedia said:

You cant! You must wait for the soft update.

# March 26, 2008 8:22 PM

Lublin ogłoszenia said:

Thanks for very interesting article. btw. I really enjoyed reading all of your posts. It’s interesting to read.

# April 15, 2008 6:04 PM

czesci do drukarek said:

it is really good article, I found the reason for my problem!!

Thanks

# April 20, 2008 5:00 PM

Darmowe Ogłoszenia said:

Thanks for the article. I really enjoyed reading all of your posts.

# June 1, 2008 6:04 PM

Domki pod sosnami said:

Thanks for very interesting article.

# June 7, 2008 4:42 PM

Padova said:

It's easy to overcome the 15 MB limit - just set the registry value HKCU\Software\Microsoft\Feeds\MaxEnclosureSize (DWORD) to a suitable value (in kB). There is a similar value for the maximum XML size (also 15 MB by default). If you are writing a podcast app that uses the common feed list, this is easy enough to adjust. More details are on MSDN.

# June 28, 2008 12:24 AM

Padova said:

@LFERC - Here's what I do:

1. From the command line, type

mklink /j %userprofile%\Music\Podcasts %localappdata%\Microsoft\Windows\Temporary Internet Files\Enclosure"

This will make the podcasts visible to WMP.

2. In WMP, create an auto playlist with a filename that contains "Music\Podcasts".

Now all you have to do is sync that playlist to your audio player. It works great!

# June 28, 2008 12:32 AM

rzeszow said:

now I get the multiple-requests subject. thanks! ;-)

# July 1, 2008 1:42 AM

ogłoszenia motoryzacyjne samochody sprzedam samochód said:

Great article. It`s realy worth reading. I wish you further successes

# July 9, 2008 6:41 AM

omaha rules said:

It's intresting what you did with the security model, but I think you should allow "infinte" sized simple HTTP GET downloads for enclosures that come from the same server who supplied the feed.

# July 25, 2008 3:44 AM
Leave a Comment

(required) 

(required) 

(optional)

(required) 

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Page view tracker