jeff's WebLog : internet explorer

last post

jeffdav — Wed, 02 May 2007 18:49:00 GMT

Readers-- I'm leaving MSFT for other opportunities. As such, this will be my last post. Thanks for reading!

I hope you will all continue to read the IE Team Blog: http://blogs.msdn.com/ie

on CComQIPtr

jeffdav — Wed, 11 Apr 2007 01:21:00 GMT

You're writing some code that uses COM. Do you use SmartPointers or not use SmartPointers? Debate on this topic flares up now and then.

The trick to answering it is to seperate the technical facts from stubbornly held beliefs. Like any technology, there is a right time and a wrong time to use it. But that's not what I want to talk about today. Let's assume you've decided to use SmartPointers. I posit that CComPtr<> is good and CComQIPtr<> is bad.

The reason for this is simple: CComQIPtr<> masks the return value of the QueryInterface() call.

Lets take a look at two snippets of code to frame the discussion:

void ExFunc(IFoo *pFoo)

{

HRESULT hr = pFoo->Init();

if (SUCCEEDED(hr))

{

CComPtr<IBar> spBar;

hr = pfoo->QueryInterface(IID_PPV_ARGS(spBar));

if (SUCCEEDED(hr))

{

hr = spBar->DoSomething();

}

void ExFunc(IFoo *pFoo)

{

HRESULT hr = pFoo->Init();

if (SUCCEEDED(hr))

{

hr = E_NOINTERFACE;

CComQIPtr<IBar> spBar = pFoo;

if (spBar)

{

hr = spBar->DoSomething();

}

Some points:

When you have a large number of people working on the same code base, you want to create patterns in the code that people recognize. CComQIPtr<> breaks the pattern of COM-call/check-HRESULT. Instead you have to check for the non-NULLness of the out parameter.
You can assume the QueryInterface() call failed with E_NOINTERFACE, but you don't know for sure. What if it failed with some other HRESULT? Nobody says QueryInterface() can't do things besides just casting this around. E.g. a function that implements IPersistFile::Load() could delay loading the file until someone does a QueryInterface() for an interface that actually needs the contents of the file it was asked to load.
CComQIPtr<> masks errors in QueryInterface(). The contract for QueryInterface() is the out parameter must be valid if it returns success, otherwise it must be set to NULL. So, in theory, if (SUCCEEDED(hr)) { ... } and if (spBar) { ... } should be equivalent. However, people often get QueryInterface() wrong. It seems to me that it is much more likely an implementer will fail to set the out parameter to NULL on failure than they will fail to return S_OK on success.
Lastly, I just don't like the fact that magic is happening in the assignment. It is easy to screw things up with COM, and it seems CComQIPtr<> just makes it easier. CComPtr<> is a lot less magic and more predictable. It makes it harder to screw things up.

In summary: use CComPtr<> and don't use CComQIPtr<>. I would especially encourage those who would write Internet Explorer extensions to use CComPtr<>.

Thoughts?

why doesn't the favicon for my site appear in IE7?

jeffdav — Fri, 02 Mar 2007 00:03:00 GMT

Today I have a mini-FAQ on Favicons.

Q: How do I make a favicon appear for my site in IE7?
A: There are two ways. The first is to put a file in the root of your domain called favicon.ico. The second is to use a <link> tag with the rel="shortcut icon" value and the href value set to the URL for the Icon you wish to display.

Q: How often does IE download the favicon?
A: IE will download the icon when a user first visits the site. The icon is stored in the Temporary Internet Files folder on the client machine. Additional metadata about the favicon is stored in the user's Url History database. If either store is cleared, or items relating to the favicon have naturally expired, then the icon will be downloaded again on the next visit. If more than one page (or site) shares the same favicon, it is only downloaded once. IE takes great pains to download the icon as few times as possible to reduce load on the server.

Q: I see the wrong favicon for some sites I visit. How do I fix this?
A: If the history database has become corrupted in some way, this can happen. The simplest solution is just to use Delete Browsing History (on the Tools menu) to clear the cache and the history store.

Q: I put a favicon.ico on my site as you described, but it still doesn't appear.
A: It must actually be a .ico (an Icon) file. Bitmaps, pngs, gifs, etc, will not work. IE7 will download your favicon to the Temporary Internet Files folder and call ExtractIcon() on the file. If this fails, we will show the default icon instead of your favicon.

Q: I verified that my favicon really is an icon, but it still doesn't appear.
A: Since IE loads your icon out of the Temporary Internet Files folder, it must be able to actually store it there. If you are setting the no-cache directive for the icon file, then IE will not be able to display your icon and will display the default icon instead. You can use Fiddler to verify.

Q: How do I create a different favicon for every page on my site?
A: Put a different <link> tag on each page, pointing to a different icon.

Q: I changed my site's favicon to a different icon, but the old one still shows in IE. How do I force IE to update?
A: If you just put the favicon.ico file in the root of your domain, IE doesn't have any way of knowing if it changed. To force an update, you need to use a <link> tag and point to a different filename than you previously used. The current filename is compared against the known filename stored in the Url History database. When IE sees the filename has changed, it will download your new icon. Alternatively, you can ask your users to clear their history and cache (Tools->Internet Options->Delete Browsing History), which will also force IE to download the new file.

That should cover most of the questions I've received about favicons in IE7. If you have more questions, feel free to ask.

Updated on Monday, 5th March to fix a spelling error and add some additional questions.

msdn blogging, ie7 and the future

jeffdav — Sun, 07 Jan 2007 01:20:00 GMT

My posting rate has declined somewhat since we shipped IE7. It seems posts come in bursts. I have a theory for this.

At the beginning of a product cycle we are focused on planning, architecting, and implementing. There is not much to talk about, since we can not talk about anything that has not been publicly announced yet. A few posts on context-free problems may trickle out.

Then we ship a Beta and the bugs start rolling in. This is where things get interesting. Lots of problems to solve translates into having lots of interesting topics to post about.

Then we lock down for the final release. We focus on fixing the last few compat issues, localizing, branding, and marketing. The devs start dreaming about the next release. Then Thanksgiving and Christmas come a long and we're all taking vacations... Now the new year has started and we are back in the planning phase, which I can't talk about.

Imagine IE is a ship. We just got back into harbor; we delivered our goods. Some of the crew has moved on; we have taken on some new members. Some are still trickling back in from shore leave. Some of us are focused on fixing damage from the last voyage. Some of us are scrubbing the decks and taking care of day-to-day business. Some are dreaming about our next destination and what the voyage there will look like.

And from there the metaphore starts to brake down, but I though it was pretty good. I just like big ships. Anyway, we are hard at work. I have a lot of hard thinking and planning to do. I will not be posting until I have something interesting to say; but I have faith that I will, indeed, post again. At the very least I will have to explain the decisions I am about to make when we ship our first Beta :) -- in the mean time, there is always the IE Blog. (If you're just interested in what I'm doing, feel free to drop in on this blog, which has an entirely different focus.)

I also wanted to do a small summary of IE7, just for some personal closure.

Overall I think IE7 is great. I personally use it at home, which is my bar for everything we build. I have worked on a product (which shall remain nameless) that I did not use at home. Those who know me know my philosophy on installing software: just don't do it. But I have great faith in IE7 and I find myself cursing machines I have to use that have not installed it (I'm looking at you CS Department...). Sure, there are some things that still need work, and I am happy to see us addressing them in our planning for the next version.

In the things-I-worked-on department, I am thrilled that the favicon code I wrote is performing well and web server administrators are not yet calling for my head on a pike.

I think it is great that the Quick Tabs feature has been so well recieved. For those who don't know, QT was originally implemented by two of the most industrious Interns I have worked with, Kevin and Eric. They wrote a lot of code. I mean a lot. They prototyped it (by which I mean they had full working versions, just minus fancy window dressing) in three different technologies. They blew way past the spec and saved a great feature from the cut list. When they left all I had to do was put some drop shadows in, tweak some aspect ratios and fix some performance-related issues (and some other bugs, as which the tester would be sure to mention :P).

I think that tabbed browsing beat everyone's expectations and is, in my opinion, the new gold standard in tabbed browsing UI. It is very usable UI. There were some very tough trade-offs, places where the behavior people expected split fifty-fifty in complete opposites. There were some incredibly challenging app-compat issues to sort out. I use it everyday, and after I changed some options (Tools->Internet Options->Tabs->Settings) it works pretty much the way I want it to.

I am still not 100% happy with the Delete Browsing History dialog layout, but we made vast improvements, the most important of which was changing it to use the copy engine infrastructure so that the deletion operations no longer happen on the UI thread. About 0.5% (which looks small, but is huge considering the size of the data set) of all IE crashes that were reported to Microsoft were people who had killed the IE process because clearing the cache on the UI thread made them think the process was hung.

And the features that got cut, well, I think we made the right decisions. May they rise again some day. And I am sure there are some other things I forgot.

In the mean time, weigh anchor -- full speed ahead!

pop-up blocker and ActiveX controls, part three: creating a user initiated action

jeffdav — Fri, 21 Jul 2006 01:19:00 GMT

Note: This is part three in a series of posts explaining how ActiveX controls and the IE Pop-up Blocker should interact. The first post is here. If you are a user (and not an ActiveX control developer), and have found this post in an attempt to track down why you are still seeing unwanted pop-up windows, please refer to this post.

Problem: You have written an ActiveX control. You have done your due diligence in making sure it does the right thing as described in parts one and two of this series. Now you find a pop-up that should not be blocked is being blocked.
Example: A real life example happened to me a few days ago; a Microsoft control, in certain circumstances, launches a dialog and then tries to open a new window in response to certain options on that dialog. The user initiated action is long past at that point.
Solution: When we shipped WindowsXP SP2 we add two new commands to CGID_MSHTML (see mshtmcid.h in the latest SDK). The first is IDM_BEGINUSERACTION and the second is IDM_ENDUSERACTION. They are pretty self-explanitory. Send the first command to tell mshtml a user initiated action is beginning. You will be able to open a new window. Once you have opened your new window, send the second command to tell mshtml that the user action has finished.

For more info on sending commands to mshtml, see this post.

pop-up blocker and ActiveX controls, part two: HlinkSimpleNavigateToMoniker()

jeffdav — Fri, 21 Jul 2006 01:00:00 GMT

Note: This is part two in a series of posts explaining how ActiveX controls and the IE Pop-up Blocker should interact. The first post is here. If you are a user (and not an ActiveX control developer), and have found this post in an attempt to track down why you are still seeing unwanted pop-up windows, please refer to this post.

Method: HlinkSimpleNavigateToMoniker(), HlinkSimpleNavigateToString().
Mitigation: I would like to discourage you from calling these functions. They are not, in my opinion, very good functions. If your control must navigate, you should use the IWebBrowser2 function. If you are already calling these functions, and cannot change things, then what you must do is pass the correct pointer in the pUnk parameter. The documentation implies that if pUnk is NULL and you are an ActiveX control then your navigation will not succeed. This is not the case--the function has no good way to even tell if you are an ActiveX control or not. You identify yourself as one or the other by passing or not passing a non-NULL value. If you pass NULL, then the pop-up blocker will not get involved. If, however, you pass a non-NULL value, you have identified yourself as an ActiveX control and pop-up blocking will be in effect. You should not lie to this function.

The documentation is also not very clear on what, exactally, you should pass in the pUnk parameter or what it is used for. Your control is being hosted by mshtml. You should pass a pointer to your client site, which is the object in mshtml hosting you. The HlinkSimpleNavigate functions will attempt to query this object for various interfaces related to pop-up blocking if your call is going to result in a new window. If they fail, your new window will be blocked and the HlinkSimpleNavigate function will return E_ACCESSDENIED. For more information about client site pointers, see this post.

Additionally the function will attempt to get pointers to various webbrowser interfaces to perform the navigation (whether it is a new window or not). If this fails (because you passed NULL) a new browser process will be created and navigated.

pop-up blocker and ActiveX controls, part one: IWebBrowser::Navigate(), IWebBrowser::Navigate2()

jeffdav — Thu, 18 May 2006 00:42:00 GMT

As I mentioned previously, one reason users may continue to experience unwanted pop-up windows while browsing is creative use of ActiveX controls that provide methods that allow web sites to open new browser windows. This series of posts will provide best-practices for ActiveX control implementors.

There are two things to keep in mind while considering this topic. The first is, as always, we are committed to application and site compatibility and thus a lot of the pop-up blocker is opt-in and in the abscence of concrete knowledge that a pop-up is unwanted, our default is often to allow the pop-up. The second thing is there are an incredible number of ways to navigate IE, so this will, necessarily, be part one in a survey of this multitude.

Note: If you are a user (and not an ActiveX control developer), and have found this post in an attempt to track down why you are still seeing unwanted pop-up windows, please refer to this post.

Method: IWebBrowser::Navigate(), IWebBrowser2::Navigate2()
Mitigation: Each of these methods takes a set of BrowserNavConstants in the Flags parameter. Controls (and applications) should pass navNewWindowsManaged if they desire pop-up blocker to be applied. Clues that you want this flag would be such things as you are also passing navOpenInNewWindow or are targeting a frame name via the TargetFrameName parameter and that frame may not exist--in which case a new browser window may be created with that name.

IOleCommandTarget, CGID_MSHTML and ActiveX controls

jeffdav — Sat, 22 Apr 2006 00:02:00 GMT

If you write an application that hosts the WebBrowser Control, and you want the control to do something, you can send commands to mshtml via the IOleCommandTarget interface.

However, if you are an ActiveX control and you want to send CGID_MSHTML commands, you may try something like this:

    ...
    IOleCommandTarget *pCommandTarget = NULL;
    hr = _punkSite->QueryInterface(IID_IOleCommandTarget, (void **)&pCommandTarget);
    if (SUCCEEDED(hr))
    {
        hr = pCommandTarget->Exec(&CGID_MSHTML, IDM_FOO, 0, NULL, NULL);
        pCommandTarget->Release();
    }
    ...

If you have tried this, you will see that it fails for CGID_MSHTML commands. It fails because your control is hosted by an object inside of mshtml.dll that delegates all incoming commands to the nearest DocHostUIHandler. For IE, that is implemented by an object in shdocvw.dll (or ieframe.dll for IE7+). That object does not recognize CGID_MSHTML commands.

In order for your call to be routed correctly, you need to get an object "above" the object that is hosting your control. To do this, you can first ask the client site for an IServiceProvider:

    ...
    IServiceProvider *pServiceProvider = NULL;
    hr = _punkSite->QueryInterface(IID_IServiceProvider, (void **)&pServiceProvider);
    if (SUCCEEDED(hr))
    {
        IOleCommandTarget *pCommandTarget = NULL;
        hr = pServiceProvider->QueryService(SID_SContainerDispatch, IID_IOleCommandTarget, &pCommandTarget);
        if (SUCCEEDED(hr))
        {
            hr = pCommandTarget->Exec(&CGID_MSHTML, IDM_FOO, 0, NULL, NULL);
            pCommandTarget->Release();
        }
        pServiceProvider->Release();
    }

By asking for the ContainerDispatch's command target, you get the correct target for MSHTML commands.

on getting IOleCommandTarget wrong (and a bit in the middle about ActiveX controls)

jeffdav — Tue, 11 Apr 2006 21:32:00 GMT

IOleCommandTarget is very useful. It provides a generic way of sending commands between objects. IE makes extensive use of IOleCommandTarget, both publically and internally. And, like IUnknown, people frequently get it wrong.

Each command is composed of a GUID (Command Group Identifier) and a DWORD (Command Identifier).

First, what is wrong with this code:

HRESULT CFoo::Exec(const GUID *pguidCmdGroup, DWORD nCmdID, DWORD nCmdexecopt, VARIANTARG *pvarargIn, VARIANTARG *pvarargOut)
{
   if (IsEqualGUID(CGID_FooCommands, *pguidCmdGroup))
   {
      ...

The answer is obvious: pguidCmdGroup might be NULL and you crash. Furthermore, pguidCmdGroup is frequently NULL, since NULL is how you specify the standard group.

You might say to yourself, "Self, I don't need to worry about NULL, since I am the only consumer of this particular implementation," and that may be the case. However, if you can be CoCreate()'ed, than it is definitively not the case. Remember, any control on your machine can be instantiated as an ActiveX control. During the ActiveX control instantiation process, mshtml will query your object for certain well-known interfaces, like IObjectSafety, IOleObject, and, you guessed it, IOleCommandTarget. If your control responds to IOleCommandTarget, mshtml will attempt to Exec() standard commands. If you do not guard for NULL, you will cause Internet Explorer (and other hosts) to crash.

The second set of common errors come in the form of returning incorrect error codes. The documentation outlines the rules and you must be careful to implement them. If you recognize the group but not the command, OLECMDERR_E_NOTSUPPORTED is the correct return value. If you do not recognize the group, OLECMDERR_E_UNKNOWNGROUP is correct. There is a special case though, which is when the group is NULL and the command is not supported; in this case the correct return value is OLECMDERR_E_NOTSUPPORTED.

on IObjectWithSite, IOleObject and ActiveX controls

jeffdav — Tue, 28 Mar 2006 19:45:00 GMT

ActiveX® controls frequently need to communicate with their containing object. For example, a control may want to QueryService for the cached InternetSecurityManager object to decide whether or not to take a particular action. Controls can obtain a pointer to their containing object (also called the object's site) in one of at least two ways.

The typical scenario is Internet Explorer hosts the ActiveX® control inside of an object implemented in mshtml.dll. Controls that need to communicate with their container implement IOleObject and mshtml will call the SetClientSite method with a pointer to itself. Communication can now proceed by querying the given IOleClientSite pointer.

However, there is another scenario that authors of ActiveX® controls must consider. Controls can be dynamically created through script, e.g.:

<script> oAX = new ActiveXObject('MyControl'); </script>

In this case, the control is now being instantiated not by mshtml.dll, but by jscript.dll. Since jscript.dll does not implement any OLE containers, it does not set a client site pointer via IOleObject. It does, however, maintain an IServiceProvider chain to mshtml that can be accessed by an ActiveX® control. The control must implement IObjectWithSite. Now jscript will be able to call SetSite and the control can query the IUnknown pointer for services it requires from its container.

Astute readers may, at this point, notice something funny in the documentation. IObjectWithSite's documentation says, without elaboration: "This interface should only be used when IOleObject is not already in use." This has to be read carefully. Notice the documentation uses the words "...in use," not "...implemented." You should keep track of the two different site pointers and prefer the one obtained from IOleObject. If, however, you find that site is NULL then IOleObject is not being used and it is acceptable to fallback to using the site obtained from IObjectWithSite.

Debugging and the C++ Renaming Problem

jeffdav — Mon, 18 Apr 2005 21:20:00 GMT

(Today we have another installment of Jeff's Real Microsoft Debugging Stories.)

Recently a minor piece of an IE feature I own stopped working in our latest internal builds, so I debugged it. Issues where something that has worked for a long time and suddenly stopped working are always a special kind of fun, especially when I have never touched the code. The problem was subtle--when I stepped through the code everything appeared to suceed, yet it still did not work. Finally I went line by line comparing the execution path in our alpha code to what we shipped in Windows XP Service Pack 2. For a long time, everything was the same, until I stepped into a call to IConnectionPoint::Advise() and ended up in two different functions on the two different machines. What happened, as Raymond later pointed out, was we had encountered the Renaming Problem, defined in The Annotated C++ Reference Manual, section 10.11c. Consider:

   Class CFoo : CConnectionPointImpl<...goo...>,
                        ...,
                        public ISomeNewInterface,
                        ...

The issue here was one of our classes inherited from a base class that did the dirty work of setting up the connection point. Then, for the next version of the product, someone added another interface (ISomeNewInterface) to CFoo and implemented all the necessary functions. Unfortunately, the new interface also had a method called Advise() that had the exact same footprint as IConnectionPoint's Advise() function. Thus the new implementation was overriding the old implementation.

So, the question is, what do we do about it? Of course there are several immediate solutions. Stroustrup's book gives an example where you have a class that inherits from two different base classes that happen to have a naming collision. He suggests implementing two wrapper classes for the two base classes with different method names that forward to their base class implementation appropriately. In my opinion this is a bit ugly. The easiest solution is to change the name of the method on the new interface, since we have not shipped it yet. I also found that explicitly declaring the new Advise() as being a member of the new interface and not IConnectionPoint in the header file worked, i.e.:

   ...
   // ISomeNewInterface
   HRESULT ISomeNewInterface::Advise(IUnknown *, DWORD *);
   HRESULT SomeOtherMethod();
   ...

However, I have not been able to find one way or the other if this is standard and legal C++ syntax or if I am getting lucky with this compiler. Thoughts?

Edit: The MSDN team informs me they are actively tracking a bug that the IConnectionPoint documentation does not work.

(Thank you for joining us. Tune in next week when we talk more about how History works.)

Index.dat Part IV - It's doing what on the UI thread?

jeffdav — Thu, 02 Dec 2004 19:19:00 GMT

Whenever someone kills an application because it is hung we get a little report with the stack and some other information (if the user chooses to send it). About 1% of the reports for IE are from users who have killed IE (or the Internet Control Panel) while it was clearing the cache. This is only slightly related to index.dat; I am using this series of postings as an excuse to talk about it on the assumption that someone might find it interesting.

In IE, Tools->Internet Options->Delete Files...->OK. Depending on how big you have set your cache and how long it has been since it was last cleared, this little operation can take a long time. I have personally seen it take as long as 20 minutes, which is a long time to stare at the little hourglass cursor. Of course the Internet Control Panel dialog is modal to the IE window, which means you cannot get back to whatever you were doing in the IE window until it is done. And if you click on things like the title bar, etc, as most people do when some program is taking an infuriating amount of time to accomplish something, Windows appends the [not responding] text to the title text. So people kill it.

The moral of this story is known to us in advance, I think: do not do blocking stuff on the UI thread.

It is an interesting problem though. The solution is not as straight-forward as one may initially think. The obvious thing to do is to spin up a background thread and have the cache-clearing done on the new thread. Assume that has been done; you open IE, clear the cache, the thread kicks off, you close the inetcpl, you close IE, you shutdown your computer. Did the cache get cleared? You asked for it to be cleared but you do not know for sure. What if it was still going 20 minutes later when you shutdown your machine? There are privacy implications here--I want to know when it has finished.

A proper solution would implement some sort of feedback, like file copy operations or the file download dialog in IE, that can live in a non-modal state even if the parent IE window has closed. Which is probably why nobody fixed it in previous releases--bigger fish to fry.

Index.dat Part III - How do I delete index.dat?

jeffdav — Fri, 19 Nov 2004 21:00:00 GMT

These data files are used by Internet Explorer and Windows Explorer. You cannot delete a file that is in use by a running program. If you feel you need to delete the file, you will have to shutdown all instances of Explorer and IE. This includes applications that may host the Webbrowser control: Outlook, Messenger, IE, Product Studio, Visual Studio, Help, Windows Media Player, etc. Your best bet is just close everything. When you are left with a desktop and a start menu, you will still need to shutdown Explorer. To cleanly shutdown Explorer: Start->Shutdown->CTRL+ALT+SHIFT+Click 'Cancel' (for more info, see this post). You can use Task Manager (CTRL+SHIFT+ESC, File->Run...) at this point to open a command window. You should be able to go delete the index.dat. I have only tried this on XPSP2, but it should work anywhere.

Index.dat: Part II - What are they used for?

jeffdav — Fri, 19 Nov 2004 00:06:00 GMT

As I mentioned in Part I, index.dat files are used by various features of IE to keep track of URLs and associated information about URLs. The two primary consumers of these files are the Temporary Internet Files folder (TIF) and History.

TIF uses an index.dat file to map URLs to the associated Cache file name. Developers can use the GetUrlCacheEntryInfoEx() function to perform this lookup.

History, which I am more familiar with, uses an index.dat file to keep track of which URLs should be displayed in History. Additionally, it stores other metadata about the url, such as the number of visits. This information is consumed by many features of IE. Off-line favorites uses it to keep track of subscription information. Shortcuts use it to find the Favicon. Auto-complete uses it to create a list of options to display in the drop-down box. Link coloring in the HTML rendering engine uses the list to decide how to color any given link on the page.

Furthermore, History actually uses more then one index.dat file. If you look in the [user profile]\Local Settings\History\History.IE5\ directory, you will see index.dat and some directories of the form MSHist###################. Those directories also contain index.dat files. The root-most index.dat file is where general history information is stored. The ones in the MSHist directories are used to support alternate views of History. If you open the History Explorer Bar (CTRL+H) and click the View drop down (ALT+W) you will see the various views that IE supports.

More to follow.

Index.dat: Part I - What is index.dat?

jeffdav — Thu, 18 Nov 2004 23:50:00 GMT

If you go poking around in [d]:\Documents and Settings\[user name]\Local Settings\History\, for example, you will find several index.dat files laying around. (Note: several of the directories that I listed in the path are hidden directories.)

We (the IE team) get a lot of questions about these files. The first question is often "What are they?" The answer is quite simple: Index.dat files are binary files that Internet Explorer uses to save information about URLs. They are basically hash tables that have been optimized to be very fast when dealing with URLs. Given an URL, these files map the URL to a binary blob of data. The binary blob of data is different, depending on what the particular index.dat file is used for.

Index.dat files are for IE's internal usage only. They are not designed or tested to be used by others. The functions for interacting with them are private, except things like the UrlCache APIs which provide a layer of abstraction for interacting with some index.dat files.