re: Web-scraping with VB's XML support

Pavel Minaev [MSFT] — Thu, 12 Mar 2009 22:27:10 GMT

Don't trust Tidy too much - its HTML parser is far from perfect, and when it fails, you do not get valid XHTML as output, even if you asked for it. One example of something it can't handle are MSOffice HTML extensions (such as VML), and the non-standard-compliant way Office uses to declare namespaces in HTML documents it produces. And there are quite a few pages on the Web made by using "Save as HTML" in word.

Apart from writing an HTML parser from scratch, the only other reasonable option is to use IE (or rather HTMLDocument coclass and IHTMLDocument interface) to parse it, and then walk its DOM. Along these lines:

http://www.codeproject.com/KB/IP/parse_html.aspx