In order for the crawler to know which page to crawl, the crawler will try to infer the site structure either using sitemap or simply crawling the links on the main page. You can generate a sitemap XML based on specification located at http://www.sitemaps.org/...( Read More...