Read the site's robots.txt and what it blocks

The rules declared in the site's robots.txt, the sitemaps it points to, and how many of the site's known pages a given crawler is allowed to fetch. Use this to find out why pages are not being crawled or indexed.