Crawling

Get the errors of a crawl job

get/crawl/{id}/errors

Response

Successful response

robotsBlockedstring[]

List of URLs that were attempted in scraping but were blocked by robots.txt

Changes