Resolve every named answer engine, search index and training crawler against a domain's robots.txt the way a crawler does: most-specific group only, longest match, allow wins a tie
get/api/tool/robots
Request
- Base URL: https://crawlcheck.io
- URL: https://crawlcheck.io/api/tool/robots
- Auth: none required
Query parameters
domainstring required
pathstring
Response
Per agent: allowed at the path, allowed_root, named, shadowed
object required
Changes
No recorded changes to this operation across all 1 revision of this API.