---
title: "Change a site's crawl settings"
method: PATCH
path: "/v1/sites/{site}/crawl-config"
tags: ["sites"]
---

# Change a site's crawl settings

`PATCH /v1/sites/{site}/crawl-config`

## Request

- Base URL: `https://frontshelf.io/api`
- URL: `https://frontshelf.io/api/v1/sites/{site}/crawl-config`
- Auth: HTTP bearer

Change a site's crawl settings: `patch` is any subset of configs_get's `config`, same snake_case keys. Sections merge key by key; a list replaces the stored list whole; in `thresholds` (rules_get's `threshold.key`, stored unit) null resets a key. Every later crawl changes, so it asks for confirmation: dry_run: true shows the change, then confirm: true. Your human can undo it in the site's settings. On an unverified site, rate_ms under 250, concurrency over 2, user_agent_preset not frontshelfbot, custom headers, robots_mode not respect, integrations.lighthouse_budget null, content.spelling.max_pages over its default answer VERIFICATION_REQUIRED naming the fields: run sites_verify first. A `custom` expression that does not compile is refused naming the field. Answers as configs_get.

## Path parameters

- `site` string, required: Site ID (site_…), origin (https://example.com), or bare domain, resolved server-side; the response carries the canonical ID.

## Request body

- object
  - `patch` object, required: The settings to change, strict at every level.
    - `mode` 'spider' | 'list' | 'serp'
    - `render_mode` 'raw' | 'adaptive' | 'always'
    - `rate_ms` integer
    - `concurrency` integer
    - `response_timeout_ms` integer
    - `max5xx_retries` integer
    - `user_agent_preset` 'frontshelfbot' | 'custom' | 'screaming-frog' | 'googlebot-desktop' | 'googlebot-desktop-chrome' | 'googlebot-smartphone' | 'googlebot-image' | 'googlebot-news' | 'googlebot-video' | 'google-favicon' | 'google-storebot' | 'google-inspectiontool' | 'googleother' | 'googleother-image' | 'googleother-video' | 'google-cloudvertexbot' | 'gptbot' | 'chatgpt-user' | 'claudebot' | 'claudebot-user' | 'perplexitybot' | 'perplexity-user' | 'ccbot' | 'oai-searchbot' | 'bingbot-desktop' | 'bingbot-smartphone' | 'yahoo-slurp' | 'duckduckbot' | 'baiduspider' | 'yandexbot' | 'applebot' | 'mj12bot' | 'seznambot' | 'android-mobile' | 'android-tablet' | 'chrome' | 'chrome-android' | 'chrome-android-tablet' | 'chrome-ios' | 'firefox' | 'ie11' | 'edge' | 'samsung-browser' | 'safari-iphone' | 'safari-ipad' | 'safari-macos'
    - `custom_user_agent` string
    - `scope` object
      - `crawl_all_subdomains` boolean
      - `crawl_outside_start_folder` boolean
      - `check_links_outside_folder` boolean
      - `includes` string[]
      - `excludes` string[]
      - `cdns` string[]
    - `limits` object
      - `max_urls` integer, nullable
      - `external_link_budget` integer, nullable
      - `max_depth` integer, nullable
      - `max_per_depth` integer, nullable
      - `max_folder_depth` integer, nullable
      - `max_query_strings` integer, nullable
      - `max_per_subdomain` integer, nullable
      - `max_redirects` integer
      - `max_url_length` integer
      - `max_links_per_page` integer
      - `max_page_size_bytes` integer
      - `url_path_limits` object[]
        - `pattern` string, required
        - `limit` integer, required
    - `rewriting` object
      - `convert_to_lowercase` boolean
      - `percent_encoding_mode` 'upper' | 'lower' | 'none'
      - `remove_parameters` string[]
      - `remove_all_parameters` boolean
      - `regex_rules` object[]
        - `pattern` string, required
        - `replace` string, required
    - `robots_mode` 'respect' | 'ignore' | 'ignore_and_report'
    - `robots_on_server_error` 'allow' | 'refuse'
    - `custom_robots` object
      - `enabled` boolean
      - `text` string
    - `cookies_mode` 'persistent' | 'session' | 'none'
    - `headers` object[]
      - `name` string, required
      - `value` string, required
    - `list_sources` union[]
      - union
        - object
          - `kind` 'urls', required
          - `urls` string[], required
        - object
          - `kind` 'sitemap', required
          - `url` string, required
        - object
          - `kind` 'blob', required
          - `ref` string, required
          - `format` 'txt' | 'csv' | 'xml', required
    - `toggles` object
      - `crawl_images` boolean
      - `store_images` boolean
      - `crawl_media` boolean
      - `store_media` boolean
      - `crawl_css` boolean
      - `store_css` boolean
      - `crawl_java_script` boolean
      - `store_java_script` boolean
      - `crawl_swf` boolean
      - `store_swf` boolean
      - `crawl_internal_links` boolean
      - `store_internal_links` boolean
      - `crawl_external_links` boolean
      - `store_external_links` boolean
      - `crawl_canonicals` boolean
      - `store_canonicals` boolean
      - `crawl_rel_next_prev` boolean
      - `store_rel_next_prev` boolean
      - `crawl_hreflang` boolean
      - `store_hreflang` boolean
      - `crawl_amp_html_links` boolean
      - `store_amp_html_links` boolean
      - `crawl_meta_refresh` boolean
      - `store_meta_refresh` boolean
      - `crawl_iframes` boolean
      - `store_iframes` boolean
      - `crawl_mobile_alt_links` boolean
      - `store_mobile_alt_links` boolean
      - `follow_internal_no_follow` boolean
      - `follow_external_no_follow` boolean
      - `crawl_sitemaps` boolean
      - `auto_discover_sitemaps` boolean
      - `sitemap_urls` string[]
      - `always_follow_redirects` boolean
      - `always_follow_canonicals` boolean
      - `respect_noindex` boolean
      - `respect_canonical` boolean
      - `respect_rel_next_prev` boolean
      - `respect_hsts` boolean
      - `respect_self_ref_meta_refresh` boolean
      - `ignore_non_indexable_urls_for_issues` boolean
      - `ignore_paginated_urls_for_duplicate_filters` boolean
      - `extract_img_src_set` boolean
      - `crawl_fragment_identifiers` boolean
      - `perform_html_validation` boolean
      - `assume_html` boolean
      - `green_web_host` boolean
      - `extract_http_headers` boolean
      - `extract_cookies` boolean
      - `store_original_html` boolean
      - `store_rendered_html` boolean
      - `store_pdf` boolean
      - `near_duplicates` boolean
      - `accessibility` boolean
      - `lighthouse` boolean
    - `rendering` object
      - `ajax_timeout_ms` integer
      - `window_resize_timeout_ms` integer
      - `window_preset` string
      - `flatten_shadow_dom` boolean
      - `flatten_iframes` boolean
      - `screenshots` boolean
      - `console_log` boolean
      - `actions` string[]
    - `structured_data` object
      - `extract_json_ld` boolean
      - `extract_microdata` boolean
      - `extract_rdfa` boolean
      - `schema_org_validation` boolean
      - `google_validation` boolean
      - `case_sensitive_validation` boolean
    - `integrations` object
      - `pull_window_days` integer
      - `lighthouse_budget` integer, nullable
      - `axe_budget` integer, nullable
      - `lighthouse_device` 'mobile' | 'desktop' | 'both'
      - `psi` boolean
      - `crawl_api_urls` boolean
    - `link_positions` object[]
      - `name` string, required
      - `xpath` string, required
    - `content` object
      - `spelling` object
        - `enabled` boolean
        - `language` union
          - 'auto'
          - string
        - `dictionary` string[]
        - `ignore_rules` string[]
        - `max_pages` integer
      - `grammar` object
        - `enabled` boolean
      - `embeddings` object
        - `enabled` boolean
        - `provider` 'workers-ai' | 'openai' | 'gemini' | 'ollama'
        - `model` string
        - `base_url` string
        - `similarity_threshold` number
        - `low_relevance_threshold` number
        - `max_pages` integer
    - `thresholds` object
      - `ai_raw_share_max` number, nullable
      - `ai_rendered_min_words` integer, nullable
      - `carbon_rating_f_mg` integer, nullable
      - `description_max_chars` integer, nullable
      - `description_max_pixels` integer, nullable
      - `description_min_chars` integer, nullable
      - `description_min_pixels` integer, nullable
      - `document_max_mb` integer, nullable
      - `field_cls_poor` number, nullable
      - `field_cls_slow` number, nullable
      - `field_inp_poor_ms` integer, nullable
      - `field_inp_slow_ms` integer, nullable
      - `field_lcp_poor_ms` integer, nullable
      - `field_lcp_slow_ms` integer, nullable
      - `ga4_high_bounce_rate` number, nullable
      - `h1_max_chars` integer, nullable
      - `h2_max_chars` integer, nullable
      - `high_crawl_depth` integer, nullable
      - `high_external_outlinks` integer, nullable
      - `high_internal_outlinks` integer, nullable
      - `image_alt_max_chars` integer, nullable
      - `image_max_kb` integer, nullable
      - `image_savings_tolerance_bytes` integer, nullable
      - `lighthouse_good_score` number, nullable
      - `lighthouse_perfect_score` number, nullable
      - `low_content_words` integer, nullable
      - `mobile_alt_max_width` integer, nullable
      - `near_duplicate_threshold` integer, nullable
      - `readability_difficult_max` integer, nullable
      - `readability_difficult_min` integer, nullable
      - `redirect_chain_min_hops` integer, nullable
      - `resource_max_mb` integer, nullable
      - `sitemap_max_bytes` integer, nullable
      - `sitemap_max_urls` integer, nullable
      - `slow_response_ms` integer, nullable
      - `soft404_max_words` integer, nullable
      - `title_max_chars` integer, nullable
      - `title_max_pixels` integer, nullable
      - `title_min_chars` integer, nullable
      - `title_min_pixels` integer, nullable
      - `url_max_chars` integer, nullable
    - `custom` object
      - `extractors` object[]
        - `name` string, required
        - `mode` 'xpath' | 'css' | 'regex'
        - `data_type` 'inner_html' | 'outer_html' | 'text' | 'function_value'
        - `expression` string, required
        - `attribute` string
      - `searches` object[]
        - `name` string, required
        - `mode` 'contains' | 'does_not_contain'
        - `data_type` 'text' | 'regex'
        - `query` string, required
        - `case_sensitive` boolean
        - `scope` 'html' | 'page_text' | 'page_text_no_anchors' | 'html_head' | 'html_body' | 'xpath' | 'content_area'
        - `xpath` string
  - `dry_run` boolean: true: check the call and say what it would do, without doing it.
  - `confirm` boolean: true: go ahead with a change that asks for confirmation.

## Response `200`

Change a site's crawl settings

- object
  - `summary` string, required: The answer in one or two sentences, with the numbers it rests on.
  - `warnings` string[], required: What limits this answer (a cap, a read that was cut short). Empty when nothing does.
  - `next_actions` object[], required: What to call next, best first. Follow them.
    - `capability` string, required: What to call next: a capability.
    - `arguments` object, required: Its arguments, complete; pass them as they are.
    - `instruction` string, required: What to do, written to you.
  - `site_id` string, required
  - `verified` boolean, required: Whether the site's ownership is verified: an unverified site crawls politely whatever `config` says (sites_verify changes it).
  - `config` object, required: What the site asked for, defaults filled. Never changed by a demotion.
    - `mode` 'spider' | 'list' | 'serp', required
    - `render_mode` 'raw' | 'adaptive' | 'always', required
    - `rate_ms` integer, required
    - `concurrency` integer, required
    - `response_timeout_ms` integer, required
    - `max5xx_retries` integer, required
    - `user_agent_preset` 'frontshelfbot' | 'custom' | 'screaming-frog' | 'googlebot-desktop' | 'googlebot-desktop-chrome' | 'googlebot-smartphone' | 'googlebot-image' | 'googlebot-news' | 'googlebot-video' | 'google-favicon' | 'google-storebot' | 'google-inspectiontool' | 'googleother' | 'googleother-image' | 'googleother-video' | 'google-cloudvertexbot' | 'gptbot' | 'chatgpt-user' | 'claudebot' | 'claudebot-user' | 'perplexitybot' | 'perplexity-user' | 'ccbot' | 'oai-searchbot' | 'bingbot-desktop' | 'bingbot-smartphone' | 'yahoo-slurp' | 'duckduckbot' | 'baiduspider' | 'yandexbot' | 'applebot' | 'mj12bot' | 'seznambot' | 'android-mobile' | 'android-tablet' | 'chrome' | 'chrome-android' | 'chrome-android-tablet' | 'chrome-ios' | 'firefox' | 'ie11' | 'edge' | 'samsung-browser' | 'safari-iphone' | 'safari-ipad' | 'safari-macos', required
    - `custom_user_agent` string
    - `scope` object, required
      - `crawl_all_subdomains` boolean, required
      - `crawl_outside_start_folder` boolean, required
      - `check_links_outside_folder` boolean, required
      - `includes` string[], required
      - `excludes` string[], required
      - `cdns` string[], required
    - `limits` object, required
      - `max_urls` integer, nullable, required
      - `external_link_budget` integer, nullable, required
      - `max_depth` integer, nullable, required
      - `max_per_depth` integer, nullable, required
      - `max_folder_depth` integer, nullable, required
      - `max_query_strings` integer, nullable, required
      - `max_per_subdomain` integer, nullable, required
      - `max_redirects` integer, required
      - `max_url_length` integer, required
      - `max_links_per_page` integer, required
      - `max_page_size_bytes` integer, required
      - `url_path_limits` object[], required
        - `pattern` string, required
        - `limit` integer, required
    - `rewriting` object, required
      - `convert_to_lowercase` boolean, required
      - `percent_encoding_mode` 'upper' | 'lower' | 'none', required
      - `remove_parameters` string[], required
      - `remove_all_parameters` boolean, required
      - `regex_rules` object[], required
        - `pattern` string, required
        - `replace` string, required
    - `robots_mode` 'respect' | 'ignore' | 'ignore_and_report', required
    - `robots_on_server_error` 'allow' | 'refuse', required
    - `custom_robots` object, required
      - `enabled` boolean, required
      - `text` string, required
    - `cookies_mode` 'persistent' | 'session' | 'none', required
    - `headers` object[], required
      - `name` string, required
      - `value` string, required
    - `list_sources` union[], required
      - union
        - object
          - `kind` 'urls', required
          - `urls` string[], required
        - object
          - `kind` 'sitemap', required
          - `url` string, required
        - object
          - `kind` 'blob', required
          - `ref` string, required
          - `format` 'txt' | 'csv' | 'xml', required
    - `toggles` object, required
      - `crawl_images` boolean, required
      - `store_images` boolean, required
      - `crawl_media` boolean, required
      - `store_media` boolean, required
      - `crawl_css` boolean, required
      - `store_css` boolean, required
      - `crawl_java_script` boolean, required
      - `store_java_script` boolean, required
      - `crawl_swf` boolean, required
      - `store_swf` boolean, required
      - `crawl_internal_links` boolean, required
      - `store_internal_links` boolean, required
      - `crawl_external_links` boolean, required
      - `store_external_links` boolean, required
      - `crawl_canonicals` boolean, required
      - `store_canonicals` boolean, required
      - `crawl_rel_next_prev` boolean, required
      - `store_rel_next_prev` boolean, required
      - `crawl_hreflang` boolean, required
      - `store_hreflang` boolean, required
      - `crawl_amp_html_links` boolean, required
      - `store_amp_html_links` boolean, required
      - `crawl_meta_refresh` boolean, required
      - `store_meta_refresh` boolean, required
      - `crawl_iframes` boolean, required
      - `store_iframes` boolean, required
      - `crawl_mobile_alt_links` boolean, required
      - `store_mobile_alt_links` boolean, required
      - `follow_internal_no_follow` boolean, required
      - `follow_external_no_follow` boolean, required
      - `crawl_sitemaps` boolean, required
      - `auto_discover_sitemaps` boolean, required
      - `sitemap_urls` string[], required
      - `always_follow_redirects` boolean, required
      - `always_follow_canonicals` boolean, required
      - `respect_noindex` boolean, required
      - `respect_canonical` boolean, required
      - `respect_rel_next_prev` boolean, required
      - `respect_hsts` boolean, required
      - `respect_self_ref_meta_refresh` boolean, required
      - `ignore_non_indexable_urls_for_issues` boolean, required
      - `ignore_paginated_urls_for_duplicate_filters` boolean, required
      - `extract_img_src_set` boolean, required
      - `crawl_fragment_identifiers` boolean, required
      - `perform_html_validation` boolean, required
      - `assume_html` boolean, required
      - `green_web_host` boolean, required
      - `extract_http_headers` boolean, required
      - `extract_cookies` boolean, required
      - `store_original_html` boolean, required
      - `store_rendered_html` boolean, required
      - `store_pdf` boolean, required
      - `near_duplicates` boolean, required
      - `accessibility` boolean, required
      - `lighthouse` boolean, required
    - `rendering` object, required
      - `ajax_timeout_ms` integer, required
      - `window_resize_timeout_ms` integer, required
      - `window_preset` string, required
      - `flatten_shadow_dom` boolean, required
      - `flatten_iframes` boolean, required
      - `screenshots` boolean, required
      - `console_log` boolean, required
      - `actions` string[], required
    - `structured_data` object, required
      - `extract_json_ld` boolean, required
      - `extract_microdata` boolean, required
      - `extract_rdfa` boolean, required
      - `schema_org_validation` boolean, required
      - `google_validation` boolean, required
      - `case_sensitive_validation` boolean, required
    - `integrations` object, required
      - `pull_window_days` integer, required
      - `lighthouse_budget` integer, nullable, required
      - `axe_budget` integer, nullable, required
      - `lighthouse_device` 'mobile' | 'desktop' | 'both', required
      - `psi` boolean, required
      - `crawl_api_urls` boolean, required
    - `link_positions` object[], required
      - `name` string, required
      - `xpath` string, required
    - `content` object, required
      - `spelling` object, required
        - `enabled` boolean, required
        - `language` union, required
          - 'auto'
          - string
        - `dictionary` string[], required
        - `ignore_rules` string[], required
        - `max_pages` integer, required
      - `grammar` object, required
        - `enabled` boolean, required
      - `embeddings` object, required
        - `enabled` boolean, required
        - `provider` 'workers-ai' | 'openai' | 'gemini' | 'ollama', required
        - `model` string, required
        - `base_url` string, required
        - `similarity_threshold` number, required
        - `low_relevance_threshold` number, required
        - `max_pages` integer, required
    - `thresholds` object, required
      - `ai_raw_share_max` number, nullable
      - `ai_rendered_min_words` integer, nullable
      - `carbon_rating_f_mg` integer, nullable
      - `description_max_chars` integer, nullable
      - `description_max_pixels` integer, nullable
      - `description_min_chars` integer, nullable
      - `description_min_pixels` integer, nullable
      - `document_max_mb` integer, nullable
      - `field_cls_poor` number, nullable
      - `field_cls_slow` number, nullable
      - `field_inp_poor_ms` integer, nullable
      - `field_inp_slow_ms` integer, nullable
      - `field_lcp_poor_ms` integer, nullable
      - `field_lcp_slow_ms` integer, nullable
      - `ga4_high_bounce_rate` number, nullable
      - `h1_max_chars` integer, nullable
      - `h2_max_chars` integer, nullable
      - `high_crawl_depth` integer, nullable
      - `high_external_outlinks` integer, nullable
      - `high_internal_outlinks` integer, nullable
      - `image_alt_max_chars` integer, nullable
      - `image_max_kb` integer, nullable
      - `image_savings_tolerance_bytes` integer, nullable
      - `lighthouse_good_score` number, nullable
      - `lighthouse_perfect_score` number, nullable
      - `low_content_words` integer, nullable
      - `mobile_alt_max_width` integer, nullable
      - `near_duplicate_threshold` integer, nullable
      - `readability_difficult_max` integer, nullable
      - `readability_difficult_min` integer, nullable
      - `redirect_chain_min_hops` integer, nullable
      - `resource_max_mb` integer, nullable
      - `sitemap_max_bytes` integer, nullable
      - `sitemap_max_urls` integer, nullable
      - `slow_response_ms` integer, nullable
      - `soft404_max_words` integer, nullable
      - `title_max_chars` integer, nullable
      - `title_max_pixels` integer, nullable
      - `title_min_chars` integer, nullable
      - `title_min_pixels` integer, nullable
      - `url_max_chars` integer, nullable
    - `custom` object, required
      - `extractors` object[], required
        - `name` string, required
        - `mode` 'xpath' | 'css' | 'regex', required
        - `data_type` 'inner_html' | 'outer_html' | 'text' | 'function_value', required
        - `expression` string, required
        - `attribute` string, required
      - `searches` object[], required
        - `name` string, required
        - `mode` 'contains' | 'does_not_contain', required
        - `data_type` 'text' | 'regex', required
        - `query` string, required
        - `case_sensitive` boolean, required
        - `scope` 'html' | 'page_text' | 'page_text_no_anchors' | 'html_head' | 'html_body' | 'xpath' | 'content_area', required
        - `xpath` string, required
  - `effective` object, required: What the next crawl will actually use: `config` held to the site's verification and the plan's caps.
    - `mode` 'spider' | 'list' | 'serp', required
    - `render_mode` 'raw' | 'adaptive' | 'always', required
    - `rate_ms` integer, required
    - `concurrency` integer, required
    - `response_timeout_ms` integer, required
    - `max5xx_retries` integer, required
    - `user_agent_preset` 'frontshelfbot' | 'custom' | 'screaming-frog' | 'googlebot-desktop' | 'googlebot-desktop-chrome' | 'googlebot-smartphone' | 'googlebot-image' | 'googlebot-news' | 'googlebot-video' | 'google-favicon' | 'google-storebot' | 'google-inspectiontool' | 'googleother' | 'googleother-image' | 'googleother-video' | 'google-cloudvertexbot' | 'gptbot' | 'chatgpt-user' | 'claudebot' | 'claudebot-user' | 'perplexitybot' | 'perplexity-user' | 'ccbot' | 'oai-searchbot' | 'bingbot-desktop' | 'bingbot-smartphone' | 'yahoo-slurp' | 'duckduckbot' | 'baiduspider' | 'yandexbot' | 'applebot' | 'mj12bot' | 'seznambot' | 'android-mobile' | 'android-tablet' | 'chrome' | 'chrome-android' | 'chrome-android-tablet' | 'chrome-ios' | 'firefox' | 'ie11' | 'edge' | 'samsung-browser' | 'safari-iphone' | 'safari-ipad' | 'safari-macos', required
    - `custom_user_agent` string
    - `scope` object, required
      - `crawl_all_subdomains` boolean, required
      - `crawl_outside_start_folder` boolean, required
      - `check_links_outside_folder` boolean, required
      - `includes` string[], required
      - `excludes` string[], required
      - `cdns` string[], required
    - `limits` object, required
      - `max_urls` integer, nullable, required
      - `external_link_budget` integer, nullable, required
      - `max_depth` integer, nullable, required
      - `max_per_depth` integer, nullable, required
      - `max_folder_depth` integer, nullable, required
      - `max_query_strings` integer, nullable, required
      - `max_per_subdomain` integer, nullable, required
      - `max_redirects` integer, required
      - `max_url_length` integer, required
      - `max_links_per_page` integer, required
      - `max_page_size_bytes` integer, required
      - `url_path_limits` object[], required
        - `pattern` string, required
        - `limit` integer, required
    - `rewriting` object, required
      - `convert_to_lowercase` boolean, required
      - `percent_encoding_mode` 'upper' | 'lower' | 'none', required
      - `remove_parameters` string[], required
      - `remove_all_parameters` boolean, required
      - `regex_rules` object[], required
        - `pattern` string, required
        - `replace` string, required
    - `robots_mode` 'respect' | 'ignore' | 'ignore_and_report', required
    - `robots_on_server_error` 'allow' | 'refuse', required
    - `custom_robots` object, required
      - `enabled` boolean, required
      - `text` string, required
    - `cookies_mode` 'persistent' | 'session' | 'none', required
    - `headers` object[], required
      - `name` string, required
      - `value` string, required
    - `list_sources` union[], required
      - union
        - object
          - `kind` 'urls', required
          - `urls` string[], required
        - object
          - `kind` 'sitemap', required
          - `url` string, required
        - object
          - `kind` 'blob', required
          - `ref` string, required
          - `format` 'txt' | 'csv' | 'xml', required
    - `toggles` object, required
      - `crawl_images` boolean, required
      - `store_images` boolean, required
      - `crawl_media` boolean, required
      - `store_media` boolean, required
      - `crawl_css` boolean, required
      - `store_css` boolean, required
      - `crawl_java_script` boolean, required
      - `store_java_script` boolean, required
      - `crawl_swf` boolean, required
      - `store_swf` boolean, required
      - `crawl_internal_links` boolean, required
      - `store_internal_links` boolean, required
      - `crawl_external_links` boolean, required
      - `store_external_links` boolean, required
      - `crawl_canonicals` boolean, required
      - `store_canonicals` boolean, required
      - `crawl_rel_next_prev` boolean, required
      - `store_rel_next_prev` boolean, required
      - `crawl_hreflang` boolean, required
      - `store_hreflang` boolean, required
      - `crawl_amp_html_links` boolean, required
      - `store_amp_html_links` boolean, required
      - `crawl_meta_refresh` boolean, required
      - `store_meta_refresh` boolean, required
      - `crawl_iframes` boolean, required
      - `store_iframes` boolean, required
      - `crawl_mobile_alt_links` boolean, required
      - `store_mobile_alt_links` boolean, required
      - `follow_internal_no_follow` boolean, required
      - `follow_external_no_follow` boolean, required
      - `crawl_sitemaps` boolean, required
      - `auto_discover_sitemaps` boolean, required
      - `sitemap_urls` string[], required
      - `always_follow_redirects` boolean, required
      - `always_follow_canonicals` boolean, required
      - `respect_noindex` boolean, required
      - `respect_canonical` boolean, required
      - `respect_rel_next_prev` boolean, required
      - `respect_hsts` boolean, required
      - `respect_self_ref_meta_refresh` boolean, required
      - `ignore_non_indexable_urls_for_issues` boolean, required
      - `ignore_paginated_urls_for_duplicate_filters` boolean, required
      - `extract_img_src_set` boolean, required
      - `crawl_fragment_identifiers` boolean, required
      - `perform_html_validation` boolean, required
      - `assume_html` boolean, required
      - `green_web_host` boolean, required
      - `extract_http_headers` boolean, required
      - `extract_cookies` boolean, required
      - `store_original_html` boolean, required
      - `store_rendered_html` boolean, required
      - `store_pdf` boolean, required
      - `near_duplicates` boolean, required
      - `accessibility` boolean, required
      - `lighthouse` boolean, required
    - `rendering` object, required
      - `ajax_timeout_ms` integer, required
      - `window_resize_timeout_ms` integer, required
      - `window_preset` string, required
      - `flatten_shadow_dom` boolean, required
      - `flatten_iframes` boolean, required
      - `screenshots` boolean, required
      - `console_log` boolean, required
      - `actions` string[], required
    - `structured_data` object, required
      - `extract_json_ld` boolean, required
      - `extract_microdata` boolean, required
      - `extract_rdfa` boolean, required
      - `schema_org_validation` boolean, required
      - `google_validation` boolean, required
      - `case_sensitive_validation` boolean, required
    - `integrations` object, required
      - `pull_window_days` integer, required
      - `lighthouse_budget` integer, nullable, required
      - `axe_budget` integer, nullable, required
      - `lighthouse_device` 'mobile' | 'desktop' | 'both', required
      - `psi` boolean, required
      - `crawl_api_urls` boolean, required
    - `link_positions` object[], required
      - `name` string, required
      - `xpath` string, required
    - `content` object, required
      - `spelling` object, required
        - `enabled` boolean, required
        - `language` union, required
          - 'auto'
          - string
        - `dictionary` string[], required
        - `ignore_rules` string[], required
        - `max_pages` integer, required
      - `grammar` object, required
        - `enabled` boolean, required
      - `embeddings` object, required
        - `enabled` boolean, required
        - `provider` 'workers-ai' | 'openai' | 'gemini' | 'ollama', required
        - `model` string, required
        - `base_url` string, required
        - `similarity_threshold` number, required
        - `low_relevance_threshold` number, required
        - `max_pages` integer, required
    - `thresholds` object, required
      - `ai_raw_share_max` number, nullable
      - `ai_rendered_min_words` integer, nullable
      - `carbon_rating_f_mg` integer, nullable
      - `description_max_chars` integer, nullable
      - `description_max_pixels` integer, nullable
      - `description_min_chars` integer, nullable
      - `description_min_pixels` integer, nullable
      - `document_max_mb` integer, nullable
      - `field_cls_poor` number, nullable
      - `field_cls_slow` number, nullable
      - `field_inp_poor_ms` integer, nullable
      - `field_inp_slow_ms` integer, nullable
      - `field_lcp_poor_ms` integer, nullable
      - `field_lcp_slow_ms` integer, nullable
      - `ga4_high_bounce_rate` number, nullable
      - `h1_max_chars` integer, nullable
      - `h2_max_chars` integer, nullable
      - `high_crawl_depth` integer, nullable
      - `high_external_outlinks` integer, nullable
      - `high_internal_outlinks` integer, nullable
      - `image_alt_max_chars` integer, nullable
      - `image_max_kb` integer, nullable
      - `image_savings_tolerance_bytes` integer, nullable
      - `lighthouse_good_score` number, nullable
      - `lighthouse_perfect_score` number, nullable
      - `low_content_words` integer, nullable
      - `mobile_alt_max_width` integer, nullable
      - `near_duplicate_threshold` integer, nullable
      - `readability_difficult_max` integer, nullable
      - `readability_difficult_min` integer, nullable
      - `redirect_chain_min_hops` integer, nullable
      - `resource_max_mb` integer, nullable
      - `sitemap_max_bytes` integer, nullable
      - `sitemap_max_urls` integer, nullable
      - `slow_response_ms` integer, nullable
      - `soft404_max_words` integer, nullable
      - `title_max_chars` integer, nullable
      - `title_max_pixels` integer, nullable
      - `title_min_chars` integer, nullable
      - `title_min_pixels` integer, nullable
      - `url_max_chars` integer, nullable
    - `custom` object, required
      - `extractors` object[], required
        - `name` string, required
        - `mode` 'xpath' | 'css' | 'regex', required
        - `data_type` 'inner_html' | 'outer_html' | 'text' | 'function_value', required
        - `expression` string, required
        - `attribute` string, required
      - `searches` object[], required
        - `name` string, required
        - `mode` 'contains' | 'does_not_contain', required
        - `data_type` 'text' | 'regex', required
        - `query` string, required
        - `case_sensitive` boolean, required
        - `scope` 'html' | 'page_text' | 'page_text_no_anchors' | 'html_head' | 'html_body' | 'xpath' | 'content_area', required
        - `xpath` string, required
  - `demotions` object[], required: Every difference between `config` and `effective`, with its reason. Empty when the crawl gets what was asked for.
    - `field` string, required: The setting held, as a snake_case path (`rate_ms`, `limits.max_urls`).
    - `reason` 'unverified' | 'entitlement_cap' | 'sensitive_header', required: `unverified`: verify the site to lift it; `entitlement_cap`: the plan caps it; `sensitive_header`: a credential stored as a plain header, no longer sent.
    - `message` string, required: The sentence that says why, for your human.
  - `max_urls` object, required
    - `value` integer, required: The page cap the next crawl honours.
    - `source` 'config' | 'site' | 'entitlement', required: Where it comes from: `limits.max_urls`, the site's page budget, or the plan.
  - `updated_at` string, nullable, required: When the settings were last changed (ISO 8601); null while the site runs on the defaults.
  - `settings_url` string, required: The site's crawl settings page in Frontshelf (app-relative), to hand your human.

## Other responses

- `400`: VALIDATION_FAILED
- `401`: UNAUTHENTICATED
- `402`: BILLING_REQUIRED
- `403`: VERIFICATION_REQUIRED, ORGANIZATION_FROZEN, SCOPE_DENIED
- `404`: SITE_NOT_FOUND
- `409`: AMBIGUOUS_SITE, AGENT_STOPPED
- `428`: CONFIRMATION_REQUIRED
- `503`: PLATFORM_READ_ONLY, DEPENDENCY_UNAVAILABLE

---

[API](https://skmtc.dev/frontshelf/apis/frontshelf-api.md) · [All operations](https://skmtc.dev/frontshelf/apis/frontshelf-api/llms.txt) · [OpenAPI document](https://skmtc.dev/frontshelf/apis/frontshelf-api/revisions/ce3602d26c7c?raw)