OnPage

Changed on

‌ OnPage API checks websites for 60+ customizable on-page parameters defines and displays all found flaws and opportunities for optimization so that you can easily fix them. It checks meta tags, duplicate content, image tags, response codes, and other parameters on every page. You can find the full list of OnPage API check-up parameters in the Pages section.

post/v3/on_page/task_post

Request

  • Base URL: https://api.dataforseo.com
  • URL: https://api.dataforseo.com/v3/on_page/task_post
  • Auth: HTTP basic

Request body

targetstring

<em>target domain</em><br><strong>required field</strong><br>domain name should be specified without <code>https://</code> and <code>www.</code><br>if you specify the page URL, the results will be returned for the domain included in the URL

max_crawl_pagesinteger

<em>crawled pages limit</em><br><strong>required field</strong><br>the number of pages to crawl on the specified domain<br><strong>Note:</strong><br>if you set <code>max_crawl_pages</code> to 1 and do not specify <code>start_url</code> or set a homepage in it, the following sitewide checks will be disabled:<br><code>test_canonicalization</code>, <code>enable_www_redirect_check</code>, <code>test_hidden_server_signature</code>, <code>test_page_not_found</code>, <code>test_directory_browsing</code>, <code>test_https_redirect</code><br>to enable them anyway, set <code>force_sitewide_checks</code> to <code>true</code>if you set <code>max_crawl_pages</code> to 1 and specify <code>start_url</code> other than a homepage, all sitewide checks will be disabled;<br>to enable them anyway, set <code>force_sitewide_checks</code> to <code>true</code>

start_urlstring nullable

<em>the first url to crawl </em><br>optional field<br><strong>Note:</strong> you should specify an absolute URL<br>if you want to crawl a single page, specify its URL in this field and additionally set the <code>max_crawl_pages</code> parameter to <code>1</code><br>you can also use the <a href="/v3/on_page/instant_pages/?bash" target="_blank" rel="noopener noreferrer">live Instant Pages endpoint</a> to get page-specific data

force_sitewide_checksboolean nullable

<em>enable sitewide checks when crawling a single page</em><br>optional field<br>set to <code>true</code> to get data on sitewide checks when crawling a single page;<br>default value: <code>false</code>

priority_urlsstring[] nullable

<em>urls to be crawled bypassing the queue</em><br>optional field<br>URLs specified in this array will be crawled in the first instance, bypassing the crawling queue;<br><strong>Note:</strong> you should specify the absolute URL;<br>you can specify up to <strong>20 URLs</strong>;<br>all URLs in the array must belong to the <code>target</code> domain;<br>subdomains will be ignored unless the <code>allow_subdomains</code> parameter is set to <code>true</code>example:<br>"priority_urls": [<br>"https://dataforseo.com/apis/serp-api",<br>"https://dataforseo.com/contact"<br>]

max_crawl_depthinteger nullable

<em>crawl depth</em><br>optional field<br>the linking depth of the pages to crawl;<br>for example, starting page of the crawl is level 0, pages that have links from that page are level 1, etc.

crawl_delayinteger nullable

<em>delay between hits, ms</em><br>optional field<br>the custom delay between crawler hits to the server<br>default value: <code>2000</code>

store_raw_htmlboolean nullable

<em>store HTML of crawled pages</em><br>optional field<br>set to <code>true</code> if you want to get the HTML of the page using the <a href="/v3/on_page/raw_html/" target="_blank" rel="noopener noreferrer">OnPage Raw HTML endpoint</a><br>default value: <code>false</code>

enable_content_parsingboolean nullable

<em>parse content on crawled pages</em><br>optional field<br>set to <code>true</code> to use the <a href="/v3/on_page/content_parsing/live/" target="_blank" rel="noopener noreferrer">OnPage Content Parsing endpoint</a><br>default value: <code>false</code>

support_cookiesboolean nullable

<em>support cookies on crawled pages</em><br>optional field<br>set to <code>true</code> to support cookies when crawling the pages<br>default value: <code>false</code>

accept_languagestring nullable

<em>language header for accessing the website</em><br>optional field<br>all locale formats are supported (xx, xx-XX, xxx-XX, etc.)<br><strong>Note:</strong> if you do not specify this parameter, some websites may deny access; in this case, pages will be returned with the <code>"type":"broken</code> in the response array

custom_robots_txtstring nullable

<em>custom robots.txt settings</em><br>optional field<br>example: <code>Disallow: /directory1/</code>

robots_txt_merge_modestring nullable

<em>merge with or override robots.txt settings</em><br>optional field<br>possible values: <code>merge</code>, <code>override</code>;<br>set to <code>override</code> if you want to ignore website crawling restrictions and other robots.txt settings<br>default value: <code>merge</code>;<br><strong>Note:</strong> if set to <code>override</code>, specify the <code>custom_robots_txt</code> parameter

custom_user_agentstring nullable

<em>custom user agent</em><br>optional field<br>custom user agent for crawling a website<br>example: <code>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36<br></code><br>default value: <code>Mozilla/5.0 (compatible; RSiteAuditor)</code>

browser_presetstring nullable

<em>preset for browser screen parameters</em><br>optional field<br>if you use this field, you don't need to indicate <code>browser_screen_width</code>, <code>browser_screen_height</code>, <code>browser_screen_scale_factor</code>possible values:<br><code>desktop</code>, <code>mobile</code>, <code>tablet</code><code>desktop</code> preset will apply the following values:<code>browser_screen_width: 1920</code><br><code>browser_screen_height: 1080</code><br><code>browser_screen_scale_factor: 1</code><code>mobile</code> preset will apply the following values:<code>browser_screen_width: 390</code><br><code>browser_screen_height: 844</code><br><code>browser_screen_scale_factor: 3</code><code>tablet</code> preset will apply the following values:<code>browser_screen_width: 1024</code><br><code>browser_screen_height: 1366</code><br><code>browser_screen_scale_factor: 2</code><br><strong>Note:</strong> to use this parameter, set <code>enable_javascript</code> or <code>enable_browser_rendering</code> to <code>true</code>

browser_screen_widthinteger nullable

<em>browser screen width</em><br>optional field<br>you can set a custom browser screen width to perform audit for a particular device;<br>if you use this field, you don't need to indicate <code>browser_preset</code> as it will be ignored;<br><strong>Note:</strong> to use this parameter, set <code>enable_javascript</code> or <code>enable_browser_rendering</code> to <code>true</code>minimum value, in pixels: <code>240</code><br>maximum value, in pixels: <code>9999</code>

browser_screen_heightinteger nullable

<em>browser screen height</em><br>optional field<br>you can set a custom browser screen height to perform an audit for a particular device;<br>if you use this field, you don't need to indicate <code>browser_preset</code> as it will be ignored;<br><strong>Note:</strong> to use this parameter, set <code>enable_javascript</code> or <code>enable_browser_rendering</code> to <code>true</code>minimum value, in pixels: <code>240</code><br>maximum value, in pixels: <code>9999</code>

browser_screen_scale_factornumber nullable

<em>browser screen scale factor</em><br>optional field<br>you can set a custom browser screen resolution ratio to perform audit for a particular device;<br>if you use this field, you don't need to indicate <code>browser_preset</code> as it will be ignored;<br><strong>Note:</strong> to use this parameter, set <code>enable_javascript</code> or <code>enable_browser_rendering</code> to <code>true</code>minimum value: <code>0.5</code><br>maximum value: <code>3</code>

respect_sitemapboolean nullable

<em>respect sitemap when crawling</em><br>optional field<br>set to <code>true</code> if you want to follow the order of pages indicated in the primary sitemap when crawling;<br>default value: <code>false</code><br><strong>Note:</strong> if set to <code>true</code>, the <code>click_depth</code> value in the API response will equal <code>0</code>;<br>the <code>max_crawl_depth</code> field of the request will be ignored, you can specify the number of pages to crawl using the <code>max_crawl_pages</code> parameter

custom_sitemapstring nullable

<em>custom sitemap url</em><br>optional field<br>the URL of the page where the alternative sitemap is located<br><strong>Note:</strong> if you want to use this parameter, <code>respect_sitemap</code> should be <code>true</code>

crawl_sitemap_onlyboolean nullable

<em>crawl only pages indicated in the sitemap</em><br>optional field<br>set to <code>true</code> if you want to crawl only the pages indicated in the sitemap<br>if you set this parameter to <code>true</code> and do not specify <code>custom_sitemap</code>, we will crawl the default sitemap<br>default value: <code>false</code><br><strong>Note:</strong> if you want to use this parameter, <code>respect_sitemap</code> should be <code>true</code>

load_resourcesboolean nullable

<em>load resources</em><br>optional field<br>set to <code>true</code> if you want to load image, stylesheets, scripts, and broken resources<br>default value: <code>false</code><br><strong>Note:</strong> if you use this parameter, additional charges will apply; learn more about the cost of tasks with this parameter <a href="https://dataforseo.com/help-center/cost-of-onpage-api-parameters" target="_blank" rel="noopener noreferrer">in our help article</a>; the cost can be calculated on the <a href="https://dataforseo.com/pricing/on-page/onpage-api" target="_blank" rel="noopener noreferrer">Pricing Page</a>

enable_www_redirect_checkboolean nullable

<em>check if the domain implemented the www redirection</em><br>optional field<br>set to <code>true</code> if you want to check if the requested domain implemented the www to non-www or non-www to www redirect;<br>default value: <code>false</code>

enable_javascriptboolean nullable

<em>load javascript on a page</em><br>optional field<br>set to <code>true</code> if you want to load the scripts available on a page<br>default value: <code>false</code><br><strong>Note:</strong> if you use this parameter, additional charges will apply; learn more about the cost of tasks with this parameter <a href="https://dataforseo.com/help-center/cost-of-onpage-api-parameters" target="_blank" rel="noopener noreferrer">in our help article</a>; the cost can be calculated on the <a href="https://dataforseo.com/pricing/on-page/onpage-api" target="_blank" rel="noopener noreferrer">Pricing Page</a>

enable_xhrboolean nullable

<em>enable XMLHttpRequest on a page</em><br>optional field<br>set to <code>true</code> if you want our crawler to request data from a web server using the XMLHttpRequest object<br>default value: <code>false</code>;if you use this field, <code>enable_javascript</code> must be set to <code>true</code>;

enable_browser_renderingboolean nullable

<em>emulate browser rendering to measure Core Web Vitals</em><br>optional field<br>by using this parameter you will be able to emulate a browser when loading a web page;<br><code>enable_browser_rendering</code> loads styles, images, fonts, animations, videos, and other resources on a page;<br>default value: <code>false</code><br>set to <code>true</code> to obtain Core Web Vitals (FID, CLS, LCP) metrics in the response;<br><strong>if you use this field, <code>enable_javascript</code>, and <code>load_resources</code> parameters must be set to <code>true</code></strong><br><strong>Note:</strong> if you use this parameter, additional charges will apply; learn more about the cost of tasks with this parameter <a href="https://dataforseo.com/help-center/cost-of-onpage-api-parameters" target="_blank" rel="noopener noreferrer">in our help article</a>; the cost can be calculated on the <a href="https://dataforseo.com/pricing/on-page/onpage-api" target="_blank" rel="noopener noreferrer">Pricing Page</a>

disable_cookie_popupboolean nullable

<em>disable the cookie popup</em><br>optional field<br>set to <code>true</code> if you want to disable the popup requesting cookie consent from the user;<br>default value:<br><code>false</code>

custom_jsstring nullable

<em>custom javascript</em><br>optional field<br><strong>Note</strong> that the execution time for the script you enter here should be 700 ms maximum, for example, you can use the following JS snippet to check if the website contains Google Tag Manager as a <code>scr</code> attribute:<br><code>let meta = { haveGoogleAnalytics: false, haveTagManager: false };rnfor (var i = 0; i < document.scripts.length; i++) {rn let src = document.scripts[i].getAttribute("src");rn if (src != undefined) {rn if (src.indexOf("analytics.js") >= 0)rn meta.haveGoogleAnalytics = true;rntif (src.indexOf("gtm.js") >= 0)rn meta.haveTagManager = true;rn }rn}rnmeta;</code>the returned value depends on what you specified in this field. For instance, if you specify the following script:<br>meta = {}; meta.url = document.URL; meta.test = 'test'; meta;<br>as a response you will receive the following data:<br>"custom_js_response": {<br>"url": "https://dataforseo.com/",<br>"test": "test"<br>}<br><strong>Note:</strong> the length of the script you enter must be no more than 2000 characters

validate_micromarkupboolean nullable

<em>enable microdata validation</em><br>optional field<br>set to <code>true</code> if you want to use the <a href="/v3/on_page/microdata/" target="_blank" rel="noopener noreferrer">OnPage API Microdata endpoint</a><br>default value: <code>false</code>

allow_subdomainsboolean nullable

<em>include pages on subdomains</em><br>optional field<br>set to <code>true</code> if you want to crawl all subdomains of a target website<br>default value: <code>false</code>

allowed_subdomainsstring[] nullable

<em>subdomains to crawl</em><br>optional field<br>specify subdomains that you want to crawl<br>example: <code>["blog.site.com", "my.site.com", "shop.site.com"]</code><br><strong>Note:</strong> to use this parameter, the <code>allow_subdomains</code> parameter should be set to <code>false</code>;<br>otherwise, the content of <code>allowed_subdomains</code> field will be ignored and the results will be returned for all subdomains

disallowed_subdomainsstring[] nullable

<em>subdomains not to crawl</em><br>optional field<br>specify subdomains that you don't want to crawl<br>example: <code>["status.site.com", "docs.site.com"]</code><br><strong>Note:</strong> to use this parameter, the <code>allow_subdomains</code> parameter should be set to <code>true</code>

check_spellboolean nullable

<em>check spelling</em><br>optional field<br>set to <code>true</code> to check spelling on a website using <a href="http://hunspell.github.io/" target="_blank" rel="noopener noreferrer">Hunspell</a> library<br>default value: <code>false</code>

check_spell_languagestring nullable

<em>language of the spell check</em><br>optional field<br>supported languages: 'hy', 'eu', 'bg', 'ca', 'hr', 'cs', 'da', 'nl', 'en', 'eo', 'et', 'fo', 'fa', 'fr', 'fy', 'gl', 'ka', 'de', 'el', 'he', 'hu', 'is', 'ia', 'ga', 'it', 'rw', 'la', 'lv', 'lt', 'mk', 'mn', 'ne', 'nb', 'nn', 'pl', 'pt', 'ro', 'gd', 'sr', 'sk', 'sl', 'es', 'sv', 'tr', 'tk', 'uk', 'vi'<br><strong>Note:</strong> if no language is specified, it will be set automatically based on page content

check_spell_exceptionsstring[] nullable

<em>words excluded from spell check</em><br>optional field<br>specify the words that you want to exclude from spell check<br>maximum word length: 100 characters<br>maximum amount of words: 1000<br>example: <code>"SERP", "minifiers", "JavaScript"</code>

calculate_keyword_densityboolean nullable

<em>calculate keyword density for the target domain</em><br>optional field<br>set to <code>true</code> if you want to calculate keyword density for website pages<br>default value: <code>false</code><br><strong>Note:</strong> if you use this parameter, additional charges will apply; learn more about the cost of tasks with this parameter <a href="https://dataforseo.com/help-center/cost-of-onpage-api-parameters" target="_blank" rel="noopener noreferrer">in our help article</a><br>once the crawl is completed, you can obtain keyword density values with <a href="/v3/on_page/keyword_density?bash" target="_blank" rel="noopener noreferrer">the Keyword Density endpoint</a>

checks_thresholdobject nullable

<em>custom threshold values for checks</em><br>optional field<br>you can specify custom threshold values for the parameters included in the <code>checks</code> object of OnPage API responses;<br><strong>Note:</strong> only integer threshold values can be modified;<br>for example, the <code>high_loading_time</code> and <code>large_page_size</code> parameters are set to 3 seconds and 1 megabyte respectively by default;<br>if you want to change these thresholds to 1 second and 1000 kbytes, use the following snippet:<br>"checks_threshold": {<br>"high_loading_time": 1,<br>"large_page_size": 1000<br>}<br>available customizable parameters with default values:<br>"title_too_short", default value: 30, type: "int"<br>"title_too_long", default value: 65, type: "int"<br>"small_page_size", default value: 1024, type: "int"<br>"large_page_size", default value: 1048576 (1024 * 1024), type: "int"<br>"low_character_count", default value: 1024, type: "int"<br>"high_character_count", default value: 256000 (250 * 1024), type: "int"<br>"low_content_rate", default value: 0.1, type: "float"<br>"high_content_rate", default value: 0.9, type: "float"<br>"high_loading_time", default value: 3000, type: "int"<br>"high_waiting_time", default value: 1500, type: "int"<br>"low_readability_rate", default value: 15.0, type: "float"<br>"irrelevant_description", default value: 0.2, type: "float"<br>"irrelevant_title", default value: 0.3, type: "float"<br>"irrelevant_meta_keywords", default value: 0.6, type: "float"

disable_sitewide_checksstring[] nullable

<em>prevent certain sitewide checks from running</em><br>optional field<br>specify the following <code>checks</code> to prevent them from running on the <code>target</code> website:<br><code>"test_page_not_found"</code><br><code>"test_canonicalization"</code><br><code>"test_https_redirect"</code><br><code>"test_directory_browsing"</code>example:<br><code>"disable_sitewide_checks": ["test_directory_browsing", "test_page_not_found"]</code>learn more on <a href="https://dataforseo.com/help-center/how-to-disable-sitewide-checks-in-onpage-api" target="_blank" rel="noopener noreferrer">our help center</a>

disable_page_checksstring[] nullable

<em>prevent certain page checks from running</em><br>optional field<br>specify certain <code>checks</code> to prevent them from running and impacting the <code>onpage_score</code>example:<br><code>"disable_page_checks": ["is_5xx_code", "is_4xx_code"]</code>

switch_poolboolean nullable

<em>switch proxy pool</em><br>optional field<br>if <code>true</code>, additional proxy pools will be used to obtain the requested data;<br>the parameter can be used if a multitude of tasks is set simultaneously, resulting in occasional <code>rate-limit</code> and/or <code>site_unreachable</code> errors

return_despite_timeoutboolean nullable

<em>return data on pages despite the timeout error</em><br>optional field<br>if <code>true</code>, the data will be provided on pages that failed to load within 120 seconds and responded with a timeout error;<br>default value: <code>false</code>

tagstring nullable

<em>user-defined task identifier</em><br>optional field<br><em>the character limit is 255</em><br>you can use this parameter to identify the task and match it with the result<br>you will find the specified <code>tag</code> value in the <code>data</code> object of the response

pingback_urlstring nullable

<em>notification URL of a completed task</em><br>optional field<br>when a task is completed we will notify you by GET request sent to the URL you have specified<br>you can use the ‘$id’ string as a <code>$id</code> variable and ‘$tag’ as urlencoded <code>$tag</code> variable. We will set the necessary values before sending the request.<br>example:<br><code>http://your-server.com/pingscript?id=$id</code><br><code>http://your-server.com/pingscript?id=$id&amp;tag=$tag</code><br><strong>Note:</strong> special characters in <code>pingback_url</code> will be urlencoded;<br>i.a., the <code>#</code> character will be encoded into <code>%23</code><p>learn more on our <a href="https://dataforseo.com/help-center/pingbacks-postbacks-with-dataforseo-api" target="_blank" rel="noopener noreferrer">Help Center</a>

Example request

[
  {
    "target": "dataforseo.com",
    "max_crawl_pages": 10,
    "load_resources": true,
    "enable_javascript": true,
    "custom_js": "meta = {}; meta.url = document.URL; meta;",
    "tag": "some_string_123",
    "pingback_url": "https://your-server.com/pingscript?id=$id&tag=$tag"
  }
]

Response

Successful operation

versionstring nullable

the current version of the API

status_codeinteger nullable

general status code you can find the full list of the response codes here

status_messagestring nullable

general informational message you can find the full list of general informational messages here

timestring nullable

total execution time, seconds

costnumber double nullable

total tasks cost, USD

tasks_countinteger nullable

the number of tasks in the tasks array

tasks_errorinteger nullable

the number of tasks in the tasks array returned with an error

Changes