OnPage

‌‌ This endpoint allows parsing the content on any page you specify and will return the structured content of the target page, including link URLs, anchors, headings, and textual content.

post/v3/on_page/content_parsing/live

Request

  • Base URL: https://api.dataforseo.com
  • URL: https://api.dataforseo.com/v3/on_page/content_parsing/live
  • Auth: HTTP basic

Request body

urlstring

<em>URL of the content to parse</em><br><strong>required field</strong><br>URL of the page to parse<br>example:<br>https://www.fujielectric.com/

custom_user_agentstring nullable

<em>custom user agent</em><br>optional field<br>custom user agent for crawling a website<br>example: <code>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36<br></code><br>default value: <code>Mozilla/5.0 (compatible; RSiteAuditor)</code>

browser_presetstring nullable

<em>preset for browser screen parameters</em><br>optional field<br>if you use this field, you don't need to indicate <code>browser_screen_width</code>, <code>browser_screen_height</code>, <code>browser_screen_scale_factor</code><p> possible values:<br><code>desktop</code>, <code>mobile</code>, <code>tablet</code><p> <code>desktop</code> preset will apply the following values:<p> <code>browser_screen_width: 1920</code><br><code>browser_screen_height: 1080</code><br><code>browser_screen_scale_factor: 1</code><p> <code>mobile</code> preset will apply the following values:<p> <code>browser_screen_width: 390</code><br><code>browser_screen_height: 844</code><br><code>browser_screen_scale_factor: 3</code><p> <code>tablet</code> preset will apply the following values:<p> <code>browser_screen_width: 1024</code><br><code>browser_screen_height: 1366</code><br><code>browser_screen_scale_factor: 2</code><p> <strong>Note:</strong> to use this parameter, set <code>enable_javascript</code> or <code>enable_browser_rendering</code> to <code>true</code>

browser_screen_widthinteger nullable

<em>browser screen width</em><br>optional field<br>you can set a custom browser screen width to perform audit for a particular device;<br>if you use this field, you don't need to indicate <code>browser_preset</code> as it will be ignored;<p> <strong>Note:</strong> to use this parameter, set <code>enable_javascript</code> or <code>enable_browser_rendering</code> to <code>true</code><p> minimum value, in pixels: <code>240</code><br>maximum value, in pixels: <code>9999</code>

browser_screen_heightinteger nullable

<em>browser screen height</em><br>optional field<br>you can set a custom browser screen height to perform audit for a particular device;<br>if you use this field, you don't need to indicate <code>browser_preset</code> as it will be ignored;<p> <strong>Note:</strong> to use this parameter, set <code>enable_javascript</code> or <code>enable_browser_rendering</code> to <code>true</code><p> minimum value, in pixels: <code>240</code><br>maximum value, in pixels: <code>9999</code>

browser_screen_scale_factornumber nullable

<em>browser screen scale factor</em><br>optional field<br>you can set a custom browser screen resolution ratio to perform audit for a particular device;<br>if you use this field, you don't need to indicate <code>browser_preset</code> as it will be ignored;<p> <strong>Note:</strong> to use this parameter, set <code>enable_javascript</code> or <code>enable_browser_rendering</code> to <code>true</code><p> minimum value: <code>0.5</code><br>maximum value: <code>3</code>

store_raw_htmlboolean nullable

<em>store HTML of a crawled page</em><br>optional field<br>set to <code>true</code> if you want to get the HTML of the page using the <a href="/v3/on_page/raw_html/" target="_blank" rel="noopener noreferrer">OnPage Raw HTML endpoint</a><br>default value: <code>false</code>

disable_cookie_popupboolean nullable

<em>disable the cookie popup </em><br>optional field<br>set to <code>true</code> if you want to disable the popup requesting cookie consent from the user;<br>default value:<br><code>false</code>

accept_languagestring nullable

<em>language header for accessing the website</em><br>optional field<br>all locale formats are supported (xx, xx-XX, xxx-XX, etc.)<br><strong>Note:</strong> if you do not specify this parameter, some websites may deny access; in this case, pages will be returned with the <code>"type":"broken</code> in the response array

enable_javascriptboolean nullable

<em>load javascript on a page</em><br>optional field<br>set to <code>true</code> if you want to load the scripts available on a page<br>default value: <code>false</code><br><strong>Note:</strong> if you use this parameter, additional charges will apply; learn more about the cost of tasks with this parameter <a href="https://dataforseo.com/help-center/cost-of-onpage-api-parameters" rel="noopener noreferrer" target="_blank">in our help article</a>; the cost can be calculated on the <a href="https://dataforseo.com/pricing/on-page/onpage-api" target="_blank" rel="noopener noreferrer">Pricing Page</a>

enable_browser_renderingboolean nullable

<em>emulate browser rendering to measure Core Web Vitals</em><br>optional field<br>by using this parameter you will be able to emulate a browser when loading a web page;<br><code>enable_browser_rendering</code> loads styles, images, fonts, animations, videos, and other resources on a page;<br>default value: <code>false</code><br>set to <code>true</code> to obtain Core Web Vitals (FID, CLS, LCP) metrics in the response;<br><strong>if you use this field, <code>enable_javascript</code>, and <code>load_resources</code> parameters must be set to <code>true</code></strong><br><strong>Note:</strong> if you use this parameter, additional charges will apply; learn more about the cost of tasks with this parameter <a href="https://dataforseo.com/help-center/cost-of-onpage-api-parameters" rel="noopener noreferrer" target="_blank">in our help article</a>; the cost can be calculated on the <a href="https://dataforseo.com/pricing/on-page/onpage-api" rel="noopener noreferrer" target="_blank">Pricing Page</a>

enable_xhrboolean nullable

<em>enable XMLHttpRequest on a page</em><br>optional field<br>set to <code>true</code> if you want our crawler to request data from a web server using the XMLHttpRequest object<br>default value:<br><code>false</code><p> if you use this field, <code>enable_javascript</code> must be set to <code>true</code>;

switch_poolboolean nullable

<em>switch proxy pool</em><br>optional field<br>if <code>true</code>, additional proxy pools will be used to obtain the requested data;<br>the parameter can be used if a multitude of tasks is set simultaneously, resulting in occasional <code>rate-limit</code> and/or <code>site_unreachable</code> errors

ip_pool_for_scanstring nullable

<em>proxy pool</em><br>optional field<br>you can choose a location of the proxy pool that will be used to obtain the requested data;<br>the parameter can be used if page content is inaccessible in one of the locations, resulting in occasional <code>site_unreachable</code> errors<br>possible values: <code>us</code>, <code>de</code>

markdown_viewboolean nullable

<em>return page content as markdown</em><br>optional field<br>if set to <code>true</code>, the markdown-formatted content of the page will be returned in the <code>page_as_markdown</code> field of the response;<br>default value: <code>false</code>

Example request

[
  {
    "url": "https://dataforseo.com/blog/a-versatile-alternative-to-google-trends-exploring-the-power-of-dataforseo-trends-api"
  }
]

Response

Successful operation

versionstring nullable

the current version of the API

status_codeinteger nullable

general status code you can find the full list of the response codes here

status_messagestring nullable

general informational message you can find the full list of general informational messages here

timestring nullable

total execution time, seconds

costnumber double nullable

total tasks cost, USD

tasks_countinteger nullable

the number of tasks in the tasks array

tasks_errorinteger nullable

the number of tasks in the tasks array returned with an error

Changes

    • breaking

      added to the /////////////////////////////////// response property oneOf list for the response status

    • breaking

      added to the ////////////////////////////// response property oneOf list for the response status

    • info

      removed from the /////////////////////////////////// response property oneOf list for the response status

    • info

      removed from the ////////////////////////////// response property oneOf list for the response status

    • info

      api operation id OnPageContentParsingLive removed and replaced with ContentParsingLive