Parameters

Request parameters for Crawl API: limits, path filters, sitemap, domain scope, output formats, geo and webhook delivery

POST endpoint

https://data.decodo.com/v1/crawl

Parameter	Type	Description
`url`	string	The URL to start crawling from.
`limit`	integer	Maximum number of pages to scrape. Maximum value is `10000`.
`max_depth`	integer	Maximum number of link hops from the starting URL. Pages at this depth are scraped, but their links aren’t followed.
`JS_retry`	boolean	when set to `true`, pages that failed to successfully scrape are retried with JavaScript rendering
`sitemap`	string	Possible values:`include, only, skip.` `include` - collect URLs from both the sitemap and page links. `only` - crawl only URLs listed in the sitemap. `skip` - ignore the sitemap and follow page links only. Default value is `include`
`select_paths`	array	Only URLs whose path matches at least one pattern are crawled. Example: `blog`
`exclude_paths`	array	URLs whose path matches any pattern are skipped. Example: `case-studies`
`domain_filter`
`geo`	string	Set the country to use when submitting the query. Example: `United States`
`locale`	string	This will change the search page web interface language (not the results). Example: – en-US – en-GB
webhook

Starting a crawl returns its crawl_id immediately, along with links to its status and results. Use it to check status, fetch results, or cancel.