Introduction

​

Key features

• Site-wide scraping, one request - start from a URL and let Decodo follow links up to a depth and page limit you set.
• Asynchronous - crawls run in the background; poll for status or get notified with a webhook when they finish.
• Sitemap + page links - include sitemap URLs, page links, or both, to decide which pages get crawled.
• Path filtering - include or exclude URLs by path pattern, so you only collect the sections you need.
• Automatic JS retry - pages that fail to scrape can be retried once with JavaScript rendering.
• Flexible output - get Markdown or HTML for every page, inline or as a list of per-page results.

​

Pricing

​

Quick start

1. Sign up - activate the free plan or choose a paid plan from the dashboard.
2. Copy your API key from the dashboard.
3. Start a crawl with the URL you want to crawl, plus any limits or filters you need.
4. Poll for status, or set a callback_url to be notified when it’s done.

​

Code example

#replace the TOKEN value with your API KEY
curl --request 'POST' \
      --url 'https://data.decodo.com/v1/crawl' \
      --header 'Accept: application/json' \
      --header 'Authorization: Bearer TOKEN VALUE' \
      --header 'Content-Type: application/json' \
      --data '
    {
    "url": "https://decodo.com",
    "limit": 2,
    "max_depth": 1,
    "sitemap": "skip"
    }
'