> ## Documentation Index
> Fetch the complete documentation index at: https://help.decodo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Introduction

> Decodo /crawl endpoint overview: site-wide crawling, sitemap handling, path filtering, webhooks, and pricing.

/crawl follows links across a website and scrapes every page it finds, starting from a URL you submit. Use it to collect content from a whole site or a section of it, without calling `/scrape` once per page yourself.

## Key features

* **Site-wide scraping, one request** - start from a URL and let Decodo follow links up to a depth and page limit you set.
* **Asynchronous** - crawls run in the background; poll for status or get notified with a webhook when they finish.
* **Sitemap + page links** - include sitemap URLs, page links, or both, to decide which pages get crawled.
* **Path filtering** - include or exclude URLs by path pattern, so you only collect the sections you need.
* **Automatic JS retry** - pages that fail to scrape can be retried once with JavaScript rendering.
* **Flexible output** - get Markdown or HTML for every page, inline or as a list of per-page results.

`/crawl` and [**/map**](/docs/web-data-api-map-introduction) work well together: use `/map` to see a site's structure and decide which paths you need, then run `/crawl` with `select_paths` to collect only those pages.

## Pricing

Each page the crawler scrapes is charged as one request. Crawls always use the premium proxy pool, with a separate rate for pages retried with JavaScript rendering. Pages that fail aren't charged. See the full rate table on the [**Billing**](/docs/web-data-api-billing) page.

## Quick start

1. **Sign up** - activate the free plan or choose a paid plan from the dashboard.
2. **Copy your API key** from the [**dashboard**](https://dashboard.decodo.com/).
3. **Start a crawl** with the URL you want to crawl, plus any limits or filters you need.
4. **Poll for status**, or set a `callback_url` to be notified when it's done.

See the [**Quick Start**](/docs/web-data-api-crawl-quick-start) guide for the full walkthrough, [**Parameters**](/docs/web-data-api-crawl-parameters) for every input option, and [**Status, Results & Cancellation**](/docs/web-data-api-crawl-status-results-cancellation) for tracking and stopping a crawl.

## Code example

<CodeGroup>
  ```shellscript cURL theme={null}
  #replace the TOKEN value with your API KEY
  curl --request 'POST' \
        --url 'https://data.decodo.com/v1/crawl' \
        --header 'Accept: application/json' \
        --header 'Authorization: Bearer TOKEN VALUE' \
        --header 'Content-Type: application/json' \
        --data '
      {
      "url": "https://decodo.com",
      "limit": 2,
      "max_depth": 1,
      "sitemap": "skip"
      }
  '
  ```
</CodeGroup>

<Note>
  ## Usage policy

  `/crawl` is intended for collecting publicly accessible web data.

  We don't support crawling:

  * Post-login or authenticated content
  * Private user information
  * Restricted or non-public data

  Users are responsible for ensuring their use of the service complies with applicable website terms and internal policies.
</Note>

***

<Columns cols={2}>
  <Card title="Support" href="https://direct.lc.chat/12092754" cta="Let's chat!">
    Need help or just want to say hello? Our support is available 24/7. \
    You can also reach us anytime via email at [support@decodo.com](mailto:support@decodo.com).
  </Card>

  <Card title="Feedback" href="mailto:feedback@decodo.com" cta="Share feedback">
    Can't find what you're looking for? Request an article! \
    Have feedback? Share your thoughts on how we can improve.
  </Card>
</Columns>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.