crawl_id and ready-to-use links for every endpoint below.
| Endpoint | Method | Purpose |
|---|---|---|
https://data.decodo.com/v1/crawl | POST | Starts a crawl. |
https://data.decodo.com/v1/crawl/{crawl_id}/status | GET | Crawl status. |
https://data.decodo.com/v1/crawl/{crawl_id} | GET | List of crawled URLs, their status and Task IDs for each URL. You can then fetch each URL content individually using Task ID. |
https://data.decodo.com/v1/crawl/{crawl_id}?type=content | GET | Scraped page content inline. |
https://data.decodo.com/v1/crawl/{crawl_id}/cancel | POST | Stop the crawl and keep collected results. |
Check crawl status
Endpoint: https://data.decodo.com/v1/crawl/{crawl_id}/status
This is the crawl_status link from the crawl response.
Get results
Results are stored for 24 hours after the crawl finishes.
- List of crawled URLs
Endpoint: https://data.decodo.com/v1/crawl/{crawl_id}
This is the results_tasks link. It returns every URL in the crawl with its own status, so you can see which pages are finished, still running, or failed.
depth is the number of link hops from the starting URL, and parent_url is the page where the URL was found. You can then fetch each page’s content individually from https://data.decodo.com/v1/task/{task_id}/results.
- Full content inline in the response
Endpoint: https://data.decodo.com/v1/crawl/{crawl_id}?type=content
This is the results_content link. It returns the scraped content of every page, in the format you chose with output. Results are split into pages of up to 10 MB. When there are more pages to fetch, the response includes a next link - send a GET request to it to get the following page. next is a ready-to-use URL with a cursor value appended, so you don’t need to build it yourself. The last page has no next.
content. total is the number of pages in the whole crawl, which can be higher than the number returned in this response.
A page that failed has the status faulted and a content_error instead of content, so the rest of the crawl isn’t blocked and you can see which pages to retry:
next link instead:
next until a response comes back without one - that’s the last batch.
Cancel a crawl
Endpoint: https://data.decodo.com/v1/crawl/{crawl_id}/cancel
Send a POST request to stop a crawl early. No new pages are started, pages already being scraped finish, and everything collected so far stays available through the results endpoints.
status set to cancelled.
Webhooks
Addcallback_url to your crawl request and we’ll send a POST request to it once all pages are processed, so you don’t need to poll the status endpoint. To confirm a webhook really came from Decodo, add a passthrough value to your request. It’s returned unchanged in the webhook payload.
POST request with a JSON body. It doesn’t include page content - use the results link to fetch it.