> ## Documentation Index
> Fetch the complete documentation index at: https://help.decodo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# CrewAI

> The decodo-crewai package: Decodo scraping, search, SERP, and ecommerce tools for CrewAI agents and crews.

`decodo-crewai` is the Python package that gives [**CrewAI**](https://www.crewai.com) agents live web data. It adds four tools to a crew: one that scrapes any public URL as Markdown, and three that return parsed JSON from search engines, Google AI Mode, and ecommerce stores. All four run on the [**Web Data API**](https://help.decodo.com/docs/web-data-api-introduction.md).

> [!NOTE]
>
> View the [**crewAI repository**](https://github.com/Decodo/crewAI) for the source code, examples, and latest releases.

## Features

* **Web scraping** – fetch any public page and hand the agent clean Markdown
* **Search** – Google web results or Amazon product listings, parsed
* **SERP** – Google, Bing, and Google AI Mode results with paging
* **Ecommerce** – product details or search results from Amazon, Walmart, and Target
* **CrewAI integration** – standard `BaseTool` classes that drop into any agent, crew, or flow
* **Failures the agent can act on** – a failed request comes back as a CrewAI `ToolFailure`, not an exception

## Requirements

* Python 3.10 to 3.13
* A Decodo [**Web Data API plan**](https://dashboard.decodo.com/web-data/pricing). Free trials are available on the [**dashboard**](https://dashboard.decodo.com/).

## Installation

```bash theme={null}
pip install decodo-crewai
```

This pulls in `crewai` 1.15.9 or newer and the [**Python SDK**](https://help.decodo.com/docs/python-sdk.md) (`decodo-sdk`) 2.5.0 or newer.

## Authentication

Copy your API key from your Web Data API subscription in the [**Web Data API playground**](https://dashboard.decodo.com/web-data/playground) and store it in the `DECODO_API_KEY` environment variable. Every tool reads it from there:

```bash theme={null}
export DECODO_API_KEY=your_api_key
```

You can also pass it to a tool directly:

```python theme={null}
DecodoWebScraperTool(api_key="your_api_key")
```

> [!WARNING]
>
> These tools take an API key only. The basic authentication token from older plans (a base64-encoded `username:password`) isn't supported, even though the Python SDK accepts one.

A tool raises `ValueError` as soon as you create it if no key is found, so a missing key fails at startup rather than on the crew's first web request.

## Quick start

A one-agent crew that searches the web and reads the pages it finds. CrewAI calls its own LLM, so you also need a key for that – it defaults to OpenAI's `gpt-4.1-mini`:

```bash theme={null}
pip install decodo-crewai
export DECODO_API_KEY=your_api_key
export OPENAI_API_KEY=your_openai_key
```

```python theme={null}
# main.py
from crewai import Agent, Crew, Task

from decodo_crewai import DecodoSearchTool, DecodoWebScraperTool

researcher = Agent(
    role="Researcher",
    goal="Find and summarize up-to-date information from the web",
    backstory="You search the web, read the most relevant pages and report what they say.",
    tools=[DecodoSearchTool(), DecodoWebScraperTool()],
)

task = Task(
    description="Find the three most recent announcements about {topic} and summarize each one.",
    expected_output="Three bullet points, each with a one-sentence summary and the source URL.",
    agent=researcher,
)

crew = Crew(agents=[researcher], tasks=[task])
print(crew.kickoff(inputs={"topic": "solid-state batteries"}))
```

To use another model, set `MODEL`, for example `MODEL=gpt-4.1`. Other providers need their CrewAI extra and key, for example `pip install "crewai[anthropic]"` with `ANTHROPIC_API_KEY` and `MODEL=anthropic/<model>`.

## Available tools

| Class | Name shown to the agent | Purpose |
| - | - | - |
| `DecodoWebScraperTool` | `Decodo Web Scraper` | Scrape any public URL and return Markdown |
| `DecodoSearchTool` | `Decodo Search` | Google or Amazon search, parsed as JSON |
| `DecodoSERPTool` | `Decodo SERP` | Google, Bing, or Google AI Mode results, parsed as JSON |
| `DecodoEcommerceTool` | `Decodo Ecommerce` | Amazon, Walmart, or Target products, parsed as JSON |

You can also call a tool directly, without an agent:

```python theme={null}
from decodo_crewai import DecodoSearchTool, DecodoWebScraperTool

print(DecodoWebScraperTool().run(url="https://ip.decodo.com"))
print(DecodoSearchTool().run(query="mechanical keyboard", source="amazon"))
```

Every tool takes `api_key` when you create it, defaulting to `DECODO_API_KEY`. The arguments below that are the ones the agent fills in on each call.

### DecodoWebScraperTool

| Argument | Type | Description |
| - | - | - |
| `url` | `str` | Full URL of the page to scrape, including `https://` |
| `geo` | `str`, optional | Country to send the request from, as a country name, e.g. `United States` |
| `js_render` | `bool`, optional | Open the page in a headless browser. Only needed for pages that load their content with JavaScript |

It returns the page as a Markdown string.

| Constructor option | Default | Description |
| - | - | - |
| `api_key` | `DECODO_API_KEY` | Web Data API key |
| `max_chars` | `45000` | Markdown longer than this is truncated, to keep the page within the agent's context |

When a page is truncated, the tool adds a last line that gives the kept and the full character counts, so the agent knows it didn't get the whole page. Lower `max_chars` when an agent reads several pages in one task:

```python theme={null}
DecodoWebScraperTool(max_chars=15_000)
```

### DecodoSearchTool

| Argument | Type | Description |
| - | - | - |
| `query` | `str` | Search query |
| `source` | `"google"` or `"amazon"`, optional | `google` (default) for web results, `amazon` for product listings |
| `geo` | `str`, optional | Google: a country name, e.g. `United States`. Amazon: a US ZIP code, e.g. `10001`, or a two-letter country code for delivery outside the US, e.g. `DE` |

### DecodoSERPTool

| Argument | Type | Description |
| - | - | - |
| `query` | `str` | Search query |
| `engine` | `"google"`, `"bing"`, or `"google_ai_mode"`, optional | `google` (default) or `bing` for a results page; `google_ai_mode` for Google's AI-written answer and the sources it cites |
| `page` | `int`, optional | Results page to fetch, starting at 1. Ignored by `google_ai_mode` |
| `geo` | `str`, optional | Country to search from, e.g. `United States` |

Bing results are rendered in a headless browser and can take over a minute.

### DecodoEcommerceTool

Pass either `product_id` or `query`, not both.

| Argument | Type | Description |
| - | - | - |
| `store` | `"amazon"`, `"walmart"`, or `"target"` | Store to look up |
| `product_id` | `str`, optional | One product's details. Amazon: the ASIN, e.g. `B09H74FXNW`. Walmart or Target: the product ID from the product URL |
| `query` | `str`, optional | Search the store's products instead |
| `zip_code` | `str`, optional | US ZIP code for prices and availability at that delivery location, e.g. `10001` |

Target pages are rendered in a headless browser and can take close to a minute.

> [!NOTE]
>
> The three JSON tools return the full parsed response, with no truncation. An Amazon search comes back as roughly 117,000 characters, which is enough to crowd a small context window and show up on your LLM bill. Give those tools to an agent whose task needs the whole result set, and keep broad research on `DecodoSearchTool` with `source="google"`.

## Errors

When a request fails – an invalid API key, a rate limit, a page that returns `404`, or a network error that never reaches Decodo – the tools return a CrewAI `ToolFailure` instead of raising. The agent sees what went wrong and can retry or change course, and CrewAI records the failure according to the agent's `tool_failure_policy`.

Each failure carries a `code`, and the ones worth retrying are marked `retryable`:

| `code` | When it happens | Retryable |
| - | - | - |
| `authentication_failed` | Decodo rejected the API key | No |
| `invalid_params` | Decodo rejected the request parameters | No |
| `rate_limited` | You've hit the rate limit for your subscription | Yes |
| `timeout` | The request to Decodo timed out | Yes |
| `network_error` | The request never reached Decodo | Yes |
| `empty_response`, `empty_content` | Decodo returned no results, or a result with no content | No |
| An HTTP status, e.g. `404` or `503` | The target responded with that status | `429` and `5xx` only |

## Example crew

The repository has a [**research crew**](https://github.com/Decodo/crewAI/tree/main/examples/research_crew): a researcher that searches Google and picks five sources, an analyst that reads each one with the scraper, and a writer that turns the notes into a cited Markdown report. Point it at any topic and it prints the report and saves it to `report.md`.

For the full list of API parameters behind these tools, see [**Parameters**](https://help.decodo.com/docs/web-scraping-api-parameters-overview.md).

***

## Related

* [**LangChain (Python)**](https://help.decodo.com/docs/langchain-python.md) – Decodo tools and a document loader for LangChain.
* [**LlamaIndex**](https://help.decodo.com/docs/llamaindex.md) – Decodo readers and agent tools for LlamaIndex.
* [**Python SDK**](https://help.decodo.com/docs/python-sdk.md) – Typed access to every Decodo target from Python.
* [**MCP Server**](https://help.decodo.com/docs/mcp.md) – Connect AI agents and MCP-compatible tools to Decodo.
* [GitHub repository](https://github.com/Decodo/crewAI) – View the source code, examples, and latest releases.

***

> [card] [Support](https://direct.lc.chat/12092754)
>
> Need help or just want to say hello? Our support is available 24/7. \
> You can also reach us anytime via email at [support@decodo.com](mailto:support@decodo.com).

> [card] [Feedback](mailto:feedback@decodo.com)
>
> Can't find what you're looking for? Request an article! \
> Have feedback? Share your thoughts on how we can improve.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.