CrewAI

decodo-crewai is the Python package that gives CrewAI agents live web data. It adds four tools to a crew: one that scrapes any public URL as Markdown, and three that return parsed JSON from search engines, Google AI Mode, and ecommerce stores. All four run on the Web Data API.

View the crewAI repository for the source code, examples, and latest releases.

Features

Web scraping – fetch any public page and hand the agent clean Markdown
Search – Google web results or Amazon product listings, parsed
SERP – Google, Bing, and Google AI Mode results with paging
Ecommerce – product details or search results from Amazon, Walmart, and Target
CrewAI integration – standard BaseTool classes that drop into any agent, crew, or flow
Failures the agent can act on – a failed request comes back as a CrewAI ToolFailure, not an exception

Requirements

Python 3.10 to 3.13
A Decodo Web Data API plan. Free trials are available on the dashboard.

Installation

pip install decodo-crewai

This pulls in crewai 1.15.9 or newer and the Python SDK (decodo-sdk) 2.5.0 or newer.

Authentication

Copy your API key from your Web Data API subscription in the Web Data API playground and store it in the DECODO_API_KEY environment variable. Every tool reads it from there:

export DECODO_API_KEY=your_api_key

You can also pass it to a tool directly:

DecodoWebScraperTool(api_key="your_api_key")

These tools take an API key only. The basic authentication token from older plans (a base64-encoded username:password) isn’t supported, even though the Python SDK accepts one.

A tool raises ValueError as soon as you create it if no key is found, so a missing key fails at startup rather than on the crew’s first web request.

Quick start

A one-agent crew that searches the web and reads the pages it finds. CrewAI calls its own LLM, so you also need a key for that – it defaults to OpenAI’s gpt-4.1-mini:

pip install decodo-crewai
export DECODO_API_KEY=your_api_key
export OPENAI_API_KEY=your_openai_key

# main.py
from crewai import Agent, Crew, Task

from decodo_crewai import DecodoSearchTool, DecodoWebScraperTool

researcher = Agent(
    role="Researcher",
    goal="Find and summarize up-to-date information from the web",
    backstory="You search the web, read the most relevant pages and report what they say.",
    tools=[DecodoSearchTool(), DecodoWebScraperTool()],
)

task = Task(
    description="Find the three most recent announcements about {topic} and summarize each one.",
    expected_output="Three bullet points, each with a one-sentence summary and the source URL.",
    agent=researcher,
)

crew = Crew(agents=[researcher], tasks=[task])
print(crew.kickoff(inputs={"topic": "solid-state batteries"}))

To use another model, set MODEL, for example MODEL=gpt-4.1. Other providers need their CrewAI extra and key, for example pip install "crewai[anthropic]" with ANTHROPIC_API_KEY and MODEL=anthropic/<model>.

Available tools

Class	Name shown to the agent	Purpose
`DecodoWebScraperTool`	`Decodo Web Scraper`	Scrape any public URL and return Markdown
`DecodoSearchTool`	`Decodo Search`	Google or Amazon search, parsed as JSON
`DecodoSERPTool`	`Decodo SERP`	Google, Bing, or Google AI Mode results, parsed as JSON
`DecodoEcommerceTool`	`Decodo Ecommerce`	Amazon, Walmart, or Target products, parsed as JSON

You can also call a tool directly, without an agent:

from decodo_crewai import DecodoSearchTool, DecodoWebScraperTool

print(DecodoWebScraperTool().run(url="https://ip.decodo.com"))
print(DecodoSearchTool().run(query="mechanical keyboard", source="amazon"))

Every tool takes api_key when you create it, defaulting to DECODO_API_KEY. The arguments below that are the ones the agent fills in on each call.

DecodoWebScraperTool

Argument	Type	Description
`url`	`str`	Full URL of the page to scrape, including `https://`
`geo`	`str`, optional	Country to send the request from, as a country name, e.g. `United States`
`js_render`	`bool`, optional	Open the page in a headless browser. Only needed for pages that load their content with JavaScript

It returns the page as a Markdown string.

Constructor option	Default	Description
`api_key`	`DECODO_API_KEY`	Web Data API key
`max_chars`	`45000`	Markdown longer than this is truncated, to keep the page within the agent’s context

When a page is truncated, the tool adds a last line that gives the kept and the full character counts, so the agent knows it didn’t get the whole page. Lower max_chars when an agent reads several pages in one task:

DecodoWebScraperTool(max_chars=15_000)

DecodoSearchTool

Argument	Type	Description
`query`	`str`	Search query
`source`	`"google"` or `"amazon"`, optional	`google` (default) for web results, `amazon` for product listings
`geo`	`str`, optional	Google: a country name, e.g. `United States`. Amazon: a US ZIP code, e.g. `10001`, or a two-letter country code for delivery outside the US, e.g. `DE`

DecodoSERPTool

Argument	Type	Description
`query`	`str`	Search query
`engine`	`"google"`, `"bing"`, or `"google_ai_mode"`, optional	`google` (default) or `bing` for a results page; `google_ai_mode` for Google’s AI-written answer and the sources it cites
`page`	`int`, optional	Results page to fetch, starting at 1. Ignored by `google_ai_mode`
`geo`	`str`, optional	Country to search from, e.g. `United States`

Bing results are rendered in a headless browser and can take over a minute.

DecodoEcommerceTool

Pass either product_id or query, not both.

Argument	Type	Description
`store`	`"amazon"`, `"walmart"`, or `"target"`	Store to look up
`product_id`	`str`, optional	One product’s details. Amazon: the ASIN, e.g. `B09H74FXNW`. Walmart or Target: the product ID from the product URL
`query`	`str`, optional	Search the store’s products instead
`zip_code`	`str`, optional	US ZIP code for prices and availability at that delivery location, e.g. `10001`

Target pages are rendered in a headless browser and can take close to a minute.

The three JSON tools return the full parsed response, with no truncation. An Amazon search comes back as roughly 117,000 characters, which is enough to crowd a small context window and show up on your LLM bill. Give those tools to an agent whose task needs the whole result set, and keep broad research on DecodoSearchTool with source="google".

Errors

When a request fails – an invalid API key, a rate limit, a page that returns 404, or a network error that never reaches Decodo – the tools return a CrewAI ToolFailure instead of raising. The agent sees what went wrong and can retry or change course, and CrewAI records the failure according to the agent’s tool_failure_policy. Each failure carries a code, and the ones worth retrying are marked retryable:

`code`	When it happens	Retryable
`authentication_failed`	Decodo rejected the API key	No
`invalid_params`	Decodo rejected the request parameters	No
`rate_limited`	You’ve hit the rate limit for your subscription	Yes
`timeout`	The request to Decodo timed out	Yes
`network_error`	The request never reached Decodo	Yes
`empty_response`, `empty_content`	Decodo returned no results, or a result with no content	No
An HTTP status, e.g. `404` or `503`	The target responded with that status	`429` and `5xx` only

Example crew

The repository has a research crew: a researcher that searches Google and picks five sources, an analyst that reads each one with the scraper, and a writer that turns the notes into a cited Markdown report. Point it at any topic and it prints the report and saves it to report.md. For the full list of API parameters behind these tools, see Parameters.

LangChain (Python) – Decodo tools and a document loader for LangChain.
LlamaIndex – Decodo readers and agent tools for LlamaIndex.
Python SDK – Typed access to every Decodo target from Python.
MCP Server – Connect AI agents and MCP-compatible tools to Decodo.
GitHub repository – View the source code, examples, and latest releases.

Support

Need help or just want to say hello? Our support is available 24/7.
You can also reach us anytime via email at support@decodo.com.

Feedback

Can’t find what you’re looking for? Request an article!
Have feedback? Share your thoughts on how we can improve.

ABOUT

RESIDENTIAL PROXIES

STATIC RESIDENTIAL (ISP) PROXIES

DATACENTER PROXIES

FREE PROXIES

SITE UNBLOCKER

WEB DATA API

VIDEO DOWNLOADER

DECODO TOOLS

AI

INTEGRATIONS

ACCOUNT & DASHBOARD

PAYMENTS

TROUBLESHOOTING

TIPS & TRICKS

USEFUL LINKS

Features

Requirements

Installation

Authentication

Quick start

Available tools

DecodoWebScraperTool

DecodoSearchTool

DecodoSERPTool

DecodoEcommerceTool

Errors

Example crew

Support

Feedback

​Features

​Requirements

​Installation

​Authentication

​Quick start

​Available tools

​DecodoWebScraperTool

​DecodoSearchTool

​DecodoSERPTool

​DecodoEcommerceTool

​Errors

​Example crew

​Related

Support

Feedback

Features

Requirements

Installation

Authentication

Quick start

Available tools

DecodoWebScraperTool

DecodoSearchTool

DecodoSERPTool

DecodoEcommerceTool

Errors

Example crew

Related