> ## Documentation Index
> Fetch the complete documentation index at: https://help.decodo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# LlamaIndex

> Decodo readers and tools for LlamaIndex: load live web pages and search results into RAG pipelines, or let agents scrape and search the web.

Decodo's LlamaIndex integration ships as two Python packages:

* **`llama-index-readers-decodo`** – readers that load web pages and search results as LlamaIndex `Document` objects for RAG pipelines
* **`llama-index-tools-decodo`** – a tool spec that lets LlamaIndex agents scrape URLs and search Google and Amazon

> [!NOTE]
>
> View the [**llama\_index repository**](https://github.com/Decodo/llama_index) for the source code, examples, and latest releases.

## Requirements

* Python 3.10+
* A Decodo [**Web Data API plan**](https://dashboard.decodo.com/web-data/pricing). Free trials are available on the [**dashboard**](https://dashboard.decodo.com/).

## Installation

Install one or both packages:

```bash theme={null}
pip install llama-index-readers-decodo llama-index-tools-decodo
```

## Authentication

Copy your Web Data API key from the [**Web Data API playground**](https://dashboard.decodo.com/web-data/playground) and store it in the `DECODO_API_TOKEN` environment variable. Every class reads it from there:

```bash theme={null}
export DECODO_API_TOKEN=your_api_key
```

Then pass `auth_mode="token"` to each class you create:

```python theme={null}
from llama_index.readers.decodo import DecodoWebReader

reader = DecodoWebReader(auth_mode="token")
```

> [!WARNING]
>
> `auth_mode` defaults to `"basic"`, not `"token"`. If you leave it out, the API key is sent as a basic authentication token and every request fails with `Username invalid.`

Older plans only have a basic authentication token (a base64-encoded `username:password`). Put that in `DECODO_API_TOKEN` and leave `auth_mode` out, since `"basic"` is the default. You can also pass the credential directly with `api_token=` instead of using the environment variable.

## Load web pages for RAG

`DecodoWebReader` scrapes each URL and returns it as a `Document` with the page as Markdown. This example indexes two pages and queries them. It uses LlamaIndex's default OpenAI models, so it also needs `llama-index-llms-openai`, `llama-index-embeddings-openai` and an `OPENAI_API_KEY`:

```python theme={null}
# main.py
from llama_index.core import VectorStoreIndex
from llama_index.readers.decodo import DecodoWebReader

reader = DecodoWebReader(auth_mode="token")
documents = reader.load_data(
    [
        "https://quotes.toscrape.com",
        "https://quotes.toscrape.com/page/2/",
    ]
)

# Drop pages that failed to scrape before indexing them
documents = [doc for doc in documents if "error" not in doc.metadata]

index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()

print(query_engine.query("Which quotes are about books or reading?"))
```

Each `Document` has `url`, `source` and `status_code` in its metadata. URLs are scraped one at a time, in order.

> [!WARNING]
>
> A URL that fails to scrape doesn't raise by default. It still returns a `Document`, with the error message as its **text** (`[Decodo scrape error] ...`) and an `error` key in its metadata. Filter these out, as in the example above, or the error text gets indexed alongside your pages. Pass `continue_on_error=False` to `load_data()` to raise on the first failure instead.

## Load search results

`DecodoSearchReader` runs a search and returns the results page as Markdown:

```python theme={null}
from llama_index.readers.decodo import DecodoSearchReader

reader = DecodoSearchReader(auth_mode="token")
documents = reader.load_data("wireless keyboard", engine="amazon")
```

| Argument | Default | Description |
| - | - | - |
| `query` | – | Search query. Required |
| `engine` | `"google"` | `"google"` calls the `google_search` target, `"amazon"` calls `amazon_search` |
| `num_results` | `10` | Sent to the API as `limit` |

Each search returns **one** `Document` holding the whole results page, not one `Document` per result. Its metadata has `query`, `engine`, `target`, `url`, `source` and `status_code`. If the search fails, the reader raises a `RuntimeError`.

> [!NOTE]
>
> `num_results` doesn't cap the results returned. A Google search always returns the full first page of results.

## Agent tools

`DecodoToolSpec` gives a LlamaIndex agent tools to scrape and search the web. This example uses OpenAI through `llama-index-llms-openai`, but any LLM LlamaIndex supports works:

```bash theme={null}
pip install llama-index-tools-decodo llama-index-llms-openai
export OPENAI_API_KEY=your_openai_key
```

```python theme={null}
# agent.py
import asyncio

from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI
from llama_index.tools.decodo import DecodoToolSpec

tools = DecodoToolSpec(auth_mode="token").to_tool_list()

agent = FunctionAgent(
    tools=tools,
    llm=OpenAI(model="gpt-5-mini"),
)


async def main():
    response = await agent.run(
        "Search Google for the 2025 NBA Finals, scrape the Wikipedia page about it and tell me who won."
    )
    print(response)


asyncio.run(main())
```

`to_tool_list()` gives the agent these tools:

| Tool | Purpose | Returns |
| - | - | - |
| `scrape_url` | Scrape any URL | The page as Markdown |
| `search_web` | Google Search | A list of results, each with `url`, `content` and `status_code` |
| `search_amazon` | Amazon Search | A list of results, each with `url`, `content` and `status_code` |
| `search_reddit` | Google Search limited to `site:reddit.com` | A list of results, each with `url`, `content` and `status_code` |

The search tools return the results page as Markdown in `content`, and raise a `RuntimeError` when the search fails so the agent can retry.

## Configuration

`DecodoWebReader`, `DecodoSearchReader` and `DecodoToolSpec` all take the same arguments:

| Argument | Default | Description |
| - | - | - |
| `api_token` | `DECODO_API_TOKEN` | API key, or basic authentication token on older plans |
| `auth_mode` | `"basic"` | Set to `"token"` for an API key |
| `timeout` | `180` | Request timeout in seconds |

For the full list of API parameters, see [**Parameters**](https://help.decodo.com/docs/web-scraping-api-parameters-overview.md).

***

## Related

* [**LangChain (Python)**](https://help.decodo.com/docs/langchain-python.md) – Decodo tools and a document loader for LangChain.
* [**Python SDK**](https://help.decodo.com/docs/python-sdk.md) – Typed access to every Decodo target from Python.
* [**MCP Server**](https://help.decodo.com/docs/mcp.md) – Connect AI agents and MCP-compatible tools to Decodo.
* [GitHub repository](https://github.com/Decodo/llama_index) – View the source code, examples, and latest releases.

***

> **[Support](https://direct.lc.chat/12092754)**
>
> Need help or just want to say hello? Our support is available 24/7. \
> You can also reach us anytime via email at [support@decodo.com](mailto:support@decodo.com).

> **[Feedback](mailto:feedback@decodo.com)**
>
> Can't find what you're looking for? Request an article! \
> Have feedback? Share your thoughts on how we can improve.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.