Skip to main content
Decodo’s LlamaIndex integration ships as two Python packages:
  • llama-index-readers-decodo – readers that load web pages and search results as LlamaIndex Document objects for RAG pipelines
  • llama-index-tools-decodo – a tool spec that lets LlamaIndex agents scrape URLs and search Google and Amazon
View the llama_index repository for the source code, examples, and latest releases.

Requirements

Installation

Install one or both packages:

Authentication

Copy your Web Data API key from the Web Data API playground and store it in the DECODO_API_TOKEN environment variable. Every class reads it from there:
Then pass auth_mode="token" to each class you create:
auth_mode defaults to "basic", not "token". If you leave it out, the API key is sent as a basic authentication token and every request fails with Username invalid.
Older plans only have a basic authentication token (a base64-encoded username:password). Put that in DECODO_API_TOKEN and leave auth_mode out, since "basic" is the default. You can also pass the credential directly with api_token= instead of using the environment variable.

Load web pages for RAG

DecodoWebReader scrapes each URL and returns it as a Document with the page as Markdown. This example indexes two pages and queries them. It uses LlamaIndex’s default OpenAI models, so it also needs llama-index-llms-openai, llama-index-embeddings-openai and an OPENAI_API_KEY:
Each Document has url, source and status_code in its metadata. URLs are scraped one at a time, in order.
A URL that fails to scrape doesn’t raise by default. It still returns a Document, with the error message as its text ([Decodo scrape error] ...) and an error key in its metadata. Filter these out, as in the example above, or the error text gets indexed alongside your pages. Pass continue_on_error=False to load_data() to raise on the first failure instead.

Load search results

DecodoSearchReader runs a search and returns the results page as Markdown:
Each search returns one Document holding the whole results page, not one Document per result. Its metadata has query, engine, target, url, source and status_code. If the search fails, the reader raises a RuntimeError.
num_results doesn’t cap the results returned. A Google search always returns the full first page of results.

Agent tools

DecodoToolSpec gives a LlamaIndex agent tools to scrape and search the web. This example uses OpenAI through llama-index-llms-openai, but any LLM LlamaIndex supports works:
to_tool_list() gives the agent these tools: The search tools return the results page as Markdown in content, and raise a RuntimeError when the search fails so the agent can retry.

Configuration

DecodoWebReader, DecodoSearchReader and DecodoToolSpec all take the same arguments: For the full list of API parameters, see Parameters.
  • LangChain (Python) – Decodo tools and a document loader for LangChain.
  • Python SDK – Typed access to every Decodo target from Python.
  • MCP Server – Connect AI agents and MCP-compatible tools to Decodo.
  • GitHub repository – View the source code, examples, and latest releases.

Support

Need help or just want to say hello? Our support is available 24/7.
You can also reach us anytime via email at support@decodo.com.

Feedback

Can’t find what you’re looking for? Request an article!
Have feedback? Share your thoughts on how we can improve.