llama-index-readers-decodo– readers that load web pages and search results as LlamaIndexDocumentobjects for RAG pipelinesllama-index-tools-decodo– a tool spec that lets LlamaIndex agents scrape URLs and search Google and Amazon
View the llama_index repository for the source code, examples, and latest releases.
Requirements
- Python 3.10+
- A Decodo Web Data API plan. Free trials are available on the dashboard.
Installation
Install one or both packages:Authentication
Copy your Web Data API key from the Web Data API playground and store it in theDECODO_API_TOKEN environment variable. Every class reads it from there:
auth_mode="token" to each class you create:
username:password). Put that in DECODO_API_TOKEN and leave auth_mode out, since "basic" is the default. You can also pass the credential directly with api_token= instead of using the environment variable.
Load web pages for RAG
DecodoWebReader scrapes each URL and returns it as a Document with the page as Markdown. This example indexes two pages and queries them. It uses LlamaIndex’s default OpenAI models, so it also needs llama-index-llms-openai, llama-index-embeddings-openai and an OPENAI_API_KEY:
Document has url, source and status_code in its metadata. URLs are scraped one at a time, in order.
Load search results
DecodoSearchReader runs a search and returns the results page as Markdown:
Each search returns one
Document holding the whole results page, not one Document per result. Its metadata has query, engine, target, url, source and status_code. If the search fails, the reader raises a RuntimeError.
num_results doesn’t cap the results returned. A Google search always returns the full first page of results.Agent tools
DecodoToolSpec gives a LlamaIndex agent tools to scrape and search the web. This example uses OpenAI through llama-index-llms-openai, but any LLM LlamaIndex supports works:
to_tool_list() gives the agent these tools:
The search tools return the results page as Markdown in
content, and raise a RuntimeError when the search fails so the agent can retry.
Configuration
DecodoWebReader, DecodoSearchReader and DecodoToolSpec all take the same arguments:
For the full list of API parameters, see Parameters.
Related
- LangChain (Python) – Decodo tools and a document loader for LangChain.
- Python SDK – Typed access to every Decodo target from Python.
- MCP Server – Connect AI agents and MCP-compatible tools to Decodo.
- GitHub repository – View the source code, examples, and latest releases.
Support
Need help or just want to say hello? Our support is available 24/7.
You can also reach us anytime via email at support@decodo.com.
You can also reach us anytime via email at support@decodo.com.
Feedback
Can’t find what you’re looking for? Request an article!
Have feedback? Share your thoughts on how we can improve.
Have feedback? Share your thoughts on how we can improve.