langchain-decodo is the Python LangChain package for the Decodo Web Data API. It gives LangChain agents two tools, one that scrapes any URL and one that searches Google or Amazon, and adds a document loader that turns web pages into LangChain Document objects for RAG pipelines.
View the langchain repository for the source code and latest releases. Building in Node.js? See LangChain (TypeScript).
Features
- Web scraping – scrape any URL and get the page back as Markdown
- Search – Google Search and Amazon Search with parsed, structured results
- Document loader – load one or more URLs as LangChain
Documentobjects for RAG - LangChain integration – standard
BaseToolandBaseLoaderclasses that plug into any agent or chain
Requirements
- Python 3.10+
- A Decodo Web Data API plan. Free trials are available on the dashboard.
Installation
Authentication
Copy your Web Data API key from the Web Data API playground and store it in theDECODO_API_TOKEN environment variable. Every class reads it from there:
auth_mode="token" to each class you create:
username:password). Put that in DECODO_API_TOKEN and leave auth_mode out, since "basic" is the default.
You can also pass the credential directly instead of using the environment variable: decodo_api_token= on the two tools, api_token= on the loader.
Quick start
A LangChain agent that searches Google, picks a page and scrapes it. This example uses OpenAI throughlangchain-openai, but any chat model LangChain supports works:
Available tools
You can also call a tool directly, without an agent:
DecodoWebScrapeTool
Takes a single url, which must include http:// or https://. It calls the universal target with markdown: true and returns the page as a Markdown string.
DecodoSearchTool
Takes these inputs:
It always sends
parse: true and returns a JSON string: a list with one entry per result, each holding content, status_code and url. The parsed results are in content, for example the Google organic results at content.results.results.organic.
If the search fails, the tool raises a RuntimeError with the status code. A failed search can come back from the API as HTTP 200, so raising lets the agent retry instead of treating it as “no results”.
Pass
engine with each call, as in the example above. Setting it when you create the tool (DecodoSearchTool(engine="amazon")) has no effect, and the search still goes to Google.num_results doesn’t cap the results returned. A Google search with num_results set to 3 still returns the full first page of organic results.Document loader
DecodoLoader scrapes one or more URLs and returns each page as a LangChain Document, ready to split, embed and store in a RAG pipeline:
Document holds the page as Markdown in page_content. Its metadata has source, url and status_code, plus task_id, created_at and updated_at when the API returns them. URLs are scraped one at a time, in order. Use load() for a list of every document, or lazy_load() to process them as they arrive.
With
continue_on_error on, a failed URL still produces a Document, with empty page_content and the error message in metadata["error"]. Filter those out before embedding. Set continue_on_error=False to raise on the first failure instead.
For the full list of API parameters, see Parameters.
Related
- LangChain (TypeScript) – The Node.js LangChain tools.
- LlamaIndex – Decodo readers and agent tools for LlamaIndex.
- Python SDK – Typed access to every Decodo target from Python.
- MCP Server – Connect AI agents and MCP-compatible tools to Decodo.
- GitHub repository – View the source code and latest releases.
Support
Need help or just want to say hello? Our support is available 24/7.
You can also reach us anytime via email at support@decodo.com.
You can also reach us anytime via email at support@decodo.com.
Feedback
Can’t find what you’re looking for? Request an article!
Have feedback? Share your thoughts on how we can improve.
Have feedback? Share your thoughts on how we can improve.