Skip to main content
langchain-decodo is the Python LangChain package for the Decodo Web Data API. It gives LangChain agents two tools, one that scrapes any URL and one that searches Google or Amazon, and adds a document loader that turns web pages into LangChain Document objects for RAG pipelines.
View the langchain repository for the source code and latest releases. Building in Node.js? See LangChain (TypeScript).

Features

  • Web scraping – scrape any URL and get the page back as Markdown
  • Search – Google Search and Amazon Search with parsed, structured results
  • Document loader – load one or more URLs as LangChain Document objects for RAG
  • LangChain integration – standard BaseTool and BaseLoader classes that plug into any agent or chain

Requirements

Installation

Authentication

Copy your Web Data API key from the Web Data API playground and store it in the DECODO_API_TOKEN environment variable. Every class reads it from there:
Then pass auth_mode="token" to each class you create:
auth_mode defaults to "basic", not "token". If you leave it out, the API key is sent as a basic authentication token and every request fails with Decodo API error: Username invalid.
Older plans only have a basic authentication token (a base64-encoded username:password). Put that in DECODO_API_TOKEN and leave auth_mode out, since "basic" is the default. You can also pass the credential directly instead of using the environment variable: decodo_api_token= on the two tools, api_token= on the loader.

Quick start

A LangChain agent that searches Google, picks a page and scrapes it. This example uses OpenAI through langchain-openai, but any chat model LangChain supports works:

Available tools

You can also call a tool directly, without an agent:

DecodoWebScrapeTool

Takes a single url, which must include http:// or https://. It calls the universal target with markdown: true and returns the page as a Markdown string.

DecodoSearchTool

Takes these inputs: It always sends parse: true and returns a JSON string: a list with one entry per result, each holding content, status_code and url. The parsed results are in content, for example the Google organic results at content.results.results.organic. If the search fails, the tool raises a RuntimeError with the status code. A failed search can come back from the API as HTTP 200, so raising lets the agent retry instead of treating it as “no results”.
Pass engine with each call, as in the example above. Setting it when you create the tool (DecodoSearchTool(engine="amazon")) has no effect, and the search still goes to Google.
num_results doesn’t cap the results returned. A Google search with num_results set to 3 still returns the full first page of organic results.

Document loader

DecodoLoader scrapes one or more URLs and returns each page as a LangChain Document, ready to split, embed and store in a RAG pipeline:
Each Document holds the page as Markdown in page_content. Its metadata has source, url and status_code, plus task_id, created_at and updated_at when the API returns them. URLs are scraped one at a time, in order. Use load() for a list of every document, or lazy_load() to process them as they arrive. With continue_on_error on, a failed URL still produces a Document, with empty page_content and the error message in metadata["error"]. Filter those out before embedding. Set continue_on_error=False to raise on the first failure instead. For the full list of API parameters, see Parameters.

Support

Need help or just want to say hello? Our support is available 24/7.
You can also reach us anytime via email at support@decodo.com.

Feedback

Can’t find what you’re looking for? Request an article!
Have feedback? Share your thoughts on how we can improve.