pobear – web scraping CLI for the terminal

pobear is an open-source web scraping CLI for extracting structured data and crawling websites from the command line. It runs in your terminal as a single Go binary – no Python or Node.js runtime to install – and keeps what you extract separate from how you fetch. Start with direct requests and no account. When a site starts blocking you, add a proxy or the Decodo Web Data API with one flag and keep the same extraction rules.

View the pobear repository for the source code, full CLI reference and latest releases. pobear is MIT-licensed.

What you can do

Extract the fields you need: CSS selectors with attribute support, or HTML-to-Markdown conversion.
Crawl a site: depth, page limits, scope, include/exclude globs and request rates, with sitemap discovery and robots.txt support.
Save reusable jobs: print any command as a JSON job spec and run it again later, from a script, a scheduler or an AI agent.
Escalate when blocked: with --ladder, pobear tries a direct request first, then your proxy, then the Web Data API.

Install

On Linux or macOS:

curl -fsSL https://raw.githubusercontent.com/Decodo/pobear/main/install.sh | sh

pobear ships as a precompiled binary for Linux, macOS and Windows, on both x64 (amd64) and ARM64. You don’t need Go installed – only to build from source. The installer downloads the latest release, verifies its checksum and installs pobear to $HOME/.local/bin. On Windows, download the windows_amd64 or windows_arm64 ZIP from the releases page and run pobear.exe. Check the install:

pobear version

Run pobear with no arguments to open an interactive terminal wizard instead of typing flags.

Quick start

Crawl a sample bookshop, extract titles and prices, and save the results. No account needed:

pobear crawl https://books.toscrape.com/ --depth 1 --max-pages 10 \
  --css 'title=h3 a@title' --css 'price=.price_color' --out books.ndjson

The run summary goes to stderr:

fetched 10, errors 0, skipped 64

Each fetched page becomes one line of NDJSON in books.ndjson. The data field holds the extracted values:

{
  "url": "https://books.toscrape.com/",
  "tier": "local",
  "status": 200,
  "data": {
    "title": ["A Light in the Attic", "Tipping the Velvet", "..."],
    "price": ["£51.77", "£53.74", "..."]
  }
}

h3 a@title reads the title attribute of each matching link, and .price_color reads the element text. Each record also includes finalUrl, parentUrl, depth and the raw body. skipped counts links the crawl found but didn’t visit. Without --out, records stream to stdout, so you can pipe them into jq or another tool.

Common tasks

Get a page as Markdown

Print one page as clean Markdown, without the JSON envelope. Useful as LLM input:

pobear scrape https://go.dev/doc/ --extract markdown --bare

Scrape a list of URLs

Put one URL per line in a text file. Blank lines and lines starting with # are skipped:

pobear scrape urls.txt --extract markdown --out pages.ndjson

Save and rerun a job

Add --print-spec to any command to print it as a JSON job spec instead of running it:

pobear crawl https://books.toscrape.com/ --depth 1 --max-pages 10 \
  --css 'title=h3 a@title' --css 'price=.price_color' \
  --out books.ndjson --print-spec > books.json

Run the saved job whenever you need fresh data:

pobear run books.json

pobear run accepts JSON or YAML. Keep job files in version control or run them on a schedule. See the job spec reference for the format.

Choose how to fetch

Tier	Flag	When to use it
Direct (`local`)	Default	Pages that load without protection. No account needed
Your proxy (`proxy`)	`--proxy '<proxy-url>'`	Requests that need to come from another IP
Decodo Web Data API (`provider`)	`--provider decodo`	Pages behind anti-bot protection, CAPTCHAs, or that need JavaScript rendering

Every record has a tier field showing which method fetched it. Direct and proxy requests can mimic Chrome, Firefox or Safari TLS fingerprints with --profile (default chrome), but they don’t run JavaScript.

Use the Web Data API

Copy your Web Data API key from the Web Data API playground. Any Web Data API plan works, and free trials are available on the dashboard.
Save the key. pobear prompts for it and stores it for later runs:
pobear auth decodo
In CI or a container, set the POBEAR_PROVIDER_DECODO_TOKEN environment variable instead.

Add --provider decodo to any scrape or crawl command:

pobear scrape https://books.toscrape.com/ \
  --css 'title=h3 a@title' --css 'price=.price_color' \
  --provider decodo --out books.ndjson

The request goes through the Web Data API universal target with the premium proxy pool. JavaScript rendering is off by default. Add --render for pages that build their content in the browser:

pobear scrape https://quotes.toscrape.com/js/ \
  --css 'quote=.quote .text' --provider decodo --render

Without --render, that page returns no quotes. With it, all 10 are extracted. Use --geo and --opt name=value to set Web Data API parameters, such as --geo "United States" or --opt device_type=mobile. See localisation for accepted geo values.

Web Data API requests are billed to your Decodo plan. pobear itself is free.

Use a target template

--template switches from universal to a dedicated Web Data API target. Amazon URLs need the amazon template – the universal target rejects them with HTTP 400:

pobear scrape 'https://www.amazon.com/dp/B09H74FXNW' \
  --provider decodo --template amazon --geo 10001 --opt parse=true

With parse=true, the body field contains parsed JSON (title, price, brand, ratings and more) instead of HTML. Available templates: universal (default), airbnb, amazon, apple_app_store, autotrader, bbb, bing, google, mobile, target, tiktok, tiktok_post and walmart. To see each template’s options:

pobear providers decodo

A template applies to the whole job, so run Amazon URLs and other sites as separate jobs.

Amazon geo: when the country matches the Amazon domain, for example US on amazon.com, use a ZIP code such as 10001. A country code returns HTTP 400. Read more
Google geo: the google template doesn’t accept --geo, so results can come from any country. For location-specific Google results, call the Web Data API google_search target directly.

Escalate automatically when blocked

--ladder starts with a direct request and moves to the next tier only when a request is blocked, rate-limited, times out, or returns a known anti-bot challenge. Pass --provider decodo with it – on its own, --ladder has nowhere to escalate to:

pobear scrape 'https://www.amazon.com/dp/B09H74FXNW' \
  --ladder --provider decodo --template amazon --geo 10001 \
  --css 'title=#productTitle' --css 'price=.a-price .a-offscreen'

Add --proxy '<proxy-url>' as well to try your proxy between the direct request and the Web Data API. Pages that load directly never use the API, so you only pay for the ones that need it.

Each tier retries 3 times with a 30-second timeout before pobear escalates. For sites you know are protected, skip the ladder and use --provider decodo directly. Tune the wait with --retries and --timeout.

Use pobear from AI agents and scripts

pobear works well as a subprocess for an agent or a script:

pobear docs --json returns the commands, output format and exit codes as JSON.
pobear providers decodo --json returns every template, option and default.
Data goes to stdout as NDJSON. Run summaries and errors go to stderr.

Pipe a job spec straight into the runner:

pobear scrape https://books.toscrape.com/ --css 'price=.price_color' \
  --print-spec | pobear run -

Check the exit code before using the results:

Exit code	Meaning
`0`	Success
`1`	Missing or rejected provider credentials
`3`	Invalid command, flag or job spec
`4`	Partial failure – some results were produced
`5`	Fatal failure – nothing usable was produced

Troubleshooting

empty 1 in the summary. The page was fetched with HTTP 200, but your selectors matched nothing. Check the selectors and the page content first. An empty result can also mean the page needs --render, or that the site served a challenge page. Catching empty extractions in pipelines. --fail-on-empty exits with code 5 only when every fetched page is empty. A run where some pages extract data and others don’t still exits 0, so check the empty count in the summary as well. 401 with Missing authorization header. No Web Data API key is saved. Run pobear auth decodo or set POBEAR_PROVIDER_DECODO_TOKEN. 400 with Validation failed. The Web Data API accepted your key but rejected the request. The most common cause is an Amazon URL without --template amazon.

FAQ

How do I scrape a website from the terminal?

Install pobear, then run pobear scrape with the URL and the fields you want:

pobear scrape https://books.toscrape.com/ --css 'price=.price_color'

It prints one NDJSON record with the extracted prices. Use pobear crawl to follow links across a site, or run pobear with no arguments to open the interactive terminal wizard.

Do I need a Decodo account to use pobear?

No. Direct requests work without any account. You only need a Web Data API key or proxy credentials when you add --provider decodo or --proxy.

Can pobear scrape JavaScript-rendered websites?

Direct and proxy requests don’t run JavaScript. Use --provider decodo --render to get the page after the browser has rendered it. See Use the Web Data API.

How is pobear different from the Decodo CLI?

They start from opposite defaults. The Decodo CLI is a Node.js client for the Web Data API: every request goes through Decodo, and every Web Data API target – search engines, eCommerce, social media – is a subcommand. pobear is a standalone Go binary that fetches directly by default and adds a proxy or the Web Data API only when you ask for it, or when --ladder hits a block. Use the Decodo CLI when you want the full Web Data API with ready-made targets. Use pobear when you want one binary that works without an account and pays for the API only on the pages that need it.

Decodo CLI – the official Web Data API command line tool, with search, map, crawl and screenshots
Web Data API introduction
pobear CLI reference

Support

Need help or just want to say hello? Our support is available 24/7.
You can also reach us anytime via email at support@decodo.com.

Feedback

Can’t find what you’re looking for? Request an article!
Have feedback? Share your thoughts on how we can improve.

ABOUT

RESIDENTIAL PROXIES

STATIC RESIDENTIAL (ISP) PROXIES

DATACENTER PROXIES

FREE PROXIES

SITE UNBLOCKER

WEB DATA API

VIDEO DOWNLOADER

DECODO TOOLS

AI

INTEGRATIONS

ACCOUNT & DASHBOARD

PAYMENTS

TROUBLESHOOTING

TIPS & TRICKS

USEFUL LINKS

pobear – web scraping CLI for the terminal

What you can do

Install

Quick start

Common tasks

Get a page as Markdown

Scrape a list of URLs

Save and rerun a job

Choose how to fetch

Use the Web Data API

Use a target template

Escalate automatically when blocked

Use pobear from AI agents and scripts

Troubleshooting

FAQ

How do I scrape a website from the terminal?

Do I need a Decodo account to use pobear?

Can pobear scrape JavaScript-rendered websites?

How is pobear different from the Decodo CLI?

Support

Feedback

​What you can do

​Install

​Quick start

​Common tasks

​Get a page as Markdown

​Scrape a list of URLs

​Save and rerun a job

​Choose how to fetch

​Use the Web Data API

​Use a target template

​Escalate automatically when blocked

​Use pobear from AI agents and scripts

​Troubleshooting

​FAQ

​How do I scrape a website from the terminal?

​Do I need a Decodo account to use pobear?

​Can pobear scrape JavaScript-rendered websites?

​How is pobear different from the Decodo CLI?

​Related

Support

Feedback

What you can do

Install

Quick start

Common tasks

Get a page as Markdown

Scrape a list of URLs

Save and rerun a job

Choose how to fetch

Use the Web Data API

Use a target template

Escalate automatically when blocked

Use pobear from AI agents and scripts

Troubleshooting

FAQ

How do I scrape a website from the terminal?

Do I need a Decodo account to use pobear?

Can pobear scrape JavaScript-rendered websites?

How is pobear different from the Decodo CLI?

Related