> ## Documentation Index
> Fetch the complete documentation index at: https://help.decodo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# pobear – web scraping CLI for the terminal

> pobear is an open-source web scraping CLI. Crawl sites, extract data with CSS selectors or Markdown, save reusable jobs, and fall back to the Decodo Web Data API when a request gets blocked.

[**pobear**](https://github.com/Decodo/pobear) is an open-source web scraping CLI for extracting structured data and crawling websites from the command line. It runs in your terminal as a single Go binary – no Python or Node.js runtime to install – and keeps *what you extract* separate from *how you fetch*. Start with direct requests and no account. When a site starts blocking you, add a proxy or the Decodo Web Data API with one flag and keep the same extraction rules.

> [!NOTE]
>
> View the [**pobear repository**](https://github.com/Decodo/pobear) for the source code, full CLI reference and latest releases. pobear is MIT-licensed.

## What you can do

* **Extract the fields you need**: CSS selectors with attribute support, or HTML-to-Markdown conversion.
* **Crawl a site**: depth, page limits, scope, include/exclude globs and request rates, with sitemap discovery and `robots.txt` support.
* **Save reusable jobs**: print any command as a JSON job spec and run it again later, from a script, a scheduler or an AI agent.
* **Escalate when blocked**: with `--ladder`, pobear tries a direct request first, then your proxy, then the Web Data API.

## Install

On Linux or macOS:

```bash theme={null}
curl -fsSL https://raw.githubusercontent.com/Decodo/pobear/main/install.sh | sh
```

pobear ships as a precompiled binary for Linux, macOS and Windows, on both x64 (amd64) and ARM64. You don't need Go installed – only to build from source. The installer downloads the latest release, verifies its checksum and installs `pobear` to `$HOME/.local/bin`. On Windows, download the `windows_amd64` or `windows_arm64` ZIP from the [**releases page**](https://github.com/Decodo/pobear/releases) and run `pobear.exe`.

Check the install:

```bash theme={null}
pobear version
```

Run `pobear` with no arguments to open an interactive terminal wizard instead of typing flags.

## Quick start

Crawl a sample bookshop, extract titles and prices, and save the results. No account needed:

```bash theme={null}
pobear crawl https://books.toscrape.com/ --depth 1 --max-pages 10 \
  --css 'title=h3 a@title' --css 'price=.price_color' --out books.ndjson
```

The run summary goes to stderr:

```text theme={null}
fetched 10, errors 0, skipped 64
```

Each fetched page becomes one line of NDJSON in `books.ndjson`. The `data` field holds the extracted values:

```json theme={null}
{
  "url": "https://books.toscrape.com/",
  "tier": "local",
  "status": 200,
  "data": {
    "title": ["A Light in the Attic", "Tipping the Velvet", "..."],
    "price": ["£51.77", "£53.74", "..."]
  }
}
```

`h3 a@title` reads the `title` attribute of each matching link, and `.price_color` reads the element text. Each record also includes `finalUrl`, `parentUrl`, `depth` and the raw `body`. `skipped` counts links the crawl found but didn't visit.

Without `--out`, records stream to stdout, so you can pipe them into `jq` or another tool.

## Common tasks

### Get a page as Markdown

Print one page as clean Markdown, without the JSON envelope. Useful as LLM input:

```bash theme={null}
pobear scrape https://go.dev/doc/ --extract markdown --bare
```

### Scrape a list of URLs

Put one URL per line in a text file. Blank lines and lines starting with `#` are skipped:

```bash theme={null}
pobear scrape urls.txt --extract markdown --out pages.ndjson
```

### Save and rerun a job

Add `--print-spec` to any command to print it as a JSON job spec instead of running it:

```bash theme={null}
pobear crawl https://books.toscrape.com/ --depth 1 --max-pages 10 \
  --css 'title=h3 a@title' --css 'price=.price_color' \
  --out books.ndjson --print-spec > books.json
```

Run the saved job whenever you need fresh data:

```bash theme={null}
pobear run books.json
```

`pobear run` accepts JSON or YAML. Keep job files in version control or run them on a schedule. See the [**job spec reference**](https://github.com/Decodo/pobear/blob/main/docs/JOB_SPEC_REFERENCE.md) for the format.

## Choose how to fetch

| Tier | Flag | When to use it |
| - | - | - |
| Direct (`local`) | Default | Pages that load without protection. No account needed |
| Your proxy (`proxy`) | `--proxy '<proxy-url>'` | Requests that need to come from another IP |
| Decodo Web Data API (`provider`) | `--provider decodo` | Pages behind anti-bot protection, CAPTCHAs, or that need JavaScript rendering |

Every record has a `tier` field showing which method fetched it. Direct and proxy requests can mimic Chrome, Firefox or Safari TLS fingerprints with `--profile` (default `chrome`), but they don't run JavaScript.

## Use the Web Data API

1. Copy your Web Data API key from the [**Web Data API playground**](https://dashboard.decodo.com/web-data/playground). Any [**Web Data API plan**](https://dashboard.decodo.com/web-data/pricing) works, and free trials are available on the [**dashboard**](https://dashboard.decodo.com/).
2. Save the key. pobear prompts for it and stores it for later runs:

   ```bash theme={null}
   pobear auth decodo
   ```

   In CI or a container, set the `POBEAR_PROVIDER_DECODO_TOKEN` environment variable instead.
3. Add `--provider decodo` to any `scrape` or `crawl` command:

   ```bash theme={null}
   pobear scrape https://books.toscrape.com/ \
     --css 'title=h3 a@title' --css 'price=.price_color' \
     --provider decodo --out books.ndjson
   ```

The request goes through the Web Data API `universal` target with the premium proxy pool. JavaScript rendering is **off** by default. Add `--render` for pages that build their content in the browser:

```bash theme={null}
pobear scrape https://quotes.toscrape.com/js/ \
  --css 'quote=.quote .text' --provider decodo --render
```

Without `--render`, that page returns no quotes. With it, all 10 are extracted.

Use `--geo` and `--opt name=value` to set Web Data API parameters, such as `--geo "United States"` or `--opt device_type=mobile`. See [**localisation**](https://help.decodo.com/docs/web-data-api-scrape-localisation.md) for accepted `geo` values.

> [!NOTE]
>
> Web Data API requests are billed to your Decodo plan. pobear itself is free.

### Use a target template

`--template` switches from `universal` to a dedicated Web Data API target. Amazon URLs need the `amazon` template – the `universal` target rejects them with HTTP `400`:

```bash theme={null}
pobear scrape 'https://www.amazon.com/dp/B09H74FXNW' \
  --provider decodo --template amazon --geo 10001 --opt parse=true
```

With `parse=true`, the `body` field contains parsed JSON (title, price, brand, ratings and more) instead of HTML.

Available templates: `universal` (default), `airbnb`, `amazon`, `apple_app_store`, `autotrader`, `bbb`, `bing`, `google`, `mobile`, `target`, `tiktok`, `tiktok_post` and `walmart`. To see each template's options:

```bash theme={null}
pobear providers decodo
```

A template applies to the whole job, so run Amazon URLs and other sites as separate jobs.

> [!WARNING]
>
> * **Amazon geo:** when the country matches the Amazon domain, for example `US` on amazon.com, use a ZIP code such as `10001`. A country code returns HTTP `400`. [Read more](https://help.decodo.com/docs/web-scraping-api-amazon-geo-parameter.md)
> * **Google geo:** the `google` template doesn't accept `--geo`, so results can come from any country. For location-specific Google results, call the Web Data API [**`google_search`**](https://help.decodo.com/docs/web-scraping-api-google-search) target directly.

## Escalate automatically when blocked

`--ladder` starts with a direct request and moves to the next tier only when a request is blocked, rate-limited, times out, or returns a known anti-bot challenge. Pass `--provider decodo` with it – on its own, `--ladder` has nowhere to escalate to:

```bash theme={null}
pobear scrape 'https://www.amazon.com/dp/B09H74FXNW' \
  --ladder --provider decodo --template amazon --geo 10001 \
  --css 'title=#productTitle' --css 'price=.a-price .a-offscreen'
```

Add `--proxy '<proxy-url>'` as well to try your proxy between the direct request and the Web Data API. Pages that load directly never use the API, so you only pay for the ones that need it.

> [!TIP]
>
> Each tier retries 3 times with a 30-second timeout before pobear escalates. For sites you know are protected, skip the ladder and use `--provider decodo` directly. Tune the wait with `--retries` and `--timeout`.

## Use pobear from AI agents and scripts

pobear works well as a subprocess for an agent or a script:

* `pobear docs --json` returns the commands, output format and exit codes as JSON.
* `pobear providers decodo --json` returns every template, option and default.
* Data goes to stdout as NDJSON. Run summaries and errors go to stderr.

Pipe a job spec straight into the runner:

```bash theme={null}
pobear scrape https://books.toscrape.com/ --css 'price=.price_color' \
  --print-spec | pobear run -
```

Check the exit code before using the results:

| Exit code | Meaning |
| - | - |
| `0` | Success |
| `1` | Missing or rejected provider credentials |
| `3` | Invalid command, flag or job spec |
| `4` | Partial failure – some results were produced |
| `5` | Fatal failure – nothing usable was produced |

## Troubleshooting

**`empty 1` in the summary.** The page was fetched with HTTP `200`, but your selectors matched nothing. Check the selectors and the page content first. An empty result can also mean the page needs `--render`, or that the site served a challenge page.

**Catching empty extractions in pipelines.** `--fail-on-empty` exits with code `5` only when *every* fetched page is empty. A run where some pages extract data and others don't still exits `0`, so check the `empty` count in the summary as well.

**`401` with `Missing authorization header`.** No Web Data API key is saved. Run `pobear auth decodo` or set `POBEAR_PROVIDER_DECODO_TOKEN`.

**`400` with `Validation failed`.** The Web Data API accepted your key but rejected the request. The most common cause is an Amazon URL without `--template amazon`.

## FAQ

### How do I scrape a website from the terminal?

Install pobear, then run `pobear scrape` with the URL and the fields you want:

```bash theme={null}
pobear scrape https://books.toscrape.com/ --css 'price=.price_color'
```

It prints one NDJSON record with the extracted prices. Use `pobear crawl` to follow links across a site, or run `pobear` with no arguments to open the interactive terminal wizard.

### Do I need a Decodo account to use pobear?

No. Direct requests work without any account. You only need a Web Data API key or proxy credentials when you add `--provider decodo` or `--proxy`.

### Can pobear scrape JavaScript-rendered websites?

Direct and proxy requests don't run JavaScript. Use `--provider decodo --render` to get the page after the browser has rendered it. See [**Use the Web Data API**](#use-the-web-data-api).

### How is pobear different from the Decodo CLI?

They start from opposite defaults. The [**Decodo CLI**](https://help.decodo.com/docs/cli.md) is a Node.js client for the Web Data API: every request goes through Decodo, and every Web Data API target – search engines, eCommerce, social media – is a subcommand. pobear is a standalone Go binary that fetches directly by default and adds a proxy or the Web Data API only when you ask for it, or when `--ladder` hits a block.

Use the Decodo CLI when you want the full Web Data API with ready-made targets. Use pobear when you want one binary that works without an account and pays for the API only on the pages that need it.

## Related

* [**Decodo CLI**](https://help.decodo.com/docs/cli.md) – the official Web Data API command line tool, with search, map, crawl and screenshots
* [**Web Data API introduction**](https://help.decodo.com/docs/web-data-api-introduction.md)
* [**pobear CLI reference**](https://github.com/Decodo/pobear/blob/main/docs/CLI_REFERENCE.md)

***

> **[Support](https://direct.lc.chat/12092754)**
>
> Need help or just want to say hello? Our support is available 24/7. \
> You can also reach us anytime via email at [support@decodo.com](mailto:support@decodo.com).

> **[Feedback](mailto:feedback@decodo.com)**
>
> Can't find what you're looking for? Request an article! \
> Have feedback? Share your thoughts on how we can improve.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.