Go SDK - Introduction

View the Go SDK GitHub repository for the latest installation instructions, examples, and source code.

Build typed scraping workflows for search engines, eCommerce platforms, social media, AI tools, and more with the official Decodo Go SDK.

Typed, target-specific request parameters with editor autocomplete
Sync, async, and batch scraping methods
Web search, site mapping, and crawling through the Web Data API
Runtime validation against the current Decodo target schema
Typed error set for safer integrations
One dependency beyond the standard library; requires Go 1.21 or later

What is the Decodo Go SDK?

The Decodo Go SDK provides a typed interface for interacting with Decodo targets like Google, Amazon, TikTok, YouTube, ChatGPT, Perplexity, and more. Instead of assembling HTTP requests and validating payloads by hand, you work with typed constructors and target-specific parameter structs directly in your editor.

Why use the SDK?

Typing and autocomplete – each target has its own parameter struct, so your editor knows which fields it accepts.
Unified scraping interface – work with search engines, eCommerce platforms, social media, and AI tools through one SDK.
Async and batch workflows – create scraping tasks, poll task status, and process batches at scale.
Typed errors – handle authentication, validation, timeout, and rate-limit failures explicitly.
Generated from the schema – target constants and parameter structs are generated from the Decodo schema, which is also used for runtime validation.
Minimal setup – one dependency for schema validation, and the standard library for everything else.

Requirements

Go 1.21 or later. Run go version to check, or install Go.

Installation

go get github.com/decodo/sdk-go

Quick start

Create a new project:

mkdir scraper
cd scraper
go mod init scraper
go get github.com/decodo/sdk-go
touch main.go

Copy your Web Data API key from your Web Data API subscription on the Decodo dashboard. Keep it out of your source by exporting it:

export DECODO_API_KEY=your_api_key

Older plans only have a basic authentication token. Set Token instead of APIKey – see Configuration.

Every target has a constructor that returns its parameter struct. Create the one you need, set its fields, and pass it to Scrape:

// main.go
package main

import (
	"context"
	"fmt"
	"log"
	"os"

	"github.com/decodo/sdk-go"
)

func main() {
	client := decodo.NewClient(decodo.Config{
		WebScrapingAPI: &decodo.WebScrapingAPIConfig{
			APIKey: os.Getenv("DECODO_API_KEY"),
		},
	})

	params := decodo.NewGoogleSearchParams()
	params.Query = decodo.Ptr("coffee shops")
	params.Geo = decodo.Ptr("United States")
	params.Parse = decodo.Ptr(true)

	result, err := client.WebScrapingAPI.Scrape(context.Background(), params)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(result.Results[0].Content)
}

Run the program:

go run main.go

Each constructor pins its own target and returns a struct that carries only the fields that target supports, so you never pass a target name or assemble a generic parameter map. Your editor flags misspelled or unsupported fields, and the SDK validates values against the target schema before sending the request.

Setting optional parameters

Optional parameters are pointer fields, so an unset field is distinguishable from a zero value. Use decodo.Ptr to set them inline:

params.Query = decodo.Ptr("shoes")
params.Parse = decodo.Ptr(true)
params.PageFrom = decodo.Ptr(1)

Configuration

client := decodo.NewClient(decodo.Config{
	WebScrapingAPI: &decodo.WebScrapingAPIConfig{
		APIKey:            "<api_key>",
		IntegrationHeader: "my-app",
	},
	TimeoutMs: 120_000, // optional, request timeout in ms (default: 180000)
})

Set either APIKey or Token, never both:

// Basic authentication token (older plans)
client := decodo.NewClient(decodo.Config{
	WebScrapingAPI: &decodo.WebScrapingAPIConfig{
		Token: "<basic_auth_token>",
	},
})

Parameter	Description
`WebScrapingAPI.APIKey`	Web Data API key from your subscription on the dashboard. Recommended
`WebScrapingAPI.Token`	Basic authentication token for older plans - the base64-encoded `user:password` string. Use it instead of `APIKey`
`WebScrapingAPI.IntegrationHeader`	Identifies the calling integration in requests (default: `sdk-go`)
`TimeoutMs`	Request timeout in milliseconds (default: 180000)
`Schema`	Target schema used for request validation (default: loaded automatically)

Setting both returns a *decodo.ConfigurationError: WebScrapingAPI accepts either Token or APIKey, not both. The credential also decides which API the SDK calls: APIKey uses the Web Data API at data.decodo.com, which is the only way to reach Search, Map, and Crawl. Token uses the older Web Scraping API at scraper-api.decodo.com, which supports scraping only.

Scraping

Access the API via client.WebScrapingAPI.

Method	Returns
`Scrape(ctx, params)`	Returns the scraping result after the request completes.
`ScrapeAsync(ctx, params)`	Returns a scraping task immediately.
`GetStatus(ctx, taskID)`	The current status of a task, as a `TaskStatus` constant
`GetResults(ctx, taskID)`	The results of a task, or `nil` if not ready
`ScrapeBatch(ctx, params)`	A batch task containing one query per input
`Search(ctx, params)`	Parsed web search results. Requires `APIKey`
`Map(ctx, params)`	The links discovered on a site. Requires `APIKey`
`Crawl(ctx, params, opts...)`	Every page of a crawl, once it settles. Requires `APIKey`
`StartCrawl(ctx, params)`	A crawl ID, returned immediately. Requires `APIKey`
`GetCrawlStatus(ctx, crawlID)`	The progress and status of a crawl. Requires `APIKey`
`GetCrawlResults(ctx, crawlID, opts...)`	The pages of a crawl. Requires `APIKey`
`CancelCrawl(ctx, crawlID)`	Stops a running crawl. Requires `APIKey`

Sync scrape

Waits for the scraping result before returning:

params := decodo.NewAmazonProductParams()
params.Query = decodo.Ptr("B09H74FXNW")
params.Parse = decodo.Ptr(true)

result, err := client.WebScrapingAPI.Scrape(context.Background(), params)
if err != nil {
	log.Fatal(err)
}
fmt.Println(result.Results[0].Content)

Async scrape

Creates a scraping task and returns immediately. You then poll for its status and results:

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
defer cancel()

params := decodo.NewGoogleSearchParams()
params.Query = decodo.Ptr("laptop reviews")
params.Parse = decodo.Ptr(true)

task, err := client.WebScrapingAPI.ScrapeAsync(ctx, params)
if err != nil {
	log.Fatal(err)
}

for {
	status, err := client.WebScrapingAPI.GetStatus(ctx, task.ID)
	if err != nil {
		log.Fatal(err)
	}
	if status.Status == decodo.TaskStatusDone {
		break
	}
	if status.Status == decodo.TaskStatusFaulted {
		log.Fatalf("Scraping task %s failed.", task.ID)
	}
	time.Sleep(2 * time.Second)
}

results, err := client.WebScrapingAPI.GetResults(ctx, task.ID)
if err != nil {
	log.Fatal(err)
}
fmt.Println(results.Results[0].Content)

GetStatus returns one of decodo.TaskStatusPending, decodo.TaskStatusDone, or decodo.TaskStatusFaulted. GetResults returns nil until results are available.

Batch scrape

Every supported target also has a batch constructor, such as NewGoogleSearchBatchParams(). For batch requests, fields such as URL or Query accept a slice, with each value becoming a separate task:

params := decodo.NewGoogleSearchBatchParams()
params.Query = []string{"coffee", "tea", "juice"}
params.Parse = decodo.Ptr(true)

batch, err := client.WebScrapingAPI.ScrapeBatch(ctx, params)
if err != nil {
	log.Fatal(err)
}

for _, query := range batch.Queries {
	for {
		status, err := client.WebScrapingAPI.GetStatus(ctx, query.ID)
		if err != nil {
			log.Fatal(err)
		}
		if status.Status == decodo.TaskStatusDone {
			break
		}
		if status.Status == decodo.TaskStatusFaulted {
			log.Fatalf("Scraping task %s failed.", query.ID)
		}
		time.Sleep(2 * time.Second)
	}
	result, err := client.WebScrapingAPI.GetResults(ctx, query.ID)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(result.Results[0].Content)
}

A batch runs against a single target, which the constructor pins for you. URL and Query are the only fields that accept multiple values; all other fields use their regular types.ScrapeBatch doesn’t support walmart_product, target_product, tiktok_shop_product, chatgpt, gemini, or perplexity. Use ScrapeAsync for those instead. Batch requests are limited to one per second and skip schema validation because the schema describes single values.

Web Data API: Search, Map, and Crawl

These methods are served by the Web Data API at data.decodo.com and require APIKey. Calling one on a client configured with a basic authentication Token returns a *decodo.ConfigurationError before any request is sent, for example: Search requires an API key. Configure WebScrapingAPIConfig.APIKey instead of Token. Match it with errors.As.

Search

serp, err := client.WebScrapingAPI.Search(ctx, decodo.RunSearchRequest{
	Query: "laptop reviews",
	Geo:   decodo.Ptr("United States"),
})

Map

site, err := client.WebScrapingAPI.Map(ctx, decodo.MapSiteRequest{
	URL:   "https://example.com",
	Limit: decodo.Ptr(100),
})

Crawl

Crawl starts a crawl, polls until it settles, and returns every page:

pages, err := client.WebScrapingAPI.Crawl(ctx,
	decodo.StartCrawlRequest{URL: "https://example.com", Limit: decodo.Ptr(50)},
	decodo.WithPollInterval(5*time.Second),
)

To drive a crawl yourself, start it and poll separately:

job, err := client.WebScrapingAPI.StartCrawl(ctx, decodo.StartCrawlRequest{URL: "https://example.com"})

status, err := client.WebScrapingAPI.GetCrawlStatus(ctx, job.CrawlID)

pages, err := client.WebScrapingAPI.GetCrawlResults(ctx, job.CrawlID, decodo.WithMetadataOnly())

_, err = client.WebScrapingAPI.CancelCrawl(ctx, job.CrawlID)

Option	Effect
`decodo.WithPollInterval(d)`	How often `Crawl` checks the status (default: 5s)
`decodo.WithMetadataOnly()`	Return page metadata without page content

GetCrawlResults follows the API’s pagination internally and returns every page, so there is no separate list-pages call. The TypeScript SDK’s listCrawlPages has no Go equivalent.Crawl wraps its failures, so match them with errors.As rather than a type switch - see Error handling.

Targets and parameter constructors

Every target has a matching constructor. The target name is converted to PascalCase, prefixed with New, and suffixed with Params – for example, google_search → NewGoogleSearchParams(). Batch constructors follow the same pattern, for example NewGoogleSearchBatchParams(). You don’t need to pass a target explicitly because each constructor pins its own target. Each parameter struct also carries a Target field, and GetTarget() returns it as a string. Each target takes one primary input field (URL, Query, ProductID, or Prompt) together with optional configuration such as Parse, Markdown, Geo, or CallbackURL. Available optional fields vary by target, so use editor autocomplete or the parameters documentation.

Search engines

Target	Description	Primary input	Constructor
`google_search`	Google Search results for a query	`Query`	`NewGoogleSearchParams()`
`google_lens`	Google Lens reverse image search	`Query`	`NewGoogleLensParams()`
`google_ads`	Google Ads results for a query	`Query`	`NewGoogleAdsParams()`
`google_travel_hotels`	Google Travel hotel listings	`Query`	`NewGoogleTravelHotelsParams()`
`bing_search`	Bing Search results	`Query`	`NewBingSearchParams()`

eCommerce

Target	Description	Primary input	Constructor
`amazon_product`	Amazon product detail page by ASIN	`Query`	`NewAmazonProductParams()`
`amazon_search`	Amazon search results	`Query`	`NewAmazonSearchParams()`
`amazon_pricing`	Amazon pricing and offers	`Query`	`NewAmazonPricingParams()`
`amazon_sellers`	Amazon seller listings	`Query`	`NewAmazonSellersParams()`
`amazon_bestsellers`	Amazon bestsellers by category	`Query`	`NewAmazonBestsellersParams()`
`walmart_product`	Walmart product page by product ID	`ProductID`	`NewWalmartProductParams()`
`walmart_search`	Walmart search results	`Query`	`NewWalmartSearchParams()`
`target_product`	Target.com product page by product ID	`ProductID`	`NewTargetProductParams()`
`target_search`	Target.com search results	`Query`	`NewTargetSearchParams()`

Target	Description	Primary input	Constructor
`youtube_search`	YouTube search results	`Query`	`NewYoutubeSearchParams()`
`youtube_metadata`	YouTube video metadata by ID	`Query`	`NewYoutubeMetadataParams()`
`youtube_subtitles`	YouTube video subtitles by ID	`Query`	`NewYoutubeSubtitlesParams()`
`youtube_channel`	YouTube channel by handle or ID	`Query`	`NewYoutubeChannelParams()`
`tiktok_post`	TikTok post by URL	`URL`	`NewTiktokPostParams()`
`tiktok_shop_search`	TikTok Shop search results	`Query`	`NewTiktokShopSearchParams()`
`tiktok_shop_product`	TikTok Shop product page by product ID	`ProductID`	`NewTiktokShopProductParams()`

AI tools

Target	Description	Primary input	Constructor
`chatgpt`	ChatGPT response for a prompt	`Prompt`	`NewChatgptParams()`
`perplexity`	Perplexity response for a prompt	`Prompt`	`NewPerplexityParams()`
`gemini`	Gemini response for a prompt	`Prompt`	`NewGeminiParams()`
`google_ai_mode`	Google AI Mode response	`Query`	`NewGoogleAiModeParams()`

Other

Target	Description	Primary input	Constructor
`airbnb`	Airbnb listing by URL	`URL`	`NewAirbnbParams()`
`apple_app_store`	Apple App Store app by URL	`URL`	`NewAppleAppStoreParams()`
`autotrader`	Autotrader listing by URL	`URL`	`NewAutotraderParams()`
`bbb`	Better Business Bureau listing by URL	`URL`	`NewBbbParams()`
`mobile`	Mobile.de listing by URL	`URL`	`NewMobileParams()`

Universal and raw URL scraping

Target	Description	Primary input	Constructor
`universal`	Any URL via the universal scraper	`URL`	`NewUniversalParams()`
`google`	Raw Google URL scraping	`URL`	`NewGoogleParams()`
`bing`	Raw Bing URL scraping	`URL`	`NewBingParams()`
`amazon`	Raw Amazon URL scraping	`URL`	`NewAmazonParams()`
`walmart`	Raw Walmart URL scraping	`URL`	`NewWalmartParams()`
`target`	Raw Target.com URL scraping	`URL`	`NewTargetParams()`
`tiktok`	Raw TikTok URL scraping	`URL`	`NewTiktokParams()`

The Go SDK ships 40 targets. Coverage differs between the Decodo SDKs, so a target available in the TypeScript SDK isn’t necessarily available here - ecommerce, universal_ecommerce, google_maps, google_suggest, google_shopping_search, youtube_video, and youtube_transcript have no constructor in Go. Use universal for any URL.

For the full target list and parameter details, see Target list and Parameters.

Schema validation

Go checks that a field exists and holds the right type at compile time. Values are checked separately, at runtime, against the Decodo target schema. When no Schema is configured, the client uses decodo.SharedDefaultSchema, which loads the latest schema on first use and caches it at ~/.decodo/decodo.ir.json for 24 hours. The load runs once per process, so a failed fetch is not retried: the SDK logs a warning and continues with validation disabled for that session.

schema, err := decodo.LoadRemoteSchema(decodo.RemoteSchemaOptions{
	TTLMs:     3_600_000, // re-fetch after an hour instead of the default 24
	CachePath: "/tmp/decodo.ir.json", // optional, overrides ~/.decodo/decodo.ir.json
})
if err != nil {
	log.Fatal(err)
}

client := decodo.NewClient(decodo.Config{
	WebScrapingAPI: &decodo.WebScrapingAPIConfig{APIKey: os.Getenv("DECODO_API_KEY")},
	Schema:         schema,
})

Error handling

The SDK returns typed errors that map to API error codes:

result, err := client.WebScrapingAPI.Scrape(ctx, params)
if err != nil {
	switch e := err.(type) {
	case *decodo.AuthenticationError:
		log.Fatalf("Invalid API key or token (HTTP %d). Check your credentials in the dashboard.", e.StatusCode)
	case *decodo.RateLimitError:
		time.Sleep(5 * time.Second)
		result, err = client.WebScrapingAPI.Scrape(ctx, params)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Println(result.Results[0].Content)
	case *decodo.ValidationError:
		fmt.Println("Payload rejected:", e.Errors)
	case *decodo.TimeoutError:
		fmt.Println("Timed out. Increase TimeoutMs or switch to ScrapeAsync for slow targets:", e.Msg)
	case *decodo.CancellationError:
		fmt.Println("Cancelled:", e.Msg)
	default:
		fmt.Printf("Request failed: %v\n", err)
	}
}

AuthenticationError, RateLimitError, and ValidationError embed DecodoError and implement Unwrap. As a result, errors.Is and errors.As also work with *decodo.DecodoError.

A type switch only matches the error’s own type. Crawl wraps its failures, so a type switch sends them all to default. Use errors.As for anything that can come back from a crawl:

pages, err := client.WebScrapingAPI.Crawl(ctx, req)
if err != nil {
	var timeout *decodo.CrawlTimeoutError
	var crawlErr *decodo.CrawlError
	switch {
	case errors.As(err, &timeout):
		log.Printf("still running, collect it later: %s", timeout.CrawlID)
	case errors.As(err, &crawlErr):
		log.Printf("crawl %s ended %s: %s", crawlErr.CrawlID, crawlErr.Status, crawlErr.Reason)
	default:
		log.Fatal(err)
	}
}

Error	Returned when	Useful fields
`AuthenticationError`	The API returns `401` or `403`	`StatusCode`, `APIStatus`, `Message`
`RateLimitError`	The API returns `429`	`StatusCode`, `APIStatus`, `Message`
`ValidationError`	The API returns `422`, or `400` with validation errors	`Errors`, plus the `DecodoError` fields
`TimeoutError`	The request exceeds `TimeoutMs`	`Msg`
`CancellationError`	The caller’s context is cancelled	`Msg`
`ConfigurationError`	Credentials are missing, both are set, or a Web Data API method is called with a basic `Token`	`Msg`
`CrawlError`	`Crawl` ends `faulted` or `cancelled`	`CrawlID`, `Status`, `Reason`
`CrawlTimeoutError`	`Crawl` exceeds its context deadline. The crawl keeps running unless `Finished` is set	`CrawlID`, `Finished`
`CrawlRequestError`	A request fails after the crawl started, for example a 5xx while polling. Unwraps to the underlying error	`CrawlID`, `Finished`, `Err`
`CrawlCancellationError`	`Crawl`’s context is cancelled. Unwraps to `*CancellationError`	`CrawlID`, `Finished`
`DecodoError`	Any other unsuccessful response - embedded by the three HTTP errors above	`StatusCode`, `APIStatus`, `Message`

TimeoutError and CancellationError sit outside the DecodoError hierarchy and only carry Msg, so checks against *decodo.DecodoError won’t match them.Schema validation may also be skipped if the target schema can’t be fetched. In that case, invalid values may reach the API and be returned as API errors instead of compile-time or local validation errors.

CLI – Run Decodo scraping workflows from the command line.
TypeScript SDK – Integrate the Decodo APIs into TypeScript and JavaScript projects.
Python SDK – Integrate the Decodo APIs into Python projects.
MCP Server – Connect AI agents and MCP-compatible tools to Decodo.
GitHub repository – View the Go SDK source code, examples, and latest releases.

Support

Need help or just want to say hello? Our support is available 24/7.
You can also reach us anytime via email at support@decodo.com.

Feedback

Can’t find what you’re looking for? Request an article!
Have feedback? Share your thoughts on how we can improve.

​What is the Decodo Go SDK?

​Why use the SDK?

​Requirements

​Installation

​Quick start

​Setting optional parameters

​Configuration

​Scraping

​Sync scrape

​Async scrape

​Batch scrape

​Web Data API: Search, Map, and Crawl

​Search

​Map

​Crawl

​Targets and parameter constructors

​Search engines

​eCommerce

​Social media

​AI tools

​Other

​Universal and raw URL scraping

​Schema validation

​Error handling

​Related

Support

Feedback

What is the Decodo Go SDK?

Why use the SDK?

Requirements

Installation

Quick start

Setting optional parameters

Configuration

Scraping

Sync scrape

Async scrape

Batch scrape

Web Data API: Search, Map, and Crawl

Search

Map

Crawl

Targets and parameter constructors

Search engines

eCommerce

Social media

AI tools

Other

Universal and raw URL scraping

Schema validation

Error handling

Related