Real-time Requests

Web Scraping API Real-time Requests

Introduction

The real-time integration puts the entire job request into one URL, which is formed by the API.

Post Endpoint:

scraper-api.decodo.com/v2/scrape

Examples
https://scraper-api.decodo.com/v2/scrape?target=google_search&query=world&domain=com&access_token=pass2021
curl -u username:password 'https://scraper-api.decodo.com/v2/scrape' -H "Content-Type: application/json" -d '{"target": "google_search", "domain": "com", "query": "world"}'
<?php 

$username = "username";
$password = "password";

$search = [
    'target' => 'google_search',
    'domain' => 'com',
    'query' => 'world',
    'parse' => true
];

$ch = curl_init();

$headers[] = 'Content-Type: application/json';

$options = [
   CURLOPT_URL => 'https://scraper-api.decodo.com/v2/scrape',
   CURLOPT_USERPWD => sprintf('%s:%s', $username, $password),
   CURLOPT_POSTFIELDS => json_encode($search),
   CURLOPT_RETURNTRANSFER => 1,
   CURLOPT_ENCODING => 'gzip, deflate',
   CURLOPT_HTTPHEADER => $headers,
   CURLOPT_SSL_VERIFYPEER => false,
   CURLOPT_SSL_VERIFYHOST => false
];
curl_setopt_array($ch, $options);

$result = curl_exec($ch);
if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close($ch);

$result = json_decode($result);
var_dump($result);

 ?>
import requests

headers = {
    'Content-Type': 'application/json'
}

task_params = {
    'target': 'google_search',
    'domain': 'com',
    'query': 'world',
    'parse': True
}

username = 'userame'
password = 'password'
  
response = requests.post(
    'https://scraper-api.decodo.com/v2/scrape',
    headers = headers,
    json = task_params,
    auth = (username, password)
)
print(response.text)

Quick Start

  1. Send a query. To specify it, you can add parameters.
    • You need to post query parameters in the same way you post JSON ones.
    • Don't forget to input your credentials (tokens). How to find it?
  2. The Web Scraping API retrieves the content you need.
  3. The data should come back with the HTTP status code 200, and it should be parsed in JSON format or contain raw HTML.

📘

Keep an Open Connection

  • If the connection is closed before the job is completed, the data is lost.
  • The timeout limit for open connections is 150 seconds. In a rare case of a heavy load, we may not be able to get the data to you.
  • You can collect data using the same connection and get an immediate response.

Support

Still can't find an answer? Want to say hi? We take pride in our 24/7 customer support. Alternatively, you can reach us via our support email at [email protected].

Feedback

Something's missing? Request an article!
Got feedback? Let us know how we're doing and what can be improved.


```