Real-time Requests
Web Scraping API Real-time Requests
Introduction
The real-time integration puts the entire job request into one URL, which is formed by the API.
Post Endpoint:
scraper-api.decodo.com/v2/scrape
Examples
https://scraper-api.decodo.com/v2/scrape?target=google_search&query=world&domain=com&access_token=pass2021
curl -u username:password 'https://scraper-api.decodo.com/v2/scrape' -H "Content-Type: application/json" -d '{"target": "google_search", "domain": "com", "query": "world"}'
<?php
$username = "username";
$password = "password";
$search = [
'target' => 'google_search',
'domain' => 'com',
'query' => 'world',
'parse' => true
];
$ch = curl_init();
$headers[] = 'Content-Type: application/json';
$options = [
CURLOPT_URL => 'https://scraper-api.decodo.com/v2/scrape',
CURLOPT_USERPWD => sprintf('%s:%s', $username, $password),
CURLOPT_POSTFIELDS => json_encode($search),
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_ENCODING => 'gzip, deflate',
CURLOPT_HTTPHEADER => $headers,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_SSL_VERIFYHOST => false
];
curl_setopt_array($ch, $options);
$result = curl_exec($ch);
if (curl_errno($ch)) {
echo 'Error:' . curl_error($ch);
}
curl_close($ch);
$result = json_decode($result);
var_dump($result);
?>
import requests
headers = {
'Content-Type': 'application/json'
}
task_params = {
'target': 'google_search',
'domain': 'com',
'query': 'world',
'parse': True
}
username = 'userame'
password = 'password'
response = requests.post(
'https://scraper-api.decodo.com/v2/scrape',
headers = headers,
json = task_params,
auth = (username, password)
)
print(response.text)
Quick Start
- Send a query. To specify it, you can add parameters.
- You need to post query parameters in the same way you post
JSON
ones. - Don't forget to input your credentials (tokens). How to find it?
- You need to post query parameters in the same way you post
- The Web Scraping API retrieves the content you need.
- The data should come back with the
HTTP
status code200
, and it should be parsed inJSON
format or contain rawHTML
.
Keep an Open Connection
- If the connection is closed before the job is completed, the data is lost.
- The timeout limit for open connections is
150
seconds. In a rare case of a heavy load, we may not be able to get the data to you.- You can collect data using the same connection and get an immediate response.
Support
Still can't find an answer? Want to say hi? We take pride in our 24/7 customer support. Alternatively, you can reach us via our support email at [email protected].
Updated 1 day ago