Skip to main content

​
Using our scraping products

This article includes common steps that may help you when facing issues with our Web Scraping API or Site Unblocker.

​
Unable to scrape the target

We are sometimes not able to scrape the requested target. Some possible reasons may be:
  • The target URL returns 500, which we interpret as a failure.
  • The target can only be reached from a specific country.
  • The target times out (we retry automatically, but we give up after several retries).
  • The target domain DNS name cannot be resolved.
It is not easy to deduce the exact reason for each specific failed request, but common troubleshooting steps include:
  • Confirm you can open the target on a browser window (unless the target is rate-limiting your IP address)
  • Enable JavaScript rendering
  • Change the geolocation (country) of the request
  • Retry the request later (target websites may go down from time to time)

​
Enable or disable JavaScript rendering

If your target does not scrape, try sending the same request with JavaScript rendering enabled or disabled. This feature is added when you include the "headless": "html" parameter to your request.

​
Web Scraping API

curl --request 'POST' \
        --url 'https://scraper-api.decodo.com/v2/scrape' \
        --header 'Accept: application/json' \
        --header 'Authorization: Basic <TOKEN>' \
        --header 'Content-Type: application/json' \
        --data '
    {
      "url": "https://ip.decodo.com",
      "headless": "html"
    }
'

​
Site Unblocker

curl -k -x https://unblock.decodo.com:60000 'https://ip.decodo.com' \
    -H 'X-SU-Headless: html' \
    -U 'Username:Password'

​
Other parameters to look out for

While not having the biggest effect, changing device_type and locale can improve the response rate for your targeted website:
  • device_type by default is set to desktop. Try changing it to mobile.
  • locale should be used in specific situations, as it will change the language of the response, possibly even the content.
Examples of how to add these parameters:

​
Web Scraping API

curl --request 'POST' \
        --url 'https://scraper-api.decodo.com/v2/scrape' \
        --header 'Accept: application/json' \
        --header 'Authorization: Basic <TOKEN>' \
        --header 'Content-Type: application/json' \
        --data '
    {
      "url": "https://ip.decodo.com",
      "headless": "html",
      "locale": "en-us",
      "device_type": "mobile"
    }
'

​
Site Unblocker

curl -k -x https://unblock.decodo.com:60000 'https://ip.decodo.com' \
    -H 'X-SU-Locale: en-us' \
    -H 'X-SU-Device-Type: mobile' \
    -H 'X-SU-Headless: html' \
    -U 'Username:Password'

​
Peak times

Scraper API is most actively used on the hour (12:00, 21:00, etc.) throughout the day. When making a significant number of synchronous requests, consider splitting the requests across minute intervals (12:05, 12:10, etc) to avoid having requests hang at peak times.

Support

Need help or just want to say hello? Our support is available 24/7.
You can also reach us anytime via email at support@decodo.com.

Feedback

Can’t find what you’re looking for? Request an article!
Have feedback? Share your thoughts on how we can improve.