> ## Documentation Index > Fetch the complete documentation index at: https://help.decodo.com/llms.txt > Use this file to discover all available pages before exploring further. # Scraping & Crawling Strategies ## Before you start... Check what your target has to offer. Many websites will have a Public API available to prevent getting hit by thousands of different scrapers. Not only will it save you time, but the functionalities that APIs now provide will let you get clearer data with low maintenance. ## Expect the unexpected As scraping & crawling are getting more and more popular, website owners tend to tighten their web security to prevent sites from going down due to the number of incoming requests. Ensure that you investigate how your target handles security, as this will be one of the biggest fallbacks if something goes wrong after your scraper/crawler is already in business. ## Work with robots.txt It's important to know what your target allows you to crawl since it can potentially show you where you will and will not meet security roadblocks. It can also save you tons of time by showing where the exact information can be found, as these files are used for SEO. Each website should have this file available, which will be mostly found in a format like `yourwebtarget.com/robots.txt` such as `https://decodo.com/robots.txt`. ## Look for traps The easiest way to detect a scraper or a crawler browsing through your website is by showing a link that cannot be seen by any other user on page load. It can only be checked by looking through the HTML code of the website. Ensure you inspect your target using built-in tools from Chrome/Firefox. You can simply hit **F12** to open **Developer Tools**. In most cases, these links are going to be hidden with an additional CSS code. ## Have your connection look human-like Each website tracks what requests they get, some taking extreme security measures and tracking the whole fingerprint of the request. When sending requests from scrapers or crawlers, make sure that you include a user agent and, if needed, send all of the required cookies. In other cases, you may need to follow a certain path for the requests to go through, since asking for some links directly may give a clear indication that the request is not genuine. ## Be responsible for the request amounts It's important to understand that sending a target request adds to its current load. Sending too many rapid requests will slow down your process and result in the website becoming unavailable for a longer period of time. Being smart about the number of your requests will help you achieve quality results faster and decrease the chance of the web owner investigating incoming requests and tightening web security, resulting in your scraper needing many additional changes. *** Need help or just want to say hello? Our support is available 24/7. \ You can also reach us anytime via email at [support@decodo.com](mailto:support@decodo.com). Can't find what you're looking for? Request an article! \ Have feedback? Share your thoughts on how we can improve.