How to scrape a cloudflare protected website
Getting past a cloudflare protected website with your scraper can be a daunting one in data engineering. Below are 3 easy steps to get past the security that redirects your scraper to a dead page;
Step 1
pip install cfscrape
Step 2
Use cfscrape in place of requests, as cfscrape is built on top of the requests library. Remember to create your headers variable with your 'user-agent' as input.
scraper = cfscrape.create_scraper()
source = scraper.get(url, headers=headers).text
soup = bs(source, 'lxml')
Step 3
Now you may run your soup object like you will normally do when scraping.
Happy scraping!
If you need assistance with your projects feel free to email me at info@airgad.com or whatsapp Jesse stay safe!