Configuration

Learn how to use Thordata’s Universal Scraping API to fetch your first piece of scraped content.

Navigate to the Universal Scraping Configuration section.

1. Scraping Configuration

You can configure what you want to scrape using the parameter options on the right side of the page. These include: URL, request method, output format, JS rendering, block resource loading, and proxy.

Click here to learn more about Unlocker parameters.

2. Integrate the Example into Your Codebase

Once all parameters are configured, you can integrate the generated code snippet on the right into your codebase.

Take data crawling from https://www.google.com/ as an example. Please don’t forget to replace “token” with your API Token.

curl --request POST --url 
https://universalapi.thordata.com/request --header 
"Authorization: Bearer xxxxxxxxxxxxxxxxxxxxxx"   --header 
"content-type: application/json" --data "
{"url":"https://www.google.com", 
"type":"html","js_render":"True"}"

Need more help? Contact us via email: [e-mail]