Query Parameters

Universal Scraping API Query Parameters

Understanding Thordata's Universal Scraping API - Scraping Parameters.

Token: This parameter defines the API token used for scraping. It plays a decisive role in your successful scraping.

Parameter	Name	Function
token	Token	API token for authentication during scraping

URL: This parameter defines the target website URL for scraping, with google.com as default. You can also change it to other search engine links.

Parameter	Name	Function
url	URL	Target website URL for scraping

Example Request:

Example with token:Token ,url:https://www.google.com parameter

curl -X POST https://universalapi.thordata.com/request -H "Authorization: Bearer Token" -H "Content-Type: application/x-www-form-urlencoded" -d "url=https://www.google.com" -d "type=html" -d "js_render=True"

js_render ,JS Rendering(Optional)

JS rendering can handle dynamically loaded content and Single Page Applications (SPA), supporting more interactive and complex pages and rendering requirements. We recommend enabling this.

Example Request:

Example with js_render:True parameter

curl -X POST https://universalapi.thordata.com/request -H "Authorization: Bearer Token" -H "Content-Type: application/x-www-form-urlencoded" -d "url=https://www.google.com" -d "type=html" -d "js_render=True"

type ,Format(Optional)

JS rendering can handle dynamically loaded content and Single Page Applications (SPA), supporting more interactive and complex pages and rendering requirements. We recommend enabling this.

Example Request:

Example with type:png parameter

curl -X POST https://universalapi.thordata.com/request -H "Authorization: Bearer Token" -H "Content-Type: application/x-www-form-urlencoded" -d "url=https://www.google.com" -d "type=png" -d "js_render=True"

block_resources,Block(Optional)

This parameter can limit unnecessary resource fetching, thus speeding up the corresponding speed.

Example Request:

Example with block_resources:script parameter

curl -X POST https://universalapi.thordata.com/request -H "Authorization: Bearer Token" -H "Content-Type: application/x-www-form-urlencoded" -d "url=https://www.google.com" -d "type=html" -d "js_render=True" -d "block_resources=script"

country,Country/Region(Optional)

This parameter defines which country/region's proxy to use during scraping, with no proxy country/region's as default.

Example Request:

Example with country:al parameter

curl -X POST https://universalapi.thordata.com/request -H "Authorization: Bearer Token" -H "Content-Type: application/x-www-form-urlencoded" -d "url=https://www.google.com" -d "type=html" -d "js_render=True" -d "country=al"

For assistance, contact [email protected].