Query Parameters

Universal Scraping API Query Parameters

Understanding Thordata's Universal Scraping API - Scraping Parameters.

Token: This parameter defines the API token used for scraping. It plays a decisive role in your successful scraping.

Parameter	Name	Function
token	Token	API token for authentication during scraping

URL: This parameter defines the target website URL for scraping, with google.com as default. You can also change it to other search engine links.

Parameter	Name	Function
url	URL	Target website URL for scraping

Example Request:

Example with token:Token ,url:https://www.google.com parameter

curl -X POST https://universalapi.thordata.com/request -H "Authorization: Bearer Token" -H "Content-Type: application/x-www-form-urlencoded" -d "url=https://www.google.com" -d "type=html" -d "js_render=True"

js_render ,JS Rendering(Optional)

JS rendering can handle dynamically loaded content and Single Page Applications (SPA), supporting more interactive and complex pages and rendering requirements. We recommend enabling this.

Example Request:

Example with js_render:True parameter

curl -X POST https://universalapi.thordata.com/request -H "Authorization: Bearer Token" -H "Content-Type: application/x-www-form-urlencoded" -d "url=https://www.google.com" -d "type=html" -d "js_render=True"

type ,Format(Optional)

JS rendering can handle dynamically loaded content and Single Page Applications (SPA), supporting more interactive and complex pages and rendering requirements. We recommend enabling this.

Example Request:

Example with type:png parameter

curl -X POST https://universalapi.thordata.com/request -H "Authorization: Bearer Token" -H "Content-Type: application/x-www-form-urlencoded" -d "url=https://www.google.com" -d "type=png" -d "js_render=True"

header ,Header(Optional)

Once enabled, the output result will include the information from the request headers.

Example Request:

Example with header:Ture parameter

curl -X POST https://universalapi.thordata.com/request \
 -H "Authorization: Bearer token" \
 -H "Content-Type: application/x-www-form-urlencoded" \
 -d "url=https://www.google.com" \
 -d "type=html" \
 -d "js_render=True" \
 -d "header=True"

block_resources,Block(Optional)

This parameter can limit unnecessary resource fetching, thus speeding up the corresponding speed.

Example Request:

Example with block_resources:script parameter

curl -X POST https://universalapi.thordata.com/request -H "Authorization: Bearer Token" -H "Content-Type: application/x-www-form-urlencoded" -d "url=https://www.google.com" -d "type=html" -d "js_render=True" -d "block_resources=script"

country,Proxies(Optional)

This parameter defines which country/region's proxy to use during scraping, with no proxy country/region's as default.

Example Request:

Example with country:al parameter

curl -X POST https://universalapi.thordata.com/request -H "Authorization: Bearer Token" -H "Content-Type: application/x-www-form-urlencoded" -d "url=https://www.google.com" -d "type=html" -d "js_render=True" -d "country=al"

clean_content,Remove(Optional)

This parameter is used to remove JS and CSS content from the returned results.

Example Request:

Example of a parameter with clean_content: JS,CSS

curl -X POST https://universalapi.thordata.com/request \
 -H "Authorization: Bearer token" \
 -H "Content-Type: application/x-www-form-urlencoded" \
 -d "url=https://www.google.com" \
 -d "type=html" \
 -d "js_render=True" \
 -d "clean_content=js,css" \
 -d "header=False"

waitfor,Wait for (in ms)(Optional)

This parameter is intended to wait for the number of milliseconds required for the page to load content. (Suitable for slower websites)

Example Request:

Example of a parameter with waitfor:1000

curl -X POST https://universalapi.thordata.com/request \
 -H "Authorization: Bearer token" \
 -H "Content-Type: application/x-www-form-urlencoded" \
 -d "url=https://www.google.com" \
 -d "type=html" \
 -d "js_render=True" \
 -d "header=False" \
 -d "waitfor=1000"

headers,Custom Headers(Optional)

Pass custom request headers to the target website. Use English commas ‘,’ to separate multiple headers. HTTP headers are key-value pairs separated by a colon (:). The parameters sent should be in JSON format.

Example Request:

If you want to send User-Agent and Content-Type, it should look like this:

curl -X POST https://universalapi.thordata.com/request \
 -H "Authorization: Bearer token" \
 -H "Content-Type: application/x-www-form-urlencoded" \
 -d "url=https://www.google.com" \
 -d "type=html" \
 -d "js_render=True" \
 -d "header=False" \
 -d "headers="User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36", "Content-Type": "application/json""

cookies,Cookies(Optional)

Pass custom cookies to the target website. Use semicolons ; to separate multiple cookies.

Cookies are small pieces of data stored on a user's device by websites through a web browser. They allow websites to retain user information, such as login status, preferences, or tracking details, improving and personalizing the browsing experience.

When making a request, you can add cookies in two ways:

1. In the headers parameter: Send them as part of the Cookie header in the following format: "Cookie": "cookie_name_1=cookie_value_1; cookie_name_2=cookie_value_2"

2. Using the dedicated cookies parameter: Pass them directly in this format: cookie_name_1=cookie_value_1; cookie_name_2=cookie_value_2

Example Request: Using the dedicated cookies parameter: Pass them directly in this format: cookie_name_1=cookie_value_1; cookie_name_2=cookie_value_2

curl -X POST https://universalapi.thordata.com/request \
 -H "Authorization: Bearer token" \
 -H "Content-Type: application/x-www-form-urlencoded" \
 -d "url=https://www.google.com" \
 -d "type=html" \
 -d "js_render=True" \
 -d "header=False" \
 -d "cookies=cookie_name_1=cookie_value_1; cookie_name_2=cookie_value_2"

For assistance, contact [email protected].