Twitter Scraping Parameters

Web Scraper API Twitter Scraping Parameters

Configure Twitter Scraping Parameters using Thordata's Web Scraper API, including URL, username, start date, end date, and other parameters.

Unique Identifier:

token ， Access token (required)

This parameter is used as the API access token to ensure the legitimacy of the scraping request.

Request examples:

Authorization: Bearer ********************

curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer ********************" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=x.com" ^
  -d "spider_id=twitter_profiles_by-url" ^
  -d "spider_parameters=[{\"url\": \"https://x.com/fabrizioromano\",\"max_number_of_posts\": \"10\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

Product - Scrape Twitter Profile information：

1. Twitter - Scrape Twitter profile information by URL

spider_id ，Scraper tool (required)

Defines which scraper tool to use.

Request examples:

spider_id=twitter_profiles_by-url

curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer Token-ID" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=x.com" ^
  -d "spider_id=twitter_profiles_by-url" ^
  -d "spider_parameters=[{\"url\": \"https://x.com/fabrizioromano\",\"max_number_of_posts\": \"10\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

url ，Profile URL (required)

This parameter specifies the Twitter profile URL to be scraped.

Request examples:

"url": "https://x.com/fabrizioromano"

curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer Token-ID" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=x.com" ^
  -d "spider_id=twitter_profiles_by-url" ^
  -d "spider_parameters=[{\"url\": \"https://x.com/fabrizioromano\",\"max_number_of_posts\": \"10\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

max_number_of_posts ，max number of posts（optional）

This parameter specifies the max number of posts to scrape.

Request examples:

"max_number_of_posts": "10"

curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer Token-ID" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=x.com" ^
  -d "spider_id=twitter_profiles_by-url" ^
  -d "spider_parameters=[{\"url\": \"https://x.com/fabrizioromano\",\"max_number_of_posts\": \"10\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

2. Twitter - Scrape Twitter profile information by username

spider_id ，Scraper tool (required)

Defines which scraper tool to use.

Request examples:

spider_id=twitter_profiles_by-username

curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer Token-ID" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=x.com" ^
  -d "spider_id=twitter_profiles_by-username" ^
  -d "spider_parameters=[{\"user_name\": \"elonmusk\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

user_name ，Twitter username（required）

This parameter specifies the profile username of posts to scrape.

Request examples:

"user_name": "elonmusk"

curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer Token-ID" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=x.com" ^
  -d "spider_id=twitter_profiles_by-url" ^
  -d "spider_parameters=[{\"url\": \"https://x.com/fabrizioromano\",\"max_number_of_posts\": \"10\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

Product - Scrape Twitter Post information

1. Twitter - Scrape Twitter post information by url

spider_id ，Scraper tool (required)

Defines which scraper tool to use.

Request examples:

spider_id=twitter_by-posturl_by-url

curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer Token-ID" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=x.com" ^
  -d "spider_id=twitter_by-posturl_by-url" ^
  -d "spider_parameters=[{\"url\": \"https://x.com/FabrizioRomano/status/1683559267524136962\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

url ，URL (required)

This parameter specifies the URL of posts to scrape.

Request examples:

"url": "https://x.com/FabrizioRomano/status/1683559267524136962"

curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer Token-ID" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=x.com" ^
  -d "spider_id=twitter_by-posturl_by-url" ^
  -d "spider_parameters=[{\"url\": \"https://x.com/FabrizioRomano/status/1683559267524136962\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

2. Twitter - Scrape Twitter post information by profile url

spider_id ，Scraper tool (required)

Defines which scraper tool to use.

Request examples:

spider_id=twitter_by-posturl_by-profileurl

curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer Token-ID" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=x.com" ^
  -d "spider_id=twitter_by-posturl_by-profileurl" ^
  -d "spider_parameters=[{\"url\": \"https://x.com/elonmusk\",\"start_date\": \"2023-01-15T16:31:04.000Z\",\"end_date\": \"2024-01-01\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

url ，Profile URL (required)

This parameter specifies the profile url of posts to scrape.

Request examples:

"url": "https://x.com/elonmusk"

curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer Token-ID" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=x.com" ^
  -d "spider_id=twitter_by-posturl_by-profileurl" ^
  -d "spider_parameters=[{\"url\": \"https://x.com/elonmusk\",\"start_date\": \"2023-01-15T16:31:04.000Z\",\"end_date\": \"2024-01-01\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

start_date ，start date（optional）

This parameter specifies the start date of post publication, in the format mm-dd-yyyy, and should be earlier than "end_date".

Request examples:

"start_date": "2023-01-15T16:31:04.000Z"

curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer Token-ID" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=x.com" ^
  -d "spider_id=twitter_by-posturl_by-profileurl" ^
  -d "spider_parameters=[{\"url\": \"https://x.com/elonmusk\",\"start_date\": \"2023-01-15T16:31:04.000Z\",\"end_date\": \"2024-01-01\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

end_date ，end date（optional）

This parameter specifies the end date of post publication, in the format mm-dd-yyyy, and should be later than "start_date".

Request examples:

"end_date": "2024-01-01"

curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer Token-ID" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=x.com" ^
  -d "spider_id=twitter_by-posturl_by-profileurl" ^
  -d "spider_parameters=[{\"url\": \"https://x.com/elonmusk\",\"start_date\": \"2023-01-15T16:31:04.000Z\",\"end_date\": \"2024-01-01\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

If you need further assistance, please contact us via email at [email protected].