Webhook Integration

Through the Webhook integration feature, you can automatically send notifications to a specified system when a specific event occurs in a Thordata Web Scraper task. This allows you to monitor task status in real time or push data to other services. Webhooks support dynamic data injection to facilitate further automation or business linkage.

Integration Configuration:

1. Integration Name Used to identify and manage the Webhook configuration. It is recommended to give a clear and descriptive name for easy maintenance and reference.

2. Event Types You can configure notifications for the following task events:

• Running: The task is in progress

• Task Succeeded: The scraping task completed successfully and returned results

• Task Failed: The scraping task did not return data or encountered an error

1. Notification Method When a selected event occurs, the system will send an HTTP POST request to your configured URL. The request body will be in JSON format and can be customized with dynamic variables.

① Payload Format: The payload template must be a valid JSON string, and variables can be referenced using double curly braces {{ }}. Example:

{
  "userId": "{{userId}}",
  "eventType": "{{eventType}}",
  "createdAt": "{{createdAt}}",
  "eventData": "{{eventData}}",
  "resource": "{{resource}}"
}

② Available Variables:

Variable	Type	Description
userId	string	Thordata user ID triggering the event
createdAt	string	Webhook trigger time (ISO format)
eventType	string	Type of triggered event
eventData	Object	Contextual data related to the event
resource	Object	Detailed event resource info (e.g. links)
globals	Object	Current event date (ISO 8601 format)

③ Example:

{
  "userId": 3406,
  "userName": "Thor84******",
  "createdAt": "2025-06-07T06:33:14Z",
  "eventType": "ACTOR.RUN.SUCCEEDED",
  "eventData": {
    "prodect_id": "3",
    "apiRunId": "2e9ddee0582a********************"
  },
  "resource": {
    "taskId": "2e9ddee0582a********************",
    "url": "{\"spider_errors\":true,\"spider_id\":\"3\",\"spider_parameters\":[{\"url\":\"https://www.amazon.com/dp/B0DPMHRZ1W?th=1\\u0026psc=1\"}]}",
    "apiKey": "289f5c53e47d616********************",
    "apiUrl": "",
    "apiCode": 400,
    "apiResponseTime": 28000,
    "apiErrorMsg": "",
    "apiResult": "",
    "errorNumber": 1,
    "successRate": 0,
    "runseconds": 12340,
    "detail": "Completed! 26 requests in total: 26 successful, 0 failed",
    "flow": 1765,
    "fileSize": 1765,
    "type": 0,
    "csvUrl": "https://th-scrapers-1303252866.cos.na-siliconvalley.myqcloud.com/scrapers/thordata/2025/06/07/2e9ddee0582a40a9****************.csv",
    "jsonUrl": "https://th-scrapers-1303252866.cos.na-siliconvalley.myqcloud.com/scrapers/thordata/2025/06/07/2e9ddee0582a40a9****************.json",
    "audioUrl": "",
    "videoUrl": "",
    "subtitleUrl": "",
    "createdAt": "2025-06-07T14:33:14+08:00",
    "finishedAt": "2025-06-07T14:33:44+08:00"
  }
}

Field Mappings:

Description	Variable
User ID	userId
Account ID	userName
Creation Time	createdAt
Status (success/fail)	eventType
Event Context Data	eventData
Product Unique ID	prodect_id
API Run ID	apiRunId
Resource	resource
Task ID	taskId
Scraping Parameters	url
Token	apiKey
API URL	apiUrl
API Response Code	apiCode
Response Time (ms)	apiResponseTime
API Error Message	apiErrorMsg
API Result	apiResult
Error Count	errorNumber
Success Rate	successRate
Run Time (seconds)	runseconds
Task Detail	detail
Data Transfer (MB)	flow
File Size	fileSize
Type	type
CSV Download URL	csvUrl
JSON Download URL	jsonUrl
Audio Download URL	audioUrl
Video Download URL	videoUrl
Subtitle Download URL	subtitleUrl
Task Created At	createdAt
Task Finished At	finishedAt

1. Headers Template (Optional) The headers template is a JSON-like string where you can add additional information to the default headers of the webhook request. Variables can be passed in the same way as in the payload template. The generated headers must be a valid JSON object with string values only.

Note: The following keys are hard-coded and will always be overwritten:

• Host: Request URL

• Content-Type: application/json

• Webhook: Tasks value

• Webhook-Dispatch-Id: Tasks ID

• Request-Origin

For further assistance, please contact us at [email protected].