Webhook Integration

Description of Webhook Integration Function

With the Webhook integration function, you can automatically send notifications to a specified system when specific events occur in Thordata Web Scraper crawling tasks, allowing you to monitor task status in real-time or push data to other services. Webhook supports dynamic data injection, facilitating subsequent automated processing or business linkage.

Integration Configuration:

1. Integration Function Name Used to identify and manage this Webhook configuration. It is recommended to customize a clear and understandable name for easy subsequent viewing and maintenance.

2. Event Types You can set up trigger notifications for the following task events:

   · In Progress: The task is being executed

   · Task Success: The crawling task has been successfully completed and results are returned

   · Task Failure: The crawling task returned no data or an error occurred

3. Notification Sending Method When the selected event occurs, the system will send an HTTP POST request to the URL you set. The request body is in JSON format, and its content can be customized with event data dynamically injected through variables.

① Payload Format: The payload template must be a valid JSON string, and variables can be referenced using double curly braces {{ }}. Example:

{
  "userId": "{{userId}}",
  "eventType": "{{eventType}}",
  "createdAt": "{{createdAt}}",
  "eventData": "{{eventData}}",
  "resource": "{{resource}}"
}

② Available Variables:

Variable	Type	Description
userId	string	Thordata user ID triggering the event
createdAt	string	Webhook trigger time (ISO format)
eventType	string	Type of triggered event
eventData	Object	Contextual data related to the event
resource	Object	Detailed event resource info (e.g. links)
globals	Object	Current event date (ISO 8601 format)

③ Example:

{
  "userId": 3406,
  "userName": "Thor84******",
  "createdAt": "2025-06-07T06:33:14Z",
  "eventType": "ACTOR.RUN.SUCCEEDED",
  "eventData": {
    "prodect_id": "3",
    "apiRunId": "2e9ddee0582a********************"
  },
  "resource": {
    "taskId": "2e9ddee0582a********************",
    "url": "{\"spider_errors\":true,\"spider_id\":\"3\",\"spider_parameters\":[{\"url\":\"https://www.amazon.com/dp/B0DPMHRZ1W?th=1\\u0026psc=1\"}]}",
    "apiKey": "289f5c53e47d616********************",
    "apiUrl": "",
    "apiCode": 400,
    "apiResponseTime": 28000,
    "apiErrorMsg": "",
    "apiResult": "",
    "errorNumber": 1,
    "successRate": 0,
    "runseconds": 12340,
    "detail": "Completed! 26 requests in total: 26 successful, 0 failed",
    "flow": 1765,
    "fileSize": 1765,
    "type": 0,
    "csvUrl": "https://th-scrapers-1303252866.cos.na-siliconvalley.myqcloud.com/scrapers/thordata/2025/06/07/2e9ddee0582a40a9****************.csv",
    "jsonUrl": "https://th-scrapers-1303252866.cos.na-siliconvalley.myqcloud.com/scrapers/thordata/2025/06/07/2e9ddee0582a40a9****************.json",
    "audioUrl": "",
    "videoUrl": "",
    "subtitleUrl": "",
    "createdAt": "2025-06-07T14:33:14+08:00",
    "finishedAt": "2025-06-07T14:33:44+08:00"
  }
}

Field Mappings:

Description	Variable
User ID	userId
Account ID	userName
Creation Time	createdAt
Status (success/fail)	eventType
Event Context Data	eventData
Product Unique ID	prodect_id
API Run ID	apiRunId
Resource	resource
Task ID	taskId
Scraping Parameters	url
Token	apiKey
API URL	apiUrl
API Response Code	apiCode
Response Time (ms)	apiResponseTime
API Error Message	apiErrorMsg
API Result	apiResult
Error Count	errorNumber
Success Rate	successRate
Run Time (seconds)	runseconds
Task Detail	detail
Data Transfer (MB)	flow
File Size	fileSize
Type	type
CSV Download URL	csvUrl
JSON Download URL	jsonUrl
Audio Download URL	audioUrl
Video Download URL	videoUrl
Subtitle Download URL	subtitleUrl
Task Created At	createdAt
Task Finished At	finishedAt

1. Headers Template (Optional) The headers template is a JSON-like string where you can add additional information to the default headers of webhook requests. You can pass variables in the same way as in the payload template (including using string interpolation and available variables). The generated headers must be a valid JSON object, and values can only be strings.

Please note that the following keys are hard-coded and will always be overwritten:

• Host: Request URL

• Content-Type: application/json

• Webhook: Tasks value

• Webhook-Dispatch-Id: Tasks ID

• Request-Origin

For further assistance, please contact us at [email protected].