Amazon S3 Integration

Amazon S3 Integration Function Description： This function sends the results of a specified Web Scraper API scraping task to Amazon S3.

Integration Function Name: Used to define the name of the integration task. For easier management, it is recommended that you customize the name.

Event Type: You can choose to send data based on a specified task ID, or you can choose to follow a task. Specified Task ID: Send the scraping results based on the task ID. If multiple IDs are needed, use commas (",") to separate them, with a maximum of 10. You can input task IDs for multiple scraping APIs of the same product. Follow Task: Follow the selected task to send the scraping results. After the integration is successfully created, all subsequent scraping results from this tool will be sent unless you turn off or delete the integration task.

Send to Amazon S3 Parameters:

1.Bucket:

Parameter	Name	Description
bucketName	Target Bucket Name	The name of the target Amazon S3 bucket.
targetPath	Target Path	The target location of Amazon S3
fileName	File Name	Used to define the file name, which defaults to the task ID. If you send multiple files after naming, the system will automatically append suffixes like -001, -002, etc. For example: filename-001.json / filename-002.json

2.Authentication Type:

Type	Parameter	Name	Description
Access Key Credentials	awsAccessKey	AWS Access Key	AWS Access Key ID used for authorization of uploads. You can obtain it from AWS Console -> IAM -> Users -> Create User/Select Existing User -> Security Credentials -> Access Keys. It functions similarly to a username.
Access Key Credentials	awsSecretKey	AWS Key	Your AWS Secret Access Key is used to authorize uploads. You can obtain it from AWS Console -> IAM -> Users -> Create User/Select Existing User -> Security Credentials -> Access Keys -> Create Access Key. After creating, the secret access key is shown only once. It functions like a password. Sua Chave de Acesso Secreta da AWS é usada para autorizar uploads. Você pode obtê-la no Console AWS -> IAM -> Usuários -> Criar Usuário/Selecionar Usuário Existente -> Credenciais de Segurança -> Chaves de Acesso -> Criar Chave de Acesso. Após a criação, a chave secreta é exibida apenas uma vez. Funciona como uma senha.

3.File Format:

Parameter	Name	Description
fileFormat	File Format	The results for Amazon products can be sent in either JSON format or CSV format. The results for YouTube products can only be sent in compressed archive format.

4.Compress Files:

Parameter	Name	Description
compressFiles	Compress File	When selecting this option, please ensure your endpoint supports compressed data.

5.Batch Size:

Parameter	Name	Description
batchSize	Batch size	Batch size is used to determine how many records are sent at a time. The minimum value is 1000.

View the Transferred File: If the status of your integration task shows “Success,” you can go to your Amazon S3 account to view the file. Alternatively, you can access it directly via the link below: https://s3.us-east-2.amazonaws.com/downloaddirectory/your-specified-path/filename

Example: If your specified path is path/to, the file name is 123, and the file format is json, then your access link will be: https://s3.us-east-2.amazonaws.com/downloaddirectory/path/to/123.json

If you need further assistance, please contact us via email at [e-mail].