> For the complete documentation index, see [llms.txt](https://doc.thordata.com/doc/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://doc.thordata.com/doc/scraping/web-scraper-api/parameter-description/github-scraping-parameters.md).

# Github Scraping Parameters

Web Scraper API Github Scraping Parameters

Configure Github Scraping Parameters using Thordata's Web Scraper API, including repository URL, search URL, code URL and other parameters.

Unique Identifier: 

<details>

<summary><code>token</code>，<strong>Access token(required)</strong></summary>

This parameter is used as the API access token to ensure the legitimacy of the scraping request.

**Request examples:**&#x20;

`Authorization: Bearer ********************`

```sh
curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer ********************" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=github.com" ^
  -d "spider_id=github_repository_by-url" ^
  -d "spider_parameters=[{\"url\": \"https://github.com/TheAlgorithms/Python/blob/master/divide_and_conquer/power.py\"},{\"url\": \"https://github.com/AkarshSatija/msSync/blob/master/index.js\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

```

</details>

**Product - Scrape repository information:**

1\. Github - Scrape repository information by repository URL

<details>

<summary><code>spider_id</code> ，<strong>Scraper tool (required)</strong></summary>

Defines which scraper tool to use.

**Request examples:**&#x20;

`spider_id=github_repository_by-repo-url`

```sh
curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer Token-ID" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=github.com" ^
  -d "spider_id=github_repository_by-repo-url" ^
  -d "spider_parameters=[{\"repo_url\": \"https://github.com/TheAlgorithms/Python\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

```

</details>

<details>

<summary><code>repo_url</code>，<strong>Repository URL (required)</strong></summary>

This parameter specifies the repository URL to be scraped.

**Request examples:**&#x20;

`"repo_url": "https://github.com/TheAlgorithms/Python"`

```sh
curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer Token-ID" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=github.com" ^
  -d "spider_id=github_repository_by-repo-url" ^
  -d "spider_parameters=[{\"repo_url\": \"https://github.com/TheAlgorithms/Python\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"
```

</details>

2.Github - Scrape repository information by search URL

<details>

<summary><code>spider_id</code> ，<strong>Scraper tool (required)</strong></summary>

Defines which scraper tool to use.

**Request examples:**&#x20;

`spider_id=github_repository_by-search-url`

```sh
curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer Token-ID" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=github.com" ^
  -d "spider_id=github_repository_by-search-url" ^
  -d "spider_parameters=[{\"search_url\": \"https://github.com/search?q=ML%26type=repositories\",\"page_turning\": \"\",\"max_num\": \"1\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

```

</details>

<details>

<summary><code>search_url</code> ，<strong>Search URL (required)</strong></summary>

This parameter specifies the Search URL to be scraped.

**Request examples:**&#x20;

`"search_url": "https://github.com/search?q=ML%26type=repositories"`

```sh
curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer Token-ID" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=github.com" ^
  -d "spider_id=github_repository_by-search-url" ^
  -d "spider_parameters=[{\"search_url\": \"https://github.com/search?q=ML%26type=repositories\",\"page_turning\": \"\",\"max_num\": \"1\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

```

</details>

<details>

<summary><code>page_turning</code> ，<strong>Page Turning (optional)</strong></summary>

This parameter specifies the limit on the number of crawled results , please enter the number of pages.

**Request examples:**&#x20;

`"page_turning": "1"`

```sh
curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer Token-ID" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=github.com" ^
  -d "spider_id=github_repository_by-search-url" ^
  -d "spider_parameters=[{\"search_url\": \"https://github.com/search?q=ML%26type=repositories\",\"page_turning\": \"1\",\"max_num\": \"1\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

```

</details>

<details>

<summary><code>max_num</code> ，<strong>Maximum number (optional)</strong></summary>

This parameter specifies the maximum number of warehouses to crawl.

**Request examples:**&#x20;

`"max_num": "1"`

```sh
curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer Token-ID" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=github.com" ^
  -d "spider_id=github_repository_by-search-url" ^
  -d "spider_parameters=[{\"search_url\": \"https://github.com/search?q=ML%26type=repositories\",\"page_turning\": \"1\",\"max_num\": \"1\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

```

</details>

3.Github - Scrape repository information by URL

<details>

<summary><code>spider_id</code> ，<strong>Scraper tool (required)</strong></summary>

Defines which scraper tool to use.

**Request examples:**&#x20;

`spider_id=github_repository_by-url`

```sh
curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer Token-ID" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=github.com" ^
  -d "spider_id=github_repository_by-url" ^
  -d "spider_parameters=[{\"url\": \"https://github.com/TheAlgorithms/Python/blob/master/divide_and_conquer/power.py\"},{\"url\": \"https://github.com/AkarshSatija/msSync/blob/master/index.js\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

```

</details>

<details>

<summary><code>url</code>，<strong>Code URL (required)</strong></summary>

This parameter specifies the code URL to search.

**Request examples:**&#x20;

`"url": "https://github.com/TheAlgorithms/Python/blob/master/divide_and_conquer/power.py"`

```sh
curl -X POST "https://scraperapi.thordata.com/builder" ^
  -H "Authorization: Bearer Token-ID" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -d "spider_name=github.com" ^
  -d "spider_id=github_repository_by-url" ^
  -d "spider_parameters=[{\"url\": \"https://github.com/TheAlgorithms/Python/blob/master/divide_and_conquer/power.py\"}]" ^
  -d "spider_errors=true" ^
  -d "file_name={{TasksID}}"

```

</details>

If you need further assistance, please contact us via email at <support@thordata.com>.