配置

如何配置Thordata 的抓取浏览器

本文将引导您完成 Thordata 抓取浏览器的整个配置与使用流程，包括凭证获取、基础配置、示例脚本运行及实时会话管理。遵循本指南，您将能够快速上手并高效地进行网页数据抓取。

在开始之前，请先准备好您的账户凭证，即用于网络自动化工具的用户名和密码。您可以在 Thordata 抓取浏览器区域的 “演示场”标签页中直接查看这些凭证信息。我们假设您已获得有效凭证，若尚未获取，请从 Thordata 处申请。

在使用抓取浏览器之前，需完成基础环境配置。我们将逐步指导您完成身份凭证的配置、API 基本参数设置，以及如何在操作控制台中管理实时浏览器会话，助您更顺畅地启用浏览器功能。

抓取浏览器快速入门示例

我们为您准备了一系列抓取示例，帮助快速入门。您只需替换脚本中的个人凭证和目标网址，即可根据实际业务需求进行调整和扩展。如需编写更复杂的抓取逻辑，可参考 Thordata 官方文档中支持的框架协议说明。

您可以在仪表板中的“演示场”中在线调试脚本，也支持在本地环境中执行实际抓取任务。若选择本地运行，请确保已安装相应依赖（参考 Thordata 支持的框架协议），正确配置身份凭证后，执行示例脚本即可获取目标数据。

import asyncio  
from playwright.async_api import async_playwright  
  
const AUTH = 'PROXY-FULL-ACCOUNT:PASSWORD';  
const SBR_WS_SERVER = `wss://{AUTH}@ws-browser.thordata.com`;  
  
async def run(pw):  
    print('Connecting to Browser API...')  
    browser = await pw.chromium.connect_over_cdp(SBR_WS_SERVER)  
    try:  
        print('Connected! Navigating to Target...')  

        page = await browser.new_page()  
        await page.goto('https://example.com', timeout= 2 * 60 * 1000) 

        # Screenshot
        print('To Screenshot from page')  
        await page.screenshot(path='./remote_screenshot_page.png')  
        # html content
        print('Scraping page content...')  
        html = await page.content()  
        print(html)  
 
    finally:  
        # In order to better use the Browser API, be sure to close the browser 
        await browser.close()  
   
async def main():  
    async with async_playwright() as playwright:  
        await run(playwright)  
  
if _name_ == '_main_':  
 asyncio.run(main())

from selenium.webdriver import Remote, ChromeOptions  
from selenium.webdriver.chromium.remote_connection import ChromiumRemoteConnection  
from selenium.webdriver.common.by import By  

# Enter your credentials - the zone name and password  
AUTH = 'PROXY-FULL-ACCOUNT:PASSWORD'  
REMOTE_WEBDRIVER = f'https://{AUTH}@hs-browser.thordata.com'  

def main():  
    print('Connecting to Browser API...')  
    sbr_connection = ChromiumRemoteConnection(REMOTE_WEBDRIVER, 'goog', 'chrome')  
    with Remote(sbr_connection, options=ChromeOptions()) as driver:  

        # get target URL
        print('Connected! Navigating to target ...')  
        driver.get('https://example.com') 

        # screenshot 
        print('screenshot to png')  
        driver.get_screenshot_as_file('./remote_page.png')  

        # html content
        print('Get page content...')  
        html = driver.page_source  
        print(html)  
  
if __name__ == '__main__':  
   main()

const puppeteer = require('puppeteer-core');  

const AUTH = 'PROXY-FULL-ACCOUNT:PASSWORD';  
const WS_ENDPOINT = `wss://{AUTH}@ws-browser.thordata.com`;  
  
(async () => {
    console.log('Connecting to Scraping Browser...');  
    const browser = await puppeteer.connect({  
        browserWSEndpoint: SBR_WS_ENDPOINT,
        defaultViewport: {width: 1920, height: 1080}  
   });  
    try {  
        console.log('Connected! Navigating to Target URL');  
        const page = await browser.newPage();  
        
        await page.goto('https://example.com', { timeout: 2 * 60 * 1000 });  

        //1.Screenshot
        console.log('Screenshot to page.png');  
        await page.screenshot({ path: 'remote_screenshot.png' }); 
        console.log('Screenshot be saved');  

        //2.Get content
        console.log('Get page content...');  
        const html = await page.content();  
        console.log("source Htmml: ", html)  

    } finally {  
        // In order to better use the Browser API, be sure to close the browser after the script is executed
        await browser.close();  
   }  
})();

const pw = require('playwright');


const AUTH = 'PROXY-FULL-ACCOUNT:PASSWORD';  
const SBR_CDP = `wss://{AUTH}@ws-browser.thordata.com`;  
  
async function main() {  
    console.log('Connecting to Browser API...');  
    const browser = await pw.chromium.connectOverCDP(SBR_CDP);  
    try {  
        console.log('Connected! Navigating to target...');  
        const page = await browser.newPage();
        // Target URL
        await page.goto('https://www.windows.com', { timeout: 2 * 60 * 1000 });  
        // Screenshot
        console.log('To Screenshot from page');  
        await page.screenshot({ path: './remote_screenshot_page.png'});  

        // html content
        console.log('Scraping page content...');  
        const html = await page.content();  
        console.log(html);  
    } finally {  
        // In order to better use the Browser API, be sure to close the browser after the script is executed
        await browser.close();  
   }  
}  
  
if (require.main === module) {  
    main().catch(err => {  
        console.error(err.stack || err);  
        process.exit(1);  
   });  
}

浏览器API初始导航

根据抓取浏览器的会话管理机制，每个会话仅允许执行一次初始导航，即首次加载目标网站以进行数据提取的操作。在此会话起点确立后，用户便可在该网站内部通过点击、滚动等交互动作自由跳转。然而，任何需要从初始导航阶段重新开始的抓取任务——无论目标是同一网站还是不同网站——都必须通过创建新会话来完成。

会话时间限制

自动超时机制：所有浏览器会话均受限于30分钟的最大存活时间。若会话未通过脚本指令主动终止，系统将在此时间后自动将其结束。
Web控制台特殊限制：在Web控制台环境中，系统强制实行单账户单一活动会话的策略。为避免资源冲突与潜在错误，请在您的自动化脚本中务必加入显式关闭会话的逻辑。

如果您需要进一步配置方面的帮助，请随时通过以下方式与我们联系： [email protected].

Previous快速开始 Next标准功能

Last updated 1 month ago

Was this helpful?