Python 爬虫进阶：如何用 Scrapy + Playwright 爬取动态网站？

### 使用Scrapy-Playwright进行网页爬取 在现代Web开发中，许多页面依赖JavaScript来加载动态内容。为了有效抓取这些类型的页面，`scrapy-playwright`提供了强大的工具集[^1]。 安装`scrapy-playwright`可以通过pip完成： ```bash pip install scrapy-playwright ``` 配置项可以在项目的settings.py文件里设置以启用Playwright下载器中间件并指定浏览器类型和其他选项: ```python # settings.py DOWNLOAD_HANDLERS = { "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler", "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler", } TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor" PLAYWRIGHT_LAUNCH_OPTIONS = {"headless": False} # 设置无头模式为关闭以便观察操作过程 ``` 编写Spider时可以利用`playwright`请求处理函数执行特定的任务比如等待元素出现或模拟用户交互行为: ```python import asyncio from scrapy import Spider from scrapy_playwright.page import PageMethod class ExampleSpider(Spider): name = 'example' start_urls = ['https://example.org'] custom_settings = { "DOWNLOADER_MIDDLEWARES": { "scrapy_playwright.middlewares.PlaywrightMiddleware": 800, }, } async def parse(self, response): await response.page.wait_for_selector('h1') # 等待某个选择器存在 yield { 'title': response.css('h1::text').get(), } _page_methods = [ PageMethod('click', selector='a.next'), # 模拟点击下一页链接 PageMethod('wait_for_navigation'), # 等待导航完成 ] new_page = await response.follow(response.urljoin('/next-page'), method="GET", page_methods=_page_methods) next_data = await (await new_page.json()) yield next_data ``` 上述代码展示了如何通过`PageMethod`对象定义一系列动作序列，在访问新页面之前依次被执行；同时也包含了基本的选择器等待逻辑以及简单的分页处理方式[^3]。