Selenium in Scrapy

Data Flow of Scrapy ¹Scrapy Doc.: Architecture Overview

Middleware

Downloader Middleware

Between ENGINE and DOWNLOADER

Spider Middleware

Between ENGINE and SPIDERS

Using Selenium in Scrapy

Scrapy with Selenium ²GitHub: Scrapy with Selenium Project

Scrapy middleware to handle javascript pages using selenium.

Installing

pip install scrapy-selenium

Setting settings.py in Scrapy

1. 打開 {scrapy_project} 中的 settings.py

 

Last Updated on 2025/04/10 by A1go

References

• 1
  Scrapy Doc.: Architecture Overview
• 2
  GitHub: Scrapy with Selenium Project