scrapy设置随机User-agent、scrapy爬取大量网页、设置爬虫请求并发数

Scrapy可以通过在settings.py文件中设置USER_AGENTS列表来实现随机user-agent。在列表中添加多个user-agent，Scrapy会随机选择一个作为请求头中的user-agent。这样可以避免被网站识别为爬虫，提高爬虫的稳定性和成功率。具体设置方法如下： 1. 在settings.py文件中添加USER_AGENTS列表，如下所示： ``` USER_AGENTS = [ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3', 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0', 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36', 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; AS; rv:11.0) like Gecko', 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36', ] ``` 2. 在middlewares.py文件中添加RandomUserAgentMiddleware中间件，如下所示： ``` from scrapy import signals import random class RandomUserAgentMiddleware(object): def __init__(self, agents): self.agents = agents @classmethod def from_crawler(cls, crawler): return cls(crawler.settings.getlist('USER_AGENTS')) def process_request(self, request, spider): request.headers.setdefault('User-Agent', random.choice(self.agents)) ``` 3. 在settings.py文件中启用RandomUserAgentMiddleware中间件，如下所示： ``` DOWNLOADER_MIDDLEWARES = { 'myproject.middlewares.RandomUserAgentMiddleware': 543, 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None, } ``` 这样就完成了随机user-agent的设置。