DEBUG: Filtered offsite request to

最新推荐文章于 2023-12-08 15:35:37 发布

m_spider

最新推荐文章于 2023-12-08 15:35:37 发布

阅读量1.2w

点赞数 21

CC 4.0 BY-SA版权

分类专栏： spider

本文链接：https://blog.csdn.net/weixin_41607151/article/details/80515030

在做爬虫项目时，出现了一个问题，解析一个网站二次爬取时没有获取到数据，就写了一个测试程序试了下，测试程序如下

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule

class ZhenaiSpider(CrawlSpider):
name = 'zhenai'
allowed_domains = ['www.zhenai.com']
start_urls = ['http://www.zhenai.com/zhenghun/beijing/1']

rules = (
Rule(LinkExtractor(allow=r'http://www.zhenai.com/zhenghun/beijing/\d+'), callback='parse_item', follow=False),
)

def parse_item(self, response):
a_list = response.xpath('//div[@class="content"]/table//tr/th/a')
for a in a_list:
item = {}
title = a.xpath('./text()').extract_first()
item['title'] = title