#解析数据 detail_selector = parsel.Selector(detail_html) title = detail_selector.css('.b_crumb_cont *:nth-child(3)::text').get() comment = detail_selector.css('.title white::text').get() view_count = detail_selector.css('.view_count::text').get() date = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.when > p > span.data::text').get() days = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.howlong > p > span.data::text').get() money = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.howmuch > p > span.data::text').get() character = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.who > p > span.data::text').get() play_list = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.how > p > span.data > span::text').getall() for play in play_list: print(play) print(title, comment, view_count, date, days, money, character, play_list, detail_url) time.sleep(random.randint(3,5))这段代码为什么只能输出最后一个网站的内容
时间: 2024-03-30 07:33:59 浏览: 72
这段代码中的 `detail_html` 变量并没有被展示出来,所以无法确定是如何获取该变量的。但是从代码逻辑上来看,可能是在循环中多次赋值了 `detail_html` 但是在代码的其他部分并没有使用该变量。因此,每一次循环时都会从最后一个网站获取数据并输出,导致输出结果只有最后一个网站的内容。要想输出所有网站的内容,应该在循环内部对 `detail_html` 进行赋值并使用。
相关问题
selector = parsel.Selector(html_data) lis = selector.css('.list_item h2 a::attr(href)').getall() details = [] for li in lis: detail_url = 'https://travel.qunar.com' + li detail_html = requests.get(detail_url).text details.append(detail_html) # 输出每个网站的 html 数据 for detail_html in details: print(detail_html) detail_html = requests.get(detail_url).text #解析数据 detail_selector = parsel.Selector(detail_html) title = detail_selector.css('.b_crumb_cont *:nth-child(3)::text').get() comment = detail_selector.css('.title white::text').get() view_count = detail_selector.css('.view_count::text').get() date = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.when > p > span.data::text').get() days = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.howlong > p > span.data::text').get() money = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.howmuch > p > span.data::text').get() character = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.who > p > span.data::text').get() play_list = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.how > p > span.data > span::text').getall() for play in play_list: print(play) print(title, comment, view_count, date, days, money, character, play_list, detail_url) time.sleep(random.randint(3,5))这段代码只能输出最后一个网站的信息 该怎么改
这段代码只能输出最后一个网站的信息是因为在循环中每次请求新的网站信息时都会覆盖前面的信息,需要将每次请求的信息存储到一个列表中,最后再一起处理。
可以尝试如下修改:
```python
selector = parsel.Selector(html_data)
lis = selector.css('.list_item h2 a::attr(href)').getall()
details = []
for li in lis:
detail_url = 'https://travel.qunar.com' + li
detail_html = requests.get(detail_url).text
details.append(detail_html)
time.sleep(random.randint(3, 5)) # 每次请求后等待一段时间,避免请求频繁被封
for detail_html in details:
# 解析数据
detail_selector = parsel.Selector(detail_html)
title = detail_selector.css('.b_crumb_cont *:nth-child(3)::text').get()
comment = detail_selector.css('.title.white::text').get()
view_count = detail_selector.css('.view_count::text').get()
date = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.when > p > span.data::text').get()
days = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.howlong > p > span.data::text').get()
money = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.howmuch > p > span.data::text').get()
character = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.who > p > span.data::text').get()
play_list = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.how > p > span.data > span::text').getall()
for play in play_list:
print(play)
print(title, comment, view_count, date, days, money, character, play_list, detail_url)
```
这样每次请求得到的网站信息都会被存储到 `details` 列表中,最后再遍历处理每个网站的信息。同时,为了避免请求频繁被封,可以在每次请求后等待一段时间。
selector = parsel.Selector(html_data) lis = selector.css('.list_item h2 a::attr(href)').getall() details = [] for li in lis: detail_url = 'https://travel.qunar.com' + li detail_html = requests.get(detail_url).text details.append(detail_html) time.sleep(random.randint(3, 5)) # 每次请求后等待一段时间,避免请求频繁被封 for detail_html in details: # 解析数据 detail_selector = parsel.Selector(detail_html) title = detail_selector.css('.b_crumb_cont *:nth-child(3)::text').get() comment = detail_selector.css('.title.white::text').get() view_count = detail_selector.css('.view_count::text').get() date = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.when > p > span.data::text').get() days = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.howlong > p > span.data::text').get() money = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.howmuch > p > span.data::text').get() character = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.who > p > span.data::text').get() play_list = detail_selector.css('#js_mainleft > div.b_foreword > ul > li.f_item.how > p > span.data > span::text').getall() for play in play_list: print(play) print(title, comment, view_count, date, days, money, character, play_list, detail_url)这段代码输出的文字可以匹配,但是网站都是同一个,要怎么改
这段代码是爬取了一个页面中多篇游记的链接,然后依次爬取每篇游记的详情页,并解析其中的数据。由于您爬取的是同一个网站的多篇游记,所以每篇游记的详情页链接都是相同的。如果您想爬取不同网站的游记,可以修改`detail_url`的值为对应的网址。
如果您想爬取同一个网站的不同分类、不同标签或不同页面的游记,可以修改`html_data`的值为对应页面的HTML源码,并修改`lis`的值为对应页面中游记链接所在的CSS选择器。例如,如果您要爬取https://travel.qunar.com/travelbook/list.htm?order=hot_heat页面中所有游记的链接,可以将`lis`的值修改为`.list_item h2 a::attr(href)`。
请注意,网站的反爬虫机制比较严格,频繁地爬取同一个页面可能会导致IP被封禁,建议使用代理IP或降低爬取频率。
阅读全文
相关推荐
















