盖得化工_采集某省份所有公司的链接网址

最新推荐文章于 2024-06-23 08:00:00 发布

转载最新推荐文章于 2024-06-23 08:00:00 发布 · 215 阅读

文章标签：

#爬虫 #python #测试

本文介绍了一种使用Python Selenium爬取湖北化工企业信息的方法，包括构造页面URL、获取公司链接等步骤。通过Firefox浏览器自动化操作实现了数据抓取。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Python爬虫视频教程零基础小白到scrapy爬虫高手-轻松入门

https://item.taobao.com/item.htm?spm=a1z38n.10677092.0.0.482434a6EmUbbW&id=564564604865

# -*- coding: utf-8 -*-
"""
Created on Mon May  9 09:14:32 2016

@author: Administrator
"""
import requests,bs4,csv,time,selenium,random
from selenium import webdriver


site_hubei="http://china.guidechem.com/suppliers/list_catid-21_area-%E6%B9%96%E5%8C%97"
pages_hubei=31
#所有公司的链接网址_存储列表
list_corporation_link=[]

def Get_sites(site,pages):
    list_pages=[]
    for page in range(1,pages+1):
        thePage=site+"-"+"p"+str(page)+".html"
        list_pages.append(thePage)
    
    return list_pages
    
#采集一页所有公司的二级网址
def Get_corporation_site():
    elems=browser.find_elements_by_class_name("dblue")
    links_list=[i.get_attribute("href") for i in elems]
    return links_list 

#构造所有湖北页面网址    
list_pages_hubei=Get_sites(site_hubei,pages_hubei)
#打开浏览器
browser=webdriver.Firefox()

for page in list_pages_hubei:
    browser.get(page)
    links_list=Get_corporation_site()
    list_corporation_link.append(links_list)
    
#测试速度太慢，主要是反复调用browser.get(page)    
'''
[['http://show.guidechem.com/hbsthcsales/',
  'http://show.guidechem.com/whdinghuichem/',
  'http://show.guidechem.com/hbeixys2001/',
  'http://show.guidechem.com/jusheng/',
  'http://show.guidechem.com/yongnuopharm/',
  'http://show.guidechem.com/gameboy0820/',
  'http://show.guidechem.com/whhxk/',
  'http://show.guidechem.com/zhuoxinghy/',
  'http://show.guidechem.com/nrcaocao/',
  'http://show.guidechem.com/hbxyhhgwy/',
  'http://show.guidechem.com/jushengkeji/',
  'http://show.guidechem.com/jushengchem4/',
  'http://show.guidechem.com/eva1210/',
  'http://show.guidechem.com/whdhwy/',
  'http://show.guidechem.com/chenjinping2014/',
  'http://show.guidechem.com/hbjltchem/',
  'http://show.guidechem.com/jlt7796/',
  'http://show.guidechem.com/saichuang123/',
  'http://show.guidechem.com/apidky/',
  'http://show.guidechem.com/finetechwh/'],
 ['http://show.guidechem.com/csw288/',
  'http://show.guidechem.com/cuizhiying123/',
  'http://show.guidechem.com/sjpxkj/',
...........  
  ttp://show.guidechem.com/hengshuochem/',
  'http://show.guidechem.com/DSHKJYXGS/',
  'http://show.guidechem.com/HUBEIBOKAI/',
  'http://show.guidechem.com/xiangdetech/']]

'''

速度太慢，尝试翻页速度