大批量爬取彼岸图网内容遇到的问题

大批量爬取彼岸图网内容遇到的问题

用多进程配套多进程短时间大量爬取图网25000张图.

问题:出现程序无结果不出问题也不报错,正常结束的异常,经多方检查调试.

原因:彼岸图网cookie隔30min刷新一次,用旧的cookie爬第一级页面只能爬到含有"跳转中"的源代码,导致xpath什么都解析不到,,而不用cookie也是如此.

解决:唯一解决办法是将旧的cookie换成新的cookie,若不用多线程多进程,则不会出现此问题.推测是因为cookie能跳过含有"跳转中"的页面,直接请求目标页面,应该不是网站站长主动反扒的措施,而就算用多进程套多线程,30min内只能爬到15000张图片,与目标数量相差甚远,所以换成使用selenium.目前正在尝试中....

这是第一版的代码,错误和算法问题最多,留录在此,方便往后再次观摩.--1.23记录

# -*- coding: UTF-8 -*-
"""
@Author: 王散 Creative
@Time: 2022/1/22 18:50
@IDE_Name/Software: PyCharm
@File: 应对彼岸图网的极限反爬
"""
import requests
from lxml import etree
import time
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
import threading
from multiprocessing import Lock


def task(url):
    # lock = threading.Lock()
    Squence = 0
    header = {
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.469"
                      "2.99 Safari/537.36",
        
    }
    resp1 = requests.get(url=url, headers=header)
    resp1.encoding = 'gbk'
    # print(resp1.text)
    tree = etree.HTML(resp1.text)
    analysis1 = tree.xpath('//*[@id="main"]/div[3]/ul/li//a/@href')
    analysis2 = tree.xpath('//*[@id="main"]/div[3]/ul/li/a/b/text()')
    for ItemTwo in analysis1:
        url_two_page = 'https://pic.netbian.com' + ItemTwo
        resp2 = requests.get(url=url_two_page, headers=header)
        # time.sleep(0.5)
        resp2.encoding = 'gbk'
        tree_two = etree.HTML(resp2.text)
        analysis3 = tree_two.xpath('//*[@id="img"]/img/@src')
        for ItemThree in analysis3:
            url_image_page = 'https://pic.netbian.com' + ItemThree
            resp3 = requests.get(url=url_image_page, headers=header)
            # lock.acquire()
            image_file = open(f'D:python_write_file爬虫NumberTwoImage彼岸网爬的好图2\{analysis2[Squence]}.jpg', 'wb')
            image_file.write(resp3.content)
            image_file.close()
            # lock.release()
            print(f'{analysis2[Squence]}==>爬取完毕')
            # lock.acquire()
            Squence = Squence + 1
            # lock.release()


# def main(num):
#     with ThreadPoolExecutor(252) as exe_Pool:
#         for item in range(num, num+126):
#             lock.acquire()
#             if item == 1:
#                 exe_Pool.submit(task, 'https://pic.netbian.com/new/')
#                 lock.release()
#             else:
#                 exe_Pool.submit(task, f'https://pic.netbian.com/new/index_{item}.html')
#                 lock.release()
#         return num+30


if __name__ == "__main__":
    # num = 1
    # lock = Lock()
    with ProcessPoolExecutor(45) as Process_Pool:
        for item in range(1, 1261):
            if item == 1:
                Process_Pool.submit(task, 'https://pic.netbian.com/new/')
            else:
                Process_Pool.submit(task, f'https://pic.netbian.com/new/index_{item}.html')
        # for number in range(1, 11):
        #     lock.acquire()
        #     num = Process_Pool.submit(main, num)
        #     lock.release()