Python – 在django随机地与芹菜一起崩溃

概述我在Ubuntu服务器上运行Django中的Scrapy项目. 问题是,即使只有一只蜘蛛在运行,Scrapy会随机崩溃. 下面是TraceBack的片段.作为一个没有专家,我用Google搜索 _SIGCHLDWaker Scrappy 但无法理解下面的片段找到的解决方案： --- <exception caught here> --- File "/home/b2b/virtualenvs/ 我在ubuntu服务器上运行Django中的Scrapy项目.
问题是,即使只有一只蜘蛛在运行,Scrapy会随机崩溃.

下面是TraceBack的片段.作为一个没有专家,我用Google搜索

_SIGCHLDWaker Scrappy

但无法理解下面的片段找到的解决方案：

--- <exception caught here> ---  file "/home/b2b/virtualenvs/venv/local/lib/python2.7/site-packages/twisted/internet/posixbase.py",line 602,in _doReadOrWrite    why = selectable.doWrite()exceptions.AttributeError: '_SIGCHLDWaker' object has no attribute 'doWrite'

我不熟悉扭曲,尽管我试图理解它,但对我来说似乎非常不友好.

以下是完整的追溯：

2015-10-10 14:17:13,652: INFO/Worker-4] Enabled downloader mIDdlewares: httpAuthMIDdleware,DownloadTimeoutMIDdleware,RandomUserAgentMIDdleware,ProxyMIDdleware,RetryMIDdleware,DefaultheadersMIDdleware,MetaRefreshMIDdleware,httpCompressionMIDdleware,RedirectMIDdleware,cookiesMIDdleware,ChunkedTransferMIDdleware,DownloaderStats[2015-10-10 14:17:13,655: INFO/Worker-4] Enabled spIDer mIDdlewares: httpErrorMIDdleware,OffsiteMIDdleware,RefererMIDdleware,UrlLengthMIDdleware,DepthMIDdleware[2015-10-10 14:17:13,656: INFO/Worker-4] Enabled item pipelines: MadePipeline[2015-10-10 14:17:13,656: INFO/Worker-4] SpIDer opened[2015-10-10 14:17:13,657: INFO/Worker-4] Crawled 0 pages (at 0 pages/min),scraped 0 items (at 0 items/min)Unhandled ErrorTraceback (most recent call last):  file "/home/b2b/virtualenvs/venv/local/lib/python2.7/site-packages/twisted/python/log.py",line 101,in callWithLogger    return callWithContext({"system": lp},func,*args,**kw)  file "/home/b2b/virtualenvs/venv/local/lib/python2.7/site-packages/twisted/python/log.py",line 84,in callWithContext    return context.call({ILogContext: newCtx},**kw)  file "/home/b2b/virtualenvs/venv/local/lib/python2.7/site-packages/twisted/python/context.py",line 118,in callWithContext    return self.currentContext().callWithContext(ctx,line 81,in callWithContext    return func(*args,**kw)--- <exception caught here> ---  file "/home/b2b/virtualenvs/venv/local/lib/python2.7/site-packages/twisted/internet/posixbase.py",in _doReadOrWrite    why = selectable.doWrite()exceptions.AttributeError: '_SIGCHLDWaker' object has no attribute 'doWrite'

以下是我根据scrapy文档实现任务的方法

from scrapy.crawler import CrawlerProcess,CrawlerRunnerfrom twisted.internet import reactorfrom scrapy.utils.project import get_project_settings@shared_taskdef run_spIDer(**kwargs):    task_ID = run_spIDer.request.ID    status = AsyncResult(str(task_ID)).status    source = kwargs.get("source")    pro,created = Project.objects.get_or_create(name="b2b")    query,_ = SearchTerm.objects.get_or_create(term=kwargs['query'])    src,_ = Source.objects.get_or_create(term=query,engine=kwargs['source'])    b,_ = Bot.objects.get_or_create(project=pro,query=src,spIDerID=str(task_ID),status=status,start_time=timezone.Now())    process = CrawlerRunner(get_project_settings())    if source == "amazon":        d = process.crawl(ComberSpIDer,query=kwargs['query'],job_ID=task_ID)        d.addBoth(lambda _: reactor.stop())    else:        d = process.crawl(MadeSpIDer,job_ID=task_ID)        d.addBoth(lambda _: reactor.stop())    reactor.run()

我也试过像这样的tutorial,但它导致了一个不同的问题,我无法追溯

为了完整性,这里是我的蜘蛛的片段

class ComberSpIDer(CrawlSpIDer):    name = "amazon"    allowed_domains = ["amazon.com"]    rules = (Rule(linkExtractor(allow=r'corporations/.+/-*50/[0-9]+\.HTML',restrict_xpaths="//a[@class='next']"),callback="parse_items",follow=True),)    def __init__(self,**kwargs):        super(ComberSpIDer,self).__init__(*args,**kwargs)        self.query = kwargs.get('query')        self.job_ID = kwargs.get('job_ID')        SignalManager(dispatcher.Any).connect(self.closed_handler,signal=signals.spIDer_closed)        self.start_urls = (            "http://www.amazon.com/corporations/%s/------------"            "--------50/1.HTML" % self.query.strip().replace(" ","_").lower(),)

解决方法 这是一个已知的Scrapy问题.有关详细信息和可能的解决方法,请参阅 issue report thread. 总结

以上是内存溢出为你收集整理的Python – 在django随机地与芹菜一起崩溃全部内容，希望文章能够帮你解决Python – 在django随机地与芹菜一起崩溃所遇到的程序开发问题。

如果觉得内存溢出网站内容还不错，欢迎将内存溢出网站推荐给程序员好友。