尝试改为导入HtmlXPathSelector。
from scrapy.selector import HtmlXPathSelector
然后使用.select()方法解析出你的html。例如,
sel = HtmlXPathSelector(response) site_names = sel.select('//ul/li')
示例将如下所示:
from scrapy.spider import baseSpider from scrapy.selector import HtmlXPathSelector class DmozSpider(baseSpider): name = "dmoz" allowed_domains = ["dmoz.org"] start_urls = [ "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/", "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/" ] def parse(self, response): sel = HtmlXPathSelector(response) sites = sel.select('//ul/li') for site in sites: title = site.select('a/text()').extract() link = site.select('a/@href').extract() desc = site.select('text()').extract() print title, link, desc
希望这可以帮助!
欢迎分享,转载请注明来源:内存溢出
评论列表(0条)