Nutch 使用metadata plugin捕获页面中的meta标签数据

Nutch 使用metadata plugin捕获页面中的meta标签数据 Nutch 使用metadata plugin捕获页面中的meta标签数据 添加index-metadata和parse-metatags

要使用Nutch的插件，就需要在conf/nutch-site.xml中指定要使用的插件，插件的选取可以使用正则匹配去选取的，插件的nutch的根目录下的apache-nutch-x.x/plugins文件夹中。

index-(basic|anchor|metadata)这句包含了index-metadata，index-anchor，index-basic。

parse-(html|metatags)这句包含了parse-html，parse-metatags

  	plugin.includes
  	protocol-http|protocol-httpclient|urlfilter-regex|parse-(html|metatags)|index-(basic|anchor|metadata)|indexer-solr|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)
	my Nutch plugin

指定index-metadata需要的参数

需要获取那些metatags的内容，可以配置下面这些参数。

metatags.names
description,keywords
 Names of the metatags to extract, separated by ','.
  Use '*' to extract all metatags. Prefixes the names with 'metatag.'
  in the parse-metadata. For instance to index description and keywords,
  you need to activate the plugin index-metadata and set the value of the
  parameter 'index.parse.md' to 'metatag.description,metatag.keywords'.

1.x版本

处理要配置要获取的metatags的名字，还要配置metatags的解析字段名

  index.parse.md
  metatag.description,metatag.keywords
  
  Comma-separated list of keys to be taken from the parse metadata to generate fields.
  Can be used e.g. for 'description' or 'keywords' provided that these values are generated
  by a parser (see parse-metatags plugin)
  

2.x版本

在2.x中官方说明可以省略metatag.

  index.metadata
  description,keywords
  
  Comma-separated list of keys to be taken from the metadata to generate fields.
  Can be used e.g. for 'description' or 'keywords' provided that these values are generated
  by a parser (see parse-metatags plugin), and property 'metatags.names'.
  

添加metatags plugin需要fields

如果你需要在solr中建立索引，可以在solr的配置文件中添加以下字段

测试

可以用随便一条网页查看爬取解析的结果

#使用indexchecker检查爬取解析处理的index
./bin/nutch indexchecker https://blog.csdn.net/qq_43203949/article/details/108550960

#结果
fetching: https://blog.csdn.net/qq_43203949/article/details/108550960
parsing: https://blog.csdn.net/qq_43203949/article/details/108550960
contentType: text/html
tstamp :	Sun Jan 02 23:19:25 CST 2022
metatag.description :	解决Could not load dynamic library ‘cudnn64_7.dll’; dlerror: cudnn64_7.dll not found如果你使用的是tensorflow或
digest :	9f1bc7879417faaa87e82a7d65ca0cab
host :	blog.csdn.net
id :	https://blog.csdn.net/qq_43203949/article/details/108550960
title :	解决Could not load dynamic library ‘cudnn64_7.dll‘； dlerror cudnn64_7.dll not found_鸭梨的药丸哥的博客-CSDN博客_c
metatag.keywords :	cudnn64_7.dll
url :	https://blog.csdn.net/qq_43203949/article/details/108550960
content :	解决Could not load dynamic library ‘cudnn64_7.dll‘； dlerror cudnn64_7.dll not found_鸭梨的药丸哥的博客-CSDN博客_c