User-agent:
Disallow: /plus/ad_jsphp
Disallow: /plus/advancedsearchphp
Disallow: /plus/carphp
Disallow: /plus/carbuyactionphp
Disallow: /plus/shops_buyactionphp
Disallow: /plus/erraddsavephp
Disallow: /plus/posttocarphp
Disallow: /plus/disdlsphp
Disallow: /plus/feedback_jsphp
Disallow: /plus/mytag_jsphp
Disallow: /plus/rssphp
Disallow: /plus/searchphp
Disallow: /plus/recommendphp
Disallow: /plus/stowphp
Disallow: /plus/countphp
Disallow: /include
Disallow: /templets
Disallow: /css
Disallow: /js
Disallow: /Public当然有必要做啦,蜘蛛来到网站第一个找的就是robotstxt文件,没有的话,就相当于一个404错误,sitemap也是有必要做的,可以让蜘蛛很简单的工作,这样蜘蛛对网站也相对会有好点。
欢迎分享,转载请注明来源:内存溢出
评论列表(0条)