Python http下载页面源代码

使用urllib2下载页面。

Google会尝试阻止所有机器人，因此会阻止此请求。 将用户代理添加到请求中。

import urllib2user_agent = 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3'headers = { 'User-Agent' : user_agent }req = urllib2.Request('http://www.google.com', None, headers)response = urllib2.urlopen(req)page = response.read()response.close() # its always safe to close an open connection

您也可以使用pyCurl

import sysimport pycurlclass ContentCallback:        def __init__(self):     self.contents = ''        def content_callback(self, buf):     self.contents = self.contents + buft = ContentCallback()curlObj = pycurl.Curl()curlObj.setopt(curlObj.URL, 'http://www.google.com')curlObj.setopt(curlObj.WRITEFUNCTION, t.content_callback)curlObj.perform()curlObj.close()print t.contents