Web Scraper用于python中的动态表单

Web Scraper用于python中的动态表单

如果您在开发人员工具中查看发送到该站点的请求，则将看到在选择状态后立即发送了POST。发送回的响应具有填入城市下拉列表中的值的形式。

因此，要将其复制到脚本中，您需要以下内容：

• 打开页面
• 选择表格
• 选择模型和状态的值
• 提交表格
• 从发送回的响应中选择表格
• 选择城市值（应立即填充）
• 提交表格
• 解析结果表的响应

看起来像：

#!/usr/bin/env pythonimport reimport mechanizefrom bs4 import BeautifulSoupdef select_form(form):    return form.attrs.get('id', None) == 'form1'def get_state_items(browser):    browser.select_form(predicate=select_form)    ctl = browser.form.find_control('ctl00$ContentPlaceHolder1$ddlState')    state_items = ctl.get_items()    return state_items[1:]def get_city_items(browser):    browser.select_form(predicate=select_form)    ctl = browser.form.find_control('ctl00$ContentPlaceHolder1$ddlCity')    city_items = ctl.get_items()    return city_items[1:]br = mechanize.Browser()br.open('http://www.marutisuzuki.com/Maruti-Price.aspx')    br.select_form(predicate=select_form)br.form['ctl00$ContentPlaceHolder1$ddlmodel'] = ['AK'] # model = Maruti Suzuki Alto K10for state in get_state_items(br):    # 1 - Submit form for state.name to get cities for this state          br.select_form(predicate=select_form)    br.form['ctl00$ContentPlaceHolder1$ddlState'] = [ state.name ]    br.submit()    # 2 - Now the city dropdown is filled for state.name        for city in get_city_items(br):        br.select_form(predicate=select_form)        br.form['ctl00$ContentPlaceHolder1$ddlCity'] = [ city.name ]        br.submit()        s = BeautifulSoup(br.response().read())        t = s.find('table', id='ContentPlaceHolder1_dtDealer')        r = re.compile(r'^ContentPlaceHolder1_dtDealer_lblName_d+$')        header_printed = False        for p in t.findAll('span', id=r): tr = p.findParent('tr') td = tr.findAll('td') if header_printed is False:     str = '%s, %s' % (city.attrs['label'], state.attrs['label'])     print str     print '-' * len(str)     header_printed = True print ' '.join(['%s' % x.text.strip() for x in td])