多个网址抓selenium

您需要做的是：

• 重用同一

  webdriver

  实例-不要在循环中初始化它
• 引入显式等待 -这肯定会使代码更加可靠和快速

实现方式：

from selenium import webdriverfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.support.ui import WebDriverWaitfrom selenium.webdriver.support import expected_conditions as ECimport pandas as pdurls = [    'http://www.oddsportal.com/hockey/austria/ebel-2014-2015/results/#/page/',    'http://www.oddsportal.com/hockey/austria/ebel-2013-2014/results/#/page/']data = []driver = webdriver.PhantomJS()driver.implicitly_wait(10)wait = WebDriverWait(driver, 10)for url in urls:    for page in range(1, 8):        driver.get(url + str(page))        # wait for the page to load        wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div#tournamentTable tr.deactivate")))        for match in driver.find_elements_by_css_selector("div#tournamentTable tr.deactivate"): home, away = match.find_element_by_class_name("table-participant").text.split(" - ") date = match.find_element_by_xpath(".//preceding::th[contains(@class, 'first2')][1]").text if " - " in date:     date, event = date.split(" - ") else:     event = "Not specified" data.append({     "home": home.strip(),     "away": away.strip(),     "date": date.strip(),     "event": event.strip() })driver.close()df = pd.Dataframe(data)print(df)

印刷品：

        away         date          event     home0   Salzburg  14 Apr 2015      Play Offs     Vienna Capitals1       Vienna Capitals  12 Apr 2015      Play Offs Salzburg2   Salzburg  10 Apr 2015      Play Offs     Vienna Capitals3       Vienna Capitals  07 Apr 2015      Play Offs Salzburg4       Vienna Capitals  31 Mar 2015      Play Offs         Liwest Linz5   Salzburg  29 Mar 2015      Play Offs          Klagenfurt6Liwest Linz  29 Mar 2015      Play Offs     Vienna Capitals7 Klagenfurt  26 Mar 2015      Play Offs Salzburg8       Vienna Capitals  26 Mar 2015      Play Offs         Liwest Linz9Liwest Linz  24 Mar 2015      Play Offs     Vienna Capitals10  Salzburg  24 Mar 2015      Play Offs          Klagenfurt11Klagenfurt  22 Mar 2015      Play Offs Salzburg12      Vienna Capitals  22 Mar 2015      Play Offs         Liwest Linz13   Bolzano  20 Mar 2015      Play Offs         Liwest Linz14        Fehervar AV19  18 Mar 2015      Play Offs     Vienna Capitals15          Liwest Linz  17 Mar 2015      Play Offs  Bolzano16      Vienna Capitals  16 Mar 2015      Play Offs       Fehervar AV1917   Villach  15 Mar 2015      Play Offs Salzburg18Klagenfurt  15 Mar 2015      Play Offs   Znojmo19   Bolzano  15 Mar 2015      Play Offs         Liwest Linz20          Liwest Linz  13 Mar 2015      Play Offs  Bolzano21        Fehervar AV19  13 Mar 2015      Play Offs     Vienna Capitals22    Znojmo  13 Mar 2015      Play Offs          Klagenfurt23  Salzburg  13 Mar 2015      Play Offs  Villach24Klagenfurt  10 Mar 2015      Play Offs   Znojmo25      Vienna Capitals  10 Mar 2015      Play Offs       Fehervar AV1926   Bolzano  10 Mar 2015      Play Offs         Liwest Linz27   Villach  10 Mar 2015      Play Offs Salzburg28          Liwest Linz  08 Mar 2015      Play Offs  Bolzano29    Znojmo  08 Mar 2015      Play Offs          Klagenfurt..       ...          ... ...      ...670       TWK Innsbruck  28 Sep 2013  Not specified   Znojmo671         Liwest Linz  27 Sep 2013  Not specified Dornbirn672  Bolzano  27 Sep 2013  Not specified          Graz 99ers673          Klagenfurt  27 Sep 2013  Not specified  Olimpija Ljubljana674       Fehervar AV19  27 Sep 2013  Not specified Salzburg675       TWK Innsbruck  27 Sep 2013  Not specified     Vienna Capitals676  Villach  27 Sep 2013  Not specified   Znojmo677 Salzburg  24 Sep 2013  Not specified  Olimpija Ljubljana678 Dornbirn  22 Sep 2013  Not specified       TWK Innsbruck679          Graz 99ers  22 Sep 2013  Not specified          Klagenfurt680     Vienna Capitals  22 Sep 2013  Not specified  Villach681       Fehervar AV19  21 Sep 2013  Not specified  Bolzano682 Dornbirn  20 Sep 2013  Not specified  Bolzano683  Villach  20 Sep 2013  Not specified          Graz 99ers684   Znojmo  20 Sep 2013  Not specified          Klagenfurt685  Olimpija Ljubljana  20 Sep 2013  Not specified         Liwest Linz686       Fehervar AV19  20 Sep 2013  Not specified       TWK Innsbruck687 Salzburg  20 Sep 2013  Not specified     Vienna Capitals688  Villach  15 Sep 2013  Not specified          Klagenfurt689         Liwest Linz  15 Sep 2013  Not specified Dornbirn690     Vienna Capitals  15 Sep 2013  Not specified       Fehervar AV19691       TWK Innsbruck  15 Sep 2013  Not specified Salzburg692          Graz 99ers  15 Sep 2013  Not specified   Znojmo693  Olimpija Ljubljana  14 Sep 2013  Not specified Dornbirn694  Bolzano  14 Sep 2013  Not specified       Fehervar AV19695          Klagenfurt  13 Sep 2013  Not specified          Graz 99ers696   Znojmo  13 Sep 2013  Not specified Salzburg697  Olimpija Ljubljana  13 Sep 2013  Not specified       TWK Innsbruck698  Bolzano  13 Sep 2013  Not specified     Vienna Capitals699         Liwest Linz  13 Sep 2013  Not specified  Villach[700 rows x 4 columns]