网上刮刮香港电子立法

Question

我想摘一下香港立法的内容。但是，我无法访问不可见的内容，除非我向下滚动页面。

网站我访问：https://www.elegislation.gov.hk/hk/cap211

from selenium import webdriver 
from selenium.webdriver.common.by import By 
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC 
from selenium.common.exceptions import TimeoutException 
from selenium.common.exceptions import ElementNotVisibleException 
from selenium.webdriver.common.action_chains import ActionChains 

def init_driver(profile): 
    driver = webdriver.Firefox(profile) 
    driver.wait = WebDriverWait(driver, 5) 
    return driver 

def convert2text2(webElement): 
    if webElement != []: 
     webElements = [] 
     for element in webElement: 
      e = element.text.encode('utf8') 
      webElements.append(e) 
    else: 
     webElements = ['NA'] 
    return webElements 

profile = webdriver.FirefoxProfile() 
driver = init_driver(profile) 
url = 'https://www.elegislation.gov.hk/hk/cap211' 
driver.get(url) 
driver.wait = WebDriverWait(driver, 5) 

content = driver.find_elements_by_xpath("//div[@class='hklm_content' or @class='hklm_leadIn' or @class='hklm_continued']") 
content = convert2text2(content) 

了解，从How can I scroll a web page using selenium webdriver in python?采取下面的代码被用于滚动浏览器：

SCROLL_PAUSE_TIME = 0.5 

# Get scroll height 
last_height = driver.execute_script("return document.body.scrollHeight") 

while True: 
    # Scroll down to bottom 
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") 

    # Wait to load page 
    time.sleep(SCROLL_PAUSE_TIME) 

    # Calculate new scroll height and compare with last scroll height 
    new_height = driver.execute_script("return document.body.scrollHeight") 
    if new_height == last_height: 
     break 
    last_height = new_height 

但我不能想出如何指定内容窗口的滚动条并滚动到底部。

Answer 1

你只要把last_height在JavaScript代码如下所示：

while True: 
    # Scroll down to 'last_height' 
    driver.execute_script("window.scrollTo(0, {});".format(last_height)) 

    # Wait to load page 
    time.sleep(SCROLL_PAUSE_TIME) 

    # Calculate new scroll height and compare with last scroll height 
    new_height = driver.execute_script("return document.body.scrollHeight;") 
    if new_height == last_height: 
     break 
    last_height = new_height 

的要对此是根本谈不上把数据无硒的另一种方式。如果您查看网页所做的调用（Chrome检查器，网络选项卡），您会看到每个新元素都使用小块xml加载到网站中。

为出发点，网址为“https://www.elegislation.gov.hk/xml?skipHSC=true&LANGUAGE=E&BILINGUAL=&LEG_PROV_MASTER_ID=181740&QUERY=.&INDEX_CS=N&PUBLISHED=true”

的PROV_MASTER_ID参数将由1对于每个大块，该网站负荷增加。

你可以抓住它，像这样全部采用请求：

import requests 
url = 'https://www.elegislation.gov.hk/xml?skipHSC=true&LANGUAGE=E&BILINGUAL=&LEG_PROV_MASTER_ID={}&QUERY=.&INDEX_CS=N&PUBLISHED=true' 
starting_count = 181740 
stop_count = "" # integer - you need to figure out, when you got all you need 
count = starting_count 
while count <= stop_count: 
    response = requests.get(url.format(count)) 
    # parse the xml and grab the parts you need... 
    count +=1