2013-04-09 60 views
1

我几乎完成了抓取表格的webcralwer。这仅输出表中的第一行。任何人都可以帮助确定为什么这不会返回表中的所有行。请忽略while循环,因为它最终会有一个循环部分。美丽的行循环只运行一次?

import urllib 
from bs4 import BeautifulSoup 

#file_name = "/user/joe/uspc-cpc.txt 
#file = open(file_name,"w") 
i=125 
while i==125: 
    url = "http://www.uspto.gov/web/patents/classification/cpc/html/us" + str(i) + "tocpc.html" 
    print url + '\n' 
    i += 1 
    data = urllib.urlopen(url).read() 
    print data 
    #get the table data from dump 
    #append to csv file 
    soup = BeautifulSoup(data) 
    table = soup.find("table", width='80%') 
    for tr in table.findAll('tr')[2:]: 
     col = row.findAll('td') 
     uspc = col[0].get_text().encode('ascii','ignore') 
     cpc1 = col[1].get_text().encode('ascii','ignore') 
     cpc2 = col[2].get_text().encode('ascii','ignore') 
     cpc3 = col[3].get_text().encode('ascii','ignore') 
     print uspc + ',' + cpc1 + ',' + cpc2 + ',' + cpc3 + '\n' 
     #file.write(record) 

#file.close() 

CODE我运行:

import urllib 
from bs4 import BeautifulSoup 

#file_name = "https://stackoverflow.com/users/ripple/uspc-cpc.txt" 
#file = open(file_name,"w") 
i=125 
while i==125: 
    url = "http://www.uspto.gov/web/patents/classification/cpc/html/us" + str(i) + "tocpc.html" 
    print 'Grabbing from: ' + url + '\n' 
    i += 1 
    #get the table data from the page 
    data = urllib.urlopen(url).read() 
    #send to beautiful soup 
    soup = BeautifulSoup(data) 
    table = soup.find("table", width='80%') 
    for tr in table.findAll('tr')[2:]: 
     col = tr.findAll('td') 

     uspc = col[0].get_text().encode('ascii','ignore').replace(" ","") 
     cpc1 = col[1].get_text().encode('ascii','ignore').replace(" ","") 
     cpc2 = col[2].get_text().encode('ascii','ignore').replace(" ","") 
     cpc3 = col[3].get_text().encode('ascii','ignore').replace(" ","").replace("more...", "") 
     record = uspc + ',' + cpc1 + ',' + cpc2 + ',' + cpc3 + '\n' 
     print record 
     #file.write(record) 

#file.close() 
+0

是什么打印? – 2013-04-09 17:34:47

+0

您没有定义“行”。 – 2013-04-09 17:36:17

+0

@Marjin Pieters:如何定义行?输出是一行:125/901,H 03H 3/02,B 28D 5/00,H 03H 3/04,B 23D 47/005,B 24B 37/08更多... – 2013-04-09 17:37:36

回答

2

您使用tr作为循环变量,而是指在row代替循环。如果你之前已经定义了row,它可能会产生令人困惑的结果。

for tr in table.findAll('tr')[2:]: 
    col = tr.findAll('td') 

作品对我来说:

125/1,B 28D 1/00,B 28D 1/221,E 01C 23/081,B 28D 1/005,B 28D 1/06more... 

125/2,B 23Q 35/10,B 22C 9/18,B 23B 5/162,B 23D 63/18,B 24B 53/07more... 

125/3,B 28D 1/18,B 28D 1/003,B 28D 1/048,B 28D 1/181,B 24B 7/22more... 

+0

我改变了tr。我仍然只打印一行。 – 2013-04-09 17:42:57

+0

@JosephLee:不在您发布的代码中。它适用于我所做的修正。 – 2013-04-09 17:43:49

+0

太奇怪了。我仍然只打印一行代码。汤后打印出来。这是最后一行,所以我不知道该怎么想...... 125/901,H03H3/02,B28D5/00,H03H3/04,B23D47/005,B24B37/08 – 2013-04-09 17:46:28