用美丽的汤刮到熊猫的HTML表格

我想刮一张使用美丽的汤的html表，并将其导入到熊猫 - http://www.baseball-reference.com/teams/NYM/2017.shtml - “团队击球”表。用美丽的汤刮到熊猫的HTML表格

查找表是没有问题的：

table = soup.find('div', attrs={'class': 'overthrow table_container'}) 
table_body = table.find('tbody')

查找数据行是不是一个问题或者：

for i in table.findAll('tr')[2]: #increase to 3 to get next row in table... 
    print(i.get_text())

而且我甚至可以找到头名：

table_head = table.find('thead') 

for i in table_head.findAll('th'): 
    print(i.get_text())

现在我很难将所有东西放在一起放入数据框中。这是我到目前为止：

header = []  
for th in table_head.findAll('th'): 
     key = th.get_text() 
     header.append(key) 

row= [] 
for tr in table.findAll('tr')[2]: 
    value = tr.get_text() 
    row.append(value) 

od = OrderedDict(zip(head, row)) 
df = pd.DataFrame(d1, index=[0])

这只适用于一次一行。我的问题是如何在同一时间对表格中的每一行执行此操作？

来源

2017-04-20 e9e9s

我已经测试过，下面的内容适用于你的目的。基本上你需要创建一个列表，循环遍历播放器，使用该列表填充DataFrame。建议不要逐行创建DataFrame，因为这可能会明显变慢。

import collections as co 
import pandas as pd 

from bs4 import BeautifulSoup 

with open('team_batting.html','r') as fin: 
    soup = BeautifulSoup(fin.read(),'lxml') 

table = soup.find('div', attrs={'class': 'overthrow table_container'}) 
table_body = table.find('tbody') 

table_head = table.find('thead') 
header = []  
for th in table_head.findAll('th'): 
    key = th.get_text() 
    header.append(key) 

# loop over table to find number of rows with '' in first column 
endrows = 0 
for tr in table.findAll('tr'): 
    if tr.findAll('th')[0].get_text() in (''): 
     endrows += 1 

rows = len(table.findAll('tr')) 
rows -= endrows + 1 # there is a pernicious final row that begins with 'Rk' 

list_of_dicts = [] 
for row in range(rows): 
    the_row = [] 
    try: 
     table_row = table.findAll('tr')[row] 
     for tr in table_row: 
      value = tr.get_text() 
      the_row.append(value) 
     od = co.OrderedDict(zip(header,the_row)) 
     list_of_dicts.append(od) 
    except AttributeError: 
     continue 

df = pd.DataFrame(list_of_dicts)

来源

2017-04-20 17:10:30 bernie

感谢您的好评。在这一行中是否有“[row]”的名称：table_row = table.findAll（'tr'）[row] ---我从来没有见过像这样在范围之前使用它。 – e9e9s

你非常欢迎。这只是这种情况下的索引。等于'table_row = table.findAll（'tr'）[0]'或'table_row = table.findAll（'tr'）[1]' – bernie

那么如果'[row]'被排除在这一行之外，迭代'table_row'你将无法做到？ – e9e9s

该解决方案使用仅pandas，但通过提前知道球队击球表是第十表欺骗了一点。有了这些知识，下面通过pandas的read_html功能和返回DataFrame对象列表抓住第十DataFrame。其余的只是一些数据清理：

import pandas as pd 


url = 'http://www.baseball-reference.com/teams/NYM/2017.shtml' 

# Take 10th dataframe 
team_batting = pd.read_html(url)[9] 

# Take columns whose names don't contain "Unnamed" 
team_batting.drop([x for x in team_batting.columns if 'Unnamed' in x], axis=1, inplace=True) 

# Remove the rows that are just a copy of the headers/columns 
team_batting = team_batting.ix[team_batting.apply(lambda x: x != team_batting.columns,axis=1).all(axis=1),:] 

# Take out the Totals rows 
team_batting = team_batting.ix[~team_batting.Rk.isnull(),:] 

# Get a glimpse of the data 
print(team_batting.head(5)) 

# Rk Pos    Name Age G PA AB R H 2B ... OBP SLG OPS OPS+ TB GDP HBP SH SF IBB 
# 0 1 C Travis d'Arnaud 28 12 42 37 6 10 2 ... .357 .541 .898 144 20 1 1 0 0 1 
# 1 2 1B  Lucas Duda* 31 13 50 42 4 10 2 ... .360 .571 .931 153 24 1 0 0 0 2 
# 2 3 2B  Neil Walker# 31 14 62 54 5 12 3 ... .306 .278 .584 64 15 2 0 0 1 0 
# 3 4 SS Asdrubal Cabrera# 31 15 67 63 10 17 2 ... .313 .397 .710 96 25 0 0 0 0 0 
# 4 5 3B  Jose Reyes# 34 15 59 53 3 5 2 ... .186 .132 .319 -9 7 0 0 0 0 0

我希望这有助于。

来源

2017-04-20 18:24:02 Abdou

用美丽的汤刮到熊猫的HTML表格

回答

相关问题