将各种字段列表的JSON响应转换为Pandas数据框

我正在使用python的Requests库从BestBuy Products API下载一些数据，并且我想将它们存储到pandas数据框中。将各种字段列表的JSON响应转换为Pandas数据框

类似的东西：

results = requests.get(url1, 
        params={'paramStuff'}, 
        headers={'User-Agent': ua}) 
products = json.loads(results.text)

一个得到了很多的服务信息各领域的，因此我只瞄准，我想在JSON特定领域：

products['products']

我：

[{'details':[{'name': 'Name of Feature', 'value':'Value Of Feature'}, 
      {'name': 'Name of Other Feature', 'value':'Value Of Other 
       Feature'}, ...], 
    'ProductId': 'Id Of Product 1', 
    'Some Other Field': 'Some Other Field Value'}, 
{same structure as above for other product}, {etc}]

因此，当你看到它就像是一个字典列表，其中又包含词典列表自己。要突出显示 - 细节词典可以有各种名称组合的名称：值（名称在产品中也不同）。

对如何处理这样的结构，以获得与这种格式的数据帧的任何想法：

+-----------+-------------------+-------------------+-------------------+------------------+ 
| ProductID | Name of Feature 1 | Name of Feature 2 | Name Of Feature 3 | Some Other Field | 
+-----------+-------------------+-------------------+-------------------+------------------+ 
| Product 1 | Value    | NULL    | Value    | Value   | 
| Product 2 | NULL    | Value    | Value    | Value   | 
+-----------+-------------------+-------------------+-------------------+------------------+

到目前为止，我只设法得到这样的：

+-----------+-----------------------------------------------------------------------------------------------------------------------------------+------------------+ 
| ProductID |                Details                | Some Other Field | 
+-----------+-----------------------------------------------------------------------------------------------------------------------------------+------------------+ 
| Product 1 | [{'name': 'Name of Feature', 'value':'Value Of Feature'},{'name': 'Name of Other Feature', 'value':'Value Of Other Feature'},...] | Value 1   | 
| Product 2 | [{'name': 'Name of Feature', 'value':'Value Of Feature'},{'name': 'Name of Other Feature', 'value':'Value Of Other Feature'},...] | Value 2   | 
+-----------+-----------------------------------------------------------------------------------------------------------------------------------+------------------+

来源

2017-06-21 Maxim Haytovich

您是否尝试过使用'pandas.read_json'？ – jhamman

@jhamman是的，可悲的是它无法处理沉重的嵌套。我最终通过编写手动解析器来解决这个问题，如果没有更好的方法，我将在这里作为答案张贴 –

好吧，我结束开发一种手动解析嵌套字段的方法。没有弄清楚是否有任何简单的方法。仅供参考，它用于解析BestBuy Products API的响应，以防有人发现它有用。

#first build the pandas DF shown in question 
df = pd.io.json.json_normalize(products) 

#fields which are not nested and not require parsing 
fields = ['sku', 'name', 'regularPrice', 'manufacturer'] 
#nested field is called 'details', as mentioned can have a lot of different subfields 
featureFields = [] 


#first build a list which will have all potential features from the nested field 
for i in range(0,len(df)): 
    row = df.iloc[i] 
    for detail in row['details']: 
     featureFields.append(detail['name'].split('>', 1)[-1]) 

#make a list unique 
featureFields = set(featureFields)  
fields = set(fields) 

#now we go over each record in dataframe and parse nested field to a dict 
records = [] 

for i in range(0,len(df)): 
    row = df.iloc[i] 
    record = dict.fromkeys(fields) 
    record['name'] = row['name'] 
    record['regularPrice'] = row['regularPrice'] 
    record['manufacturer'] = row['manufacturer'] 
    record['sku'] = row['sku'] 
    for detail in row['details']: 
     record[detail['name'].split('>', 1)[-1]] = detail['value'].split('>', 1)[-1] 
    records.append(record) 

#finally we have not nested list of dictionaries with records 
dfFinal = pd.DataFrame.from_dict(records)

来源

2017-06-22 02:18:10

将各种字段列表的JSON响应转换为Pandas数据框

回答

相关问题