比列表追加方法更有效地结合Python Pandas Dataframe

Question

我一直不得不做下面的事情来从一个处理单个json行的小型流水线中构建数据框。有没有更有效的方法来做到这一点，而不是依赖将它们附加到列表然后连接？此外，我不甚至需要在列标签下方表示为“钥匙”，但不知道如何将它们排除在外没有得到数据帧构造错误：

def readfiles(pattern, textfile): 
    for line in open(textfile): 
     try: 
      parsed = ujson.loads(line.rstrip('\n').rstrip(',')) 
      if pattern in parsed: 
       yield parsed 
     except ValueError, e: 
      pass 

def convertodf(lines): 
    dfs = [] 
    for line in lines: 
     dfs.append(pd.DataFrame({'key1':line['value'], 
             'key2':line['value']['value'], 
             'key3':line['value'], 
             'key4':line['value']['value'], 
             'key5':line['value']['value']})) 

    pd.concat(dfs, ignore_index=True).to_csv('testdf2.csv', index=False, header=None) 

def main(pattern, filenames): 
    lines = readfiles(pattern, filenames) 
    convertodf(lines) 

上述实施最酷的部分是，一个行[“值”]元素实际上是逗号分隔的整数，例如[1,2,3]的列表和它结束了相应地自动复制的其他值，例如：

'key1' 'key2' 
    1  california 
    2  california 
    3  california 
     ... 

这是我的最终工作版本我去感谢unutbu的帮助。

def readfiles(pattern, filedir): 
     for f in glob.glob(filedir+'*.zip'): 
      try: 
       with zipfile.ZipFile(f, 'r') as myzip: 
        for logfile in myzip.namelist(): 
         for line in myzip.open(logfile): 
          try: 
           line = ujson.loads(line.rstrip('\n').rstrip(',')) 
           if pattern in line: 
            for i in line['key1']: 
             yield i, line['key1']['key2'],\ 
            line['key3'], line['key4']['key5'],\ 
            line['key6']['key7'] 
          except ValueError as err: 
           pass 
      except zipfile.error, e: 
       pass 

def convertdfcsv(lines): 
     df = pd.DataFrame.from_records(lines) 
     df.to_csv('testdf2.csv', index=False, header=None) 

def main(pattern): 
     lines = readfiles(pattern) 
     convertdf(lines) 

Answer 1

您可以使用DataFrame.from_records从行迭代器构建DataFrame。 一个简单的例子展示了如何from_records工作原理是：

iterator = (item for item in [[1, 2, 3], [2, 3, 4, 5]]) 
df = pd.DataFrame.from_records(iterator, 
           columns=list('abcd')) 
print(df) 
# a b c d 
# 0 1 2 3 NaN 
# 1 2 3 4 5 

---

适用于您的情况，该代码可能看起来是这样的：

def readfiles(pattern, filenames): 
    for textfile in filenames: 
     with open(textfile, 'rb') as f: 
      for line in f: 
       try: 
        line = ujson.loads(line.rstrip('\n').rstrip(',')) 
        if pattern in line: 
         yield line['value'], line['value']['value'], line['value'], line['value']['value'], line['value']['value'] 
       except ValueError as err: 
        pass 

def convertodf(lines): 
    df = pd.DataFrame.from_records(lines) 
    df.to_csv('testdf2.csv', index=False, header=None) 

def main(pattern, filenames): 
    lines = readfiles(pattern, filenames) 
    convertodf(lines)