使用lxml和xpath加速xml解析过程

Question

<?xml version="1.0" encoding="UTF-8" standalone="yes"?> 
    <document DateTime="2017-06-23T04:27:08.592Z"> 
     <PeakInfo No="1" mz="505.2315648572003965" 
       Intensity="4531.0000000000000000" 
       Rel_Intensity="3.2737729673489735" 
       Resolution="1879.5638812957554364" 
       SNR="14.0278637770897561" 
       Area="1348.1007591467391649" 
       Rel_Area="2.3371194184605959" 
       Index="238.9999999999976694"/> 
     <PeakInfo No="2" mz="522.1330917856538463" 
       Intensity="3382.0000000000000000" 
       Rel_Intensity="2.4435886505350317" 
       Resolution="3502.9921209527169594" 
       SNR="10.4705882352940982" 
       Area="881.4468100654634100" 
       Rel_Area="1.5281101521284057" 
       Index="925.0000000000000000"/> 
    </document> 

上面是我最近一直在使用的xml文件的一部分。每个文件都包含超过400分PeakInfo的，我也做一个Python脚本来分析每个文件：

from lxml import etree 
import pandas as pd 
import tkinter.filedialog 
import os 
import pandas.io.formats.excel 

full_path = tkinter.filedialog.askdirectory(initialdir='.') 
newfolder = full_path+'\\xls files' 
os.chdir(full_path) 
os.makedirs(newfolder) 

data = {} 
for files in os.listdir(full_path): 
     if os.path.isfile(os.path.join(full_path, files)): 
      plist = pd.DataFrame() 
      filename = os.path.basename(files).rpartition('.')[0] 

      if len(filename) == 2: 
       filename = filename[:1]+'0'+filename[1:] 

      xmlp = etree.parse(files) 
      for p in xmlp.xpath('//PeakInfo'): 
       data['Exp. m/z'] = p.attrib['mz'] 
       data['Intensity'] = p.attrib['Intensity'] 
       plist = plist.append(data, ignore_index=True) 
       plist['Exp. m/z'] = plist['Exp. m/z'].astype(float) 
       plist['Exp. m/z'] = plist['Exp. m/z'].map('{:.4f}'.format) 
       plist['Intensity'] = plist['Intensity'].astype(float) 
       plist['Intensity'] = plist['Intensity'].map('{:.0f}'.format) 
       pandas.io.formats.excel.header_style = None 
       plist.to_excel(os.path.join(newfolder, filename+'.xls'),index=False) 

这段代码改变，如果它只有两个字符（即A1至A01）的文件名，然后再换MZ和强度并保存为xls文件。问题是解析每个文件需要很长时间。是否有任何提示显着加快过程的技巧？

Answer 1

from lxml import etree 
import pandas as pd 
import tkinter.filedialog 
import os 
import pandas.io.formats.excel 

full_path = tkinter.filedialog.askdirectory(initialdir='.') 
newfolder = full_path+'\\xls files' 
os.chdir(full_path) 
os.makedirs(newfolder) 

data = {} 
for files in os.listdir(full_path): 
     if os.path.isfile(os.path.join(full_path, files)): 
      plist = pd.DataFrame() 
      filename = os.path.basename(files).rpartition('.')[0] 

      if len(filename) == 2: 
       filename = filename[:1]+'0'+filename[1:] 

      xmlp = etree.parse(files) 
      for p in xmlp.xpath('//PeakInfo'): 
       data['Exp. m/z'] = p.attrib['mz'] 
       data['Intensity'] = p.attrib['Intensity'] 
       plist = plist.append(data, ignore_index=True) 
plist['Exp. m/z'] = plist['Exp. m/z'].astype(float) 
plist['Exp. m/z'] = plist['Exp. m/z'].map('{:.4f}'.format) 
plist['Intensity'] = plist['Intensity'].astype(float) 
plist['Intensity'] = plist['Intensity'].map('{:.0f}'.format) 
pandas.io.formats.excel.header_style = None 
plist.to_excel(os.path.join(newfolder, filename+'.xls'),index=False) 

只是改变了空间，你的代码一样to_excel执行了太多的时间，它的速度慢，而“astype”将复制元素，并采取过多的内存，然后放慢速度。

Answer 2

你可以尝试使用C语言实现的ElementTree http://effbot.org/zone/celementtree.htm

的按照DOC比 的ElementTree的Python版本更快

cElementTree是15-20倍，并且使用2-5倍的内存更少。