Python，删除文件中的特定长行文本

Question

我只在过去的8个月里用Python进行了编程，所以请原谅我对Python的noob方法。

我的问题是以下，我希望有人能帮我解决。

我有很多数据在一个文件中，例如像这样（只是剪断）：

SWITCH MGMT IP;SWITCH HOSTNAME;SWITCH MODEL;SWITCH SERIAL;SWITCH UPTIME;PORTS NOT IN USE 
10.255.240.1;641_HX_3560X;WS-C3560X-24P-S;FDO1601V031;12 weeks, 3 days, 23 hours, 33 minutes;1 
10.255.240.7;641_HX_LEFT_2960x;WS-C2960X-24PS-L;FOC1750S2E5;12 weeks, 4 days, 7 minutes;21 
10.255.240.8;641_UX_BASEMENT_2960x;WS-C2960X-24PS-L;FOC1750S2AG;12 weeks, 4 days, 7 minutes;12 
10.255.240.9;641_UX_SPECIAL_2960x;WS-C2960X-24PS-L;FOC1750S27M;12 weeks, 4 days, 8 minutes;25 
10.255.240.2;641_UX_OFFICE_3560;WS-C3560-8PC-S;FOC1202U24E;2 years, 30 weeks, 3 days, 16 hours, 43 minutes;2 
10.255.240.3;641_UX_SFO_2960x;WS-C2960X-24PS-L;FOC1750S2BR;12 weeks, 4 days, 7 minutes;14 
10.255.240.65;641_HX_3560X;WS-C3560X-24P-S;FDO1601V031;12 weeks, 3 days, 23 hours, 34 minutes;1 
10.255.240.5;641_HX_RIGHT_2960s;WS-C2960S-24PS-L;FOC1627X1BF;12 weeks, 4 days, 12 minutes;16 
10.255.240.6;641_HX_LEFT_2960x-02;WS-C2960X-24PS-L;FOC1750S2C4;12 weeks, 4 days, 7 minutes;15 
10.255.240.4;641_UX_BASEMENT_2960s;WS-C2960S-24PS-L;FOC1607Z27T;12 weeks, 4 days, 8 minutes;3 
10.255.240.62;641_UX_OFFICE_3560CG;WS-C3560CG-8PC-S;FOC1646Y0U2;15 weeks, 5 days, 12 hours, 15 minutes;6 

我想通过文件中的所有数据，如果出现更多的序列号来运行和检查比一次。如果是这样，我想删除找到的重复。结果可能多次包含相同交换机或路由器的原因是它可能有多个第3层接口，可以在其中进行管理。

所以在上面的例子中。由于文件中的第二行已包含在同一个交换机和序列号

10.255.240.65;641_HX_3560X;WS-C3560X-24P-S;FDO1601V031;12 weeks, 3 days, 23 hours, 34 minutes;1 

：我已经通过数据运行后，它会删除线。

我花了好几天的时间试图弄清楚，如何实现这一点，它开始让我头疼。

我的基本代码如下所示：

if os.stat("output.txt").st_size != 0: 
    with open('output.txt','r') as file: 
     header_line = next(file) # Start from line 2 in the file. 

    data = [] # Contains the data from the file. 
    sn = [] # Contains the serial numbers to check up against. 
    ok = [] # Will contain the clean data with no duplicates. 

    data.append(header_line.split(";")) # Write the head to data. 

    for line in file: # Run through the file data line for line. 
     serialchk = line.split(";") # Split the data into a list 
     data.append(serialchk) # Write the data to data list. 
     sn.append(serialchk[3]) # Write the serial number to sn list. 

end = len(data) # Save the length of the data list, so i can run through the data 
i = 0 # For my while loop, so i know when to stop.' 

while i != end: # from here on out i am pretty lost on how to achieve my goal. 
     found = 0 
     for x in range(len(data)): 
      if sn[i] == data[x][3]: 
       found += 1 
       print data[x] 
       ok.append(data[x]) 
      elif found > 1: 
       print "Removing:\r\n" 
       print data[x-1] 
       del ok[-1] 
       found = 0 
     i += 1 

有没有更Python的方式做到这一点？我非常确定这里所有的人才，有人可以告诉我如何实现这一目标的线索。

非常感谢您提前。

Answer 1

我的建议：

if os.stat("output.txt").st_size != 0: 
    with open('output.txt','r') as file: 
     header_line = next(file) # Start from line 2 in the file. 

    srn = set() # create a set where the seen srn will be stored 
    ok = [] # Will contain the clean data with no duplicates. 

    ok.append(header_line.split(";")) # Write the head to ok. 

    for line in file: # Run through the file data line for line. 
     serialchk = line.split(";") # Split the data into a list 
     if serialchk[3] not in srn: # if the srn hasn't be seen 
      ok.append(serialchk) # add the row to ok 
      srn.add(serialchk[3]) # add the srn to seen set 
     else: # if the srn has already be seen 
      print "Removing: "+";".join(serialchk) # notify the user it has been skipped 

最终你会与仅含有uniq的SRN行确定，并打印已删除的行 希望它可以帮助

Answer 2

你正在做的方式更加复杂比它必须是，而不是内存友好（你不必在内存中加载整个文件来过滤重复项）。

简单的方法是逐行读取文件，每行检查是否已经看到序列号。如果是，则跳到行，否则存储的序列号和行写入到输出文件：

seen = set() 
with open('output.txt','r') as source, open("cleaned.txt", "w") as dest: 
    dest.write(next(source)) # Start from line 2 in the file. 
    for line in src: 
     sn = line.split(";")[3] 
     if sn not in seen: 
      seen.add(sn) 
      dest.write(line) 
     # else, well we just ignore the line ;) 

注：我假设你想要写回重复数据删除线到文件中。如果您想将它们保留在内存中，则算法大部分是相同的，只需将重复数据删除行附加到list，但是如果您有大量文件，请注意内存使用情况。

Answer 3

我会引导你完成我将要做的改变。

我会做的第一件事就是使用csv模块来解析输入。既然你可以遍历DictReader，为了简洁起见，我也选择了这个。 listdata将包含最终（清理）的结果。

from csv import DictReader 
import os 

if os.stat("output.txt").st_size != 0: 
    with open('output.txt', 'r') as f: 
     reader = DictReader(f, delimiter=';') # create the reader instance 

     serial_numbers = set() 
     data = [] 
     for row in reader: 
      if row["SWITCH HOSTNAME"] in serial_numbers: 
       pass 
      else: 
       data.append(row) 
       serial_numbers.add(row["SWITCH HOSTNAME"]) 

数据的格式将通过我的方法已经改变，从list个list到dict的list一个S，但如果你想在清理的数据保存到一个新的CSV文件时，DictWriter类应该是一个简单的方法来做到这一点。