在预先存在的sqllite表中防止与sqlalchemy重复的条目

Question

我有一个预先存在的sqllite表，我正在使用sqlalchemy访问。我意识到存在许多重复的“案例”数字。如果我理解正确的话，似乎你不能表的创建与sqllite后唯一约束添加到表中，除去使用复本后：

DELETE FROM mytable 
WHERE id NOT IN 
(
SELECT MIN(id) 
FROM judgements 
GROUP BY "case" 

我决定使用SQLAlchemy防止添加额外的DUP。我与scrapy工作，并有管线元件，看起来像：

class DynamicSQLlitePipeline(object): 

    def __init__(self,table_name): 
     db_path = "sqlite:///"+settings.SETTINGS_PATH+"\\data.db" 
     _engine = create_engine(db_path) 
     _connection = _engine.connect() 
     _metadata = MetaData() 
     _stack_items = Table(table_name, _metadata, 
          Column("id", Integer, primary_key=True), 
          Column("case", Text , unique=True), 
           ....) 
     _metadata.create_all(_engine) 
     self.connection = _connection 
     self.stack_items = _stack_items 



    def process_item(self, item, spider): 

     try: 
      ins_query = self.stack_items.insert().values(
      case=item['case'], 
      .... 
      ) 
      self.connection.execute(ins_query) 
     except IntegrityError: 
       print('THIS IS A DUP') 
     return item 

我所做的唯一变化是唯一=真加列“案例”。但是在测试中，dups仍然被添加/我如何才能使这个工作？

Answer 1

下面的代码片段使用python版本2.7和sqlalchemy版本1.0.9和sqlite版本3.15.2工作。

from sqlalchemy import create_engine, MetaData, Column, Integer, Table, Text 
from sqlalchemy.exc import IntegrityError 


class DynamicSQLlitePipeline(object): 

    def __init__(self, table_name): 
     db_path = "sqlite:///data.db" 
     _engine = create_engine(db_path) 
     _connection = _engine.connect() 
     _metadata = MetaData() 
     _stack_items = Table(table_name, _metadata, 
          Column("id", Integer, primary_key=True), 
          Column("case", Text, unique=True),) 
     _metadata.create_all(_engine) 
     self.connection = _connection 
     self.stack_items = _stack_items 

    def process_item(self, item): 

     try: 
      ins_query = self.stack_items.insert().values(case=item['case']) 
      self.connection.execute(ins_query) 
     except IntegrityError: 
       print('THIS IS A DUP') 
     return item 

if __name__ == '__main__': 

    d = DynamicSQLlitePipeline("pipeline") 
    item = { 
     'case': 'sdjwaichjkneirjpewjcmelkdfpoewrjlkxncdsd' 
    } 
    print d.process_item(item) 

和输出为第二次运行将是这样的：

THIS IS A DUP 
{'case': 'sdjwaichjkneirjpewjcmelkdfpoewrjlkxncdsd'} 

我没有看到你的代码逻辑多大区别。唯一的区别可能是我猜的版本。