scrapy在上传到db时抛出一个错误

我得到的错误消息是：exceptions.TypeError：'NoneType'对象没有属性'getitem'。这里是我的代码：scrapy在上传到db时抛出一个错误

# -*- coding: utf-8 -*- 
import scrapy,MySQLdb,codecs 
from scrapy import Request 

class MyItems(scrapy.Item): 
    topLinks = scrapy.Field() #links at top of home page to grab artist page 
    artists = scrapy.Field() # list of all the artists 
    artists_urls = scrapy.Field() #url of the page for the artist + songs 
    song_name = scrapy.Field() #name of each song 
    song_duration = scrapy.Field() #duration of the song, duh 
    song_dl = scrapy.Field() #dl link for the proxy site for songs 
    rd = scrapy.Field() 


class UpdaterSpider(scrapy.Spider): 
    name = "updater" 
    allowed_domains = [ 
     'myfreemp3.cc', 'unref.eu', 'safeurl.eu' 
    ] 
    start_urls = (
     'http://www.example.com/', 
    ) 

    def __init__(self, *a, **kw): 
     super(UpdaterSpider, self).__init__(*a, **kw) 

     self.item = MyItems() 
     self.db = MySQLdb.connect(#setup my SQL database info 
      "127.0.0.1", #host 
      "root", #user 
      "uconn3", #password 
      "Music" #database 
     ) 

     self.cursor = self.db.cursor() #create a cursor to execute SQL statements 

     self.item = MyItems() 
     self.y = 0 

    def parse(self, response): 
     sql = "SELECT * FROM Songs" 

     self.cursor.execute(sql) 

     result = self.cursor.fetchone() 

     while result is not None: 
      yield Request(url=result[3], callback= lambda r: self.parse_dl(r, result[1], result[2])) 

      result = self.cursor.fetchone() 

    def parse_dl(self, response, songname, duration): 
     self.item['rd'] = response.xpath("//head/script/text()").extract() 

     for x in range(len(self.item['rd'])): 
      self.item['rd'][x] = self.item['rd'][x].split('"')[1] 


      sql = "INSERT INTO Songs (" \ 
       "Artist, songName, Duration, URL, Genre) " \ 
       "VALUES ('Placeholder', '%s', '%s', '%s', 'Placeholder')" % \ 
       (songname.decode('utf-8'), duration.decode('utf-8'), self.item['rd'][x]) 

      self.cursor.execute(sql) 

      self.db.commit()

当我删除的sql语句行的错误是固定的，但我想不通这是为什么事情搞乱了。

EDIT1：回溯

Traceback (most recent call last): 
     File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop 
     self.runUntilCurrent() 
     File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilCurrent 
     call.func(*call.args, **call.kw) 
     File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382, in callback 
     self._startRunCallbacks(result) 
     File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks 
     self._runCallbacks() 
    --- <exception caught here> --- 
     File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks 
     current.result = callback(current.result, *args, **kw) 
     File "/home/user/PycharmProjects/untitled/mp3_scraper/mp3_scraper/spiders/updater.py", line 48, in <lambda> 
     yield Request(url=result[3], callback= lambda r: self.parse_dl(r, result[1], result[2])) 
    exceptions.TypeError: 'NoneType' object has no attribute '__getitem__'

来源

2014-10-03 johnc31

当你从一个循环返回lambda时，你并没有捕获到你认为你所做的事情，请参阅http://stackoverflow.com/questions/7514093/lambda-function-dont-closure-the-p arameter-in-python或http://stackoverflow.com/questions/2295290/what-do-lambda-function-closures-capture-in-python。 – immerrr 2014-10-03 05:02:14

ahhh yep，得使用scrapy提供的词典元。 – johnc31 2014-10-03 05:22:48

它看起来像

self.item = MyItems()

没有它由括号[] 我想调用的方法的GetItem - 问题是不是与数据库，请提供全部回溯

来源

2014-10-03 04:37:12

k上传了traceback – johnc31 2014-10-03 04:40:17

对不起 - 我没有机会在env中运行此代码。我可以说的唯一一件事 - 问题不在数据库中，它也在大括号中调用也是 - 在异步程序中使用阻塞数据库调用是不好的做法 - 使用tx-mongo + mongo可以节省更多时间 – 2014-10-03 04:55:03

scrapy在上传到db时抛出一个错误

回答

相关问题