如何将队列中的项目与集合中的项目进行比较？

Question

REDIT：试图避免只是把对论坛整个代码块，说我对其进行修复，但在这里它是，简单地确定错误的过程：

#! /usr/bin/python2.6 
import threading 
import Queue 
import sys 
import urllib 
import urllib2 
from urlparse import urlparse 
from lxml.html import parse, tostring, fromstring 

THREAD_NUMBER = 1 


class Crawler(threading.Thread): 

def __init__(self, queue, mal_urls, max_depth): 
    self.queue = queue 
    self.mal_list = mal_urls 
    self.crawled_links = [] 
    self.max_depth = max_depth 
    self.count = 0 
    threading.Thread.__init__(self) 

def run(self): 
    while True: 
     if self.count <= self.max_depth: 
      self.crawled = set(self.crawled_links) 
      url = self.queue.get() 
      if url not in self.mal_list: 
       self.count += 1 
       self.crawl(url) 
      else: 
       #self.queue.task_done() 
       print("Malicious Link Found: {0}".format(url)) 
       continue 
     else: 
      self.queue.task_done() 
      break 
    print("\nFinished Crawling! Reached Max Depth!") 
    sys.exit(2) 

def crawl(self, tgt): 
    try: 
     url = urlparse(tgt) 
     self.crawled_links.append(tgt) 
     print("\nCrawling {0}".format(tgt)) 
     request = urllib2.Request(tgt) 
     request.add_header("User-Agent", "Mozilla/5,0") 
     opener = urllib2.build_opener() 
     data = opener.open(request) 

    except: # TODO: write explicit exceptions the URLError, ValueERROR ... 
     return 

    doc = parse(data).getroot() 
    for tag in doc.xpath("//a[@href]"): 
     old = tag.get('href') 
     fixed = urllib.unquote(old) 
     self.queue_links(fixed, url) 


def queue_links(self, link, url): 

    if link.startswith('/'): 
     link = "http://" + url.netloc + link 

    elif link.startswith("#"): 
     return 

    elif not link.startswith("http"): 
     link = "http://" + url.netloc + "/" + link 


    if link not in self.crawled_links: 
     self.queue.put(link) 
     self.queue.task_done() 
    else: 
     return 


def make_mal_list(): 
"""Open various malware and phishing related blacklists and create a list 
of URLS from which to compare to the crawled links 
""" 

hosts1 = "hosts.txt" 
hosts2 = "MH-sitelist.txt" 
hosts3 = "urls.txt" 

mal_list = [] 

with open(hosts1) as first: 
    for line1 in first: 
     link = "http://" + line1.strip() 
     mal_list.append(link) 

with open(hosts2) as second: 
    for line2 in second: 
     link = "http://" + line2.strip() 
     mal_list.append(link) 

with open(hosts3) as third: 
    for line3 in third: 
     link = "http://" + line3.strip() 
     mal_list.append(link) 

return mal_list 

def main(): 
    x = int(sys.argv[2]) 
    queue = Queue.Queue() 

    mal_urls = set(make_mal_list()) 
    for i in xrange(THREAD_NUMBER): 
     cr = Crawler(queue, mal_urls, x) 
     cr.start() 


    queue.put(sys.argv[1]) 

    queue.join() 


if __name__ == '__main__': 
    main() 

所以我”网络蜘蛛首先创建了一组包含“恶意链接”的文本文件的行。然后启动一个线程，传递一组坏链接和sys.argv [1]。启动的线程然后调用从sys.argv [1]中检索lxml.html解析的抓取函数，然后在解析出该初始页面之外的所有链接之后，将它们放入队列中。循环继续，每个链接放置在队列中，并用self.queue.get（）删除。然后将相应的链接与SUPPOSED进行比较，并与一组不良链接进行比较。如果链接发现不好，则应该将循环输出到屏幕，然后继续到下一个链接，除非它已经爬过该链接。

如果没有问题，抓取它，解析它，把它的链接放入队列等等，每次链接被抓取时递增一个计数器，直到计数器达到由作为sys传递的值.argv [2]。问题在于，它应该触发'if url not in mal_list'的if/else语句的项目不是，并且已放置在“crawled_already”列表中的链接正在被第2次，第3次和第4次抓取无论如何。

Answer 1

我不明白这个代码的一个细节：队列被标记为task_done如果在self.queue_links发现任何新的链接，但不作为当然的self.crawl的问题。我还以为，该代码会更有意义：

def crawl(self, tgt): 
    try: 
     url = urlparse(tgt) 
     self.crawled_links.append(tgt) 
     print("\nCrawling {0}".format(tgt)) 
     request = urllib2.Request(tgt) 
     request.add_header("User-Agent", "Mozilla/5,0") 
     opener = urllib2.build_opener() 
     data = opener.open(request) 
     doc = parse(data).getroot() 
     for tag in doc.xpath("//a[@href]"): 
      old = tag.get('href') 
      fixed = urllib.unquote(old) 
      self.queue_links(fixed, url) 
     self.queue.task_done() 
    except: # TODO: write explicit exceptions the URLError, ValueERROR ... 
     pass 

def queue_links(self, link, url): 
    if not link.startswith("#"): 
     if link.startswith('/'): 
      link = "http://" + url.netloc + link 
     elif not link.startswith("http"): 
      link = "http://" + url.netloc + "/" + link 
     if link not in self.crawled_links: 
      self.queue.put(link) 

我不能说，不过，我有一个完整的回答你的问题。

---

后来：在docs为Queue.task_done建议task_done应为1：1 Queue.get电话：

Queue.task_done()¶

Indicate that a formerly enqueued task is complete. Used by queue consumer threads. For each get() used to fetch a task, a subsequent call to task_done() tells the queue that the processing on the task is complete.

If a join() is currently blocking, it will resume when all items have been processed (meaning that a task_done() call was received for every item that had been put() into the queue).

Raises a ValueError if called more times than there were items placed in the queue.

是你得到[未捕获] ValueError异常？看起来这可能是这样的。