Python简单的多线程下载文件损坏

Question

这是我的第一篇文章。我一直在进行python编程，最近正在开发一个多线程下载器。但问题是我的文件（jpg是我的目标）被损坏。另外随着followinf输入：http://www.aumathletics.com/images_web/headerAUMLogo.jpg

它显示错误

，而与输入： http://www.nasa.gov/images/content/607800main_kepler1200_1600-1200.jpg

的文件被损坏。

下面是代码： -

import os, sys, requests 
import threading 
import urllib2 
import time 

URL = sys.argv[1] 

def buildRange(value, numsplits): 
    lst = [] 
    for i in range(numsplits): 
    if i == 0: 
     lst.append('%s-%s' % (i, int(round(1 + i * value/(numsplits*1.0) + value/(numsplits*1.0)-1, 0)))) 
    else: 
     lst.append('%s-%s' % (int(round(1 + i * value/(numsplits*1.0),0)), int(round(1 + i * value/(numsplits*1.0) + value/(numsplits*1.0)-1, 0)))) 
return lst 

def main(url=None, splitBy=5): 
    start_time = time.time() 
    if not url: 
     print "Please Enter some url to begin download." 
     return 

fileName = "image.jpg" 
sizeInBytes = requests.head(url, headers={'Accept-Encoding': 'identity'}).headers.get('content-length', None) 
print "%s bytes to download." % sizeInBytes 
if not sizeInBytes: 
    print "Size cannot be determined." 
    return 

dataDict = {} 

# split total num bytes into ranges 
ranges = buildRange(int(sizeInBytes), splitBy) 

def downloadChunk(idx, irange): 
    req = urllib2.Request(url) 
    req.headers['Range'] = 'bytes={}'.format(irange) 
    dataDict[idx] = urllib2.urlopen(req).read() 

# create one downloading thread per chunk 
downloaders = [ 
    threading.Thread(
     target=downloadChunk, 
     args=(idx, irange), 
    ) 
    for idx,irange in enumerate(ranges) 
    ] 

# start threads, let run in parallel, wait for all to finish 
for th in downloaders: 
    th.start() 
for th in downloaders: 
    th.join() 



print 'done: got {} chunks, total {} bytes'.format(
    len(dataDict), sum((
     len(chunk) for chunk in dataDict.values() 
    )) 
) 

print "--- %s seconds ---" % str(time.time() - start_time) 

if os.path.exists(fileName): 
    os.remove(fileName) 



# reassemble file in correct order 
with open(fileName, 'w') as fh: 

    for _idx,chunk in sorted(dataDict.iteritems()): 
     fh.write(chunk) 

print "Finished Writing file %s" % fileName 
print 'file size {} bytes'.format(os.path.getsize(fileName)) 

if __name__ == '__main__': 
    main(URL) 

这里的缺口可能是错了，所以这里是代码引擎收录（点）的COM/wGEkp878

我会很感激，如果有人能指出错误

编辑：由人提议

def buildRange(value, numsplits): 
    lst = [] 
    for i in range(numsplits): 
     first = i if i == 0 else buildRange().start(i, value, numsplits) 
     second = buildRange().end(i, value, numsplits) 
     lst.append("{}-{}".format(first, second)) 
    return lst 

谁能告诉我锄保持part1 part2等名称下载的零件文件等

Answer 1

事实证明，文件必须以二进制模式打开，'wb'而不是'w'。如果用'w'打开，会写入一堆额外的字符。这与derpy windows与linux新行语义有关。如果你使用'wb'，它会准确地写入你放入文件的内容。

编辑： 如果要存储个人文件部分可以改变

# reassemble file in correct order 
with open(fileName, 'w') as fh: 
    for _idx,chunk in sorted(dataDict.iteritems()): 
     fh.write(chunk) 

print "Finished Writing file %s" % fileName 
print 'file size {} bytes'.format(os.path.getsize(fileName)) 

要

# reassemble file in correct order 
for _idx,chunk in sorted(dataDict.iteritems()): 
    with open(fileName + str(".part-") + str(_idx), 'wb') as fh: 
     fh.write(chunk) 

print "Finished Writing file %s" % fileName 
#print 'file size {} bytes'.format(os.path.getsize(fileName))