2016-10-04 142 views
0

我研究了许多pool.map并仍然无法找到任何提示我的问题。Python Pool.map() - 在本地工作,在服务器上失败

我在每个.py文件中都有if __name__ == '__main__'。我在每个包含import multiprocessing的.py文件中都有freeze_support(),我仍然对发生的事情感到不知所措。我已经在我的代码中移动了freeze_support(),并得到了相同的不成功结果。

脚本A调用脚本B,脚本B调用脚本C(发生多处理的地方)。本地这个场景完美地工作,但是当我将它加载到Windows Server 2008计算机时,奇怪的事情开始发生。

在服务器上,我可以看到第一个迭代打印到解释器,但它然后跳回到脚本B并继续处理。有列表中的脚本C.

脚本B代码51个的其他项目:

if not arcpy.Exists(MergedDataFC): 
    ScriptC.intersect_main(input1, input2) 

if not arcpy.Exists(MergedDataSHP): 
    shpList = arcpy.ListFields(*.shp) # output of multiprocess 
    # Merge all shapefiles into single shapefile 
    # Being executed before the multiprocess finishes all 52 items 

脚本的C代码:

import multiprocessing as mp 

def intersect_main(input1,input2):  
try: 
    mp.freeze_support() 
    # Create a list of states for input1 polygons 
    log.log("Creating Polygon State list...") 
    fldList = arcpy.ListFields(input1) 
    flds = [fld.name for fld in fldList] 
    idList = [] 
    with arcpy.da.SearchCursor(input1, flds) as cursor: 
     for row in cursor: 
      idSTATE = row[flds.index("STATE")] 
      idList.append(idSTATE) 

    idList = set(idList) 
    log.log("There are " + str(len(idList)) + " States (polygons) to process.") 

    log.log("Sending to pool") 
    # declare number of cores to use, use 1 less than the max 
    cpuNum = mp.cpu_count() -1 

    # Create the pool object 
    pool = mp.Pool(processes=cpuNum) 

    # Fire off list to worker function. 
    # res is a list that is created with what ever the worker function is returning 
    log.log ("Entering intersectWork") 
    res = pool.map((intersectWork(input1, input2, idSTATE)),idList) 
    pool.close() 
    pool.join() 

    # If an error has occurred report it 
    if False in res: 
     log.log ("A worker failed!") 
     log.log (strftime('[%H:%M:%S]', localtime())) 
     raise Exception 
    else: 
     log.log("Finished multiprocessing!") 
     log.log (strftime('[%H:%M:%S]', localtime())) 
except Exception, e: 
    tb = sys.exc_info()[2] 
    # Geoprocessor threw an error 
    log.log("An error occurred on line " + str(tb.tb_lineno)) 
    log.log (str(e)) 

def intersectWork(input1,input2, idSTATE): 
try: 
    if idSTATE == None: 
     query = "STATE IS NULL" 
     idSTATE = 'pr' 
    else: 
     query = "STATE = '" + idSTATE + "'" 

    DEMOlayer = arcpy.MakeFeatureLayer_management(input1,"input1_" + idSTATE) 

    log.log (query) 
    arcpy.SelectLayerByAttribute_management(DEMOlayer,"NEW_SELECTION",query) 

    # Do the Intersect 
    outFC = r'C:/EclipseWorkspace' + '/INTER_' + idSTATE.upper() + '.shp' 
    strIntersect = str(DEMOlayer) + ";" + str(input2) 
    arcpy.Intersect_analysis(strIntersect, outFC, "ALL", "", "LINE") 
    return True 
except: 
    # Some error occurred so return False 
    log.log(arcpy.GetMessage(2)) 
    return False 

if __name__ == '__main__': 
    intersect_main(input1, input2) 

编辑

上的所有数据服务器存储在本地,没有跨网络处理。

回答

0

问题是数据的完整路径没有正确传递到服务器上的pool.map(),从以前的模块。我不得不在import语句下添加所有文件路径。不是很优雅,但它的工作。

相关问题