我在Bluemix中的Spark服务中使用pySpark来转换我的数据中的某些内容,然后将它写入DashDB中,也是在Bluemix中。但是当我尝试加载数据时,我收到以下错误:bluemix pyspark在调用jdbc时发生错误
Py4JJavaError:调用o111.jdbc时发生错误。 :org.apache.spark.SparkException:由于阶段失败导致作业中止:阶段4.0中的任务0失败10次,最近失败:阶段4.0中丢失的任务0.9(TID 23,yp-spark-dal09-env5-0045) :com.ibm.db2.jcc.am.BatchUpdateException:[jcc] [t4] [102] [10040] [4.19.26]批量失败。该批次已提交,但该批次的单个成员中至少发生一次异常。 使用getNextException()检索特定批处理元素的异常。 ERRORCODE = -4229,SQLSTATE = null
我已经尝试创建一个新的DashDB,但收到相同的错误。然后,我试图创建另一个笔记本,但它也不起作用。我试图停止所有内核,并且每次只运行一个内核,都不起作用。
我试图从数据框中写入数据的所有笔记本都收到相同的错误,但在相同的情况下,数据已加载,而其他数据不是。
这是我用来写代码:
for num in range(0,22):
(shared_df_first
.select(
Func.col('Customer').alias("CUSTOMER"),
Func.col('Customer Environment').alias("CUST_ENV"),
Func.col('Device').alias("DEVICE"),
Func.col('User ID').alias("USER_ID"),
Func.col('Date').alias("DATE"),
Func.col('Time').alias("TIME"),
Func.col('Requester').alias("REQUESTER"),
Func.col('Manager').alias("MANAGER"),
Func.col('Manager Mail').alias("MANAGER_MAIL"),
Func.col('Ticket').alias("TICKET"),
Func.col('Request').alias("REQUEST"),
Func.col('Teste').alias("TESTE"),
Func.col('Approver USERID').alias("APPROVER_USERID"),
Func.col('Approver Name').alias("APPROVER_NAME"),
Func.col('Period in hours').alias("PERIOD"),
Func.col('Business Need').alias("BUSINESS_NEED"),
Func.col('Password Periodic Changable').alias("PASSWORD_PERIODIC_CHANGABLE"),
Func.col('Is Pre Approved?').alias("IS_PRE_APPROVED"),
Func.col('Has Personal User ID?').alias("HAS_PERSONAL_USER_ID"),
Func.col('Check in way').alias("CHECK_IN_WAY"),
Func.col('SLA').alias("SLA"),
Func.col('Invalid Business Need').alias("BUSINESS_NEED_INVALID")
)
.write
.jdbc("jdbc:db2://bluemix05.bluforcloud.com:50000/BLUDB", "DASH014638.WATSON_UAT_DEV", "append", propertiesDBDash
)
print num + 1
df = ds_clean.toDF(["account_id","customer","device_name","device_os","user_id","user_id_type","creation_date","last_logon","password_is_never_expires","responsible",
"privileges","user_id_status"])
propertiesDBDash = {
"user":"dash014638",
"password":"pwd"}
df.write.mode("append").jdbc("jdbc:db2://bluemix05.bluforcloud.com:50000/BLUDB", "DASH014638.DORMANT_PROD",properties=propertiesDBDash)
df = ds_clean.toDF(["REQUEST_NUMBER","TYPE_TICKET","SOLUTIONER","CUSTOMER","DELIVERY","OPEN_DATE","OPEN_TIME","CLOSE_DATE","CLOSE_TIME","SERVICE","DEVICE","PLATFORM","REQUESTER","REQUESTER_MANAGER_MAIL","SLA","ELAPSED_TIME","SLA_STATUS","URGENCY","ACTION","REQUEST_STATUS"])
propertiesDBDash = {
"user":"dash014638",
"password":"pwd"}
df.write.mode("append").jdbc("jdbc:db2://bluemix05.bluforcloud.com:50000/BLUDB", "DASH014638.WATSON_REQUEST_NEW",properties=propertiesDBDash)