我有一个需要访问DynamoDB表的应用程序。每个工作人员自己建立与数据库的连接。从Spark Worker访问环境变量
我已将AWS_ACCESS_KEY_ID
和AWS_SECRET_ACCESS_KEY
同时添加到主人和工作人员spark-env.sh
文件中。我还使用sh
运行该文件以确保变量已导出。
当代码运行时,我总是得到错误:
Caused by: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:131)
at com.amazonaws.http.AmazonHttpClient.getCredentialsFromContext(AmazonHttpClient.java:774)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:800)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:695)
at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:447)
at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:409)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:358)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2051)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2021)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.describeTable(AmazonDynamoDBClient.java:1299)
at com.amazon.titan.diskstorage.dynamodb.DynamoDBDelegate.describeTable(DynamoDBDelegate.java:635)
... 27 more
看来,AWS SDK未能加载即使它们导出的证书。我应该尝试什么类型的解决方案?
你可以尝试在代码中明确地设置它吗? http://stackoverflow.com/questions/33475931/spark-streaming-checkpoint-to-amazon-s3 – Knight71
@ Knight71,我可以但不是不安全? –