当我使用flinks流API写S3:弗林克S3写入失败无法从任何供应商加载链AWS凭据
// Set StreamExecutionEnvironment
final StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
// Set checkpoints in ms
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
// Add source (input stream)
DataStream<String> dataStream = StreamUtil.getDataStream(env, params);
// Sink to S3 Bucket
dataStream.writeAsText("s3a://test-flink/test.txt").setParallelism(1);
我收到以下错误:
Unable to load AWS credentials from any provider in the chain
我配置为:
# flink --version
Version: 1.3.1, Commit ID: 1ca6e5b
Hadoop的config目录加入到弗林克-conf.yaml
# cat flink/config/flink-conf.yaml | head -n1
fs.hdfs.hadoopconf: /root/hadoop-config
flink-conf.yaml的其余内容与发行版本相同。加入
下到/root/hadoop-config/core-site.xml
# cat /root/hadoop-config/core-site.xml
<configuration>
<property>
<name>fs.s3a.impl</name>
<value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
</property>
<property>
<name>fs.s3a.buffer.dir</name>
<value>/tmp</value>
</property>
<property>
<name>fs.s3a.access.key</name>
<value>MY_ACCESS_KEY</value>
</property>
<property>
<name>fs.s3a.secret.key</name>
<value>MY_SECRET_KEY</value>
</property>
</configuration>
JAR的AWS-Java的SDK-1.7.4.jar,Hadoop的AWS-2.7.4。其中加入弗林克/ LIB /从http://apache.mirror.anlx.net/hadoop/common/hadoop-2.7.4/hadoop-2.7.4.tar.gz
# ls flink/lib/
aws-java-sdk-1.7.4.jar
flink-dist_2.11-1.3.1.jar
flink-python_2.11-1.3.1.jar
flink-shaded-hadoop2-uber-1.3.1.jar
hadoop-aws-2.7.4.jar
httpclient-4.2.5.jar
httpcore-4.2.5.jar
log4j-1.2.17.jar
slf4j-log4j12-1.7.7.jar
注罐,HttpClient的-4.2.5.jar,的HttpCore-4.2.5.jar的AWS-java的SDK-1.7.4.jar是1.7.4而不是1.7.2,因为它在文档中here
pom.xml具有以下构建依赖关系。
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-filesystem_2.10</artifactId>
<version>1.3.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
<version>2.7.2</version>
</dependency>
我的基准是(https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/aws.html#set-s3-filesystem)
我能够写使用核心与site.xml中的awscli凭据S3桶。
谢谢维奈,后'jar xf jar-file'这个JAR文件,然后'find。 -name“* .xml”',我只能找到./core-default.xml,更新这个文件并不能解决问题,我错过了什么? – 64k
应该将'core-site.xml'放置在jar的根目录下还是放在哪里? – 64k
这意味着您正在运行的节点配置不正确。您可以请检查您正在运行flink作业的节点上是否存在.aws/credentials文件。如果不存在,您将不得不运行aws s3 configure命令,如下所示http://docs.aws.amazon.com/cli/latest/reference/configure/ –