0
Spark Spark RDD具有saveAsTxtFile
功能。但是,我如何打开一个文件并向Hadoop商店写入一个简单的字符串?在Spark中,如何在没有RDD的情况下在Hadoop上编写文件?
val sparkConf: SparkConf = new SparkConf().setAppName("example")
val sc: SparkContext = new SparkContext(sparkConf)
sc.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", "...")
sc.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey", "...")
val lines: RDD[String] = sc.textFile("s3n://your-output-bucket/lines.txt")
val lengths: RDD[Int] = lines.map(_.length)
lengths.saveAsTextFile("s3n://your-output-bucket/lenths.txt")
val numLines: Long = lines.count
val resultString: String = s"numLines: $numLines"
// how to save resultString to "s3n://your-output-bucket/result.txt"
sc.stop()
thx。我可以使用“s3n://your-output-bucket/result.txt”网址,而不是“hdfs:// localhost:9000 // tmp/hello.txt”吗? –