我们有一个运行火花命令,但我们正面临着一些磁盘空间问题 所以我们想利用分配纱构造的火花命令EMR
conf,yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-
percentage=99.0
,但我们得到了我们的日志Warning: Ignoring non-spark config property: yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage=99.0
?
我们如何得到它使火花借此配置
""Spark_Command": "command-runner.jar,spark-submit, --class,com.go.ccc.ew.modules.fred.fmodel, --name,\\\"f module Module\\\", --master,yarn, --deploy-mode,client, --executor-memory,36G, --executor-cores,4, --conf,spark.sql.shuffle.partitions=640, --conf,spark.yarn.am.memoryOverhead=5120, --conf,yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk- percentage=99.0,/home/hadoop/linking.jar,jobId=# {myJobId},environment=dev"