如何知道Spark应用程序的执行者IP？

Question

我想在运行时获取执行程序的所有IP地址，我应该使用Spark中的哪个API？或者在运行时获取IP的任何其他方法？

Answer 1

Apache Spark中有一个类，即ExecutorInfo，它具有返回执行主机IP的方法executorHost（）。

Answer 2

您应该使用SparkListener抽象类并拦截两个执行程序特定事件 - SparkListenerExecutorAdded和SparkListenerExecutorRemoved。

override def onExecutorAdded(executorAdded: SparkListenerExecutorAdded): Unit = { 
    val execId = executorAdded.executorId 
    val host = executorAdded.executorInfo.executorHost 
    executors += (execId -> host) 
    println(s">>> executor id=$execId added on host=$host") 
} 

override def onExecutorRemoved(executorRemoved: SparkListenerExecutorRemoved): Unit = { 
    val execId = executorRemoved.executorId 
    val host = executors remove execId getOrElse "Host unknown" 
    println(s">>> executor id=$execId removed from host=$host") 
} 

整个工作项目在我的Spark Executor Monitor Project。