在mesos上运行kafka发布消息的问题DCOS

Question

我有一个运行DCOS的小型集群。我可以在this指南中成功安装kafka。运行

$ dcos kafka connection 

给

{ 
    "address": [ 
    "10.131.17.126:9475", 
    "10.131.24.6:9655", 
    "10.131.14.192:9181" 
    ], 
    "zookeeper": "master.mesos:2181/dcos-service-kafka", 
    "dns": [ 
    "broker-0.kafka.mesos:9475", 
    "broker-1.kafka.mesos:9655", 
    "broker-2.kafka.mesos:9181" 
    ] 
} 

我可以创建主题，我已经研究饲养员用CLI工具和状态似乎不错

get /dcos-service-kafka/brokers/ids/0 
{"jmx_port":-1,"timestamp":"1474206074029","endpoints":["PLAINTEXT://10.131.17.126:9475"],"host":"10.131.17.126","version":3,"port":9475} 
get /dcos-service-kafka/brokers/ids/1 
{"jmx_port":-1,"timestamp":"1474206120002","endpoints":["PLAINTEXT://10.131.24.6:9655"],"host":"10.131.24.6","version":3,"port":9655} 
get /dcos-service-kafka/brokers/ids/2 
{"jmx_port":-1,"timestamp":"1474206122985","endpoints":["PLAINTEXT://10.131.14.192:9181"],"host":"10.131.14.192","version":3,"port":9181} 

然而，当我尝试发布

echo "Hello, World." | ./kafka-console-producer.sh --broker-list 10.131.17.126:9475, 10.131.24.6:9655, 10.131.14.192:9181 --topic topic1 

我得到

[2016-09-18 18:49:32,909] ERROR Error when sending message to topic topic1 with key: null, value: 13 bytes with error: Failed to update metadata after 60000 ms. (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) 

我怀疑这可能是与私人与公共IP地址，也许host.name在server.properties。

任何人都可以提出一些建议，我可能会调试（并希望修复！）的问题，所以我可以成功地发布消息？

感谢

AJ

Answer 1

编辑：对于任何人都希望在未来。 This was a problem in /etc/hosts caused by a terraform script。

您的怀疑是正确的。这些是私人IP地址，不能从群集外寻址。为了与Kafka进行通信，您需要设置一个VPN，以使这些IP地址变得可访问，或者在集群中的某台计算机上运行发布命令。

另外，它看起来像是运行在1.8以前的DC/OS版本上。如果使用1.8，则无论动态分配的IP地址如何，您都可以使用更简单的代理端点。您可以使用命名的VIP broker.kafka.l4lb.thisdcos.directory:9092，但是这只能从群集中的机器寻址。

将haproxy或nginx设置为指向命名VIP也是一种轻松外部访问在DC/OS集群上运行的服务（本例中为Kafka）的方法。您需要确保这些代理在公共代理上运行。 See here for more details。

这里是安装，生产和从默认安装卡夫卡耗费的例子：

~ $ dcos package install kafka 
Installing Marathon app for package [kafka] version [1.1.11-0.10.0.0] 
Installing CLI subcommand for package [kafka] version [1.1.11-0.10.0.0] 
New command available: dcos kafka 
DC/OS Kafka Service is being installed. 

     Documentation: https://docs.mesosphere.com/usage/services/kafka/ 
     Issues: https://docs.mesosphere.com/support/ 
~ $ dcos kafka connection 
{ 
    "address": [ 
    "10.0.3.64:9951", 
    "10.0.3.68:9795", 
    "10.0.3.66:9531" 
    ], 
    "zookeeper": "master.mesos:2181/dcos-service-kafka", 
    "dns": [ 
    "broker-0.kafka.mesos:9951", 
    "broker-1.kafka.mesos:9795", 
    "broker-2.kafka.mesos:9531" 
    ], 
    "vip": "broker.kafka.l4lb.thisdcos.directory:9092" 
} 
~ $ dcos kafka topic create topic0 
{ 
    "message": "Output: Created topic \"topic0\".\n" 
} 
~ $ dcos node ssh --master-proxy --leader 
core@ip-10-0-7-56 ~ $ wget http://download.nextag.com/apache/kafka/0.10.0.1/kafka_2.11-0.10.0.1.tgz 
core@ip-10-0-7-56 ~ $ tar xf kafka_2.11-0.10.0.1.tgz 
core@ip-10-0-7-56 ~ $ cd kafka_2.11-0.10.0.1 
core@ip-10-0-7-56 ~/kafka_2.11-0.10.0.1 $ bin/kafka-console-producer.sh --broker-list broker.kafka.l4lb.thisdcos.directory:9092 --topic topic0 
This is a message 
This is another message 
^Ccore@ip-10-0-7-56 ~/kafka_2.11-0.10.0.1 $ bin/kafka-console-consumer.sh --zookeeper master.mesos:2181/dcos-service-kafka --topic topic0 --from-beginning 
This is a message 
This is another message 
^CProcessed a total of 2 messages 
$ bin/kafka-console-producer.sh --broker-list 10.0.3.64:9951,10.0.3.68:9795,10.0.3.66:9531 --topic topic0 
foo 
bar 
baz 
^Ccore@ip-10-0-7-56 ~/kafka_2.11-0.10.0.1 $ bin/kafka-console-consumer.sh --zookeeper master.mesos:2181/dcos-service-kafka --topic topic0 --from-beginning 
This is a message 
This is another message 
foo 
bar 
baz 
^CProcessed a total of 5 messages 

Answer 2

更新 - 这确实出现了已被丢失在/ etc/hosts中的条目引起的。我已经更新了我的terraform脚本以在安装过程中编写这些脚本，上面的示例现在按预期工作。

感谢您的帮助