使用Kafka实现多个数据源的流式传输

Question

我们计划使用apache kafka构建一个实时监控系统。总体思路是将数据从多个数据源推送到kafka并执行数据质量检查。我有这个架构

1. 几个问题，什么是流媒体从多个来源主要包括Java应用程序，Oracle数据库数据的最佳方法，REST API的，日志文件到Apache卡夫卡？注意每个客户端部署都包含每个这样的数据源。因此，将数据推送到kafka的数据源的数量将等于客户数* x，其中x是我列出的数据源类型。理想情况下，推送方式最适合，而不是拉式方法。在拉方法中，目标系统必须配置各种不同源系统的证书，这是不实际的
2. 我们如何处理故障？
3. 我们如何对传入消息执行数据质量检查？对于例如如果某个消息不具备所有必需的属性，则可能会丢弃该消息，并且可能会提醒维护团队检查。

请让我知道你的专家意见。谢谢 ！

Answer 1

我觉得这里最好的办法是用卡夫卡连接：link 但它是一个拉的方法： Kafka Connect sources are pull-based for a few reasons. First, although connectors should generally run continuously, making them pull-based means that the connector/Kafka Connect decides when data is actually pulled, which allows for things like pausing connectors without losing data, brief periods of unavailability as connectors are moved, etc. Second, in distributed mode the tasks that pull data may need to be rebalanced across workers, which means they won't have a consistent location or address. While in standalone mode you could guarantee a fixed network endpoint to work with (and point other services at), this doesn't work in distributed mode where tasks can be moving around between workers.阿雯