Both Apache Kafka and Flume systems can be scaled and configured to suit different computing needs.

Kafka is the durable, scalable and fault-tolerant public-subscribe messaging system.

The publish-subscribe architecture was initially developed by LinkedIn to overcome the limitations in batch processing of large data and to resolve issues on data loss.

The architecture in Kafka will disassociate the information provider from the consumer of information.

Hence, the sending application and the receiving application will not know anything about each other for that data sent and received.

It is a distributed streaming platform with capabilities similar to an enterprise messaging system but has unique capabilities with high levels of sophistication.

With Kafka, users can publish and subscribe to information as and when they occur.

It allows users to store data streams in a fault-tolerant manner.

Irrespective of the application or use case, Kafka easily factors massive data streams for analysis in enterprise Apache Hadoop.

Kafka also can render streaming data through a combination of Apache HBase, Apache Storm, and Apache Spark systems and can be used in a variety of application domains.

A subscriber requests for a subscription and Kafka forwards the data to the requested subscriber.

Typically, there can be numerous publishers and subscribers on different topics on a Kafka cluster.

Likewise, an application can act as both, a publisher and subscriber.

Flume is highly reliable, configurable and manageable distributed data collection service which is designed to gather streaming data from different web servers to HDFS.

Flume offers highly fault-tolerant, robust and reliable mechanism for fail-over and recovery with the capability to collect data in both batch and in stream modes.

Flumes capabilities are leveraged by enterprises to manage high volume streams of data to land in HDFS.

For instance, data streams include application logs, sensors and machine data and social media, and so on.

These data, when landed in Hadoop, can be analyzed by running interactive queries in Apache Hive or serve as real-time data for business dashboards in Apache HBase.

However, Kafka is a more general purpose system where multiple publishers and subscribers can share multiple topics.

Contrarily, Flume is a special purpose tool for sending data into HDFS.

Kafka replicates data in the cluster, whereas Flume does not replicate events.

Hence, when a Flume agent crashes, access to those events in the channel is lost till the disk is recovered, on the other hand, Kafka makes data available even in case of single point failure.

On the other hand, Flume supports a large set of source and destination types to land data on Hadoop.

For example, e-commerce, online retail portals, social media, etc.

Flume The Difference