What is the use of __consumer_offsets and _schema topics in Kafka?

apache-kafka confluent-schema-registry
Anveshak picture Anveshak · Sep 16, 2016 · Viewed 21.3k times · Source

After setting up the Kafka Broker cluster and creating few topics, we found that the following two topics are automatically created by Kafka:

  1. __consumer_offsets
  2. _schema

What is the importance and use of these topics ?

Answer

Natalia picture Natalia · Sep 16, 2016

__consumer_offsets is used to store information about committed offsets for each topic:partition per group of consumers (groupID). It is compacted topic, so data will be periodically compressed and only latest offsets information available.

_schema - is not a default kafka topic (at least at kafka 8,9). It is added by Confluent. See more: Confluent Schema Registry - github.com/confluentinc/schema-registry (thanks @serejja)

Related questions

TimeoutException: Timeout expired while fetching topic metadata Kafka

I have been trying to deploy Kafka with schema registry locally using Kubernetes. However, the logs of the schema registry pod show this error message: ERROR Server died unexpectedly: (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain:51) org.apache.kafka.common.errors.…

Read More
KafkaAvroSerializer for serializing Avro without schema.registry.url

I'm a noob to Kafka and Avro. So i have been trying to get the Producer/Consumer running. So far i have been able to produce and consume simple Bytes and Strings, using the following : Configuration for the Producer : Properties …

Read More
Kafka schema registry not compatible in the same topic

I'm using Kafka schema registry for producing/consuming Kafka messages, for example I have two fields they are both string type, the pseudo schema as following： {"name": "test1", "type": "string"} {"name": "test2", "type": "string"} but after sending and consuming a …

Read More