Is it ok to copy the topic with Replicator when the target schema registry has the same schema ID but with a different schema definition?

Photo by Oskar Yildiz on Unsplash

In this article, we are going to uncover how Confluent replicator will behave and what to expect when it's been asked to copy topic from source to target Kafka cluster where source and target have their own schema registries. In this scenario, we will see what happens when the same schema ID already exists somewhere on the target schema registry instance.

Before covering this scenario, there are certain knowledge and environment assumptions that I’m going to make:

Knowledge:

  1. Kafka connect framework and replicator as a connector

Part 2 (Conflicting Schema Exists On Target, Same Subject)

What to do when Replicator fails to insert messages to the target topic because the topic’s subject has a version with conflicting schema?

Photo by Charl Folscher on Unsplash

This is a follow-up article based on this, where we discussed what to expect when replicator tries to copy topic with Avro messages to target but target schema registry already have same schema ID (which is embedded into messages) residing in different subject and that schema object is completely different with what it had on the source. In this article, we are going to see how Confluent Replicator is going to behave when a subject with the same name already exists on the target schema registry that has a version with a conflicting schema.

Before going forward with this article…


<span>Photo by <a href=”https://unsplash.com/@markusspiske?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=credit
<span>Photo by <a href=”https://unsplash.com/@markusspiske?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=credit
Photo by Markus Spiske on Unsplash

While maintaining Kafka stack for quite some time, I have come across situations many times when a developer would seek your help to manage the Avro schemas in schema-registry of the Confluent Platform. I usually redirect them to this useful schema-registry API reference doc, which I also find very useful to manage the life cycle of Avro schemas by myself.

In this short article, I will discuss some commonly used methods in my office working hours.

List all subjects

curl -s -X GET http://schema-registry-1:8081/subjects/

RanjitNagi

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store