Commonly used confluent schema-registry API methods

Photo by Markus Spiske on Unsplash

While maintaining Kafka stack for quite some time, I have come across situations many times when a developer would seek your help to manage the Avro schemas in schema-registry of the Confluent Platform. I usually redirect them to this useful schema-registry API reference doc, which I also find very useful to manage the life cycle of Avro schemas by myself.

In this short article, I will discuss some commonly used methods in my office working hours.

List all subjects

curl -s -X GET http://schema-registry-1:8081/subjects/

This will list all the subjects registered in the schema registry. The above command assumes schema-registry is accessible at address schema-registry-1 with port 8081 and no auth i.e. SSL enabled. You could also get the output in multi-line for more readability, but assuming you have python and JSON module available.

curl -s -X GET http://schema-registry-1:8081/subjects/ | python -m jsob.tool

List all versions registered for a subject

curl -s -X GET http://schema-registry-1:8081/subjects/<subject-name>/versions/
e.g:
curl -s -X GET http://schema-registry-1:8081/subjects/test-value/versions
[1]

This should list all the versions registered under a subject name, there could be 1 or 100 or n..

Please note the following method will return an error when executed, so make sure to specify “versions” after the subject name

curl -s -X GET http://schema-registry-1:8081/subjects/test-value/
{“error_code”:405,”message”:”HTTP 405 Method Not Allowed”}

Get the schema from a specific version under subject

curl -s -X GET http://schema-registry-1:8081/subjects/<subject name>/versions/<version number>
e.g:
curl -s -X GET http://schema-registry-1:8081/subjects/test-value/versions/1 | python -m json.tool
{
    "id": 1,
    "schema": "{\"type\":\"record\",\"name\":\"myrecord\",\"fields\":[{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"age\",\"type\":\"int\"}]}",
    "subject": "test-value",
    "version": 1
}

This will give you the schema with metadata i.e. under which subject this schema is been referred, what is the version and the globally unique id of the schema itself (which is 1 in this case)

List the schema by schema id

curl -s -X GET http://schema-registry-1:8081/schemas/ids/<schema id>
e.g:
curl -s -X GET http://schema-registry-1:8081/schemas/ids/1 | python -m json.tool 
{
 “schema”: “{\”type\”:\”record\”,\”name\”:\”myrecord\”,\”fields\”:[{\”name\”:\”name\”,\”type\”:\”string\”},{\”name\”:\”age\”,\”type\”:\”int\”}]}”
}

If you happen to already know the schema id number that you want to lookup, the above command should be straightforward.

However, there are many other ways to get the schema id of the schema. The easy one would be to just run the previous GET command for a given subject and version combination, another one, rather not so intuitive, is running the kafka-avro-console-consumer with print.schema.ids=true been passes as a property. For example,

/usr/bin/kafka-avro-console-consumer --property print.schema.ids=true --property schema.id.separator=: --property schema.registry.url=http://schema-registry-1:8081 --bootstrap-server broker-1:9092 --topic test --from-beginning 
{“name”:”test2",”age”:5}:1
{“name”:”test2",”age”:5}:1
{“name”:”test2",”age”:5}:1

This will tell you what schema id was embedded with each of these messages in the test topic, then you can take these ids (1 in this example) and run curl on /schemas/ids api endpoint. Note that, we could also have some messages in the same topic referring to a different schema, this is totally acceptable as long as these are schemas are compatible based on compatibility settings. For example, in “BACKWARD” compatibility settings where we are allowed to send a message by dropping a field, that message will refer to a different schema. Let take a further look.

Send a message with just one field i.e. name

/usr/bin/kafka-avro-console-producer \
--broker-list broker-1:9092 — topic test \
--property schema.registry.url=http://schema-registry-1:8081 \
--property value.schema=’{“type”:”record”,”name”:”myrecord”,”fields”:[{“name”:”name”,”type”:”string”}]}'
{"name":"test123"}

Run the consumer to check whether this message is referring to a different schema?

usr/bin/kafka-avro-console-consumer --property print.schema.ids=true --property schema.id.separator=: --property schema.registry.url=http://schema-registry-1:8081 --bootstrap-server broker-1:9092 --topic test --from-beginning
{“name”:”test2",”age”:5}:1
{“name”:”test2",”age”:5}:1
{“name”:”test2",”age”:5}:1
{“name”:”test123"}:2

And there you can see the last message sent to this topic is referring to a different schema

Get top-level of compatibility config

curl -s -X GET http://schema-registry-1:8081/config
{“compatibilityLevel”:”BACKWARD”}

Whatever you have set the top-level compatibility setting, it will show that up. Please note that “BACKWARD” is the default setting.

The above methods will only get you the information from schema-registry, however other methods like registering a new schema or updating the config of a subject, or deleting the subject or its versions, we will cover in another article.

Hope this helps!