Pushing data in JSON format to an Apache Kafka topic is very common.
However, dealing with messages not having a predefined structure can create some problems, specifically when trying to sink the data via connectors, like the JDBC sink, which require the knowledge of the message structure.
Transforming the messages from JSON to AVRO can enforce a schema on messages and allow the usage of a bigger variety of connectors.

In this video we’ll showcase how to convert messages from JSON to AVRO in streaming mode via Apache Flink.

Check out these resources to learn more:

Build your first data pipeline with Aiven for Apache Flink https://www.youtube.com/watch
AVRO Confluent data format https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/formats/avro-confluent/
Karapace - open source Schema registry and REST API for Apache Kafka https://www.karapace.io/

Francesco Tisiot on Twitter: https://twitter.com/FTisiot
Francesco Tisiot on LinkedIn: https://www.linkedin.com/in/francescotisiot/

CHAPTERS

00:38 Check the data in Apache Kafka in JSON format

2:12 Create the Flink Application

3:10 Map the source Apache Kafka topic in JSON format as Apache Flink table

4:40 Map the target Apache Kafka topic in AVRO format as Apache Flink table

5:25 Define the schema registry endpoint in the Apache Flink table

6:35 Define the message Key with Apache Flink ‘key-fields’

6:57 Define the transformation SQL

7:55 Check the data in Apache Kafka in AVRO format

9:05 Summary

ABOUT AIVEN
Aiven’s cloud data platform helps your business reach its highest potential by making your data work for you.

It provides fully managed open source data infrastructure on all major clouds, helping developers focus on what they do best: innovate and create without worrying about the limitations of technology.

We like to think that Aiven is not only a cloud data platform but also an
extension of your team. We are dedicated to helping you to succeed by removing barriers and finding the right solutions – with the help of the best data technology there is.

CONNECT WITH US
Website: http://aiven.io​​​
LinkedIn: https://linkedin.com/company/aiven​​​
GitHub: https://github.com/aiven​​​
Twitter: https://twitter.com/aiven_io​​​