Kafka

lenses

SELECT ApacheKafka WITH StreamingSQL FROM RealTimeData

In another life, I taught the Book of Genesis to high school students, including The Tower of Babel excerpt. It struck me ironic that God’s wrath strikes down the tower, cofounds the universal language and scatters humans around the globe to teach King Nimrod a lesson in hubris; meanwhile, the boys in my class were texting their girlfriends across the country and playing video games with friends in Europe and Asia.

lenses

New Apache Kafka to AWS S3 Connector

Many in the community have been asking us to develop a new Kafka to S3 connector for some time. So we’re pleased to announce it's now available. It’s been designed to deliver a number of benefits over existing S3 connectors. Like our other Stream Reactors, the connector extends the standard connect config adding a parameter for a SQL command (Lenses Kafka Connect Query Language or “KCQL”). This defines how to map data from the source (in this case Kafka) to the target (S3).

lenses

Deploy turn-key DataOps for AWS MSK

Running your own Kafka is starting to feel like wading through oatmeal. We’re not the only ones thinking that. The majority of organizations we speak to have or are in the process of moving their Kafka to a managed service. If you’re already an AWS-shop, Managed Streaming for Apache Kafka (MSK) is a no-brainer. It is the same Kafka that we know and love and integrated with other AWS services such as IAM, Cloudwatch, Cloudtrail, KMS, VPC and more.

lenses

On the importance of load testing Kafka

Socrates preached, “To know thyself is the beginning of wisdom.” This ancient Greek anecdote applies to your modern Apache Kafka project: developers, go forth and load test your real-time application to understand the capacity and limitations of your project before deployment. Failure to do so will cost you time and money (e.g. Robinhood’s outage on a historic trading day). Load testing your real-time applications has three main objectives.