This is a short blog post about a pattern that we’ve observed more frequently among some of the large enterprises: the use of AWS S3 as both an observability lake and a data bus. AWS S3’s simple API, ubiquitous language support, unmatched reliability and durability, retention options, and numerous pricing plans have made it the de facto standard for storing massive amounts of data.

The cloud native revolution brought by Kubernetes has transformed the way we build and deliver software, but the world of big data has for too long been left on the side of this transformation. Thanks to many contributions from the open source community, Apache Spark integration on top of Kubernetes is now officially generally available with the recent releases this year.

Today, data is a bit of an esoteric concept. Data is contextual and you need to understand the data that’s important to you. For instance, cost data. Cost insight has given companies the ability to make informed engineering, product, and business decisions like rearchitecting software or pricing your product for higher profitability. However, to do this optimally, you’ll need to establish the right systems and practices. This is where business intelligence (BI) tools come in.

Like many cool tools out there, this project started from a request made by a customer of ours. Having recently migrated to our service, this customer had ~30TB of historical logging data. This is a considerable amount of operational data to leave behind when moving from one SaaS platform to another. Unfortunately, most observability solutions are built around the working assumption that data flows are future-facing.

Aiven is happy to support the newly released Apache Kafka version 3.0, announced on September 21, 2021. This new major version comes with a number of enhancements across the board - read on!

The recent surge in internet usage and its corresponding increase in data has triggered a new awareness among business owners. One of the main questions from marketers is how their businesses can benefit from Big Data. As a business owner, you should know that Big Data is indeed one of the most incredible things to happen to the marketing industry. If used right, it can enhance your company’s ability to serve its consumers and increase revenue.

We’re excited to announce that a beta version of our Aiven for Apache Flink service is now publicly available for anyone to try. Find out more!