At Datadog, we rely heavily on Kubernetes, and we’re facing some interesting challenges as we use Kubernetes to scale further and strive for greater efficiency. To address these challenges, we’ve been working on solutions to help us better control how our clusters scale, and to make it easier to deploy and manage the Datadog Agent. Today, we’re open sourcing these solutions to share them with the rest of the Kubernetes community.
AWS CloudFormation is a service that enables you to build infrastructure as code, similar to Terraform. You can create CloudFormation templates to provision and manage all of the resources for your stacks, such as EC2 instances, load balancers, and security groups. These templates automate the process of building infrastructure, creating repeatable steps that you can easily check into version control. This ensures that your configurations do not drift with each new environment you spin up.
AWS re:Invent is an annual gathering of tens of thousands of AWS staff, partners, and users for a full week of keynote sessions, feature announcements, customer case studies, hands-on workshops, and more. As in years past, we will be there with dozens of engineers, ready to answer your monitoring questions and show you the newest additions to Datadog.
Datadog provides full visibility into your environment through a wide variety of features, ranging from host and container maps of your dynamic infrastructure to customizable dashboards that provide a unified view of every layer of your stack. And now we’re pleased to announce that you can enjoy these visualization features and the rest of the Datadog platform in dark mode.
MapR is an Apache Hadoop distribution that enables organizations to manage, analyze, and store all their data at scale. MapR handles a wide range of data types across infrastructures and locations by leveraging dataware, an abstraction layer in the enterprise software stack that separates data from any dependencies. We’re excited to announce that our new integration provides comprehensive visibility across all the moving parts of your MapR deployment.
Amazon MQ is a managed ActiveMQ messaging service hosted on the AWS cloud. Amazon MQ’s brokers route messages between the nodes in a distributed application. Each broker is a managed AWS instance, so your messaging infrastructure doesn’t require the maintenance and upfront costs of a self-hosted solution.
In Part 1 of this series, we saw how Amazon MQ routes messages between services in a distributed application, and we looked at some of the key metrics that describe the performance of the message broker and its destinations. Now that we’ve introduced the metrics and their meaning, we’ll look at some tools you can use to collect and query metrics from Amazon MQ:
In Part 2 of this series, we showed you how to use CloudWatch to monitor metrics and logs from Amazon MQ. With CloudWatch, you can easily create ad-hoc graphs to visualize the performance of your messaging infrastructure and other AWS services you use (such as EC2, Lambda, and S3). But to monitor your Amazon MQ brokers, destinations, and clients alongside the rest of your applications and infrastructure, you need a monitoring platform that easily integrates with your whole technology stack.
To centralize logging from your entire stack—from traditional infrastructure to serverless components—Datadog is announcing native support for the launch of FireLens for Amazon ECS. FireLens streamlines logging by enabling you to configure a log collection and forwarding tool such as Fluent Bit directly in your Fargate tasks. We’ve partnered with AWS to provide built-in Fluent Bit support for Datadog so that you can now seamlessly route container logs from AWS Fargate.
Since Kubernetes was open sourced by Google in 2014, it has steadily grown in popularity to become nearly synonymous with Docker orchestration. Kubernetes is being widely adopted by forward-thinking organizations such as Box and GitHub for a number of reasons: its active community, rapid development, and of course its ability to schedule, automate, and manage distributed applications on dynamic container infrastructure.
