Apache Airflow is an open source system for programmatically creating, scheduling, and monitoring complex workflows including data processing pipelines. Originally developed by Airbnb in 2014, Airflow is now a part of the Apache Software Foundation and has an active community of contributing developers. Airflow represents workflows as Directed Acyclic Graphs (DAGs), which are made up of tasks written in Python. This allows Airflow users to programmatically build and modify their workflows.

Since StatsD was introduced by Etsy in 2011, it has become a mainstay of infrastructure monitoring. But as dev and ops teams rely more and more on containerized microservices, they have pushed the limits of StatsD’s design. One major shortcoming is that StatsD has no built-in support for tagging your metrics with key-value pairs.

Datadog operates large-scale Kubernetes clusters in production across multiple clouds. Along the way, audit logs have been extremely helpful for tracking user interactions with the API server, debugging issues, and getting clarity into our workloads. In this post, we’ll show you how to leverage the power of Kubernetes audit logs to get deep insight into your clusters.

Monitoring your application from end to end is important for ensuring that core functionalities work as designed. Datadog’s browser tests help you verify that key user workflows—such as signing up for a new account—are consistent across devices and locations. Within these workflows, email often plays a key role in onboarding users and providing customers with important information about their accounts and application activity, such as profile changes and order confirmations.

As your on-premise network infrastructure grows in size and complexity, monitoring thousands of devices becomes a challenge. Whether you’re monitoring firewalls in a branch office or the routing and switching fabric in your datacenter over which all customer transactions are performed, visibility into all points of your infrastructure is critical for network maintenance.

Hashicorp Vault is a tool for managing secrets—sensitive data such as passwords, certificates, and API keys. Vault allows you to encrypt your secrets, control access to them, and audit activity to see who has requested data from your Vault. Datadog already monitors the status of your Vault servers—for example, you can configure the Vault integration to automatically notify you if a Vault server is unexpectedly sealed, or if there is a leader change in your Vault cluster.

ClickHouse is an open source database management system, and was originally developed as a backend for Yandex’s Metrica analytics platform. ClickHouse is column oriented, meaning that it can quickly scan through ranges of values in a single column without touching irrelevant values in other columns. This makes ClickHouse well suited for online analytical processing (OLAP).

AWS Lambda is a compute service that enables you to build serverless applications without the need to provision or maintain infrastructure resources (e.g., server capacity, network, security patches). AWS Lambda is event driven, meaning it triggers in response to events from other services, such as API calls from Amazon API Gateway or changes to a DynamoDB table.

In Part 1 of this series, we discussed AWS Lambda functions and some key metrics for monitoring them. In this post, we’ll look at using Amazon’s native tooling to query those metrics. We’ll also show you how to collect logs and traces that provide further visibility into your Lambda functions. Amazon provides built-in monitoring functionality through CloudWatch and X-Ray.

In Part 2 of this series, we looked at how Amazon’s built-in monitoring services can help you get insights into all of your AWS Lambda functions. In this post, we’ll show you how to use Datadog to monitor all of the metrics emitted by Lambda, as well as function logs and performance data, to get a complete picture of your serverless applications. In this post, we will: Datadog integrates with AWS Lambda and other services such as Amazon API Gateway, S3, and DynamoDB.

AWS Lambda decouples the need to provision and maintain a runtime environment from running code, allowing developers to focus on applications rather than infrastructure. But, by abstracting away the underlying infrastructure of an application, serverless architectures introduce new challenges into monitoring and observability.

Since AWS Lambda was launched in 2014, serverless has transformed the way applications are built, deployed, and managed. By abstracting away the underlying infrastructure, developers are able to shift operational responsibilities to the cloud provider and focus on solving customer problems.