To build applications in AWS Lambda, you often need to use third party libraries and packages in your function code. Previously, these packages had to be included in a function’s deployment package. Today, Amazon Web Services released a new feature called Layers to simplify this process for Lambda developers. Layers allow you to deploy common components that you can reuse across functions, such as machine learning models, SDKs, or instrumentation libraries.

Envoy proxies communication among microservices. It is a key component in many service-oriented architectures—and one that offers a unique opportunity to gain visibility into your service mesh. We’re pleased to announce that Datadog integrates with Envoy as well as AWS App Mesh, a new hosted service based on Envoy that dynamically configures your service mesh proxies.

To make serverless architectures more observable, we’re excited to introduce the new Cloud Functions view in Datadog. You can now search, filter, and explore all your AWS Lambda functions in one central view, and dive straight into detailed performance data that is scoped to a single function. The Cloud Functions view brings together Lambda metrics and logs with distributed request traces from your functions, which are now available in Datadog thanks to our new integration with AWS X-Ray.

Datadog is pleased to announce the upcoming availability of Synthetics, a whole new layer of visibility on the Datadog platform. By monitoring your applications and API endpoints via simulated user requests, Synthetics helps you ensure uptime, identify regional issues, and track application performance. By unifying Synthetics with your metrics, traces, and logs, Datadog allows you to observe how all your systems are performing as experienced by your users.

Each November, AWS re:Invent draws thousands of AWS staff, partners, and users to Las Vegas for an intense week featuring all things AWS and AWS-related. As always, Datadog will be there and we’d love to meet you in person. Our engineers are excited to show off the new features they’ve been building and to answer your monitoring questions!

We recently implemented a Spark streaming application, which consumes data from from multiple Kafka topics. The data consumed from Kafka comprises different types of telemetry events generated by mobile devices. We decided to host the Spark cluster using the Amazon EMR service, which manages a fleet of EC2 instances to run our data-processing pipelines.