In distributed application environments, to solve problems in code you need to be able to connect the dots between all the different places where your code runs, including frameworks like Databricks and Apache Airflow. The Databricks pipeline may be one of the most crucial places where your code runs, but the visibility you’re getting there is limited.

In this guide, we’ll set up an OpenTelemetry Collector that collects logs and sends them to Grafana Loki running in Grafana Cloud. We will consider two examples for sending logs to Loki via OpenTelemetry Collector. The first one shows how to collect container logs with a Fluent Forward receiver. The second one shows how to collect system logs with a Filelog receiver.

This article was published in The New Stack. For most developers, software development means there is an API for almost everything, hardware is provisioned via the cloud and the core focus is on building only the features most crucial to your business. Of course, all these integrations and modern distributed architectures create their own set of problems. Having full insight into your application has become even more important and is now commonly known as observability.

Testing today’s environments is more challenging than it was a few years ago. The transition to distributed environments has created complexity, overhead, and friction when writing and running new backend tests. These tests require a lot of preparation, infrastructure building, and maintenance since many services communicate asynchronously and they often miss exceptions that are thrown on the “deeper layer” of the system architecture and it’s hard to make it testable.

Monitoring and observability have increased with software applications moving from monolithic to distributed microservice architectures. While observability and application monitoring share similar definitions, they also have some differences. The purpose of both monitoring and observability is to find issues in an application. However, monitoring aims to capture already known issues and display them on a dashboard to understand their root cause and the time they occurred.