Operations | Monitoring | ITSM | DevOps | Cloud

Platform Engineering leaders are caught between two competing imperatives. You’re under pressure to flatten cloud spend but your team is still provisioning defensively because nobody wants to be the person who causes a production incident. You try to optimize, but six months later, when someone pulls a report, nothing has changed.

In this blog, we’re going to take a look at how you can set up a fully-functioning, robust data pipeline to centralize your data into an InfluxDB instance by collecting and sending messages with the MQTT protocol. We’ll start with a brief overview of the technologies and protocols used in the pipeline, then dive into how you can connect, configure, and test them to ensure your data pipeline is fully functional. It’s going to be a long post, so let’s jump right in.

Last month I was making a change to sx, our CLI. I updated a core flow, adding external catalogs as a source for sx add. Small change. Then came the testing. I knew I was messing with a core flow and wanted to be sure I hadn't broken anything. I spent about forty-five minutes setting up an isolated environment. Spinning up Docker. Fighting with tmux. Getting a clean install state I could run through the TUI a few times. Forty-five minutes of my afternoon that produced zero code. I complained in Slack.