As cloud deployments scale, Amazon Web Services (AWS) VPC flow logs become an invaluable network visibility and security tool. They are also one of the most voluminous classes of data, making them an expensive choice to add to analytics platforms. With growing infrastructure and traffic, managing these logs presents significant challenges. ‍In part 1 of this series, we took a look at common use cases and problems associated with storing and processing VPC Flow Logs.

Observability pipelines solve some critical problems IT is facing today: the cloud environment has generated an unprecedented amount of data in recent years; enterprises now have multiple SaaS/cloud-based applications running; it’s becoming tough to know which of this massive volume of data needs to be processed for analysis vs. stored (often for regulatory reasons) cheaply; and dealing with growing numbers of source data makes the meaningful management of the problem only harder.

In the last few years, fintech enterprises have disrupted the financial services and banking industry by taking everything computing technology offers – from machine learning to blockchain – and turning it up a notch. Traditional financial institutions must now compete with challenger banks offering electronic payment alternatives, peer-to-peer lending, and investment apps.

Artificial intelligence for IT Operations (or AIOps) has been playing an expanding role in helping SREs, DevOps, and developers effectively navigate the challenges around application and infrastructure complexity, pace of change, and data volume that characterize the operations landscape.

The software industry has seen many evolutions. There is a new disruption in the market every five years or so. Software testing cannot remain isolated from all the latest trends and technologies. Testing strategies need to keep up with agile development, faster deployments and increasing customer demand for reliability and user-friendly interfacing. They need to be able to grow just as quickly and just as reliably as the business logic.

How is your organization handling Kubernetes observability? What tools are you using to monitor Kubernetes? Is it a time-consuming, manual process to collect, store and visualize your logging, metrics and tracing data? And, what are you actually getting out of all that investment? At Logz.io we’re trying to make this process easier for customers who are serious about Kubernetes observability. We’ve made significant investments in this area for Kubernetes use cases.

If you’ve ever seen Indiana Jones and the Last Crusade, you might remember the scene where Indy and his dad are in a room replete with the most ornate chalices possible, only to realize that the Holy Grail is the most plain, utilitarian one in the room. Windows event logs are the IT version of the plain-looking clay cup that holds the key to answering your service questions and system issues.