Kafka and the ELK Stack — usually these two are part of the same architectural solution, Kafka acting as a buffer in front of Logstash to ensure resiliency. This article explores a different combination — using the ELK Stack to collect and analyze Kafka logs. As explained in a previous post, Kafka plays a key role in our architecture. As such, we’ve constructed a monitoring system to ensure data is flowing through the pipelines as expected.

Data breaches come in all shapes, sizes, and levels of exposure. They can range from a couple of log files unintentionally left available to the public to the leak of hundreds of thousands of users’ personally identifiable information (PII). Don’t think that just because you have a secure network, a leak can’t happen to you.

Let’s start with the happy ending — after a long search, we managed to identify a Netty memory leak in one of our log listeners and were able to troubleshoot and fix the issue on time before the service crashed.

Phishing happens. It is probably happening as you read this. Right now, some well-defended company is having data under its care exposed. This data may contain sensitive information, such as login credentials, and in many cases, it is only known that an attack of this type has taken place after the fact. Protecting yourself and your employer against phishing attacks relies foremost on critical thinking; however, there are some business processes and technologies that can help.

Microsoft Azure has long proven it’s a force to consider in the world of cloud computing. Over the past year, Azure has made some significant steps in bridging the gap with AWS by offering new services and capabilities as well as competitive pricing.

SIEM has been with us for almost two decades now and is seen as a proven approach to dealing with potential threats as well as actual attacks on business critical systems. But today, it is becoming clear that changes in IT infrastructure and deployment practices are giving rise to new challenges that cannot be met by existing SIEM platforms.

Gaining visibility into modern IT environments is a challenge that an increasing number of organizations are finding difficult to overcome. Yes–the advent of cloud computing and virtually “unlimited” storage has made it much easier to solve some of the traditional challenges involved in gaining visibility. However, architecture has evolved into microservices, containers and scheduling infrastructure.

API access is available at the Enterprise tier of our product. With it, you can create a whole range of heavily customized use cases to further expand our suite of offerings. For example, you can hit our API to send customized query results to a third party service like Nagios, or you can automate the creation and deletion of sub-accounts.

In many product development workflows, there are three main concerns: building, testing, and deployment. In this scenario, every change that is made to the code means something could accidentally go wrong, so to lessen the likelihood of this happening, developers assume many strategies to reduce incidents and bugs. One strategy is to adopt continuous integration tools (CI): used together with a source version software to verify if something has gone wrong for every update.

In part 1 of this series, we tried to outline what data retention is and why it is needed to overcome increasing requirements for various regulatory standards. As detailed, there are some clear guidelines for organizations to take what we called a “data retention approach for compliance”. In this follow up post, outline some specific technological and procedural challenges you might face as well as some practical guidelines and strategies to overcome them.