Operations | Monitoring | ITSM | DevOps | Cloud

In this lesson, we’ll learn how we can use Elasticsearch Hadoop to process very large amounts of data. For our exercise, we’ll use a simple Apache access log to represent our “big data”. We’ll learn how to write a MapReduce job to ingest the file with Hadoop and index it into Elasticsearch.

Prometheus is the cornerstone of many monitoring solutions, and sooner or later, prometheus federation will appear on your radar. A well monitored application with flexible logging frameworks can pay enormous dividends over a long period of sustained growth. However, once you begin to scale your prometheus stack, it becomes difficult to keep up with your application’s demands. Prometheus is an extremely popular choice when it comes down to collecting and querying real-time metrics.

Modern day Security Information and Event Management (SIEM) tooling enterprise security technology combine systems together for a comprehensive view of IT security. This can be tricky, so we’ve put together a simple SIEM tutorial to help you understand what a great SIEM provider will do for you. A SIEM’s responsibility is to collect, store, analyze, investigate and report on log and other data for incident response, forensics and regulatory compliance purposes.

Availability and quality are the biggest differentiators when people opt for a service or product today. You should be aware of the impact of your customers alerting you to your own problems, as well as how to stop this from becoming the norm. To make sure you don’t become an organization known for its bugs, understanding the organizational changes required to deliver a stable service is key.

Logz.io has a comprehensive API that allows you to send and query logs or metrics easily. Now, that process is getting even easier for infrastructure monitoring. Logz.io has added support for the open source grafana API. In other words, you can use the OSS grafana’s integration within Logz.io.

Across the technology and IT infrastructure domain, log files are recognized as often time-stamped files that can virtually record all critical information about events occurring within the purview of your IT network, OS, or other software applications. Some log files are humanly interpretable, while others are largely meant for machines to consume.

While all cybersecurity professionals agree that log management is integral for robust proactive and reactive security, managing the enormous amount of data logs can be a challenge. While you might be tempted to collect all logs generated from your systems, software, network devices, and users, this “fear of missing out” on an important notification ultimately leads to so much noise that your security analysts and threat hunters cannot find the most important information.