Full-text searches are a marvel of modern computing. In less than a second, search engines can match a query against hundreds of millions of documents. In the early days of search engines, you often had to use specific search operators and terms to get accurate results.

In Elasticsearch parlance, a document is serialized JSON data. In a typical ELK setup, when you ship a log or metric, it is typically sent along to Logstash which groks, mutates, and otherwise handles the data, as defined by the Logstash configuration. The resulting JSON is indexed in Elasticsearch.

Elasticsearch is a highly scalable, distributed, open-source RESTful search and analytics engine that offers log analytics, real-time application monitoring, click stream analytics, and more. Elasticsearch stores and retrieves data structures in real time. It has multi-tenant capabilities with an HTTP web interface, presents data in the form of structured JSON documents, makes full-text search accessible via RESTful API, and maintains web clients for languages like PHP, Ruby, .Net, and Java.

If you’ve been paying attention, you know that although collecting and reviewing metrics and logs is a core part of running a stable and successful service, access to raw events and the ability to search and pivot on any dimension of your production environment, no matter how high-cardinality, is what will help your team debug and troubleshoot new problems and outages more quickly.

Elasticsearch 6.3 included some major new features, including rollups and Java 10 support, but one of the most intriguing additions in this version is SQL support.