Hello world, it’s been a while! We are super excited today to announce the open-sourcing of one of the exciting new projects we’ve been working behind the scenes at the intersection of big-data and computation platforms – YuniKorn! Yunikorn is a new standalone universal resource-scheduler responsible for allocating/managing resources for big-data workloads including batch jobs and long-running services. Let’s dive right in!

We know that data is a key driver of success in today data-driven world. In fact, according to Forrester, data and insight-driven businesses are growing at an average of more than 30% annually. However, becoming a data driven organization is not easy. Companies often struggle with speed in accessing and analyzing their data, as well with ensuring delivery of trustworthy data that is free of critical errors.

Fig 1 – Architecture Many of Cloudera’s customers set up Cloudera Manager to collect their clusters’ diagnostic data on a regular schedule and automatically send that data to Cloudera.

The integration landscape is changing. According to Gartner, “Two-thirds of all business leaders believe that their companies must pick up the pace of digital transformation to remain competitive.” One of the byproducts of this increasing pace is the desire to get results quickly. In a cloud-first world, that means expectations are changing for how products are trialed, procured, and billed. People expect things to be simpler, faster, and more intuitive.

Authentication is a basic security requirement for any computing environment. In simple terms, users and services must prove their identity (authenticate) to the system before they can use system features. Kerberos provides strong authentication which is used in the exchange between requesting user or process and service during authentication. When a user authenticates to a particular Hadoop component, the user’s Kerberos principal is presented. The principal is presented in the form user@REALM.

As machine learning continues to address common use cases it is crucial to consider what it takes to operationalize your data into a practical, maintainable solution. This is particularly important in order to predict customer behavior more accurately, make more relevant product recommendations, personalize a treatment, or improve the accuracy of research.

As many companies move to a subscription model, one of the most important tools to provide for customers is a way to justify that their subscription is still valuable. To do so, many SaaS companies offer access to usage data to allow users to easily answer questions about the value they are getting from a product.

Data lakes: smooth sailing or choppy waters? In May, Talend announced its support for Databricks’ open source Delta Lake, “a storage layer that sits on top of data lakes to ensure reliable data sources for machine learning and other data science-driven pursuits.” What does this mean for your company, and is Delta Lake right for you?