Big data applications require distributed systems to process, store and analyze the massive amounts of information that companies are collecting. Apache Spark has become a go-to framework for this, powering use cases from AI and machine learning to data analysis, by providing a unified interface for distributing data processing tasks across a cluster of machines. Spark requires other services to manage the cluster, with YARN and Mesos as two well-known cluster management tools.

Although it is no surprise that Gartner is forecasting that overall public cloud consumption will grow by 18%, hitting $304.9 billion in 2021, Gartner’s research also indicates that the vast majority of organizations overshoot their cloud infrastructure budgets due to a lack of cost optimization approaches.

Even though January 1 has passed, it’s not too late to commit to a New Year’s resolution. “Right–sizing” might sound like the latest fad diet that promises easy weight loss results. But you didn’t come to this particular blog because (like me) your pandemic diet sometimes consisted of nothing more than take-out and never-ending snacks.

Spot by NetApp’s Ocean continuously optimizes Kubernetes clusters with a wide feature-set tackling different aspects of running and managing Kubernetes containers in a cloud environment. One such aspect are the container resource requests defined in the cluster (upon which Ocean intelligently bin-packs pods on the underlying cloud VMs). Incorrect assumptions regarding the CPU and Memory required for an application, can incur unnecessary and costly cloud infrastructure waste.

Should you run big data workloads on Kubernetes? Not so long ago, most people would have said certainly not. Among the reasons you might have heard: Although early adoption of Kubernetes was dominated by stateless services rather than data-centric applications, more and more teams responsible for big data platforms are now looking to adopt Kubernetes, especially those looking to deploy and operate big data workloads in the cloud.

Continuous integration and deployment (CI/CD) processes are becoming the standard for many organizations operating in the cloud—including many Spot by NetApp customers—with Jenkins as a core tool for building, testing and deploying code through CI/CD pipelines. Spot has had an integration with Jenkins since our early days, allowing our users to configure Elastigroups managed by Jenkins and deploy jobs running on spot instances.

AWS’s annual re:Invent wrapped up this week, and although we didn’t get to see customers, colleagues and, friends in Las Vegas this year, it was still an eventful three weeks. Spot by NetApp sponsored the virtual conference and hosted a number of activities including our AWS JAM session on Kubernetes and infrastructure scaling, an expert session on cloud compute optimization, and an interview with Spot by NetApp’s CEO and founder, Amiram Shachar.