One of the most exciting implementations to come out of Apache Spark™ on Kubernetes is the dynamic creation, mounting, and remounting of a PersistentVolumeClaim (PVC) within a Spark application. We’ve spoken previously about how PVCs can be used to recover shuffle data and prevent application failure after a spot kill has occurred, negating one of the biggest drawbacks of spot instance usage in Spark workloads.

GitHub Actions is a CI/CD platform that allows you to define custom workflows and automate various tasks within your software development projects. With GitHub Actions, you can set up continuous integration and deployment pipelines, run tests, perform code analysis, and execute other actions whenever specific events occur, such as pushing code to a repository or creating a pull request.

Cloud computing was originally pitched as a cost-savings alternative to traditional IT infrastructure. The concept was simple: only purchase what you need and adjust those resources up and down based on demand. This eliminated large up-front costs and made it possible to right-size the amount of compute and storage infrastructure based on a specific workload. While that might have been true in the beginning, CIOs are finding themselves in a different situation today.

Effective CloudOps teams can control costs, improve security, optimize resources, and provide superior services across their cloud environments. These teams can also leverage FinOps, the cross-functional cloud financial management discipline and cultural practice that promotes visibility and collaboration to help organizations manage, optimize, and predict their cloud spend.

Spot by NetApp is once again a leader and outperformer in FinOps, according to tech industry analyst firm GigaOm.

The AWS spot instances marketplace provides many options to generate savings. AWS’s cloud ecosystem offers several different instance types, zones, and architectures to support almost any application demand. The abundance of options can overwhelm even experienced cloud professionals and may prevent the organization from fully reaping the benefits of working with spot instances.

NetApp is pleased to announce General Availability of its Spot Ocean for Apache Spark™ product on the Microsoft Azure cloud. Customers can now choose to have Spot optimize and manage their AI and data engineering projects at scale on many kinds of infrastructure. Spot’s SaaS product removes the effort, complexity, and risk of setting up and running Apache Spark yourself. It also improves the performance and efficiency of data scientists’ Jupyter Notebooks.

In an uncertain and turbulent economy, organizations have made it a critical priority to ensure they are managing cloud spend to stay within their cloud budgets and efficiently optimize that spend while leveraging the cloud to achieve business growth.

AWS spot instances represent AWS’s excess capacity, available for any surge in customer demand. To offset the loss of idle infrastructure, AWS offers this excess capacity at a massive discount to drive usage. With up to 90% in savings when compared to on-demand pricing, spot instance pricing is an affordable option for organizations looking to significantly reduce their cloud bills. One of the challenges with spot instances, however, is that they can disappear at a moment’s notice.

In the beginning, cloud cost savings were the focus for Spot. As we grew, our customers helped us understand that cloud cost reduction is just one of the many challenges they face when operating in the public cloud. Spot by NetApp has since invested in and developed multiple solutions that can promote FinOps best practices and equip DevOps with comprehensive infrastructure automation and optimization solutions for VMs and containers.