Spark Cluster Performance When Using a Cloud Service vs. a Typical Configuration

Abstract

Apache Spark was developed as a free and open-source cluster computing platform for handling massive amounts of data. This article focuses on setting up a Spark cluster in the cloud using OpenStack. Spark cluster as a service is compared to a traditional Spark cluster using the HiBench test suite. Using Spark as a cloud service yields more promising results in terms of time, effort, and throughput, as shown by the results.