Master/Slave assignment optimization for high performance computing in an EC2 cloud using MPI

Abstract

For high performance computing, cloud services offer highly scaleable infrastructures on demand. Without requiring a great deal of maintenance and financial resources, which a datacenter would need, it is possible to obtain a huge set of computing instances from, e.g., the Amazon Elastic Computing Cloud (Amazon EC2). The downsides of cloud computing are that the set of computing instances is arbitrary and that communication speed varies greartly and affects the running time of algorithms.
We present a master/slave selection algorithm for the EC2 cloud service based on a benchmark to counteract this disadvantage. The benchmark measures communication speed and the variance of communication in a NxN network consisting of N cloud instances. With the results of the benchmark, we perform a master/slave selection algorithm to maximize the number of messages between master and slaves. The results show that our proposed method greatly increases efficiency.