Data-driven workflows in multi-cloud marketplaces

Abstract

Cloud computing is emerging as a viable platform for scientific exploration. The ideas of on-demand access to resources, 'unlimited' resources as well as interesting pricing models are making scientist to move their workflows into cloud computing. However, the amount of services and different pricing models offered by the providers often overwhelm users when deciding which option is best for them. Moreover, interoperability across providers remains an open topic that forces users to develop specific solutions for each provider. In this paper, we present a service framework that enables the autonomic execution of dynamic workflows in multi-cloud environments. It also allows users to customize scheduling policies to use those resources that best fit their needs. To demonstrate the benefits of this framework, we study the execution of a real scientific workflow, with data dependencies across stages, in a multi-cloud federation using different policies and objective functions.