Elasticity and Auto-scaling for Cloud and Big Data Applications - overview

This page contains pointers on (i) management techniques for big data and analytics cloud platforms that require dynamic resource allocation and (ii) recommendations on how to perform analytics in the cloud. Resource management techniques for cloud here explored novel inputs such as user's patience and application data and a novel metric to measure the quality of elasticity/auto-scaling operations in the cloud.

The efforts in this field are:

(i) a better understanding on elasticity/auto-scaling for cloud environments. The work contains: (a) auto-scaling/elasticity policies based on user patience and Prospect Theory (this is a novel way that performs resource management based on information of user expectation from cloud performance), (b) deep analysis on proactive vs reactive auto-scaling/elasticity triggers for resource management in cloud, (c) methods for determining the step size of resource scaling operations under bounded and unbounded maximum capacity, (d) the introduction of a new metric called Auto-scaling Demand Index, which represents the difference between the target utilization interval and the actual values, e) auto-scaling/elasticity in cloud based on user application data which is able to speed the trigger of resource provisioning faster than monitoring system resource usage (e.g. CPU, memory, network), which is the classical state of the art solution.

(ii) a structured access and detailed analysis of existing solutions to handle cloud-based big data and analytics and identification of important research gaps in this area for the community considering four major areas: data management and supporting architectures; model development and scoring; visualization and user interaction; and business models.