Autonomic operations in cooperative stream processing systems

Abstract

System S is a large-scale distributed streaming data analysis environment, designed to handle extreme data rates. Multiple System S sites can cooperate to further improve the scale, breadth and depth of data analysis. We describe three autonomic features in the operation of such a cooperative stream processing environment: interoperation models, planning, and failover, They enable the distributed environment to deal with dynamic and rapid changes in imposed workload, available resources, and the priorities of administrators and users. Thus, the system can minimize the human effort needed to operate such large, complex systems.