Echocardiography provides important morphological and functional details of the heart which can be used for the diagnosis of various cardiac diseases. Most of the existing automatic cardiac disease recognition systems that use echocardiograms are either based on unreliable anatomical region detection (e.g. left ventricle) or require extensive manual labeling of training data which renders such systems unscalable. In this paper we present a novel system for automatic cardiac disease detection from echocardiogram videos which overcomes these limitations and exploits cues from both cardiac structure and motion. In our framework, diseases are modeled using a configuration of novel salient features which are located at the scale-invariant points in the edge filtered motion magnitude images and are encoded using local spatial, textural and motion information. To demonstrate the effectiveness of this technique, we present experimental results for automatic cardiac Hypokinesia detection and show that our method outperforms the existing state-of-the-art method for this task. © 2010 IEEE.