This paper describes our latest work on identifying anomalous tie plates to automate railroad inspection using machine vision technology. Specifically, we have developed a completely automatic detection scheme to recognize tie plates with anomalous spiking patterns using various video analytics. In particular, each tie plate is first represented by four characteristic regions-of-interest (ROI), then each ROI is fed into a pre-trained SVM (Support Vector Machine) model, and classified to be either spike- or spike hole-related. Next, the dissimilarity between the current tie plate and a reference set of tie plates in a sliding window is measured and analyzed. Based on that, it is finally recognized as either an anomalous or a normal tie plate. Preliminary experiments conducted on a set of videos captured by our own designed imaging system, has achieved an average precision, recall and false alarm rates of 88%, 92.8% and 2.16%, respectively. This validates the promising direction of applying machine vision technology to assist in railroad inspection. © 2012 ICPR Org Committee.