Most collaborative filtering models assume that the interaction of users with items take a single form, e.g., only ratings or clicks or views. In fact, in most real-life recommendation scenarios, users interact with items in diverse ways. This in turn, generates complex usage data that contains multiple and diverse types of user feedback. In addition, within such a complex data setting, each user-item pair may occur more than once, implying on repetitive preferential user behaviors. In this work we tackle the problem of building a Collaborative Filtering model that takes into account such complex datasets. We propose a novel factor model, CDMF, that is capable of incorporating arbitrary and diverse feedback types without any prior domain knowledge. Moreover, CDMF is inherently capable of considering user-item repetitions. We evaluate CDMF against stateof- the-art methods with highly favorable results.