We study the problem of item recommendation using complex usage data. We assume that users may interact with items in various ways, each such interaction generates a usage point which may be accompanied with multiple feedback types. In addition, each user may interact with each item multiple times. We propose a generic framework that re-models the user vectors as a post-processing step that can be applied to any Matrix Factorization (MF) method. Using an evaluation on several heterogeneous real-world datasets, we demonstrate the effectiveness of the approach and demonstrate its superiority over two alternative methods.