Understanding the operation of a building is key for improving it and reducing energy waste. However, today this is a mostly manual task for which domain experts use visual tools to analyze the large amounts of building data. We show how to automate this task by means of pattern extraction techniques. These allow human operators to simply consider well defined data patterns rather than vast amounts of data. Here the manual effort consists in identifying the context in which the patterns occur. In this paper we go one step further and show how we can automatically derive also the context in which building operation patterns occur by considering the influencing factor that govern building operation. We have evaluated our approach for two pattern extraction methods: matrix factorization and clustering. Our experimental results using real world data demonstrate the applicability of our work.