#VisualHashtags: Visual summarization of social media events using mid-level visual elements

Abstract

The data generated on social media sites continues to grow at an increasing rate with more than 36% of tweets containing images making the dominance of multimedia content evidently visible. This massive user generated content has become a reflection of world events. In order to enhance the ability and effectiveness to consume this plethora of data, summarization of these events is needed. However, very few studies have exploited the images attached with social media events to summarize them using "mid-level visual elements". These are the entities which are both representative and discriminative to the target dataset besides being human-readable and hence more informative. In this paper we propose a methodology for visual event summarization by extracting mid-level visual elements from images associated with social media events on Twitter (#VisualHashtags). The key research question is Which elements can visually capture the essence of a viral event?, hence explain its virality, and summarize it. Compared to the existing approaches of visual event summarization on social media data, we aim to discover #VisualHashtags, i.e., meaningful patches that can become the visual analog of a regular text hashtag that Twitter generates. Our algorithm incorporates a multi-stage filtering process and social popularity based ranking to discover mid-level visual elements, which overcomes the challenges faced by direct application of the existing methods. We evaluate our approach on a recently collected social media event dataset, comprising of 20,084 images. We evaluate the quality of #VisualHashtags extracted by conducting a user-centered evaluation where users are asked to rate the relevance of the resultant patches w.r.t. the event and the quality of the patch in terms of how meaningful it is. We also do a quantitative evaluation on the results. We show a high search space reduction of 93% in images and 99% in patches after summarization. Further, we get a 83% of purity in the resultant patches with a data coverage of 18%.