Automated Visual Analysis for the Study of Social Media Effects: Opportunities, Approaches, and Challenges

Abstract

To advance our understanding of social media effects, it is crucial to incorporate the increasingly prevalent visual media into our investigation. In this article, we discuss the theoretical opportunities of automated visual analysis for the study of social media effects and present an overview of existing computational methods that can facilitate this. Specifically, we highlight the gap between the outputs of existing computer vision tools and the theoretical concepts relevant to media effects research. We propose multiple approaches to bridging this gap in automated visual analysis, such as justifying the theoretical significance of specific visual features in existing tools, developing supervised learning models to measure a visual attribute of interest, and applying unsupervised learning to discover meaningful visual themes and categories. We conclude with a discussion about future directions for automated visual analysis in computational communication research, such as the development of benchmark datasets designed to reflect more theoretically meaningful concepts and the incorporation of large language models and multimodal channels to extract insights.