The Techno-Politics of Conversational AI's Moral Agency: Examining ChatGPT and ErnieBot as Examples

Abstract

This chapter explores and problematizes the moralization of conversational AI tools. Instead of narrowly defining moral agency as a machine’s capacity for autonomous moral decision-making, our conceptualization focuses on the system’s ability to adhere to predefined ethics and values. Using ChatGPT and ErnieBot as case studies, we examine moral agency as a technological and a political construct, highlighting how definitions of morality are shaped by societal power struggles. We also discuss users’ role in challenging the moral agency of conversational AI, focusing on chatbot jailbreaking. The chapter concludes by addressing governance challenges, including tech firms’ inherent self-interest and their simultaneous aspirations for societal benefits—whether genuine or cosmetic—and existing societal discord and polarization.