Towards AI-Sympathy Using Agents

Abstract

As AI systems become collaborative partners rather than passive tools effective, human-AI teams require reciprocal understanding between systems and their users. Across established team-efficiency models, four foundations consistently emerge: shared understanding, decision-making models, effective communication, and trust. However, these foundations are difficult to achieve as humans often lack accurate mental models of the AI and AI systems have limited representations of human cognitive structures and their own capabilities. As with any new tool, humans must be trained to use AI effectively. Yet AI is unique in its ability to interact and potentially train humans to use itself. In this paper, we introduce three new concepts as essential for genuine human-AI teaming: AI-sympathy (humans understanding AI capabilities and limitations), Human-sympathy (AI systems understanding human context and constraints) and Self-sympathy (the AI system’s understanding of its own limitations and capabilities), and present a roadmap for the development of effective human-AI teamwork based on these concepts which identifies four main stages in this evolution.