What makes us different: Understanding the differences between human and machine-generated text

Abstract

In an era where Large Language Models (LLMs) generate text that closely mimics human language, the utility of machine-generated text spans diverse applications, from news composition to critical fields like law and education. However, the proliferation of LLMs also raises significant risks, such as fake news, fraudulent schemes, and academic dishonesty. To combat these threats, it is crucial to distinguish between human and AI-generated text. This study explores the efficacy and robustness of various AI text detectors, focusing on their ability to withstand adversarial paraphrasing attacks. We also investigate how text perplexity, a measure of unpredictability of text for a model, can serve as a reliable metric for detection and introduce a perplexity-based detector that competes with more complex models. Additionally, we examine the role of explainable artificial intelligence (XAI) methods in understanding and improving detection mechanisms. Through a user survey, we compare human and AI performance in text detection, understand the cognitive decisions of humans in the task of and assess the potential of XAI techniques to enhance human decisionmaking. This comprehensive analysis aims to bolster the development of robust and interpretable AI text detection systems, ensuring their reliability in real-world applications.