A Comparative Analysis of Classification Methods Across Document Granularities and Unseen Large Language Models

Abstract

The rapid public adoption of large language models (LLMs) has transformed written communication and creative production, while simultaneously creating growing concerns regarding authenticity, misinformation, and the preservation of human authorship. As AI-generated literary content becomes increasingly sophisticated, the ability to reliably distinguish machine-generated prose from authentic human writing has become a significant challenge. This thesis evaluates the effectiveness of multiple machine learning and deep learning classification methods in detecting AI-generated literary imitations and examines how document granularity and cross-model generalization impact classification performance. A curated dataset was constructed using eight English-language novels obtained from Project Gutenberg alongside stylistically constrained imitation chapters generated by three modern LLMs: OpenAI's ChatGPT, Anthropic's Claude, and Google's Gemini. To minimize contextual leakage, prompts enforced consistency in setting, characters, and authorial style, requiring classifiers to rely primarily on lexical and structural characteristics rather than obvious narrative cues. Textual data were transformed using TF-IDF vectorization and evaluated using Logistic Regression, Random Forest, Support Vector Machine (SVM), Naive Bayesian Classifier (NBC), a custom Neural Network, and the transformer-based RoBERTa model.

Experiments were conducted across three document granularities: chapters, pages, and paragraphs. Results demonstrate that document segmentation substantially influences classifier behavior. RoBERTa consistently achieved the strongest performance, maintaining near-perfect accuracy across all document sizes, while Random Forest and NBC achieved perfect chapter-level accuracy but degraded significantly as document size decreased. Logistic Regression and SVM exhibited improved performance at intermediate page-level segmentation, suggesting that some linear models benefit more from increased corpus size than from larger contextual windows. Across multiple models, a persistent false negative bias emerged, with AI-generated text frequently misclassified as authentic human writing. Further experiments explored model robustness through repeated random-seed evaluation, attribution of AI-generated text to its originating LLM, and Leave-One-Out (LOO) testing against unseen language models. While several classifiers successfully identified the originating LLM with near-perfect accuracy, most models demonstrated substantial degradation when evaluated against previously unseen AI architectures, revealing limited cross-model generalization. Among the evaluated methods, NBC demonstrated the strongest robustness under zero-shot conditions.

Overall, this thesis established that AI-generated literary data contains detectable statistical signatures, but that classifier effectiveness is highly dependent on document structure and exposure to diverse generator models. These findings contribute a scalable benchmark framework for AI text detection and highlight the importance of document granularity and out-of-distribution evaluation in the development of future detection systems.

• author: O'Leary, Declan
• supervisor: Hamed Sabahno
• organization: Department of Statistics
• course: STAN40 20261
• year: 2026
• type: H1 - Master's Degree (One Year)
• subject: Mathematics and Statistics
• keywords: Machine Learning, Statistics, AI Detection, Classification, Logigistic Regression, Random Forest, Support Vector Machine, Naive Bayesian Classifier, Neural Network, RoBERTa, Deep Learning, Natural Language Processing, Textual Analysis
• language: English
• id: 9231831
• date added to LUP: 2026-06-08 14:33:46
• date last changed: 2026-06-08 14:33:46