AI Summary & Key Takeaways
AI InsightThe findings support local validation and explicit reporting of human false-positive rates in consequential academic use.
📌 3 Key Takeaways
Abstract
AI text detectors are used in academic integrity settings. Their validity may change across time, providers, and writing conditions. This study examined AI-text detection in Turkish academic writing using a leakage-audited historical corpus and a prespecified provider-known benchmark. The historical audit included 50,913 human-written documents, including 40,381 dated from 2000 to 2019. The benchmark contained 300 project-lineage-held-out human sources and 2,398 valid variants. Variants were produced by OpenAI, Gemini, DeepSeek, and Claude under four writing conditions: full generation, AI polishing, deeply mixed human–AI writing, and humanization. Six supervised detectors, one lineage-unresolved deployed system, and a frozen zero-shot baseline were evaluated at locked thresholds. Human false-positive rates and AI recall were estimated with exact binomial intervals. Score uncertainty was estimated with a source-seed clustered bootstrap. XLM-R achieved the highest held-out AUROC at 0.9154. Its human false-positive rate was 0.0400, with AI recall of 0.6952. BERTurk had a human false-positive rate of 0.0267 and AI recall of 0.6197. The deployed system reached 0.8603 recall and falsely flagged 35.0% of human texts. OpenAI outputs were the most difficult provider condition for all project-lineage-held-out supervised detectors. Deeply mixed text produced the lowest recall for the transformer models. AI-polished text was the most difficult condition for the TF-IDF baselines. The zero-shot baseline remained near chance at an AUROC of 0.5069. Its recall was zero at the locked threshold. Historical threshold transfer varied by architecture. The findings support local validation and explicit reporting of human false-positive rates in consequential academic use.