https://arxiv.org/pdf/2306.05524.pdf
Title: Check Me If You Can: Detecting ChatGPT-Generated Academic Writing using CheckGPT
Authors: Zeyan Liu, Zijun Yao, Fengjun Li, Bo Luo
Word Count: Approximately 7,600
Estimated Read Time: 26-28 minutes
Github: https://github.com/progressionnetwork/CheckGPT_RestAPI
Summary:
The paper aims to investigate the use and misuse of ChatGPT in academic writing as well as the difficulty of detecting ChatGPT-generated text.
First, the authors collect a dataset of 600,000 human-written and ChatGPT-generated research paper abstracts in three disciplines. They identify three scenarios for ChatGPT usage: writing from scratch, completing partial text, and polishing existing text.
Second, the authors evaluate state-of-the-art detectors on the dataset and find that they provide unsatisfactory results, especially for polished text. A user study with 150+ participants shows that humans, including experienced researchers, are unable to accurately identify ChatGPT-generated abstracts.
The authors then propose CheckGPT, a novel detector that uses a pre-trained transformer model for representation and an attentive LSTM for classification. It achieves high accuracy (>98%) on the dataset and demonstrates transferability to new domains and models.
The key strengths of CheckGPT are:
-
Affordability: It reuses pre-trained transformer models and requires less computation to deploy compared to fine-tuning the full transformer.
-
Transferability: By learning generalized features, CheckGPT can be quickly adapted to new domains and tasks with minimum data.
-
Interpretability: The authors conduct analyses to reveal how CheckGPT detects ChatGPT-generated writing.
In summary, CheckGPT provides an effective solution to the challenge of detecting ChatGPT-generated academic writing. The dataset, code and tool will be shared publicly for further research.
CheckGPT can be used to help monitor and enforce policies regarding the use of AI tools in academic publications. The elaborated investigation and insight in this study can inspire future research to combat the misuse of ChatGPT and similar large language models.
