AI Detection: Principles and Approaches

Introduction

Introduction

Hi everyone, I am Jessica Parker, the co-founder and CEO of Moxy. Alongside my co-founder Kimberly Becker, we are diving deep into AI detection in this second webinar of a three-part series. If you missed the first webinar, you can find it on our YouTube channel through the link on our flow page. This series aims to build from the basic foundational knowledge of large language models to understanding AI detectors and eventually, rhetorical AI literacy.

Agenda

Today, we’ll cover the following:

• Transparency in AI
• How AI detectors work
• Ethical implications associated with AI detection
• Characteristics of AI-generated vs. human-generated text
• Q&A session

About Us

I, Jessica Parker, am the co-founder and CEO of Moxy and also a lecturer at the Massachusetts College of Pharmacy and Health Sciences. Moxy is an AI ed-tech company focusing on comprehensive research and writing support. We build specialized AI tools and offer professional development sessions, a community practice on Slack, and contribute to ongoing research.

My co-founder, Kimberly Becker, has a background in Applied Linguistics and technology, focusing on generative AI from a natural language processing perspective. She supports academic research writing and communication.

Transparency in AI: Black Box Vs. Glass Box

Transparency in AI has gained much traction lately. The term "black box" is often used to describe large language models because while we understand the input and output, the decision-making process within remains opaque. However, we propose the term "glass box" to highlight certain transparent aspects, like the type of data used and potential biases.

Input and Learning Process

Large language models learn from diverse examples on the internet, using neural networks for pattern recognition. While we know they use data like websites, Wikipedia, and books, the exact sources can sometimes be opaque, making them a "black box." However, understanding the input data’s representation helps us view them as a "glass box," highlighting the biases and stereotypes these models might perpetuate.

Perplexity and Burstiness in AI Detection

When an AI detector analyzes text, it considers two measures: perplexity and burstiness. Perplexity measures how predictable the text is, while burstiness evaluates sentence variation. Text with low perplexity and burstiness is more likely to be AI-generated because it's more predictable and uniform.

Ethical Implications

AI detection methods are not foolproof. They are plagued with false positives, inconsistencies, and biases against non-native English speakers. Vanderbilt University stopped using Turnitin’s AI detector because even a 1% error rate was unacceptable for their volume of student work.

Alternatives to AI Detection

• Watermarks: Invisible marks during the creation process.
• Stylometric Analysis: Analyzing writing patterns subtly.
• Context Awareness: Considering the author's background and writing history.
• Multimodal Analysis: Using multiple types of data for analysis.

Historical Context

The concerns around AI detection echo past anxieties about writing technologies like typewriters, spell checkers, and Wikipedia. Just as we overcame those fears, it's likely we'll adapt to and become comfortable with AI in due course.

Rethinking Assessment

AI challenges us to rethink curricula and assessments. Moving towards process-oriented approaches and emphasizing critical thinking over final products can help mitigate the ethical issues surrounding AI detection.

Conclusion

We support Sarah Eaton’s view of a hybrid human-AI writing future, where AI enhances creativity without replacing human responsibility. The goal is to help students use AI tools wisely while acknowledging their contributions appropriately.

Join us next week for our final webinar in this series on rhetorical AI literacy, where we will explore the characteristics of human vs. AI-generated writing.

Keywords

• AI Detection
• Large Language Models
• Transparency
• Perplexity
• Burstiness
• Ethical Implications
• Rethinking Assessment
• Hybrid Human-AI Writing

FAQ

What is the difference between a black box and a glass box in AI?

A black box refers to AI systems where the internal workings are not visible or understandable, while a glass box has transparent aspects that help us understand biases and input data.

How do AI detectors measure the likelihood of AI-generated text?

AI detectors use perplexity and burstiness to measure text predictability and sentence variation, respectively.

Are AI detectors 100% accurate?

No, AI detectors are prone to false positives and inaccuracies, making them unreliable for definite conclusions.

What are some alternatives to AI detection?

Alternatives include watermarks, stylometric analysis, context awareness, and multimodal analysis.

How should educators adapt to the rise of AI writing tools?

Educators should consider process-oriented approaches, emphasize critical thinking, and rethink assessments to better incorporate AI tools.

What is the view on hybrid human-AI writing?

Hybrid human-AI writing is seen as the future, where AI tools can enhance creativity and output while humans retain responsibility and proper attribution for their use.