ChatGPT Data Extraction: A quick demonstration

Introduction

In the realm of data journalism, extracting data from messy documents, particularly PDFs, presents a myriad of challenges. Though government entities are obligated to publish public documents, they are not required to clean them up or format them into user-friendly spreadsheets. As a data journalist, I often face requests from reporters asking me to convert these complex documents into easily analyzable data formats. My typical approach involves writing Python scripts to parse and clean the data, which can be time-consuming and prone to complications. However, I recently discovered that ChatGPT can significantly streamline the data extraction process. Here’s a step-by-step guide on using ChatGPT for effective data extraction from messy documents.

Step 1: Convert PDFs to Text

The first step in this process involves converting the PDF document into a regular text file. For this demonstration, I utilized a tool called PDFPlumber to extract text and pasted it into ChatGPT, framing the request to return a JSON representation of the text. This method dramatically simplified the data parsing process, allowing ChatGPT to handle much of the heavy lifting.

Step 2: Handling Complex Formats

Next, I tackled a more complex document resembling a police use-of-force report in tabular format. This table was challenging due to its irregular structure — field names and values were not clearly aligned in a systematic manner. After copying the text from this document into ChatGPT and requesting a JSON representation while omitting unnecessary complaint information, I received a remarkably accurate output. Even better, ChatGPT effectively recognized the potential for multiple officers per complaint and structured the data accordingly.

Step 3: Utilizing JSON Schema

In another test, I worked with a project containing a weirdly formatted table with split data entries. Rather than manually specifying fields for extraction, I leveraged ChatGPT’s understanding of JSON schema to request a more organized representation. I defined the expected schema for the data and provided clear instructions to split numerical and percentage data accurately. While the JSON output was truncated due to response limits, the results demonstrated how effective ChatGPT could be in such scenarios.

Step 4: Scaling Data Extraction

Finally, I addressed the challenge of extracting data from numerous documents. With thousands of police memos to analyze, I created a Python script called “ChatGPT Document Extraction." This script allows users to ingest input data as either text or JSON, define a corresponding JSON schema, and automate the extraction process for all records in the file, creating a comprehensive output.

While ChatGPT presents a powerful tool for data extraction, it's essential to recognize its limitations. Errors and inaccuracies may occur during data extraction, which necessitates careful verification before sharing the results with the audience. I plan to delve deeper into this topic in an upcoming article for Open News, where I will discuss the nuances and potential pitfalls of using AI for data journalism.

For further insights into the intersection of journalism, technology, and data, feel free to visit my website at bxroberts.org.

Keywords

ChatGPT, data extraction, JSON representation, PDF, data journalism, Python script, data analysis, JSON schema, document parsing, police use-of-force report.

FAQ

Q: What does ChatGPT do for data extraction?
A: ChatGPT can analyze messy documents, extracting data and converting it into a structured JSON format, reducing the need for complex scripting.

Q: Can ChatGPT handle complicated document formats?
A: Yes, ChatGPT can manage complex formats and relationships within the data, making it easier to extract relevant information.

Q: Is ChatGPT error-free in data extraction?
A: No, ChatGPT can introduce inaccuracies. It's essential to review and verify results before sharing them with others.

Q: How can I use ChatGPT for large-scale data extraction?
A: You can utilize scripts that interface with ChatGPT, allowing for batch processing of multiple documents and defined JSON schemas for structured output.

Q: Where can I find more information on data journalism and technology?
A: You can visit the author's website, bxroberts.org, for articles and resources related to journalism technology and data analysis.