ChatGPT, R and Me: A Deep Dive into AI-Assisted Image Analysis

ByThompson

In the rapidly evolving landscape of artificial intelligence and data science, the synergy between powerful language models like ChatGPT and programming languages such as R is opening up exciting new possibilities for researchers and analysts. This blog post delves into a fascinating experiment that combines the capabilities of ChatGPT with R programming to tackle complex image processing tasks, exploring how AI can assist in analyzing congressional headshots, extracting valuable data, and even attempting to measure the "smiliness" of politicians.

The Journey Begins: Downloading Congressional Headshots

Our adventure begins with a seemingly simple task: downloading headshots of members of Congress. However, the challenge lies in the fact that these images are embedded within PDF documents, each containing multiple images per page. Many data analysts might find this task daunting, but ChatGPT rose to the occasion with a surprisingly effective solution.

ChatGPT generated a code snippet using the httr library to download the PDF files:

library(httr)

base_url <- "https://www.govinfo.gov/content/pkg/GPO-PICTDIR-117/pdf/GPO-PICTDIR-117-10-"
dir_name <- "117th_congress_pdfs"

if(!dir.exists(dir_name)){
  dir.create(dir_name)
}

for(i in 1:50){
  url <- paste0(base_url, i, ".pdf")
  filename <- paste0(dir_name, "/", i, ".pdf")
  GET(url, write_disk(filename))
}

This code efficiently downloads PDF files for each state, demonstrating ChatGPT's ability to generate functional code for specific tasks. As an AI prompt engineer, I was impressed by the model's understanding of the problem and its ability to provide a working solution. However, it's crucial to note that human oversight is still necessary to ensure the code aligns with the intended goals and follows best practices.

From PDFs to Images: The Conversion Process

With the PDFs successfully downloaded, the next step was to convert them into individual image files. ChatGPT once again provided a code snippet to accomplish this task:

library(pdftools)

input_dir <- "117th_congress_pdfs"
output_dir <- "117th_congress_images"

if (!dir.exists(output_dir)) {
  dir.create(output_dir)
}

for (i in 1:50) {
  input_path <- paste0(input_dir, "/", i, ".pdf")
  output_path <- paste0(output_dir, "/", i, "_page_")
  
  pages <- pdf_info(input_path)$pages
  for (j in 1:pages) {
    png_file <- paste0(output_path, j, ".png")
    pdf_convert(input_path, dpi = 300, pages = j, filenames = png_file)
  }
}

This code successfully converted the PDF pages into PNG images, showcasing ChatGPT's ability to provide solutions for multi-step processes. As an AI expert, I find it fascinating how ChatGPT can chain together multiple operations to solve complex problems. However, it's important to note that as tasks become more complex, the likelihood of encountering errors or limitations in ChatGPT's responses increases.

The Challenge of Image Parsing

The next phase of the project involved parsing the individual headshots from the converted images. Each page contained up to four images, presenting a new challenge. While ChatGPT initially provided code for this task, it didn't work as expected. This highlights an important limitation of AI assistants: they may sometimes generate plausible-looking but non-functional code, especially for more specialized tasks.

After some trial and error and human intervention, a working solution using the 'magick' package was developed:

library(magick)

png_path <- "117th_congress_images"
output_path <- "117th_congress_images_parsed"

if (!dir.exists(output_path)) {
  dir.create(output_path)
}

png_files <- list.files(png_path, full.names = TRUE)

for (png_file in png_files) {
  img <- image_read(png_file)
  
  left = c(150, 690, 150, 690)
  top = c(165, 165, 940, 940)
  
  for (i in 1:4) {
    cropped_img <- image_crop(img, geometry = paste0("450x570+", left[i], "+", top[i]))
    output_file <- file.path(output_path, gsub(".png$", paste0("_", i, ".png"), basename(png_file)))
    image_write(cropped_img, path = output_file)
  }
}

This experience underscores the importance of human expertise in fine-tuning and troubleshooting AI-generated code, especially for tasks requiring precise parameters or domain-specific knowledge. As an AI prompt engineer, I often encounter situations where the initial AI output needs refinement, and this case serves as an excellent example of the iterative process of working with AI assistants.

Extracting Text Data with OCR

With the images successfully parsed, the next step was to extract text data from the captions below each image. ChatGPT suggested using the 'tesseract' package for Optical Character Recognition (OCR), which proved to be quite effective:

library(tesseract)

caption_dir <- "117th_congress_images_parsed/"
congress_info <- data.frame(name = character(), title = character(), party = character(), statecode = numeric(), dist = character(), stringsAsFactors = FALSE)

caption_files <- list.files(caption_dir, pattern = "caption", full.names = TRUE)

for (i in seq_along(caption_files)) {
  caption_file <- caption_files[i]
  caption_text <- tesseract::ocr(caption_file)
  
  # Code to parse and structure the extracted text
  ...
}

This code successfully extracted and structured information from the image captions, demonstrating ChatGPT's ability to suggest appropriate tools and generate code for text processing tasks. As an AI expert, I find it remarkable how ChatGPT can seamlessly integrate different libraries and techniques to solve complex problems.

Venturing into Facial Recognition

The project then took an ambitious turn towards facial recognition, specifically focusing on analyzing smiles in the headshots. ChatGPT introduced the 'opencv' package for facial recognition, but its assistance became more limited as the task grew more specialized:

library(opencv)
library(magick)

crop_smile <- function(sourcename, outname) {
  img <- ocv_read(sourcename)
  facemask <- ocv_facemask(img)
  
  if (nrow(attr(facemask,'faces'))==0) return(NULL)
  
  # Code to crop and save the smile region
  ...
}

smile_dir <- "117th_congress_smiles/"
if (!dir.exists(smile_dir)) {
  dir.create(smile_dir)
}

for (j in 1:nrow(congress_info)) {
  outnm = paste(smile_dir, "/", j, ".jpg", sep = "")
  crop_smile(sourcename = congress_info$image[j], outname = outnm)
  congress_info$smile[j] = outnm
}

This phase of the project highlighted the current limitations of AI in highly specialized tasks, requiring more human intervention and domain knowledge. As an AI prompt engineer, I recognize that while AI models like ChatGPT have broad knowledge, they may struggle with cutting-edge or highly specific applications. This underscores the importance of human expertise in guiding and supplementing AI-generated solutions.

Data Preparation and Dimensionality Reduction

The final stages of the project involved converting the cropped smile images into workable data and reducing its dimensionality. ChatGPT provided guidance on these steps, though the approach suggested (converting images to greyscale vectors) might not be the most sophisticated method for image analysis:

library(jpeg)
library(class)

image_to_vector <- function(image_paths, size = c(150, 70)) {
  # Code to convert images to vectors
  ...
}

image_paths <- list.files(smile_dir, full.names = TRUE)
image_matrix <- image_to_vector(image_paths, size = c(150, 70))
image_matrix = t(image_matrix)

# Dimensionality reduction
pca_matrix = princomp(image_matrix)
data_scaled <- scale(image_matrix)
pca <- prcomp(data_scaled, scale = FALSE)
n_components <- 100
data_reduced <- as.data.frame(pca$x[, 1:n_components])

As an AI expert, I find it interesting how ChatGPT suggested using Principal Component Analysis (PCA) for dimensionality reduction. While this is a valid approach, it's worth noting that more advanced techniques like convolutional neural networks (CNNs) might be more suitable for image analysis tasks. This highlights the importance of staying updated with the latest advancements in machine learning and computer vision.

Building a Smile Classifier

The experiment concluded with an attempt to build a classifier for smile intensity. ChatGPT assisted in creating a function to collect training data and implement a basic k-nearest neighbors classifier:

collect_smile_size <- function(img_paths) {
  # Code to manually label smile sizes
  ...
}

training_data = collect_smile_size(names(train_data))

k <- 5
classifier <- knn(train = train_data, test = test_data, cl = training_data$smile_size, k = k)

While this approach provides a starting point, it's important to note that more sophisticated methods, such as deep learning models specifically designed for facial expression recognition, could yield more accurate results. As an AI prompt engineer, I would recommend exploring state-of-the-art computer vision models for tasks like this.

The Role of AI in Research and Analysis

This experiment showcases both the exciting possibilities and current limitations of using AI assistants like ChatGPT in data science projects. As an AI expert, I've observed that ChatGPT excels in providing quick solutions and suggesting relevant tools for a wide range of tasks. However, it's crucial to understand that AI is not a replacement for human expertise but rather a powerful tool to augment and accelerate research processes.

The iterative collaboration between human researchers and AI assistants is key to achieving optimal results. While ChatGPT can generate code snippets and provide general guidance, human oversight is essential for:

  1. Ensuring code correctness and efficiency
  2. Adapting solutions to specific project requirements
  3. Integrating domain knowledge and best practices
  4. Critically evaluating and refining AI-generated suggestions

Moreover, as we delve into analyzing personal images and data, it's crucial to consider the ethical implications. Researchers must ensure proper consent, data handling practices, and privacy protection measures are in place. AI models like ChatGPT can assist in generating code, but the responsibility for ethical considerations lies with the human researchers.

Future Directions and Potential Applications

Looking ahead, the integration of AI assistants in research workflows holds immense potential. As language models continue to evolve, we can expect even more sophisticated code generation and problem-solving capabilities. However, it's important to remember that the most effective approach will likely remain a synergy between AI assistance and human expertise.

Some exciting future directions for AI-assisted research include:

  1. Automated literature reviews and data synthesis
  2. Real-time code optimization and debugging suggestions
  3. Intelligent experimental design and data collection strategies
  4. Advanced natural language interfaces for data analysis and visualization

As an AI prompt engineer, I'm particularly excited about the potential for more intuitive and natural interactions between researchers and AI assistants. This could lead to more efficient workflows and enable researchers to focus on high-level problem-solving and creative thinking.

Conclusion: The Future of AI-Assisted Research

In conclusion, this experiment with ChatGPT and R programming for image analysis demonstrates the powerful potential of AI assistants in accelerating research processes. From generating functional code for data downloading and processing to suggesting relevant tools and approaches, ChatGPT proved to be a valuable ally in tackling complex tasks.

However, the project also highlighted the current limitations of AI, particularly in highly specialized domains. As we continue to push the boundaries of AI-assisted research, it's crucial to maintain a balance between leveraging AI capabilities and applying human expertise, critical thinking, and ethical considerations.

The future of research lies in the seamless integration of AI assistants and human intelligence, creating a symbiotic relationship that enhances our ability to explore, analyze, and innovate. As AI technology continues to advance, researchers who can effectively harness these tools while maintaining their domain expertise will be at the forefront of scientific discovery and technological innovation.

By embracing AI assistants like ChatGPT as powerful tools in our research arsenal, we can unlock new frontiers in data analysis, accelerate scientific progress, and tackle increasingly complex challenges across various fields of study. The journey of AI-assisted research is just beginning, and the possibilities are truly exciting for those willing to explore this new frontier of human-AI collaboration.

Similar Posts