fathom: Qualitative Coding and Measurement with Local Language Models

The fathom R package is a toolkit for qualitative coding and measurement of a corpus of text, images, audio, or video with local large language models (LLMs), where every answer comes back as a probability rather than a word.

With the function measure(), R users can ask an Ollama-hosted LLM a constrained set of labeled questions about a corpus document and get back the model's next-token probability distribution for those constrained labels. The questions themselves are built and bundled with the function instrument(). The function item_report() provides a clear summary of how each question behaved across the entire corpus, while the function inspect_cases() displays documents that the model answered with a specific label. The function as_wide() prepares the output for any downstream analysis by pivoting the results so that each row is one document and each question becomes a column. To audit the model's answers, the function coding_sample() samples a random set of document-question pairs for human validation, which is then executed through the function validate_sample().

Motivation

Getting an LLM's response about a question provides no insight as to how confident it was in determining that answer. This is further obfuscated if a prompt does not provide a clear and explicit instruction to answer with a set of constrained response labels. Constraining a labeled set of responses and extracting their log probabilities allows a user to use the responses as a classification tool or continuous measurement item. Additionally, being able to read a model's probability distribution allows a user to assess how much probability mass was placed on each of the labels, revealing the presence of a compound question, a missing category, or a bad prompt.

The fathom workflow

0. Prepare the corpus (non-text corpora)

prepare_image()

Turns a set of image files into a valid corpus, where each row is one image filepath alongside its file size.

prepare_audio()

Turns a set of audio files into a valid corpus by using an Automatic Speech Recognition (ASR) model to transcribe each recording to text.

prepare_video()

Turns a set of video files into a valid corpus, where each row is one video holding its extracted frames alongside an ASR transcript of its audio.

1. Prepare a local model

ollama_backend()

Connects to a local Ollama server and confirms the model is installed.

2. Write the coding instrument

instrument()

Bundles the questions with the single-word labels the model may answer with.
Questions can be structured as (1) independent items or (2) exclusive items.
(1) Independent items ask a separate binary question for each category, meaning that a document can match several at once. (2) Exclusive items make the categories the labels of a single question, meaning that they share a single distribution.

template_items() (optional)

Builds an instrument of independent items by inserting each category from a list into one question template.

preview() (optional)

Prints a preview of the exact string that the model will receive.

3. Extract response probabilities

measure()

Calls the LLM with an instrument of questions and labels over a data frame or a character vector of documents, where each call processes one document and one question.
Returns an object containing (1) a table holding the probability of every label for each document-question pair and (2) a table holding details for each document-question pair, including the model's first response token, the share of probability that fell outside the constrained labels, and the probability gap between the top two labels.

4. Summarize the results

item_report()

Summarizes how each question was answered by the model across the corpus.

inspect_cases() (optional)

Prints the full text of a sample of documents where a question's target label won (e.g., text where the model answered "yes"), allowing a user to gut-check the model's answers.

as_wide()

Pivots the first table from measure() into one row per document and one column per question, each holding the probability of the target label (e.g., the probability of "yes"). For exclusive items, each category gets its own column instead.

5. Validate response probabilities

coding_sample()

Draws a seeded random sample of document-question pairs into an exportable data frame coding sheet.

coder_reference() (optional)

Consolidates two or more completed coding_sample() sheets into one, allowing a group of coders to serve as the single reference for validate_sample().

validate_sample()

Validates the model's answers from measure() against a completed coding_sample() sheet in terms of discrimination and ranking, with calibration on request. Given a named list of sheets, it also reports intercoder reliability and how far the model's responses are from the coders.

6. Get measurement diagnostics (optional)

threshold_sensitivity()

Reports endorsement rates at a range of cut points.

The package website carries the reference index and several articles.

Requirements

A running Ollama server at version 0.12.11 or later, and a compatible Ollama instruction-following model.

ffmpeg for measuring audio and video.

whisperx or whispermlx (Apple Silicon optimized) for measuring audio, with a HuggingFace User Access Token and access to the Community-1 model to use speaker diarization.

See the Quick start page for information on how to set each of these up.

Installation

fathom is distributed through R-Universe.

install.packages(
  "fathom",
  repos = c("https://alcocer-jj.r-universe.dev", "https://cloud.r-project.org")
)

The development version installs from GitHub.

# install.packages("pak")
pak::pak("alcocer-jj/fathom")

Acknowledgments

Development of this package was assisted by Claude Code, an AI coding assistant by Anthropic, for code review, documentation drafting, and the validation components.