A complete run

This article provides an example of how to run the whole fathom package pipeline. The Quick start page covers package installation and Ollama setup.

The pipeline uses a dataset available in the package (parlasent) of one hundred UK parliamentary sentences that were sampled from the English portion of ParlaSent 1.0 by Mochtak, Rupnik, Meden, and Ljubešić (2023).

Overview

1. Select a model with ollama_backend().
2. Code the qualitative instrument with instrument() and use preview() to see what the local large language model (LLM) will read.
3. Extract response probabilities with measure().
4. Read the results with item_report() for descriptive summaries, use inspect_cases() to manually validate some entries, and apply as_wide() to hand off for downstream analyses.
5. Validate the LLM results with coding_sample() and validate_sample().

The corpus

parlasent contains data on two annotators who coded every parliamentary sentence independently based on a six-level ordered scale. In addition, it contains a reconciliation column resulting from those two. annotator1 and annotator2 are the two codings, reconciliation is what they settled on.

library(dplyr)
library(fathom)
library(pander)

data(parlasent)
parlasent %>% select(id, sentence, annotator1, annotator2, reconciliation)
#> # A tibble: 100 × 5
#>    id     sentence                          annotator1 annotator2 reconciliation
#>    <chr>  <chr>                             <chr>      <chr>      <fct>         
#>  1 uk_001 First, the UK Coalition against … N_Neutral  N_Neutral  N_Neutral     
#>  2 uk_002 The thought that on top of that … Negative   Negative   Negative      
#>  3 uk_003 Thanks to their many victories, … P_Neutral  P_Neutral  P_Neutral     
#>  4 uk_004 Even the then Commissioner for P… M_Positive M_Positive M_Positive    
#>  5 uk_005 The Secretary of State told the … N_Neutral  N_Neutral  N_Neutral     
#>  6 uk_006 The family do not know where the… Negative   Negative   Negative      
#>  7 uk_007 I cannot go into the outcomes of… Positive   P_Neutral  Positive      
#>  8 uk_008 Gentleman’s references to those … M_Positive N_Neutral  M_Positive    
#>  9 uk_009 However, there are many problems… Negative   M_Negative M_Negative    
#> 10 uk_010 Member for Leicester West (Liz K… P_Neutral  P_Neutral  P_Neutral     
#> # ℹ 90 more rows

levels(parlasent$reconciliation)
#> [1] "Negative"   "M_Negative" "N_Neutral"  "P_Neutral"  "M_Positive"
#> [6] "Positive"

The Positive and Negative labels are used for sentences that emit these emotions entirely or predominantly. The M_Positive and M_Negative labels are used for sentences that emit these emotions in a mixed or leaning manner. The P_Neutral and N_Neutral labels carry no sentiment and differ only in which way they tilt. Overall, the sample of 100 documents holds around sixteen or seventeen of each.

Step 1. Select a model

The ollama_backend() function accepts an existing local model name and returns an object holding the settings that every model call will use.

oll <- ollama_backend("gemma4:31b")

oll
#> Ollama backend
#>
#> Model         gemma4:31b
#> Host          http://localhost:11434
#> Logprobs      top 20
#> Modalities    text, image

Step 2. Code the qualitative instrument

The instrument() function constructs a codebook containing the question and response categories the local LLM may answer that question with, and it returns an object that the measure() function accepts. The annotators who coded the parliamentary speeches categorized each sentence on one of the previously discussed six ordinal levels, and the article uses fathom to ask a local LLM that same question using those same six categories.

b <- instrument(
  items = "Where does this sentence fall on the six-point sentiment scale?",
  labels = c("1", "2", "3", "4", "5", "6"),
  definitions = c(
    "1" = "entirely or predominantly negative",
    "2" = "ambiguous, or a mixture of sentiments, leaning negative",
    "3" = "no sentiment of its own, leaning negative",
    "4" = "no sentiment of its own, leaning positive",
    "5" = "ambiguous, or a mixture of sentiments, leaning positive",
    "6" = "entirely or predominantly positive"
  ),
  exclusive = TRUE
)

b
#> Instrument
#> 
#> Shape         exclusive choice, so the labels share one softmax
#> Items         1 (category)
#> Labels        6 (1, 2, 3, 4, 5, 6)
#> Definitions   1  entirely or predominantly negative
#>               2  ambiguous, or a mixture of sentiments, leaning negative
#>               3  no sentiment of its own, leaning negative
#>               4  no sentiment of its own, leaning positive
#>               5  ambiguous, or a mixture of sentiments, leaning positive
#>               6  entirely or predominantly positive
#> Cue           \n\nAnswer with exactly one word: 1, 2, 3, 4, 5, or 6.
#> Matching      by prefix, so a token counts when it opens a label

The items argument holds the question that the local LLM reads exactly as written. The labels argument holds the one-word labels that the local LLM is instructed to answer with. Because scoring reads a single token, the labels chosen must have differing first characters. Given that the actual scheme’s own names do not meet this requirement (e.g., Negative and N_Neutral both start with N), the codebook uses numerals instead. The definitions argument holds the definitions of the labels that the local LLM reads in tandem with the question. Setting exclusive = TRUE places every category in one question, causing their probabilities to share a single distribution.¹ The other mechanism is independent binary items, which ask one question per category. The article What Mexicans ask their government for shows that kind of example.

Previewing the model call

The preview() function prints what the local LLM would receive without executing the call.²

preview(b, text = parlasent$sentence[1])
#> Prompt preview
#> 
#> Item          category, exclusive, one answer from 6
#> Scoring       generated position 1
#> 
#> ----------------------------------------------------------------------
#> Text: "First, the UK Coalition against Neglected Tropical Diseases said that there must be national government leadership to integrate programmes with other health, water, sanitation and education initiatives."
#> 
#> Where does this sentence fall on the six-point sentiment scale?
#> 
#> 1 = entirely or predominantly negative
#> 2 = ambiguous, or a mixture of sentiments, leaning negative
#> 3 = no sentiment of its own, leaning negative
#> 4 = no sentiment of its own, leaning positive
#> 5 = ambiguous, or a mixture of sentiments, leaning positive
#> 6 = entirely or predominantly positive
#> 
#> Answer with exactly one word: 1, 2, 3, 4, 5, or 6.
#> 
#> Answer:
#> ----------------------------------------------------------------------
#> 
#> Label         Counts as
#> 1             1
#> 2             2
#> 3             3
#> 4             4
#> 5             5
#> 6             6

Step 3. Extract response probabilities

As in the preview() function, the measure() function takes every document in the corpus and pairs it with each question and its labels from the codebook instrument, sending each pairing to the local LLM configured in Step 1. The function then extracts the probability that the local LLM assigned to each label at its first generated token, and returns an object holding all probabilities along with a record of each call.

res <- measure(
  parlasent,                     # the data frame
  backend    = oll,              # the model, from Step 1
  instrument = b,                # the codebook, from Step 2
  id         = "id",             # column holding the document identifier
  text       = "sentence",       # column holding the document
  keep       = "reconciliation"  # existing columns to carry into `res`
)
#> Constrained-label measurement
#>
#> Model               gemma4:31b
#> Context Ceiling     2,048 tokens
#> Longest doc         'uk_063', 32 words, 157 tokens
#> First token         '6', a label, from 20 log probabilities
#> Documents           100
#> Items               1 (category)
#> Labels              6 (1, 2, 3, 4, 5, 6)
#> Checkpoints         off
#>
#> Measuring           [===============]  100/100  100%  0.7 s/doc  ETA 0.0 s
#>
#> Done                100 processed, 0 skipped, 0 item failures
#> Elapsed             1.2 min, 0.7 s per document

Before the very first call, measure() by default establishes the context window that the run needs by measuring the token count of the longest document-question pair and rounding it up with headroom, so that no prompt is cut short and Ollama does not reset its cache between calls. The help page for measure() covers setting the window by hand with num_ctx.

Step 4. Read the results

The results object holds two data frames.

• $probs is one row per document, item and label, and contains the results themselves, the probability that the local LLM assigned to each label of each document-question pair.

• $calls is one row per document and item, and contains call-level information for checking the run, along with any columns keep carried through.

Probabilities ($probs)

pander(head(res$probs), n = 6)

Here, the prob_renorm and prob columns convey the probabilities that the local LLM assigned to a label. However, prob_renorm is renormalized to the codebook instrument’s label set, making it the main measure of interest.

Calls ($calls)

pander(head(res$calls), split.table = Inf)

Given that this data frame is at the call level, the prob_other column conveys the raw probability mass that the local LLM gave to all other tokens not corresponding to the codebook instrument’s label set. Additionally, the error and truncated columns flag whether certain calls would benefit from further examination or a re-run.

For further information, the documentation page for measure() outlines every column found in both data frames.

Summarizing the results by question and label

The item_report() function summarizes the run measure() produced. On a forced choice codebook instrument it returns one row per category and prints a header above the table.

item_report(res)
#> Choice instrument
#> 
#> Documents     100
#> Items         1
#> Labels        6
#> Leaked mass   max 0.0001, mean 0.0000
#> Winning       probability min 0.686
#> # A tibble: 6 × 4
#>   item     label n_won share_won
#>   <chr>    <chr> <int>     <dbl>
#> 1 category 1        17      0.17
#> 2 category 2        15      0.15
#> 3 category 3        13      0.13
#> 4 category 4        18      0.18
#> 5 category 5        18      0.18
#> 6 category 6        19      0.19

Here, the n_won column counts the number of documents where each label had the highest probability, whereas the share_won column expresses it as a proportion of the documents scored.

The console also outputs two diagnostics, Leaked mass and Winning. Leaked mass depicts the max and mean probability mass that landed on no category at all, while Winning depicts the lowest probability that any winning category received, conveying the floor of the local LLM’s confidence.

Inspecting question-document pairs

The inspect_cases() function can be used to print a question alongside the corpus documents it was answered on, by the codebook instrument label of interest and by whether that label won or the local LLM’s answer was a close call between two labels.

inspect_cases(
  res,                  # model results
  "category",           # which item to read
  parlasent,            # data frame containing corpus
  text   = "sentence",  # corpus documents
  id     = "id",        # document ids
  which  = "endorsed",  # documents where `target` won
  target = "1",         # most negative sentiment
  n      = 2,           # number of documents to print
  seed   = 444          # fixed seed
)

#> 
#> # category  (2 of 17 endorsed, from 100 documents)
#> 
#> **Question**
#> 
#> ```
#> Where does this sentence fall on the six-point sentiment scale?
#> 
#> 1 = entirely or predominantly negative
#> 2 = ambiguous, or a mixture of sentiments, leaning negative
#> 3 = no sentiment of its own, leaning negative
#> 4 = no sentiment of its own, leaning positive
#> 5 = ambiguous, or a mixture of sentiments, leaning positive
#> 6 = entirely or predominantly positive
#> ```
#> 
#> ---
#> 
#> **uk_031** | P(1) = 1.0000, other = 0.00e+00, 18 words
#> 
#> These were huge changes about which the British people were never asked and to which they never consented. 
#> 
#> ---
#> 
#> **uk_012** | P(1) = 1.0000, other = 0.00e+00, 18 words
#> 
#> We have a virtual oil slick, like the pollution streaming out into the Gulf of Mexico in 2010.

The analysis matrix

Finally, the as_wide() function takes the results object and creates a wide data frame where each document is one row and each category is a column, which is where the package hands off.

as_wide(res, prefix = "p_") %>%
  head(6)
#> 
#> Note     Exclusive instrument. Rows are a distribution, sums 1.000 to 1.000.
#> # A tibble: 6 × 7
#>   id       p_1      p_2      p_3       p_4           p_5       p_6
#>   <chr>  <dbl>    <dbl>    <dbl>     <dbl>         <dbl>     <dbl>
#> 1 uk_001 0     0        6.65e- 6 1.000e+ 0 0.0000000147  3.17 e- 7
#> 2 uk_002 1.000 5.26e-11 0        0         0             1.28 e-10
#> 3 uk_003 0     0        0        4.87 e-10 0.00000000110 1.000e+ 0
#> 4 uk_004 0     5.54e-10 6.46e-10 1.35 e- 7 1.000         1.58 e- 6
#> 5 uk_005 0     6.08e- 8 1.00e- 1 9.00 e- 1 0.00000143    1.98 e- 6
#> 6 uk_006 1.000 0        0        0         0             1.39 e-10

Step 5. Validate the LLM results

An LLM cannot entirely discern whether its answers are right. Some degree of human validation is often necessary. To do this, the coding_sample() function draws a sample of document-question pairs that a human coder can then fill out as the first step.

sheet <- coding_sample(
  res,                  # model results
  parlasent,            # data frame containing corpus
  text   = "sentence",  # corpus documents
  id     = "id",        # document ids
  n      = 20,          # document-item pairs to sample
  seed   = 444          # fixed seed
)
#> Blinded coding sample
#> 
#> Pairs         20 across 1 item
#> Drawn         20 of 100
#> 
#> Tip      Pass the completed sheet to validate_sample() together with these
#>          results.

pander(head(sheet, n = 2), split.table = Inf)

Here, each row of the sample sheet pairs a corpus document with one of the codebook instrument’s questions, named in the item column and written out with its label definitions in the question column. Since this coding instrument asks a single question, every row carries the same question and each pair is simply a document. However, on a coding instrument of several questions, a document appears once per question and n counts pairs either way. The human_label column is left blank for human coders to fill in using the labels that the allowed column lists.

Validating the sample using two coders

Handing the validate_sample() function a named list of coding_sample() sheets will get it to score the local LLM against each of the human coders, calculate the ceiling the coders set between themselves, and return the distance between the two. Because the categories are ordinal, the call sets level = "ordinal". Additionally, because the parlasent dataset already has two independent hand codings, this article uses those for validation.

v <- validate_sample(
  list(
    annotator1 = coding_sample1,
    annotator2 = coding_sample2
  ),
  res,
  level = "ordinal",
  labels = as.character(1:6),
  seed = 444
)

pander(v$ceiling$pairs, split.table = Inf)

The v$ceiling$pairs table establishes a ceiling between the two coders on the same corpus documents. The agreement column denotes the proportion of documents where the two coders wrote the same label, while the alpha column corresponds to Krippendorff’s alpha (Krippendorff 2019). Rather than counting coding matches, Krippendorff’s alpha compares the disagreement among the coders with how much they would disagree if each label were assigned at random, in the proportions the coders actually used. Alpha is calculated as one minus the ratio of the two, where 0 is chance-level agreement, 1 is a perfect match, and a value in between is the share of chance-level disagreement the coders avoided. Calculating alpha when using ordinal labels accounts for distance between disagreeing labels by weighting those that disagree by fewer levels less and vice versa. Given that Krippendorff’s alpha is a chance-corrected index, it usually sits below raw agreement. Here, however, an alpha of 0.79 sits above agreement of 0.7, suggesting that most of the disagreements between the coders were by few levels rather than by being far off between them. Additionally, the precision of alpha depends on the sample size n.

Validating the model against the coders

pander(v$gap, split.table = Inf)

The v$gap table compares the model with the ceiling the coders set, with one row per question. The model_alpha column is the model’s average alpha with each of the coders, while the coder_alpha column is the coders’ average alpha with each other, which with two coders is simply their one pairwise alpha. The gap column subtracts the second from the first, so it reads in the same units as alpha (see v$discrimination table for individual level comparisons between the model and each coder). A gap value below zero would indicate that the model on average disagrees with the coders more than they on average disagree with each other. Conversely, a gap value above zero would indicate that the model had a higher average agreement with the coders than the coders had with each other. The gap_ci column holds a percentile confidence interval around the gap (default of 95 percent), built from an empirical bootstrap. Here, the local LLM’s average alpha of 0.835 against the human coders’ average alpha of 0.79 has a gap of 0.045 with an interval of [-0.16, 0.25]. Both alpha values sitting at or near Krippendorff’s 0.80 threshold along with its gap interval crossing zero suggest that the local LLM’s judgment matched the annotators’ about as closely as their judgments match each other, further suggesting that the LLM’s coding can serve as a viable alternative to human coders for this corpus.

References

Krippendorff, K. (2019). Content Analysis: An Introduction to Its Methodology (4th ed.). Sage. https://doi.org/10.4135/9781071878781

Mochtak, M., Rupnik, P., Meden, K., & Ljubešić, N. (2023). The multilingual sentiment dataset of parliamentary debates ParlaSent 1.0. Slovenian language resource repository CLARIN.SI. http://hdl.handle.net/11356/1868