Data coding for researchers

Data Coding School · Introductory module · draft for review

Level: introductory. No statistics background assumed. Lab work uses R or Python; the lab can also be followed by reading the supplied outputs.

Estimated time: five to six hours, including the replication lab.

Learning goals

By the end of the module a participant can:

1. Explain what "coding" means in quantitative research and in qualitative research, and say where the two uses meet.
2. Assign a level of measurement to a coded variable and choose summary statistics that suit it.
3. Write a codebook entry that another researcher could apply without asking the author.
4. Describe the phases of a thematic analysis and the difference between a code and a theme.
5. Plan a check of agreement between coders and name statistics suited to it.
6. Reproduce a published analysis from an open data package and audit the coding decisions behind it.

Lesson 1 · Two meanings of "coding"

Researchers use the same word for two different activities.

In quantitative work, coding assigns a value to each observation according to a scheme fixed before analysis. A death recorded in a case file becomes NATURAL_DEATH = 0; a rating of how unexpected it was becomes a number from 0 to 4. The scheme lives in a codebook, and the goal is that any trained person applying it to the same record would write down the same value.

In qualitative work, coding labels segments of text (an interview passage, a field note) with short descriptors as one step in an interpretive analysis. Braun and Clarke define a code as the most basic segment of raw data that can be assessed in a meaningful way about the phenomenon under study. Codes are then grouped into themes, broader patterns of meaning that answer the research question. The researcher's interpretation is part of the method.

O'Connor and Joffe describe the disagreement between the two traditions over reliability. Quantitative reliability is tied to objectivity and stable findings. Many qualitative positions reject a single objective reality and treat the researcher's reading as a resource. They argue that agreement measures still have a place in qualitative work when coding is categorical and applied through a frame, as in content analysis.

The two meanings meet whenever narrative material is converted into variables. The replication lab in Lesson 6 is an example: interview-based case files from a large research archive were coded into a table of numbers and then analysed statistically.

Check your understanding - A researcher reads 40 interviews and tags passages "distrust of institutions". Is this quantitative or qualitative coding? What would have to change for it to feed a statistical test? - Give one quality question that applies to both kinds of coding.

Lesson 2 · Levels of measurement

Stevens classified scales by the empirical operations behind them.

A coded rating from 0 to 4 is ordinal: we know 4 is more unexpected than 3, but not that the step from 3 to 4 equals the step from 0 to 1. Stevens wrote that means and standard deviations should strictly not be applied to ordinal scales, while conceding that such use often gives useful results. His warning was that the results are inaccurate to the extent that the intervals are unequal.

A common response is to run the analysis both ways. In the lab, the authors report a Pearson correlation (which treats the rating as interval) and a Spearman correlation (which uses ranks only). If the two agree, the conclusion does not depend on the equal-interval assumption.

Exercise. Classify each variable in the lab dataset (CORT_IT_codebook.txt) as nominal, ordinal, interval or ratio. Two of them are judgement calls; say which and why.

Lesson 3 · Codebooks and tidy tables

A coded dataset is only as usable as its documentation. The DataWiz knowledge base of the Leibniz Institute for Psychology lists what a codebook entry should hold:

• variable name (letters, digits, underscores, with a consistent naming scheme)
• variable label (a short description)
• variable type (numeric, string, date)
• valid values, and the range for continuous variables
• value labels for every valid code
• distinct codes for each type of missing value (not applicable, not answered, missing by design)

It adds two recommendations that the lab will test: when a rating scale labels only its endpoints, give labels for the middle values too; and avoid blank cells as the missing-value code, using one standard code per type of missingness.

The table itself should follow a few rules. Wickham defines tidy data as a table where each variable is a column, each observation is a row, and each type of observational unit is its own table. Broman and Woo give practical rules for spreadsheets, among them: be consistent, write dates as year-month-day, put one thing in each cell, keep the data as a single rectangle with one header row, keep a data dictionary, keep calculations out of the raw data file, never use colour to carry information, and save in plain text.

Exercise. Write a codebook entry for a variable you would code from an interview, for example "participant's stated reason for leaving a job". Include value labels and at least two missing-value codes.

Lesson 4 · Coding qualitative material

Braun and Clarke set out thematic analysis in six phases:

1. Familiarise yourself with the data: transcribe, read and re-read, note initial ideas.
2. Generate initial codes across the whole dataset and collate the extracts for each code.
3. Search for themes by sorting codes into candidate themes.
4. Review themes against the coded extracts and then against the full dataset.
5. Define and name each theme.
6. Produce the report, choosing vivid extracts and writing an analytic narrative tied to the research question.

They distinguish two choices the analyst has to make early. An inductive analysis lets themes come from the data without fitting them to a prior frame; a theoretical analysis is driven by a specific question or theory and tends to give a detailed account of one aspect of the data. Semantic themes stay with what participants said; latent themes look for the assumptions and ideas that shape what was said.

Exercise. Take a short passage of interview text (the tutor supplies three). Generate initial codes, then group them into one or two candidate themes. Note which codes you dropped and why.

Lesson 5 · Agreement between coders

O'Connor and Joffe give practical guidance for checking intercoder reliability (ICR):

• Use at least two coders working independently, without conferring during the round.
• Double-code a share of the data, typically 10 to 25 per cent of units, chosen at random or by another stated method. A small pilot first catches obvious problems.
• Report a chance-corrected statistic. Percentage agreement does not correct for chance. Cohen's kappa and Krippendorff's alpha are recommended; chi-square, Cronbach's alpha and Pearson's r are unsuitable for this purpose.
• Set the acceptable threshold before coding, matched to the stakes of the study.
• Report agreement for each code. A pooled figure can hide codes that perform badly.
• Revise codes that fall below the threshold and test the revised frame on a fresh subset.

Interview data adds a difficulty: the units to be coded are often not given in advance. Campbell and colleagues propose standardising the text units coders work on and refining the scheme until agreement reaches an acceptable level. They note the approach suits projects where one knowledgeable coder will code all transcripts once the scheme is settled.

O'Connor and Joffe also record the objection that numerical agreement can give false precision to interpretive work, and conclude that ICR is a means to a sound analysis and never an end in itself.

Exercise. Two coders rate 20 case summaries on a 0 to 4 scale. Plan the agreement check: which statistic, which threshold, what happens to cases where they disagree.

Lesson 6 · Replication lab: from case files to coefficients

Study

Sadeghi, Pehlivanova and Cozzolino (2026) analysed cases from the University of Virginia Division of Perceptual Studies collection of children who report memories of a previous life. For cases where a previous person was identified, they measured the interval between that person's death and the child's birth, and asked whether it is shorter when the death was unexpected. The open data package (Zenodo, DOI 10.5281/zenodo.22677480, CC BY 4.0) holds 2,254 de-identified cases, a codebook, a README and the authors' R script.

The lab tests whether the published statistics follow from the coded data and examines the coding decisions. It does not evaluate the authors' interpretation of the phenomenon.

The variables were produced by researchers reading case files built from interviews and documents. A rating of how unexpected a death was, a judgement about whether the previous person was identified, and a score for how closely two families were linked are all qualitative judgements turned into numbers. The lab treats them that way.

Step 1 · Reproduce

Run Analysis Code.R against CORT_IT_open_data.csv, then run the independent Python script replicate.py. Compare your output with the reference results on the replication lab page.

Reference values (verified cases, N = 1,678):

Step 2 · Audit the codebook

Check CORT_IT_codebook.txt and README.md against the DataWiz list in Lesson 3. Questions to answer:

• The unexpectedness rating labels 0 and 4 only. What would a second coder need to know to choose between 1, 2 and 3?
• Missing values are blank cells. How many kinds of missingness could a blank stand for here?
• The README names the analysis file CORT_IT_analysis.R. What file is in the package?
• Does the package report how the ratings were made, by how many coders, or with what agreement?

Step 3 · Levels of measurement

Unexpectedness is ordinal. Compare the Pearson and Spearman results. Does the conclusion depend on treating the scale as interval?

Step 4 · Missing codes

Among verified cases with a known interval, 244 have no unexpectedness rating. Their median interval is 24 months, against 14 months for rated cases. Discuss what this could mean for the main result, and what you would ask the authors.

Step 5 · Overlapping codes

Cross-tabulate naturalness with unexpectedness. Of 838 unnatural deaths with both codes, 728 are rated 4. When both enter one regression, naturalness is no longer significant. Explain why two codes that track the same events are hard to separate.

Step 6 · Sensitivity

Run the extra checks in replicate.py: excluding the one negative interval, excluding intervals above 20 years, splitting by family association, changing the log constant, and a median regression. Decide whether any of them changes the direction or significance of the main result.

Step 7 · Design the qualitative side

Write anchor descriptions for ratings 1, 2 and 3 of unexpectedness, with one example and one exclusion each. Then plan a double-coding exercise on a sample of case files following Lesson 5.

Assessment

A short written piece (800 to 1,200 words) reporting the replication: what reproduced, what the codebook leaves undefined, and one change to the coding procedure you would recommend, with its expected effect on the analysis. Graded against the learning goals.

Sources

Register of sources not yet used

• ICPSR, Guide to Social Science Data Preparation and Archiving, 6th edition. Only the table of contents was read; the sections on qualitative data and metadata look relevant to Lessons 3 and 4.
• Campbell et al. (2013), full text, for the unitisation procedure in Lesson 5.
• The published APA article for Lesson 6, to check reported coding procedures and any agreement statistics.