Text Outcomes for RCTs

Guide researchers in analyzing open-ended text outcomes in randomized experiments using a structured detection-description-causation workflow. Use this when analyzing open-ended survey responses, chat logs, or free-text deliverables from field experiments or A/B tests.

The problem it solves

We increasingly collect text data when running field experiments, how can we analyze it?

Rem Koning Website Scholar

Text Treatment Effects

A recipe for turning open-ended text from a randomized experiment into a defensible empirical claim.

Distilled from the mechanism analysis in Otis, Clarke, Delecourt, Holtz & Koning, The Uneven Impact of Generative AI on Entrepreneurial Performance (Management Science, 2026) — including the paths that were tried and abandoned along the way — and generalized so it applies to any experiment that produces text. For the deeper conceptual foundations, pair this recipe with Grimmer, Roberts & Stewart, Text as Data: A New Framework for Machine Learning and the Social Sciences (Princeton University Press, 2022): its discovery / measurement / inference distinction maps directly onto the three questions below, and its central command — validate every text measure against human judgment — runs through every step here.

The problem

You ran an experiment. Alongside your primary outcome (revenue, test scores, retention) you collected text: open-ended survey answers, chat logs with a tool, written deliverables, free descriptions of what people did. You want the text to explain the experiment.

Three questions get conflated constantly. They are different questions, they need different methods, and the order matters:

	Question	Method family	Failure mode if you skip ahead
Q1 Detection	Is there any systematic difference in the text across arms or subgroups?	Supervised prediction on a high-dimensional representation	You run 40 dictionary measures and report the 3 that popped
Q2 Description	What is the difference?	Group-mean contrasts in embedding space, then interpretable measures	You cherry-pick quotes that confirm the story you already have
Q3 Causation	Did text A cause text B?	DAG + conditioning on the mediating text	You report a correlation between what the treatment said and what people did and call it influence

Almost every mistake in this literature comes from starting at Q2. Start at Q1. The predictive exercise is the omnibus test that disciplines everything downstream: it is one test instead of thousands, and it forces you to commit to "there is / is not a difference" before you start narrating what the difference is.

Step 0 — Before any text touches a model

Do all of this before you pull a single embedding. It takes an afternoon and it determines whether anything you do later is interpretable.

0.1 Inventory your text objects: which exist in both arms?

This is the single most consequential design fact, and it is almost always discovered too late. For every text object in your study, ask who produces it:

Text object type	Control counterfactual?	What it can identify
Text only the treated produce (queries to a tool, the tool's outputs, session logs)	No	Only within-treatment contrasts (e.g., do subgroups of treated units use the tool differently?)
Text both arms produce (open-ended survey answers, deliverables, reports of behavior)	Yes	Actual treatment effects on text; treatment × subgroup interactions

Treated-only text cannot identify a treatment effect. Ever. It can only tell you whether types of treated people behave differently — a real and useful question, but not a treatment effect, and the writeup must not blur the two. (In the source study, treated-only chat logs answered "did low performers ask worse questions?" — no — while a both-arms question about business changes carried the actual treatment effect.)

If you are still designing the experiment, build in a parallel text stream for the control arm — a matched open-ended prompt asked identically in both arms. It also keeps reminders and demand effects symmetric.

0.2 Choose the unit of analysis: concatenate to the randomization unit

Concatenate all text of a given type per unit (person, firm, team) into a single blob, then represent the blob. One row per unit per text type.

Why: the unit of randomization is the unit of inference. Document- or message-level analysis silently weights the sample by verbosity — the person who wrote 60 messages counts 60 times the one who wrote 1 — and treatment may itself have changed how much people wrote. Clustering standard errors at the unit level fixes the inference, not the estimand. (An early version of the source pipeline ran message-level regressions and needed an ad hoc word-count subsample to make them behave; concatenating made the problem disappear.)

0.3 Test for selection into having text — and report it

Before comparing content, show that the availability and length of text are not themselves treatment outcomes. If treatment changes who writes anything, every content comparison is contaminated by composition.

m1 <- feols(has_text ~ treat * subgroup, data = d)    # LPM: who wrote anything?
m2 <- fepois(n_words ~ treat * subgroup, data = d)    # Poisson: counts are skewed
etable(m1, m2, vcov = "HC1")

Winsorize word counts at the 99th percentile. Precisely-estimated zeros here license everything that follows. If you find selection, you are doing a bounding exercise, not a comparison.

0.4 Filter the text — humans define it, AI can scale it

Raw text streams are full of fragments, menu-selection replies ("3"), generic follow-ups ("tell me more"), and off-topic content. Filtering to substantive text matters more than any modeling choice downstream, because filtering errors propagate into every representation.

The rule: humans define the filter; AI can scale it; labelled examples connect the two. Read a meaningful sample of the raw text yourself — there is no substitute for knowing what your corpus actually contains. Have human coders label a few hundred texts (substantive vs. not, on-topic vs. off). If the corpus is too large for humans to label completely, prompt an LLM with your labelled examples and let it do the rest — but hold out part of the human-labelled set as a gold standard, report the agreement rate (percent agreement or Cohen's κ) between the LLM and the humans, and spot-read the LLM's decisions near the boundary. An LLM filter you never validated is a heuristic wearing a lab coat. Report counts before and after filtering, by arm.

If different analyses use different corpora (e.g., content analyses on the full concatenated blob, topic analyses on filtered substantive messages), say explicitly which corpus each number comes from.

0.5 Write down what "no difference" will mean

You are likely to find nulls — in the source study the headline mechanism finding is three nulls followed by a positive. Decide in advance how you will defend a null: a permutation baseline, an equivalence test, or a benchmark effect size from prior literature (§1.6). Deciding after you see the null is motivated reasoning.

Step 1 — The predictive exercise (always first)

The move: turn "are these texts different?" into "can a classifier tell them apart?"

Represent each unit's text as a high-dimensional vector, train a classifier to predict the unit's experimental label from that vector, and compare held-out accuracy to the no-information rate (the accuracy of always guessing the majority class).

This is the right first step for one reason: text does not vary along a single dimension. You do not know a priori which of the thousands of semantic dimensions the treatment moved. Testing them one at a time is a multiple-comparisons disaster and an invitation to specification search. The prediction test is a single omnibus test over all of them at once — and if it fails, you have a principled reason to stop rather than a license to keep fishing.

1.1 Represent the text

Use a modern sentence/document embedding model (e.g., a 1,500–3,000 dimension embedding API model, or a strong open-source encoder): one vector per unit per text type. Embeddings rather than bag-of-words, because semantically similar phrases with no shared words — "reduce prices" and "offer discounts" — land near each other. That equivalence is exactly what a treatment that changes vocabulary will exploit to hide from a word-count method.

Three rules regardless of provider:

Pin the model and dimension in the paper. Results are not comparable across embedding models, and providers deprecate them.
Cache the vectors to disk and never re-pull.
Watch the context window. If you truncate long texts, and treatment changed how much people wrote, truncation is differential by arm — report the truncation rate per arm.

1.2 Fit the classifier

Random forest is a sensible default when p ≫ n (thousands of features, hundreds of units): no a priori feature selection, captures nonlinear interactions, degrades gracefully when most features are noise. A reasonable setup: 60/40 train/test split; k-fold CV inside the training set to tune hyperparameters (e.g., mtry); a few hundred trees. Regularized logistic regression is a good companion (§4.5). Whatever you use: tune inside the fold, touch the test set once per split.

1.3 Repeat the split many times — this is not optional

Report the mean accuracy across ≥100 Monte Carlo train/test splits and the 95% empirical interval (2.5th and 97.5th percentiles of the accuracy distribution). §1.5 is what happens if you don't.

1.4 The ladder: sequence the prediction tests as an argument

Run the tests in an order that closes off explanations one at a time. The general pattern:

Within-treatment contrasts on treated-only text. Do subgroups of treated units use the treatment differently (different inputs, different outputs received)? Nulls here rule out "the subgroups got different treatments."
Treatment vs. control on both-arms text. Did the treatment move what people wrote at all? This is the treatment effect on text.
Arm × subgroup as a single multi-class prediction. Predict the cell (e.g., 4 classes for 2 arms × 2 subgroups) directly. This is the text-space analog of a heterogeneous-treatment-effect test — and the design trick worth stealing: the no-information rate drops to ~25% for four balanced classes, giving you room to detect an interaction that a two-class test would smear out.

In the source study this ladder read: subgroups didn't ask different questions, didn't receive different advice, but made different changes; treatment moved the changes; and the four-way cell was predictable — so the divergence sat at the implementation step, not the treatment-delivery step. Present your ladder the same way: each row is a hypothesis, and the table is an argument.

1.5 The single-split trap

A single train/test split is one draw from a noisy distribution. With a few hundred units and a 40% holdout, the split-to-split standard deviation of accuracy is roughly 4–5 percentage points — easily the difference between "significantly above chance, p = 0.004" and "suggestively above the benchmark." In the source study, moving from a single seeded split (in the working paper) to 100 Monte Carlo splits (in the published version) softened one headline claim and flipped another row that had looked below chance — that wasn't a finding, it was split noise. Two rules:

Never report a single-split accuracy. ≥100 splits; report the mean and the empirical interval.
Don't report a binomial test on one test set (e.g., caret::confusionMatrix's P-Value [Acc > NIR]) as your uncertainty — it conditions on the split and ignores split variance, the dominant source of variance at these sample sizes. The empirical interval across splits is the honest uncertainty.

1.6 How to defend a null

A null in a prediction test means "at this sample size, with this representation, we cannot detect a difference." That is weaker than "there is no difference," and the writeup must say so. Strengthen it four ways:

(a) Always print the no-information rate next to the accuracy. 60% accuracy is a finding if the NIR is 50% and meaningless if the NIR is 62%.

(b) Permutation null. Shuffle the labels, refit, repeat. This gives you the accuracy distribution under the sharp null with your actual sample size and feature count — far more convincing than a nominal p-value, and cheap.

(c) Equivalence testing on scalar margins. Run TOST on the quantities you can put in scalar form (message counts, word counts, human ratings), with equivalence bounds set to effect sizes from prior literature rather than arbitrary bounds: "we can reject effects as large as the ones the prior literature found."

(d) A second, human-legible representation that agrees. Two different representations reaching the same null is much stronger than one — e.g., an LLM-generated topic taxonomy (§4.3) showing the same topic distributions across groups, plus raw quantity checks (counts, lengths, and a K-S test on the full distribution rather than only a difference in means).

Step 2 — Once you find a difference, describe it

The prediction test tells you a difference exists. It tells you nothing about content. Feature importance over thousands of anonymous embedding dimensions is not interpretable and should not be reported as if it were.

2.1 The treatment-effect embedding

The most portable move in this recipe.

Construction. Average the embedding vector within each experimental cell, then take the element-wise difference — a difference in means, just in thousands of dimensions instead of one:

TEE = mean(embedding | treated) − mean(embedding | control)

Compute it separately by subgroup whenever you have heterogeneity — the interesting finding is often that the subgroup TEEs point in different directions.

Interpretation. The vector itself is uninterpretable, so make the data interpret it. Form treated × control pairs, compute the pairwise embedding difference for each, and rank pairs by cosine similarity to the TEE. The top pairs are the ones whose difference points the same way as the average treatment effect. Read them.

# cell means → TEE per subgroup
tee <- colMeans(emb[treated, ]) - colMeans(emb[control, ])

# rank all treated x control pairs by cosine(pair difference, TEE);
# z-score the similarities and keep the top ~1% (z > 2.33)

Practical notes: the Cartesian product of pairs × dimensions is the expensive step (chunk it, parallelize, cache — a run-overnight job, not an interactive one); group by the treated unit and keep each unit's top matches before the global cut so one loud control text doesn't dominate every displayed pair.

What this is and is not. It is a principled, reproducible way to sample illustrative examples that represent the average difference — replacing cherry-picked quotes. It is not an individual-level counterfactual: there is no matched-pair causal claim. Write the disclaimer sentence: you are finding representative cases that capture systematic patterns while acknowledging you cannot observe individual counterfactuals.

2.2 Then quantify what you saw — this is mandatory

Reading top pairs generates a hypothesis. It does not test it. Convert the qualitative read into a small number of simple, transparent, documented measures and run ordinary experimental regressions on them. Label clearly in the paper which step is abductive and which is a test. Good measure types:

Keyword/regex indicators for the constructs you saw (e.g., mentions of the treatment tool; mentions of the specific strategies that seemed to differ). Document every inclusion and exclusion decision and the reason — these are researcher degrees of freedom, and someone will ask.
A specificity/rarity measure — e.g., the count of words appearing in fewer than k units' texts — estimated with Poisson (counts are skewed) and controlling for total word count (longer texts mechanically contain more rare words).
A placebo measure — always. Run the same pipeline on a keyword family that shouldn't move (e.g., mentions of the control arm's materials). A clean null here is a cheap negative control against "people just describe whatever intervention they got."

Estimate the same quantities for every measure so the text results read in parallel with your main outcomes:

m1 <- feols(y ~ treat, d)                          # full sample
m2 <- feols(y ~ treat, filter(d, subgroup == 0))   # subgroup A
m3 <- feols(y ~ treat, filter(d, subgroup == 1))   # subgroup B
m4 <- feols(y ~ treat * subgroup, d)               # interaction

Scale binary outcomes by 100 so coefficients read as percentage points; plot as a coefficient chart with nested 90% and 95% bars.

Step 3 — Text → text causal claims

The hardest question: did the text the treatment delivered (advice, feedback, messages) cause the text the subjects produced (behavior descriptions, essays, decisions)?

The obstacle is that delivered text is not exogenous. What the system said is a function of what the subject asked, and what the subject asked is a function of who they are. Someone already planning a marketing campaign asks about advertising, receives advertising advice, and reports launching an ad campaign. Naive correlation between delivered content and outcome content is confounded by everything about the person.

3.1 The backdoor condition — institutional, not statistical

        U (subject characteristics)
       ╱                          ╲
      ▼                            ▼
   Q (subject input) ───► A (delivered text) ───► C (subject outcome text)

Conditioning on Q blocks every backdoor path from A to C if and only if the delivery mechanism is a pure function of an observed, logged input — e.g., a stateless AI assistant with no profile, no memory, no covariates, that sees only the message text. State that condition explicitly, not just the criterion. It fails with personalization from a user profile, memory across sessions, retrieval over user data, a human advisor who can see the person, or any input the system observes that you did not log. If you are designing the experiment: build the delivery system stateless and log every input — it buys you identification for free.

3.2 The long-format embedding regression

Reshape the embeddings to long — one row per (unit × embedding dimension) — and regress the outcome-text value on the delivered-text value in the same dimension, controlling for the input-text value in that dimension:

m1 <- feols(outcome_z ~ delivered_z,                          data = long)
m2 <- feols(outcome_z ~ delivered_z + input_z,                data = long)  # backdoor
m3 <- feols(outcome_z ~ delivered_z + input_z | unit_id,      data = long)  # + unit FE
etable(m1, m2, m3, cluster = c("unit_id", "dim"))

Expect the raw association to fall substantially once input content is controlled — in the source study it halved (0.26 → 0.12), cleanly splitting "confounding" from "the causal effect of delivered content." Interactions with subgroup indicators then test whether groups act on the delivered text differently.

Specification notes and honest caveats:

Unit fixed effects absorb "some people write more, or more distinctively, across all dimensions." Include them.
Two-way clustering (unit and dimension) is required. Dimensions are correlated across units because of embedding geometry; dimensions within a unit are correlated because it's one text.
Row counts are not sample size. Hundreds of thousands of rows = a few hundred units × thousands of dimensions. Clustering handles the inference; say it in words so no reader mistakes rows for power.
The diagonal assumption. Regressing dimension d on dimension d assumes the relationship operates dimension-wise — a "same basis on both sides" assumption, defensible when both texts come from the same embedding model, but an assumption. Say so.
Standardization choice. Global vs. within-dimension standardization reweight dimensions differently. Pick one, say which, check the other as a robustness.

3.3 The bag-of-words validation

Run the same regression at the word level — one row per (unit × word), does a word's appearance in the delivered text predict its appearance in the outcome text, controlling for its appearance in the input — and report it as a robustness check. In the source study, a word appearing in the delivered advice raised its probability of appearing in the outcome text ~15× over base rate. Keep it because it is transparent and legible to a reader who doesn't trust embeddings. Don't lead with it, because it assumes each word affects only itself: advice about pricing that causes someone to write discounts is invisible to it — and that is exactly where a treatment that shifts vocabulary is most interesting.

Step 4 — The menu: other approaches, and when each is right

Not everything below is wrong. Each has a setting where it is the right tool. This is the honest record of what was tried in the source project and why the pipeline ended up where it did.

4.1 Anchor-vector / semantic projection — tried first, abandoned

Write two anchor paragraphs at opposite poles of a construct, embed both, define direction = e_pole1 − e_pole2, project every text onto that axis, and regress the position on your moderator.

Attractive because it yields one continuous, interpretable measure per text — binscatters beautifully. Dropped because it requires naming the dimension in advance (you learn about the axis you guessed, not whether the texts differ), the anchor wording carries enormous researcher degrees of freedom, and a null is uninformative — "wrong axis" and "no difference" are indistinguishable. Right when theory names the dimension ex ante and you pre-register the anchor text verbatim. As exploration, it is a trap.

4.2 LLM-as-judge / construct scoring — tried, dropped from the main text

Prompt an LLM to score each text on a construct (0–10), with multiple independent ratings per text.

Attractive because it measures the construct you care about and reads like human coding at scale. Dropped because score distributions are lumpy and prompt-sensitive, model-version drift breaks replication, and — the serious one — the judge can see treatment-revealing cues: a text saying "the AI suggested…" scores differently because it names the treatment. That is leakage dressed as measurement. Right when the construct is genuinely defined by human judgment and you (a) take multiple ratings and report reliability, (b) blind the text by stripping treatment-revealing tokens, (c) validate against a human-coded subsample with good labelled examples — read the texts yourself first, (d) pin the model version and temperature and archive the exact prompt, (e) report the score distribution, not just the mean.

4.3 LLM-generated topic taxonomy — kept, as support for nulls

A disciplined multi-stage procedure (following Choi et al. 2023): human-filter to substantive texts; generate candidate topics with an LLM at increasing granularity and pool them; merge and prune with the LLM, then refine by hand; classify each text into exactly one topic with a fixed prompt; manually review a ~10% sample of assignments. Use as an interpretable second opinion on a predictive null. Not as the primary test: the taxonomy is partly hand-chosen, one-topic-per-text discards information, and topic shares can hide a real difference. Archive the prompts verbatim in the appendix.

4.4 Bag of words / log-odds / tf-idf — kept, as validation

Cheap, perfectly transparent, replicable in ten years with no API. Correct role: the validation layer under an embedding result (§3.3). Fails on synonymy — the failure mode that matters most when a treatment changes how people talk.

4.5 Alternative classifiers

Regularized logistic regression on embeddings is the most useful companion to a random forest: similar accuracy in the p ≫ n regime, and its coefficient vector is a direction in semantic space you can project onto and read the extremes — partially solving the "RF gives you no direction" problem. Gradient boosting is fine if you can tune it honestly inside the training fold; more hyperparameters, more ways to leak the test set.

4.6 Human coding — and scaling it with AI

Human reading is still the gold standard for small corpora and the validation backstop for everything above. Spend the human hours where errors propagate furthest — the filtering step — and on building labelled example sets. The scaling pattern is always the same: humans read and label a sample; an LLM extends the labels to the full corpus; a held-out human-labelled gold set measures agreement; the researcher reads the disagreements. There is no version of this recipe where you never read the text. The Text as Data framing is the right discipline: every automated measure is a stand-in for a human judgment, and it is only as good as the validation connecting the two.

Pitfalls checklist

Design and data

Text that exists in only one arm cannot identify a treatment effect. Know which text objects have a control counterfactual before planning any test.
Test selection into having text (has_text ~ treat × subgroup) and publish the table. If it fails, you're bounding, not comparing.
Control for length everywhere, and separately verify treatment didn't change length. Poisson for counts, winsorize at 99%.
Concatenate to the randomization unit. Document-level analysis weights by verbosity.
Filter with human-defined rules; if an LLM scales the filter, validate it against a human-labelled gold set and report agreement.
PII. Raw open-ended text is identifying — ship derived variables and embeddings in replication packages, not raw text. And embeddings are not anonymous (inversion attacks recover substantial content); think before posting them publicly.

Modeling

No single train/test splits. ≥100 Monte Carlo splits, mean + empirical interval.
Don't report a single-test-set binomial p-value as your uncertainty — split variance dominates at these sample sizes.
Tune inside the training fold. Test set touched once per split.
Always print the no-information rate next to accuracy, especially with class imbalance.
Watch treatment leakage into the text. If treated subjects can name the treatment, your classifier learns "says the treatment's name," not "changed behavior." Bound the leakage (how many texts mention it? how big is the keyword effect relative to the accuracy gap?) or strip the tokens and re-run.
Pin the embedding model and dimension. Cache vectors.
Check truncation rates by arm if you truncate to fit a context window.
Set and record seeds, and treat the seed as a nuisance parameter you average over, not a result you keep.

Inference

Cluster at the randomization unit — and at the dimension/word level in long-format regressions.
Don't let row counts masquerade as power. Rows = units × dimensions; report N in units.
Correct for multiple hypotheses across text families (e.g., Benjamini–Hochberg).
Diagonal-only (dim d on dim d) regressions are an assumption. Say so.

Interpretation

Abductive ≠ tested. Reading top pairs generates the hypothesis; the keyword and specificity regressions test it. Do both, in that order, and label which is which in the prose.
Cosine-similar pairs are illustrations, not matched counterfactuals. Write the disclaimer sentence.
A null is "we cannot detect a difference at this sample size," unless backed by a permutation baseline, an equivalence test, or a converging second representation.
Regex and dictionary measures embed judgment calls. Document every included and excluded term and the reason.

The order of operations, on one page

Hand this to a student.

0. Inventory text objects. Which exist in BOTH arms? ──► only those identify treatment effects

1. Test selection: has_text ~ treat × subgroup; n_words ~ treat × subgroup (Poisson)

2. Filter (humans define, AI scales, gold set validates); concatenate to one blob per unit

3. Embed (pin the model; cache to disk; log truncation by arm)

4. PREDICT — classifier, ≥100 MC splits, mean accuracy + 95% empirical interval vs. no-info rate
     4a. within-treatment contrasts (does the input differ by subgroup?)
     4b. treatment vs control (did the treatment move the text?)
     4c. arm × subgroup, multi-class (did it move the groups DIFFERENTLY?)  ◄── the HTE test

5. If NULL  → defend it: permutation null + equivalence bounds from prior literature
              + a second representation (topics) + quantity checks (counts, length, K-S)

6. If SIGNAL → treatment-effect embedding: cell means → element-wise difference → rank
               treated×control pairs by cosine similarity to the TEE → read the top 1%

7. Convert what you read into 3–4 simple, documented measures (keyword indicators, rarity
   counts) + one PLACEBO measure. Estimate: full / subgroups / interaction. Coefficient plot.

8. If you need text→text causation: draw the DAG, argue the delivery mechanism is a pure
   function of a logged input, regress long-format outcome_d ~ delivered_d + input_d | unit,
   cluster by unit AND dimension. Validate with bag-of-words. State the diagonal assumption.

9. Multiple-hypothesis correction across families. Report N in units, not rows.

Reporting template

Five exhibits, in this order. Together they make the argument self-contained.

Selection table — has-text and word-count regressions by arm × subgroup. Licenses everything after it.
Prediction table — one column per text × label. Rows: data, prediction target, number of classes, N units, no-information rate, mean accuracy, 95% empirical interval. The omnibus test.
Text→text table — raw / backdoor-controlled / + unit FE / interacted. The causal chain.
Example-pairs table — control and treated columns, one panel per subgroup, drawn from the top pairs by cosine similarity to the TEE. The abductive interpretation, with the disclaimer sentence.
Coefficient plots — the interpretable measures, estimated for the full sample, each subgroup, and the interaction, with 90% and 95% bars. The quantitative test of what you read in exhibit 4.

Prose ordering that works with referees: "The mechanisms we test are conceptually straightforward but empirically challenging for two reasons. First, text varies along many semantic dimensions, so detecting differences requires high-dimensional semantic representations rather than single-variable content coding. Second, [the delivered text] is endogenous; it depends on [the subject's input], so naive correlations risk confounding." Then: embeddings → supervised prediction → backdoor.