An enrichment result says which pathways moved. It does not say what moved them. The natural next question — “which regulator drives this programme?” — is the one regulator enrichment answers by finding transcription-factor target sets that overlap the gene list. This chapter takes a different route, because that one was measured and did not hold up.
The alternative is to stop asking an enrichment result to name a regulator, and instead test a candidate against it. Give the function a driver you suspect, and it asks a measurable question: if this gene drives the programme, knocking it down should move these pathway members.
The question has a wrong answer, which is what makes it worth asking. It does not have a unique right answer, and the chapter says so below rather than at the end: a knockdown that moves the pathway is consistent with driving it, and is also what several unrelated knockdowns do.
27.1 Why this chapter tests rather than ranks
Asking an enrichment result which regulator explains a gene list means ranking candidate regulators. Two candidate-ranking routes were benchmarked, and neither separates the answers from the background.
The first is the regulator-collection route documented in regulator enrichment. In an earlier benchmark of transcription-factor collections, the admission gate admitted only about 21% of the entities that could be tested, rising to 55% when the collections were pooled into a union. More importantly, the correct answer’s rank lift was 1.75 against a baseline of 1; the pooled union scored 0.92 on one benchmark set. A lift below 1 means the true regulator ranked no better than chance.
The second is the signed target-network ranking used by the discovery gate. Its audit is frozen: 17 condition–perturbation profiles, the same signed score, no size weighting, no rank-dependent choice. All 17 declared labels are scorable, so the denominator is the full corpus rather than a convenient subset. The signed top-1 candidate matched the condition-declared driver in 6/17, top-5 in 10/17, and top-10 in 13/17.
That is better than chance and still not identification. The labels themselves are condition-informed, not exhaustive causal gold standards: a nonmatching transcription factor can be a real parallel or downstream response factor. And repeated studies are present — three profiles share one accession and two share another — so the seventeen are not seventeen independent biological replicates.
The failure is also structural. A large fraction of the input genes are not transcription factors at all, and no target-set library will place the right factor first when the competitive set contains dozens of plausible ones separated by small margins. So this chapter’s function does not rank. It tests. The candidate comes from the caller — from domain knowledge, from a regulator result worth following up, or from a hypothesis a reviewer asked about.
27.2 The shape of an answer
The output is one row per pathway. Six things travel with that row:
the pathway, its direction and its member count;
an evidence state — support, conflict, mixed, undetermined or noncomputable;
an availability label and an incomplete flag;
the attempted/computable denominator and the counts behind it;
the legacy grade (A, B or C) and its reason;
a statement written so that a perturbation experiment could contradict it.
The sentence is the point:
If HIF1A drives this programme, knocking it down should move these 33 members; D23437 does (padj 2.1e-08) and 3 cell lines agree.
A statement that nothing could refute is not a mechanism claim, so the function refuses to produce one.
27.3 Read the evidence state before the grade
The state is computed from every retained candidate/dataset test for that pathway, not from the best row and not from the grade.
State
What it means
support
at least one supporting test and no conflicting test
conflict
at least one conflicting test and no supporting test
mixed
both supporting and conflicting tests
undetermined
tests ran, but none was significant in either direction
noncomputable
only unusable statistics; nothing was tested
The retained tests live in the perturbation_tests attribute and keep one row per pathway, driver, dataset and cell line. The printed table shows the best row per pathway; the state does not. A pathway can therefore be support while its grade is B, because the member-support floor failed; and a pathway can be undetermined while its grade is C, because the tests ran and came back nonsignificant rather than missing.
The grade is not the evidence. The evidence is not the grade.
27.3.1 Availability is a separate question
availability
What happened
available
the retained tests are present and complete
partial
some tests failed, some lookups were empty or failed, or the ledger contains an unrecognised state
failed
tests were attempted and every one failed
unknown
a ledger exists but its status values cannot be interpreted
missing-ledger
no ledger is attached; the grade cannot be converted into an evidence state
lookup-unavailable
the dataset lookup failed, so coverage is unknown
unqueried
a driver was requested but never looked up
no-dataset
the lookup succeeded and found no usable dataset
not-tested
no test was attempted and no lookup explains why
incomplete is TRUE whenever availability is not available or at least one test is noncomputable. Counts are reported alongside: n_attempted, n_computable, n_support, n_conflict, n_nonsignificant, n_noncomputable and n_failed.
Three distinctions do the work:
a missing ledger is not a support claim. Without the retained tests, an A grade cannot be turned into support;
noncomputable is not nonsignificant. A test that produced no usable statistic has not been run in the sense that would let it count as negative evidence;
a failed lookup is not an empty database. “Could not ask” and “asked and found nothing” are different statements.
Support may coexist with a failed member-support floor. That is why support is not a grading pass, and why the report keeps the legacy grade visible as a secondary field rather than replacing it.
27.3.2 The report carries the same states
mechanism_report() applies the same ledger-derived states to its per-pathway verdicts and adds a candidate-level table (candidate_evidence). Each candidate row keeps its support, conflict, nonsignificant, noncomputable and attempted counts, together with an assessment of support-only, conflict-only, mixed or no-significant-support. The report’s availability and incomplete flags come from the retained ledger too, so a report cannot claim support that the tests do not contain. The cached tables in this chapter were produced with development version 4.21.5.005; the book’s build environment is listed in REPRODUCIBILITY.md.
27.4 The legacy grades, and what they add
Grade
What it means
verdict
A
A knockdown moves the pathway the right way, in at least two cell lines
Level A requires two cell lines because one is not enough. In an earlier measurement, three of four supported drivers stopped being supported when a second cell line was used, and the effect size flipped sign. A grade that means “one dataset said so” is too weak to build a claim on.
27.4.1 What support and level A do not establish
On the hypoxia pilot, thirty knockdowns sampled from the database were tested against the same pathway: four of them also reached level A, and all four are essential genes — a splicing factor, an RNA helicase, a Polycomb component, an RNA-processing factor. Knocking those down disrupts transcription broadly, so a twelve- or thirty-member pathway shifts with them.
So a support row means the pathway moved under this knockdown, in the right direction, in at least the reported cell lines. It does not mean this driver is the one responsible, and it does not separate the driver from a general perturbation. The same caveat applies when the state is mixed: both support and conflict are real observations, and neither settles the causal question.
Three attempts to sharpen it were measured and none worked. Requiring a larger effect size does separate the true driver on the hypoxia data — HIF1A at 0.89 against a background maximum of 0.78 — but the threshold that would do it also discards softer true positives elsewhere. Dividing by the knockdown’s transcriptome-wide impact made the ranking worse, because a strong true knockdown also disturbs the transcriptome. Restricting the pathway to its core members let the same essential genes through.
What the evidence is genuinely good at is exclusion. In the same twelve-driver test, five unrelated factors came back direction-conflict — they push the pathway further the way it is already running — and that verdict was right every time. Treat support as “worth following up”, and treat a conflict or a thin reason as the more informative outcome.
27.5 Looked and found nothing is not the same as not looked
The reason column exists for one distinction that most write-ups lose:
looked, and it is not there — this is evidence. The source does not explain these genes.
did not look, or could not test — this is not evidence. Nothing has been ruled out.
The function keeps the two apart deliberately. no-members means the pathway’s genes are absent from the differential table, which is a real absence. no-test-completed means the calls failed, so the question was never put. too-small means the pathway had too few members for the test to run at all. Only not-significant means a test ran and came back negative.
A sentence that reports “no evidence for X” without saying which of these it means has not said anything. This is also what makes the structural warnings worth having: when nothing reaches level A because a driver has a single cell line, saying so is the difference between “the driver has no support” and “the driver could not be assessed”. The first is a finding; the second is an absence of one.
27.6 The hypoxia run
## A cached run: the calls go to GPSAdb over the network, so the table is stored## and read here. It is the fixed implementation at 8 pathways / 4 datasets,## with the condition-declared driver HIF1A and all 17 inputs retained upstream.pilot <-read.delim("cache/pathway-mechanism-hypoxia.tsv")pilot %>%mutate(evidence =sprintf("%d/%d/%d", n_support, n_conflict, n_nonsignificant)) %>%select(pathway, state = evidence_status, evidence, n_attempted, level) %>%head(8) %>% knitr::kable(col.names =c("pathway", "state", "support/conflict/NS","tested", "legacy grade"))
pathway
state
support/conflict/NS
tested
legacy grade
response to hypoxia
support
3/0/1
4
A
response to decreased oxygen levels
support
3/0/1
4
A
response to oxygen levels
support
3/0/1
4
A
cellular response to hypoxia
support
2/0/2
4
A
cellular response to decreased oxygen levels
support
2/0/2
4
A
cellular response to oxygen levels
support
2/0/2
4
A
vasculature development
support
1/0/3
4
B
muscle cell proliferation
undetermined
0/0/4
4
C
Eight pathways come back: seven are support, one is undetermined, and none is mixed or conflict. Two rows show why the state is not the grade. vasculature development is graded B but its ledger is support — the evidence is real and the grade is capped because only one cell line supports it. muscle cell proliferation is graded C but is undetermined, not noncomputable: four tests ran and came back nonsignificant, which is different from never having tested.
The tumour in this dataset is a pancreatic line, and no HIF-family knockdown in the database comes from that line, so the input experiment and the evidence are independent by construction. What comes back is the biology the field would expect: response to hypoxia on HIF1A, cellular response to hypoxia on HIF1A, and pyruvate metabolic process on HIF1A, each with a negative effect size — knocking the factor down pulls the programme back, which is what an activator should do.
27.7 Does the code change the conclusion? A frozen paired comparison
The .005 development branch changed how evidence is assembled and reported, so the obvious question is whether the conclusions moved with it. Live service responses cannot answer that: rerunning the same query is not the same observation. The comparison therefore froze the responses first.
The design crossed two factors:
implementation: the historical source at 978eaba and the fixed source at fe7c060;
Every condition was run on the same 17 profiles, all retained in the denominator — 15 graded and two with no enriched pathway. Each exact request (same driver, same gene vector, same dataset) reused the first retained response for that snapshot, and replay was offline. The final snapshot contains 235 unique requests — 13 dataset lookups and 222 perturbation tests — with 0 request failures and 57 noncomputable test responses. Those 57 stay in the record as noncomputable; they are not silently dropped or counted as negative.
At a fixed cap, the two implementations agree: zero grade changes and zero verdict changes across 120 common pathways at 8/4 and 75 at 5/2. The reason column does change — eight rows at 8/4 and five at 5/2 — and all of them are the same story: STAT1 previously came back no-dataset, and the fixed implementation records the real tests it found, which are nonsignificant. That corrects a false statement about the database. It does not turn STAT1 into a supported candidate.
The cap comparison is less comfortable. Holding the implementation fixed and shrinking 8/4 to 5/2 removes 45 pathway rows; among the 75 that remain, 3 change from A to B. Their best rows are unchanged; what disappears is the additional supporting context that made two-cell-line replication. Three other hypoxia rows switch their best dataset and statistics but stay at A. The contrast changes both the pathway cap and the dataset cap, so it does not estimate either one separately.
A single profile shows why the full ledger matters:
cell morphogenesis involved in neuron differentiation
D24948
conflicts
5-2
external encapsulating structure organization
D24948
conflicts
5-2
extracellular matrix organization
D24948
conflicts
5-2
extracellular structure organization
D24948
conflicts
5-2
skeletal system development
D24948
conflicts
5-2
tissue morphogenesis
D24948
conflicts
At 8/4 the SMAD3 ledger holds 2 support and 7 conflict rows; at 5/2 it holds 0 support and 5 conflict. The two supporting rows have negative effect sizes (−0.62 and −0.57), the conflicting rows positive values (0.49–0.60). The best row at both caps is a conflicting dataset, so a table that showed only the best row would hide the two supporting observations entirely. This is mixed evidence, not a confirmed driver, and it is exactly the case the old grade-only table could not display.
Across the fixed implementation’s full ledger, 8/4 retains 328 pathway/dataset outcomes — 75 support, 15 conflict, 122 nonsignificant and 116 noncomputable — and 5/2 retains 135 — 41, 12, 41 and 41. The number of profiles with any support falls from 9/17 to 8/17; profiles with any conflict stay at 3/17. A historical implementation has no retained full ledger, so those cells are NA, not zero.
None of this measures accuracy. The code comparison includes the lookup and dataset-selection policy, because that is part of the implementation; the cap comparison changes the evidence population; and the corpus is reused. What the experiment does establish is narrower and still useful: the reporting fix did not move the grades on these inputs, the coverage diagnosis for STAT1 was wrong, and the cap decides how much supporting and conflicting evidence is visible.
27.8 What the fixed audit says about ranking and modules
The frozen audit behind the discovery gate is worth reading alongside the paired run, because it keeps the same denominator throughout.
The signed top-1 rate is 6/17. A module is the signed top-1 plus its direct Jaccard neighbours at 0.10, restricted to that profile’s historical top-five candidate set; it is not transitive clustering and not a search over the whole network. Its label coverage is 7/17 against a size-matched random expectation of 6.5/17, with mean set size 1.882. The labels sit inside the top-five shortlist for 10/17 profiles; restricting the denominator to those ten would make the same coverage look like 7/10, which is why the full corpus is the reported denominator.
The set is larger than the singleton, so coverage rises a little. That is set coverage, not PPV, not calibrated confidence, and not an independent module test. A target-overlap neighbourhood is not evidence that its members are indistinguishable in every cell line, and it is not evidence that the true driver lies inside it. The threshold was explored on this same corpus historically; fixing it now does not make it independent.
These limits do not disappear because the ledger is now more informative. They are why the chapter tests a named candidate instead of reporting a ranked answer.
27.9 Where it works, and where it does not
Nine condition–perturbation pairs were run end to end in an earlier pilot. Four reached level A — hypoxia, sulforaphane with NRF2, DHT with AR, IL-6 with STAT3 — and they come from four different transcription-factor families: bHLH-PAS, CNC-bZIP, a nuclear receptor, and STAT. The family is not what decides it. What those four share is a single clear stimulus producing a single concentrated transcriptional programme, with the driver known independently to mediate that programme.
That last clause carries the weight. These four were chosen because the answer was already known, so they show the method agrees with established biology when the driver is right. They are not evidence that it would have picked the same driver unaided, and the ranking and specificity results above are why.
The five that fail each lack a different requirement:
Failure
Datasets
Cause
Driver has one cell line
airway, GSE286890
Level A needs two; a data-supply limit, not a method limit
Too few differentially expressed genes
MCF-7 knockout, GSE285498
Pathways of two members cannot establish direction or be tested
Condition produces no concentrated signal
GSE311462
550 up-regulated genes spread across no enriched pathway
A useful reading of one failure: dexamethasone and a GR knockdown both push extracellular-matrix genes up, so the driver moves the pathway the way it already runs. Whether that is real repression or a tissue difference — the GR knockdown available is from a kidney line, not airway — cannot be settled from this evidence, and the function declines to call it support rather than guessing.
27.10 What it takes to run
Three conditions have to hold at once, and the measurements above are what fixes them:
The condition experiment concentrates its response — dozens of enriched pathways, not hundreds of scattered genes.
The driver has knockdown data from at least two cell lines, or level A is unreachable by construction.
The pathways are large enough for the test to produce a p-value, which in practice means roughly a dozen members.
The function warns when a run cannot reach level A for a structural reason, because otherwise the output is silent about it:
Warning: Level A is unreachable: `GR` (1) has perturbation data in a single cell
line, and level A requires two. Results here cannot exceed B.
A run costs network calls proportional to pathways times drivers times datasets. On the hypoxia data, 25 pathways took 96 calls and about a minute; 60 took 210 calls and about three minutes. Results are memoised by the dataset and member set, which helps when pathways share members and not otherwise, so max_pathways is worth setting deliberately.
27.11 What this does not establish
The function tests a candidate. It does not discover one, and it should not be read as evidence that the named driver is the only driver, or even the most important one. It reports that the data are consistent with a claim and grades how strong that consistency is.
A second endpoint was implemented and then removed. The plan was to corroborate each pathway against an independent annotation source. Three attempts were made: PPI density said nothing (hypoxia pathway members had 67 edges against a null median of 67, because a GO term is not a protein complex); Reactome member overlap picked IL-4/IL-13 signalling as the counterpart of every hypoxia pathway, which shares HIF1A and glycolytic genes but is not the same pathway; and the official GO-to-Reactome cross-reference covered only 8% of pathways and dropped to zero once Reactome became the pathway source. Rather than keep a column that was constant and empty, it was removed — a permanent “no signal” reads as a check that ran, when what happened is that no usable second source was found.
No external accuracy, causal adjudication or independent validation was performed. The evidence states are a presentation of retained tests, not a new statistical test; A/B/C grading and the discovery score are unchanged. The paired comparison and the fixed audit are development evidence on a reused corpus, not a benchmark of biological truth. That is the honest state of it.
27.12 The gate sequence
The functions form one sequence, and each answers a different question. Keeping them separate is what stops a ranking from being read as a verdict.
Gate
Function
Question it answers
Preflight
mechanism_preflight()
Can this input and context support an attribution claim at all?
Discovery
signed target-network ranking
Which candidates are worth testing? A ranking, not identification.
Verification
pathway_evidence()
Does a named candidate move the pathway, and what is the evidence state?
Report
mechanism_report()
What is the candidate-level support, conflict and coverage gap?
The preflight gate reports input and coverage limits; passing it permits attribution work but does not make the inference correct. The discovery ranking retains its target-count disclosure and remains a proposal, not a verdict. The verification gate tests the caller’s candidate. The report gate carries the states, counts and gaps into one table without collapsing them into a pass label. adjudicate() and evidence-guided interpretation sit on the regulator-centred line and keep their own evidence types; the two lines are not merged into a single score.
27.13 Next steps
For the regulator-centred route and its measured limits, see regulator enrichment.
To turn a graded row into prose, evidence-guided interpretation takes the same claim and keeps the driver, evidence type and direction explicit.
For the gene-set collections an enrichment result comes from, return to GO, KEGG and Reactome.