To decide which statistical test to use, answer three questions: what type of outcome you measured (continuous, ordinal, categorical or time-to-event), how many groups you are comparing, and whether the observations are paired or independent. Then check the test's assumptions about distribution and sample size. The decision table below maps each combination to a standard test.
A statistical test is a procedure that asks how compatible your data are with a stated null hypothesis. Every test makes assumptions about the data it receives. Choosing one is therefore a design decision, made before the data arrive, not a search for the test that gives the smallest p-value.
This guide is a starting point, not a substitute for a statistician. It covers the standard comparisons in clinical research and shows where the simple choices stop working.
Question one: what type of outcome did you measure?
The outcome type narrows the options more than anything else. Classify the primary outcome before you look at any other variable.
- Continuous — a measurement on a scale, such as haemoglobin, blood pressure or length of stay.
- Ordinal — ordered categories whose gaps are not necessarily equal, such as a pain score or a Clavien–Dindo grade.
- Categorical — groups with no order, or a binary yes/no, such as wound infection or 30-day readmission.
- Time-to-event — the time until something happens, where some patients have not had the event when follow-up ends. Those patients are censored, and ordinary tests cannot handle them.
Do not convert a continuous outcome into a binary one just to use a simpler test. Splitting at a cut-off discards information and reduces power.
Question two: how many groups are you comparing?
Two groups call for one family of tests; three or more call for another. Running every pairwise two-group test instead of one overall test inflates the chance of a false positive.
A study may also ask about an association rather than a group difference, such as whether age relates to operating time. That question needs correlation or regression, not a group comparison.
Question three: are the observations paired or independent?
Observations are paired when each value in one set is linked to a specific value in the other. Before-and-after measurements on the same patients are paired. So are two eyes of one patient and matched case-control sets.
Independent groups contain different people with no link between them. Treating paired data as independent ignores the correlation and usually loses power. Treating independent data as paired is simply wrong.
Parametric or non-parametric: checking a statistical test's assumptions
A parametric test, such as the t-test, assumes a distribution for the data — usually approximately normal. A non-parametric test, such as the Mann–Whitney U test, relies on ranks and makes weaker assumptions. It is not assumption-free.
Judge normality with a histogram or a quantile–quantile (Q–Q) plot and with what you know about the variable. Do not rely only on a significance test for normality. In small samples such a test has little power to detect a real departure. In large samples it flags trivial departures that do not matter.
Three further points prevent frequent errors:
- For a paired t-test, the assumption applies to the differences, not to each set of measurements.
- Welch's t-test does not assume equal variances and loses little when the variances are in fact equal, so it is a sensible default for two independent groups. Choose it in advance rather than after a preliminary test for equal variances.
- The Mann–Whitney U test asks whether values in one group tend to be larger than values in the other. It compares medians only if the two distributions have the same shape. Report it as a comparison of distributions unless that condition holds.
Chi-square or Fisher's exact test when counts are small
The chi-square test compares proportions between independent groups. It relies on an approximation that fails when expected cell counts are small. A widely used rule of thumb is to avoid it when more than 20% of cells have an expected count below 5, or any cell has an expected count below 1.
The rule concerns expected counts, calculated from the row and column totals — not the observed counts you see. When the rule is broken, use Fisher's exact test. For paired binary data, such as a result before and after an intervention in the same patients, use McNemar's test instead. McNemar's test uses only the discordant pairs.
Which statistical test to use: the full decision table
Copy the table below into your analysis plan. Each row is a standard choice. The last column gives the rank-based, exact or more specialised option when the main test's assumptions do not hold.
| Outcome type | Comparison | Independent groups test | Paired test | Non-parametric or small-sample alternative |
|---|---|---|---|---|
| Continuous | Two groups | Independent-samples t-test (Welch's version as the default) | Paired t-test | Mann–Whitney U (independent); Wilcoxon signed-rank (paired) |
| Continuous | Three or more groups | One-way analysis of variance (ANOVA) | Repeated-measures ANOVA or mixed-effects model | Kruskal–Wallis (independent); Friedman (paired) |
| Continuous | Association between two measures | Pearson correlation (linear relationship) | Not applicable — both measures come from the same people | Spearman rank correlation (monotonic relationship) |
| Continuous | Adjusting for other variables | Multiple linear regression | Mixed-effects linear model | Transformation or robust methods — agree with a statistician |
| Ordinal | Two groups | Mann–Whitney U | Wilcoxon signed-rank | Already rank-based; ordinal logistic regression to adjust |
| Ordinal | Three or more groups | Kruskal–Wallis | Friedman | Already rank-based |
| Binary or categorical | Two or more groups | Chi-square test | McNemar's test (two paired binary measures) | Fisher's exact test when expected counts are small; exact McNemar's test when discordant pairs are few |
| Binary | Three or more matched measurements | Not applicable | Cochran's Q test | Agree with a statistician |
| Binary | Adjusting for other variables | Logistic regression | Conditional logistic regression (matched designs) | Exact or penalised logistic regression when events are few |
| Time-to-event | Two or more groups | Log-rank test with Kaplan–Meier curves | Stratified methods — agree with a statistician | The log-rank test is already non-parametric |
| Time-to-event | Adjusting for other variables | Cox proportional hazards regression | Stratified Cox or frailty model | If hazards are not proportional: time-varying effects or restricted mean survival time — agree with a statistician |
Two cautions apply to the correlation row. Correlation measures association, not agreement: two methods can correlate strongly and still disagree. For method comparison, a Bland–Altman analysis is the usual approach. And a correlation coefficient says nothing about cause.
A worked example of choosing a statistical test
The study below is invented to illustrate the structure; its numbers are not real data.
A retrospective cohort compares 60 patients who had a laparoscopic repair with 55 who had an open repair. Primary outcome: surgical site infection within 30 days (binary). Groups: two. Pairing: independent. First choice: chi-square. One arm has 3 infections and the other 9, so 12 events in 115 patients. The expected infection counts are about 6.3 and 5.7 — both above 5, so the chi-square test is acceptable here. Had there been 7 infections in total, both expected counts would fall below 5 (2 of 4 cells, 50%), and Fisher's exact test would be used instead. Secondary outcome: length of stay in days, right-skewed on the histogram — Mann–Whitney U, reported with medians and interquartile ranges. Adjusted analysis: logistic regression, but 12 events cannot support more than the repair type and one or two pre-specified covariates, so a statistician is consulted, and penalised logistic regression may be preferred.
Every choice in the example follows from the three questions and one assumption check. None depends on which test produced a smaller p-value.
Plan regression, multiple comparisons and the protocol in advance
Observational studies rarely end with a simple two-group test. Groups differ at baseline, so the comparison needs adjustment. Linear, logistic and Cox regression are the standard tools for continuous, binary and time-to-event outcomes. For logistic and Cox regression, the number of covariates a model can support depends on the number of events (or non-events, if fewer), not the total number of patients.
Every extra test adds a chance of a false positive. Name one primary outcome. Label the rest secondary or exploratory, and report every test you ran. Where several comparisons carry equal weight, state the correction you used, such as Bonferroni or Holm. After an ANOVA, use a pairwise procedure designed for the purpose, such as Tukey's test.
Write all of this into the protocol or statistical analysis plan before the data are analysed. A test chosen after seeing the data is hard to defend in peer review and weakens the reproducibility of your findings. The test also drives the sample size calculation for a small clinical study, so the two decisions belong together.
When to involve a statistician before choosing a test
Ask a statistician before data collection if your design includes clustering (patients within hospitals), repeated measurements over time, more than trivial missing data, or matched sets. Ask too if the outcome does not fit any row of the table. Help at the analysis stage is better than none, but analysis cannot repair a design flaw.
The Statistical Analyses and Methods in the Published Literature (SAMPL) guidelines, listed by the EQUATOR Network, describe how to report the tests you chose. For presenting results, see our guide to reporting p-values, confidence intervals and effect sizes. If a reviewer questions your choice of test, our guide to decoding statistical reviewer comments sets out your options. If you spot an error in this guide, please report the problem to us.