Calculator overview
Inputs and outputs
This summary comes from the calculator's published input and output contract.
Inputs
- Study layout (required)
-
Default Two independent groups Allowed One sample or paired, Two independent groups
About this input
One sample or paired (one set of values, or of paired differences), or two independent groups.
- Standard deviation (required)
-
Unit units Default 10 Range 0.1 to 1000
About this input
The standard deviation sigma of single values (of the differences when paired), from 0.1 to 1000 units.
- Significance level alpha (required)
-
Default 0.05 Range 0.001 to 0.2
About this input
The significance level alpha, the false-alarm risk accepted, from 0.001 to 0.2.
- Sides of the test (required)
-
Default Two-sided Allowed Two-sided, One-sided
About this input
Two-sided, or one-sided in the direction of the effect.
- Test family (required)
-
Default Means, known SD (z test) Allowed Means, known SD (z test), Means, estimated SD (t test), Proportions
About this input
The test family: Means, known SD (z test); Means, estimated SD (t test), whose power is computed exactly; or Proportions (normal approximation without a continuity correction).
- Target power (required)
-
Default 0.8 Range 0.5 to 0.99
About this input
The power the required size must reach, from 0.5 to 0.99.
- Power prediction (required)
-
Default Not yet predicted Allowed Not yet predicted, Below 0.1, 0.1 to 0.2, 0.2 to 0.3, 0.3 to 0.4, 0.4 to 0.5, 0.5 to 0.6, 0.6 to 0.7, 0.7 to 0.8, 0.8 to 0.9, 0.9 or more
About this input
Challenge 1: your prediction of the power, as a range of width 0.1; Not yet predicted until you choose.
- Baseline proportion (required)
-
Default 0.5 Range 0.01 to 0.99
About this input
The baseline proportion: the null value (one sample) or group 2's proportion (two groups), from 0.01 to 0.99.
- Alternative proportion (required)
-
Default 0.6 Range 0.01 to 0.99
About this input
The alternative proportion: the true proportion (one sample) or group 1's proportion (two groups), from 0.01 to 0.99.
- Group size ratio k = n2/n1 (required)
-
Default k = 1 Allowed k = 1/4, k = 1/3, k = 1/2, k = 2/3, k = 1, k = 3/2, k = 2, k = 3, k = 4
About this input
Group 2's size as a multiple k of group 1's, from the list k = 1/4 to k = 4 (two groups only; group 2 is k x group 1, rounded up).
- Planned size of group 1 (required)
-
Unit participants Default 40 Range 2 to 10000
About this input
The planned size of group 1 (or of the single sample), a whole number from 2 to 10000.
- Mean difference to detect (required)
-
Unit units Default 5 Range 0 to 1000
About this input
The mean difference Delta to detect (the means families), from 0 to 1000 units. Set with the handle on the sampling picture.
- Challenge (required)
-
Default 0 Range 0 to 5
About this input
Selects a challenge: 0 for none, 1 to 5 for a predefined challenge, which fixes the settings except the input it leaves to you.
Outputs
- Power at the required size
-
No unit declared
About this output
The power at the required size (at least the target). Blank as the required size.
- Power at the planned size
-
No unit declared
About this output
The power at the planned size: the chance the study detects the assumed effect. Blank while challenge 1 awaits a prediction and for a t test planned below its smallest size.
- Power against the effect at the planned size
-
No unit declared
About this output
Power against the effect at the planned size, at 61 effects from none to 2.5 times the plan's (for proportions, within 0.005 and 0.995). The power column is withheld while challenge 1 awaits a prediction and during challenge 4.
- Required size of group 1
-
Unit participants
About this output
The smallest group-1 size whose power reaches the target, from a 16-row integer bisection up to 40,000. Blank when no size reaches it and in challenge 2.
- Power against the size of group 1, with the exact binomial power (one-sample proportions)
-
No unit declared
About this output
Power against the size of group 1 at 61 sizes from the smallest allowed, with the exact binomial power for one-sample proportions. The power columns are withheld while challenge 1 awaits a prediction and during challenge 2.
- Planned total size
-
Unit participants
About this output
The planned total size.
- Noncentrality at the planned size
-
No unit declared
About this output
lambda, the effect in standard errors at the plan (the means families).
- Formula module versions
-
No unit declared
About this output
The formula modules the model is built from, with their versions.
- Standard error if there is no effect
-
No unit declared
About this output
The standard error of the estimated difference if there is no effect, at the plan.
- Planned size of group 1 used
-
Unit participants
About this output
The planned size of group 1 the calculation used (in challenge 3, 120 split by the ratio). Blank when the inputs are not valid.
- Planned size of group 2
-
Unit participants
About this output
The planned size of group 2, k x group 1 rounded up (two groups only).
- Study layout used
-
No unit declared
About this output
The layout the calculation used. Blank when the inputs are not valid.
- Standard deviation used
-
Unit units
About this output
The standard deviation the calculation used. Blank when the inputs are not valid.
- Target power used
-
No unit declared
About this output
The target power the calculation used. Blank when the inputs are not valid.
- Sides used
-
No unit declared
About this output
The sides the calculation used. Blank when the inputs are not valid.
- Test family used
-
No unit declared
About this output
The test family the calculation used (the challenge value when a challenge is selected). Blank when the inputs are not valid.
- Smallest allowed size of group 1
-
Unit participants
About this output
The smallest group-1 size the lab allows for this design (at least 2 in each group, and 4 degrees of freedom for the t test).
- Required total size
-
Unit participants
About this output
The required total size. Blank as the required size.
- Required size of group 2
-
Unit participants
About this output
Group 2 at the required size (two groups only). Blank as the required size.
- The sampling picture at the planned size: the estimated difference if there is no effect and if the effect is real, and the region code (1 reject, 2 a critical value, 0 keep the null)
-
No unit declared
About this output
The estimated difference at the planned size at 121 sorted points (including both critical values exactly): its density if there is no effect, its density if the effect is real (an approximate shape for the t test), and the region code (1 reject, 2 a critical value, 0 keep the null).
- Significance level used
-
No unit declared
About this output
The significance level the calculation used. Blank when the inputs are not valid.
- The search for the smallest size: each bisection row
-
No unit declared
About this output
The 16 bisection rows of the search for the smallest size: the range before the row, its middle, the power there and whether it reaches the target. Blank when no size reaches the target and in challenge 2.
- Model version
-
No unit declared
About this output
The model version.
- Challenge goal
-
No unit declared
About this output
The goal of the selected challenge; blank with no challenge.
- Challenge feedback
-
No unit declared
About this output
The met text when the challenge is met, otherwise a hint; in challenge 1 the prompt until a prediction is made; blank with no challenge.
- Challenge: the value scored
-
No unit declared
About this output
The value the selected challenge scores: the power at the plan (1 and 3), the planned size (2), the mean difference (4), or the change in exact power from one more participant (5). Blank with no challenge, before a prediction in challenge 1, or when the inputs are not valid.
- Challenge result
-
No unit declared
About this output
No challenge selected, MET or NOT MET.
- Challenge score
-
No unit declared
About this output
100 when the selected challenge is met, else 0.
- Chance of missing a real effect at the planned size (β)
-
No unit declared
About this output
The chance of missing a real effect at the planned size, beta = 1 - power. Blank as the power.
- Critical values at seven significance levels, at the plan's degrees of freedom and sides
-
No unit declared
About this output
The critical value at seven significance levels (0.001 to 0.2), at the plan's degrees of freedom and sides.
- Group size ratio used
-
No unit declared
About this output
The group size ratio the calculation used. Blank when the inputs are not valid.
- Alternative proportion used
-
No unit declared
About this output
The alternative proportion the calculation used. Blank when the inputs are not valid.
- Baseline proportion used
-
No unit declared
About this output
The baseline proportion the calculation used. Blank when the inputs are not valid.
- Standard error if the effect is real
-
No unit declared
About this output
The standard error of the estimated difference if the effect is real, at the plan (equal to the null one for the means families).
- Effect size h
-
No unit declared
About this output
Cohen's h = 2 asin(sqrt(p1)) - 2 asin(sqrt(pb)), signed (proportions).
- Effect size d
-
No unit declared
About this output
Cohen's d = Delta/sigma (the means families).
- Exact binomial power at the planned size
-
No unit declared
About this output
The exact binomial power of the one-sample proportion test at the planned size (up to 2,000), with equal tails of alpha/2. Blank otherwise and while challenge 1 awaits a prediction.
- Model Status
-
No unit declared
About this output
NOT VALID with the reason when an input cannot be used; CHECK when a t test is planned below its smallest size, there is no effect, the target is out of reach within 40,000, the required size sits at the edge of the target, or the normal approximation for proportions is rough; otherwise OK.
- Mean difference used
-
Unit units
About this output
The mean difference the calculation used. Blank when the inputs are not valid.
- Proportion difference used (alternative less baseline)
-
No unit declared
About this output
The proportion difference the calculation used, alternative less baseline (signed), for the proportions family; blank for the means families and when the inputs are not valid. The canvas places the plan point with it.
- Closed-form size (about)
-
Unit participants
About this output
The closed-form group-1 size, not rounded: a teaching comparison that ignores the far tail (and, for the t test, uses z). Blank with no effect, above a size of 1E12 and in challenge 2.
- Required size less the closed form
-
Unit participants
About this output
The required size less the closed-form size. Blank when either is blank.
- Critical difference
-
No unit declared
About this output
The difference that just reaches significance: the critical value x the standard error if there is no effect.
- Degrees of freedom at the planned size
-
No unit declared
About this output
The degrees of freedom of the t test at the plan.
- Critical value of the test statistic
-
No unit declared
About this output
The critical value of the test statistic at the plan (z, or t at the plan's degrees of freedom).
LogicCommons is in beta. If a result, label, or reference looks wrong, tell us here; we read every message.
Methodology
Purpose and model boundary
This lab plans a study before any data are collected. You describe what is measured, the effect you hope to detect, how noisy the measurements are, how many participants each group gets, and the risks you accept of a false alarm (the significance level, alpha) and of a miss (beta). The model returns the power at the planned size, the chance that the study detects the assumed effect; the smallest size that reaches a target power, found by a fixed integer search; a closed-form approximate size for comparison; a picture of the estimated difference if there is no effect and if the effect is real; and power against size and against effect. At Advanced it adds the search's steps, the critical value at seven significance levels, and the exact binomial power of a one-sample proportion test. Five scored challenges use the same model.
It covers three test families: means with a known standard deviation (z test), means with an estimated standard deviation (t test), and proportions (normal approximation). The study is one sample (or paired differences), or two independent groups whose sizes are in a chosen ratio, tested two-sided or one-sided in the direction of the effect. Nothing is simulated: every power is computed directly.
It is a lab for learning how power, effect, noise, size and alpha trade against each other, and why a study can be too small to find a real effect. It is not a substitute for a statistician's plan: it assumes the effect and the standard deviation you enter are right, covers only these tests, and does not adjust for dropouts, clustering or multiple comparisons.
The model is the calculation authority. The page sends the values you set to the model and draws only what it returns: the two sampling curves with the rejection region, alpha, beta and power as areas between the model's points, and the two power curves. The page computes no power, size or critical value itself.
Inputs and units
The mean difference and the standard deviation are in your own measurement units. Proportions, alpha and power are fractions from 0 to 1. Sizes are numbers of participants.
| Input | Meaning and unit |
|---|---|
| Test family | Means, known SD (z test) (default), Means, estimated SD (t test), or Proportions. |
| Study layout | One sample or paired, or Two independent groups (default). |
| Sides of the test | Two-sided (default) or One-sided, in the direction of the effect. |
| Significance level alpha | From 0.001 to 0.2 (default 0.05): the risk of a false alarm. |
| Target power | From 0.5 to 0.99 (default 0.8): the power the search must reach. |
| Mean difference to detect | From 0 to 1,000 units (default 5): Δ, the effect (means families). |
| Standard deviation | From 0.1 to 1,000 units (default 10): σ of single values, or of the differences when paired (means families). |
| Baseline proportion | From 0.01 to 0.99 (default 0.5): pb, the value under no effect (one sample), or group 2's proportion (two groups). |
| Alternative proportion | From 0.01 to 0.99 (default 0.6): p1, the true proportion (one sample), or group 1's (two groups). |
| Group size ratio k = n2/n1 | One of 1/4, 1/3, 1/2, 2/3, 1 (default), 3/2, 2, 3 or 4 (two groups only). |
| Planned size of group 1 | A whole number from 2 to 10,000 (default 40): n1, or the size of the single sample. |
| Challenge | 0 for free exploration, or 1 to 5 to load a challenge. |
| Power prediction | Not yet predicted, or a band of width 0.1 from Below 0.1 to 0.9 or more: your answer in challenge 1. |
Every input is checked on its entered value, whether or not the selected family uses it.
Where each input is set:
- On the canvas only: the mean difference to detect, dragged as the handle of the effect curve on the sampling picture, and your power prediction in challenge 1.
- In the input panel: every other input: the standard deviation and the planned size, then the family, layout, sides, alpha, target power, the two proportions, the group size ratio and the challenge.
Governing relationships
Below, D is the effect on the difference scale (Δ for means, |p1 - pb| for proportions), n is the size of group 1, and group 2 has n2 = k x n rounded up to a whole number. The critical value is z_c = z(1 - α/2) two-sided or z(1 - α) one-sided for the z test and proportions, and the t quantile at the same tail with df degrees of freedom for the t test. Φ is the standard normal distribution function.
Means, known SD (z test). The standard error of the estimated difference is SE = σ/√n (one sample) or σ√(1/n + 1/n2) (two groups), and the noncentrality is λ = D / SE. Then
power = Φ(λ - z_c) + Φ(-λ - z_c),
where the second term (the far tail) is dropped for a one-sided test.
Proportions (normal approximation, no continuity correction). Under no effect the standard error uses the pooled proportion; under the effect it uses each group's own proportion:
- one sample: SE0 = √(pb(1 - pb)/n), SE1 = √(p1(1 - p1)/n);
- two groups: p̄ = (p1 + k·pb)/(1 + k), SE0 = √(p̄(1 - p̄)(1/n + 1/n2)), SE1 = √(p1(1 - p1)/n + pb(1 - pb)/n2).
Then power = Φ((D - z_c·SE0)/SE1) + Φ((-D - z_c·SE0)/SE1), the second term two-sided only. The pooled proportion uses the planned ratio k, the usual planning convention.
Means, estimated SD (t test). The test statistic follows a noncentral t distribution with df = n - 1 (one sample) or n + n2 - 2 (two groups) and noncentrality λ = D / SE. The exact power is
power = E[ Φ(λ - t_c·W/√df) + Φ(-λ - t_c·W/√df) ],
where W follows a chi distribution with df degrees of freedom and the second term is dropped one-sided.
The model evaluates this average by a fixed 32-point Gauss-Legendre quadrature over two windows of W, with an error below 2×10⁻⁹. The t test needs at least 4 degrees of freedom. No normal approximation is used for any t power.
The required size: integer bisection. The search starts from the smallest allowed size and 40,000 and halves the range 16 times: at each middle size it computes the power by the same formula as the plan's and keeps the half that still contains the first size reaching the target. The result n* reaches the target, and the size one below does not (or n* is the smallest allowed size). A power counts as reaching the target within 1×10⁻¹⁰. Only whole numbers move through the search, so it is exact.
The closed-form approximate size (for comparison only). For means, one sample, ((z_c + z_β)σ/Δ)², with z_β the normal quantile at the target power; for two groups the same times (1 + 1/k). For proportions, one sample, (z_c√(pb(1 - pb)) + z_β√(p1(1 - p1)))² / D², with a pooled two-group version. It ignores the far tail and, for the t test, uses z instead of t, so it runs a little low; it is shown, never rounded, beside the searched size.
Effect sizes. Cohen's d = Δ/σ for the means families, and Cohen's h = 2·asin(√p1) - 2·asin(√pb), signed, for proportions.
The exact binomial power (one-sample proportions, up to 2,000). The rejection region is the set of whole counts whose tail chance under pb is within α/2 on each side (α one-sided). The exact power is the chance of landing in it under p1. Because the region changes in whole steps, the exact power rises in a saw-tooth and can fall when one participant is added.
The sampling picture. At the planned size, the estimated difference is drawn at 121 sorted points, including both critical differences exactly. For the z test and proportions the curves are normal, centred at 0 with SE0 and at D with SE1. For the t test they are t densities in difference units, and the curve for the real effect is a shifted central t: an approximate shape, labelled as such; the power card is exact. Each point is coded as inside the rejection region, on its edge, or outside, so the shaded areas begin and end on model points.
The curves. Power against size runs over 61 whole sizes from the smallest allowed size to about three times the larger of the planned and required sizes (at most 40,000). Power against effect runs over 61 effects from none to 2.5 times the planned effect, at the planned size; with no effect its power equals alpha.
Calculation sequence
- Check the inputs in the order of the status list under Errors and warnings. In a challenge, take the challenge's fixed values, except the inputs it leaves to you.
- Find the effect, the group 2 size, the degrees of freedom, the smallest allowed size and the critical values.
- Compute the power at the planned size, and the standard errors, noncentrality and critical difference there.
- Check that 40,000 in group 1 reaches the target; if so, run the 16-step bisection for the required size.
- Compute the closed-form approximate size and its gap to the searched size.
- Compute the two power curves, the exact binomial power where it applies, the sampling picture and the critical values at seven significance levels.
- Set the status and, in a challenge, score it.
Outputs and interpretation
- Power at the plan and beta at the plan (1 - power).
- Planned group 2 size and planned total.
- Required group 1 size, group 2 size and total, and the power at the required size.
- Closed-form size and its gap to the required size.
- Critical statistic (z or t), critical difference (the smallest estimated difference that is significant), and the standard errors if there is no effect and if the effect is real.
- Noncentrality (means families) and degrees of freedom (t test) at the plan.
- Effect size d (means) or h (proportions).
- Exact binomial power at the plan (one-sample proportions, up to 2,000).
- Smallest allowed size: at least 2 in each group, and for the t test enough for 4 degrees of freedom.
- Settings used: every model input in effect.
- Tables: the sampling picture, power against size (with the exact binomial power), power against effect, the 16 search rows, and the critical value at alpha = 0.001, 0.005, 0.01, 0.02, 0.05, 0.1 and 0.2.
Reading the canvas: the curve for no effect and the curve for the real effect overlap; the area of the no-effect curve beyond the critical line is alpha, the area of the effect curve beyond it is the power, and the rest of the effect curve is beta. A larger effect, a smaller standard deviation or a larger study pulls the curves apart.
Challenges
Each challenge fixes its own design, except the input it leaves to you. The model scores each one MET (100) or NOT MET (0) and gives a one-line hint while it is not met.
- Predict the power. Predict the power of this two-group study with 20 per group, then reveal it. It is met when your band holds the power at the plan.
- Smallest size. Find the smallest number of participants that gives this paired t test a power of at least 0.9. It is met when your planned size equals the required size.
- Split 120. Split 120 participants between two groups to get the most power. The group size ratio sets the split, and it is met when the power is the largest of the nine ratios (within 1×10⁻¹² relative).
- Smallest effect. With 100 per group, find the smallest effect this study detects with power 0.9, to the nearest 0.1. It is met when the power reaches 0.9 at your effect but not at an effect 0.1 smaller.
- The saw-tooth. Find a size where adding one participant lowers the exact power of this proportion test. It is met when the exact power at one more participant is lower by more than 1×10⁻⁹ (sizes up to 1,999).
Three challenges hold results back. In challenge 1, until you predict, the power, beta, the exact power and the power columns of both curves are blank, and the challenge reads NOT MET with a prompt to predict; the sampling picture stays, because estimating the shaded area is the skill. In challenge 2 the required size, its power, the closed-form size and gap, the search rows and the power-against-size curve are blank: you move the planned size until the power card passes 0.9. In challenge 4 the power-against-effect curve is blank.
What to take from the lab: halving the effect roughly quadruples the size needed; equal groups give the most power for a fixed total; the t test needs a few more participants than the z shortcut suggests; and exact power for counts does not always rise with size.
Validation and status logic
The status reads the first match in the list under Errors and warnings. Items 1 to 9 are checked on the values you entered; the CHECK items on the values in use, and they apply inside challenges too. A challenge number that is not a whole number from 0 to 5 counts as no challenge.
Assumptions and limitations
- The effect and the standard deviation (or the two proportions) are taken as known and correct; the power is only as good as these guesses.
- Observations are independent; the t test assumes normal data with equal spread in both groups; the proportions test uses the normal approximation without continuity correction.
- Group 1 runs up to 40,000 in the search and the curves, and the planned size up to 10,000; group 2 is k times group 1, rounded up.
- The closed-form size is a teaching comparison: it ignores the far tail and, for the t test, uses z.
- The t sampling curve for the real effect is an approximate shape; the t power itself is exact to within 2×10⁻⁹.
- The exact binomial power is computed for one-sample proportions up to 2,000 participants only.
- No adjustment is made for dropouts, clustering, multiple tests or sequential looks.
Restrictions and non-computing states
Under NOT VALID every result and every table is blank, apart from the model version, the challenge status (No challenge selected, or NOT MET when a challenge is selected), a score of 0, and the goal and hint of a selected challenge.
For a t test planned below its smallest size, the results at the plan, the sampling picture, the effect curve and the critical values are blank. With no effect, or when even 40,000 in group 1 does not reach the target, the required size, its power, the closed-form gap and the search rows are blank. The group 2 sizes are blank for one sample.
In challenges 1, 2 and 4, results are held back as described under Challenges.
Errors and warnings
The status messages, first match wins:
- NOT VALID: choose a test family from the list
- NOT VALID: choose a study layout from the list
- NOT VALID: choose Two-sided or One-sided from the list
- NOT VALID: the significance level must be from 0.001 to 0.2 and the target power from 0.5 to 0.99
- NOT VALID: the mean difference must be from 0 to 1000 and the standard deviation from 0.1 to 1000
- NOT VALID: each proportion must be from 0.01 to 0.99
- NOT VALID: choose a group size ratio from the list
- NOT VALID: the planned size must be a whole number from 2 to 10000
- NOT VALID: the challenge number must be a whole number from 0 to 5, and the power prediction one of the listed choices
- CHECK: a t test here needs at least 4 degrees of freedom, so the planned size must be at least m in group 1; the results at the plan are blank: m is the smallest allowed size.
- CHECK: there is no effect to detect (the two means or the two proportions are equal), so the power equals the significance level and no size can reach the target
- CHECK: even 40,000 in group 1 does not reach the target power; the effect is too small to detect within this lab's range
- CHECK: the power at the required size is within 0.000001 of the target, so a tool that computes power another way may report one more or one fewer: the required size sits at the edge of the target.
- CHECK: at the planned size the normal approximation for proportions is rough (fewer than 5 expected successes or failures in a group), so treat the power as approximate
- OK
References
This laboratory is an educational model of statistical power and sample size planning. It computes the power of the z test for means, of the t test for means through the noncentral t distribution (evaluated by Gauss-Legendre quadrature), and of the normal-approximation test for proportions, and it finds the smallest size that reaches a target power by an integer bisection, beside the familiar closed-form sample size formula. It reports Cohen's effect sizes d and h, and, for a one-sample proportion, the exact binomial power, whose saw-tooth shape shows that power need not rise with every added participant.
Sources and further reading
- The sample size needed to detect a shift in a mean with given risks of a false alarm and a miss, for known and estimated standard deviations: NIST/SEMATECH, e-Handbook of Statistical Methods, section 7.2.2.2, Sample sizes required. NIST
- The sample size for a test of a proportion by the normal approximation, with the null and alternative proportions: NIST/SEMATECH, e-Handbook of Statistical Methods, section 7.2.4.2, Sample sizes required. NIST
- The noncentral t distribution and its use for the power of the t test: Wikipedia, Noncentral t-distribution, Use in power analysis. Wikipedia
- Gauss-Legendre quadrature: NIST, Digital Library of Mathematical Functions, section 3.5, Quadrature, subsection 3.5(v), Gauss Quadrature. NIST DLMF
- Power analysis and the effect sizes d and h: J. Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd edition, Lawrence Erlbaum Associates, 1988.
- The saw-tooth behaviour of exact binomial power: M. R. Chernick and C. Y. Liu, "The saw-toothed behavior of power versus sample size and software solutions: single binomial proportion using exact methods", The American Statistician 56(2), 2002, pages 149-155.
No table or text from any of these publications is reproduced, and none of their authors or publishers has reviewed or endorses this laboratory.
This laboratory is for learning. Its results hold only within its stated ranges; a plan is only as good as the effect and the spread assumed, so a real study should be planned with a statistician.
Links were current at build time. If one moves, search the document's title rather than reporting a broken link as a defect.
Found a problem, or have an idea?
Tell us if a result looks wrong, a label is unclear, or something is missing. We read every message.