Calculator overview
Inputs and outputs
This summary comes from the calculator's published input and output contract.
Inputs
- Significance level alpha (required)
-
Default 0.05 Range 0.001 to 0.2
About this input
The significance level alpha, from 0.001 to 0.2. Set with the marker on the threshold chart.
- Observations per experiment n (required)
-
Default 10 Range 2 to 16
About this input
The number of observations n in each experiment, a whole number from 2 to 16.
- Random seed (required)
-
Default 8080 Range 1 to 2147483646
About this input
The seed of the random stream, a whole number from 1 to 2147483646. The same seed always gives the same experiments.
- True mean μ₁ (required)
-
Unit units Default 105 Range -1000 to 1000
About this input
The true mean mu1 of the effect world, from -1000 to 1000 units and within 5 population SDs of the null mean.
- Tails (required)
-
Default Two-sided Allowed Two-sided, Upper tail, Lower tail
About this input
Two-sided, Upper tail or Lower tail: where the rejection region lies. Set with the chip on the canvas.
- Test (required)
-
Default z test (known SD) Allowed z test (known SD), t test (sample SD), z test with the sample SD (misused)
About this input
The test: z test (known SD), t test (sample SD), or z test with the sample SD (misused), which judges the t statistic against normal critical values.
- Predicted false alarms, low end
-
When omitted Blank
Default Not set Range 0 to 200
About this input
Challenge 1: the low end of your predicted false-alarm count, a whole number from 0 to 200. Blank until you predict; 0 is a real prediction.
- Inspected experiment (required)
-
Default 1 Range 1 to 200
About this input
The experiment whose observations are shown, a whole number from 1 to 200; above the number of experiments it reads the last one. Set by clicking a dot.
- Number of experiments R (required)
-
Default 200 Range 20 to 200
About this input
The number of experiments R, a multiple of 20 from 20 to 200. Raising it adds experiments and keeps the earlier ones unchanged.
- Challenge (required)
-
Default 0 Range 0 to 5
About this input
Selects a challenge: 0 for none, 1 to 5 for a predefined challenge, which fixes the settings except the input it leaves to you.
- Predicted false alarms, high end
-
When omitted Blank
Default Not set Range 0 to 200
About this input
Challenge 1: the high end of your predicted false-alarm count, a whole number from 0 to 200. Blank until you predict.
- Population SD sigma (required)
-
Unit units Default 10 Range 0.1 to 1000
About this input
The population SD sigma, from 0.1 to 1000 units. The z test uses it; the t-based tests estimate it from each sample.
- Null mean μ₀ (required)
-
Unit units Default 100 Range -1000 to 1000
About this input
The null mean mu0: the mean the null hypothesis states, from -1000 to 1000 units.
Outputs
- Significance level used
-
No unit declared
About this output
The significance level the calculation used. Blank when the inputs are not valid.
- Seed used
-
No unit declared
About this output
The seed the calculation used (the challenge value when a challenge is selected). Blank when the inputs are not valid.
- Sample size used
-
No unit declared
About this output
The sample size the calculation used. Blank when the inputs are not valid.
- Statistic axis, low end
-
No unit declared
About this output
The low end of the statistic axis: the smaller of 0 and lambda, less 5.
- Statistic axis, high end
-
No unit declared
About this output
The high end of the statistic axis: the larger of 0 and lambda, plus 5.
- Effect in population SDs (delta)
-
No unit declared
About this output
delta, the effect in population SDs: (true mean - null mean)/population SD. Blank when the inputs are not valid.
- Population SD used
-
Unit units
About this output
The population SD the calculation used. Blank when the inputs are not valid.
- Null mean used
-
Unit units
About this output
The null mean the calculation used. Blank when the inputs are not valid.
- Formula module versions
-
No unit declared
About this output
The formula modules the model is built from, with their versions.
- The running rates: the share of experiments rejected so far, every 10 experiments
-
No unit declared
About this output
The share of experiments rejected so far in each world, every 10 experiments (rows past the number of experiments blank); the last row equals the rates at your alpha. Withheld while challenge 5 is active and not met.
- The p-value histogram: 20 bins of width 0.05, per world
-
No unit declared
About this output
The p-values of each world in 20 bins of width 0.05: flat under a true null for the z and t tests, piled near 0 under an effect.
- Theory power
-
No unit declared
About this output
The theory power at your alpha: a closed form for the z test; a fixed 32-point quadrature of the noncentral t for the t-based tests (about 1E-9).
- The curves of the statistic: the null and effect densities, the density the test assumes, and the rejection region
-
No unit declared
About this output
The statistic axis at 101 points: the null density, the effect density, the density the test assumes (the normal curve for the misused test) and whether the point is in the rejection region.
- Exact Type I rate
-
No unit declared
About this output
The exact Type I rate: alpha for the z and t tests; for the misused test the chance that a t statistic passes the normal critical value.
- False alarms (null-world rejections)
-
No unit declared
About this output
False alarms: the null-world experiments that reject at your alpha. Blank when the inputs are not valid.
- Type I rate, 95% band low
-
No unit declared
About this output
The low end of the 95% Wilson band of the Type I rate (exactly 0 with no false alarm).
- Simulated Type I rate
-
No unit declared
About this output
The simulated Type I rate: false alarms over the number of experiments. Blank when the inputs are not valid.
- Theory Type II rate
-
No unit declared
About this output
The theory Type II rate, 1 - theory power.
- Simulated Type II rate (misses)
-
No unit declared
About this output
The simulated Type II rate: the effect-world experiments that do not reject, over the number of experiments.
- Tails used
-
No unit declared
About this output
The tails the calculation used. Blank when the inputs are not valid.
- Test used
-
No unit declared
About this output
The test the calculation used. Blank when the inputs are not valid.
- The statistic histogram: 30 bins across the axis and the experiments in each, per world
-
No unit declared
About this output
The 30 statistic bins across the axis (the end bins also hold every statistic beyond it) and the experiments of each world in each.
- Type I rate, 95% band high
-
No unit declared
About this output
The high end of the 95% Wilson band of the Type I rate (exactly 1 when every experiment rejects).
- True mean used
-
Unit units
About this output
The true mean the calculation used. Blank when the inputs are not valid.
- The threshold sweep: seven fixed significance levels and yours, with the simulated and theory error rates and their 95% bands
-
No unit declared
About this output
Seven fixed significance levels and yours: the critical value, the simulated and exact Type I rates, the simulated and theory Type II rates, their 95% Wilson bands and the borderline count. Withheld while challenge 3 is active and not met.
- Model version
-
No unit declared
About this output
The model version.
- Degrees of freedom
-
No unit declared
About this output
n - 1, the degrees of freedom of the t distribution (shown for every test). Blank when the inputs are not valid.
- Lower critical value
-
No unit declared
About this output
The lower critical value at your alpha (minus the critical value); blank for an upper-tail test or when the inputs are not valid.
- Upper critical value
-
No unit declared
About this output
The upper critical value at your alpha; blank for a lower-tail test or when the inputs are not valid.
- Detections (effect-world rejections)
-
No unit declared
About this output
Detections: the effect-world experiments that reject at your alpha. Blank when the inputs are not valid.
- Detection rate, 95% band low
-
No unit declared
About this output
The low end of the 95% Wilson band of the detection rate.
- Detection rate, 95% band high
-
No unit declared
About this output
The high end of the 95% Wilson band of the detection rate.
- Challenge goal
-
No unit declared
About this output
The goal of the selected challenge; blank with no challenge.
- Challenge feedback
-
No unit declared
About this output
The met text when the challenge is met, otherwise a hint; blank with no challenge.
- Decisions within rounding of a critical value
-
No unit declared
About this output
The (experiment, world, threshold) triples whose statistic sits within 1E-9 relative of a critical value, at your alpha and the seven sweep thresholds; above 0 the status reads CHECK.
- Challenge result
-
No unit declared
About this output
No challenge selected, MET or NOT MET.
- Challenge score
-
No unit declared
About this output
100 when the selected challenge is met, else 0.
- Challenge: the value scored
-
No unit declared
About this output
The value the selected challenge scores: the chance the predicted range holds the count (1), the theory Type II rate (2), alpha plus the theory Type II rate (3), the theory power (4) or the inspected null-world p-value (5). Blank with no challenge, before both counts are predicted, or when the inputs are not valid.
- Simulated power (detection rate)
-
No unit declared
About this output
The simulated power: detections over the number of experiments. Blank when the inputs are not valid.
- Inspected statistic, effect world
-
No unit declared
About this output
The inspected experiment's statistic in the effect world. Blank as the null-world statistic.
- Inspected sample SD
-
Unit units
About this output
The inspected experiment's sample SD (dividing by n - 1), in units; the same in both worlds.
- Inspected sample mean (null world)
-
Unit units
About this output
The inspected experiment's sample mean in the null world, in units.
- Model Status
-
No unit declared
About this output
NOT VALID with the reason when an input cannot be used; CHECK when an experiment's statistic sits within rounding of a critical value, or a t statistic is undefined; otherwise OK. A misused test with a high false-alarm rate reads OK: it is the lesson.
- Experiment inspected
-
No unit declared
About this output
The experiment inspected: your inspected experiment, or the number of experiments when it is larger.
- Inspected statistic, null world
-
No unit declared
About this output
The inspected experiment's statistic in the null world. Blank when its t statistic is undefined or the inputs are not valid.
- The experiments: each one's statistic in the null world and the effect world, their histogram bins, and the outcome (0 neither rejects, 1 a false alarm, 2 a detection, 3 both)
-
No unit declared
About this output
One row per experiment (rows past the number of experiments blank): the null-world and effect-world statistics, their histogram bins and the outcome code (0 neither world rejects, 1 only the null world, 2 only the effect world, 3 both). The outcome column is withheld while challenge 5 is active and not met.
- Experiments used
-
No unit declared
About this output
The number of experiments the calculation used. Blank when the inputs are not valid.
- Shift of the statistic (lambda)
-
No unit declared
About this output
lambda = delta x SQRT(n), the shift of the effect-world statistic's centre in standard errors. Blank when the inputs are not valid.
- Inspected p-value, null world
-
No unit declared
About this output
The inspected experiment's null-world p-value (shown, never used to decide). Blank as the statistic.
- Inspected p-value, effect world
-
No unit declared
About this output
The inspected experiment's effect-world p-value. Blank as the statistic.
- The inspected experiment's observations in both worlds
-
No unit declared
About this output
The inspected experiment's n observations in the null world and the effect world (the same draws shifted by the true mean less the null mean).
LogicCommons is in beta. If a result, label, or reference looks wrong, tell us here; we read every message.
Methodology
Purpose and model boundary
This lab repeats one small experiment many times and tests each repeat, so that the two kinds of error a hypothesis test can make become counts you can see. Each of up to 200 experiments draws 2 to 16 observations from a normal population with a known standard deviation and tests the null hypothesis "the mean is μ₀". Every experiment is tested twice on the same draws: once in a null world, where the mean really is μ₀, and once in an effect world, where the mean is the true mean μ₁. A rejection in the null world is a false alarm (a Type I error); a non-rejection in the effect world is a miss (a Type II error). Three tests are offered: the z test with the known SD, the t test with the sample SD, and the misused z test, which divides by the sample SD but still uses normal critical values. Each can be two-sided or one-sided.
For the run the lab returns the false alarms and detections with 95% bands, the exact Type I rate and the theory power beside them, the same rates at seven fixed significance levels, the histograms of the statistic and of the p-value in each world, the running rates, one inspected experiment in full, and five scored challenges.
It is a learning tool for what a significance level, a Type I rate, a Type II rate and power mean, and why a test must match how the standard deviation is known. It is not a statistics tool for real decisions. Two-sample, paired and proportion tests, non-normal populations, a search over the sample size, and more than 200 experiments or 16 observations are not modelled.
The model is the calculation authority. The page sends the values you set to the model and draws only what it returns: the experiment dots in both worlds, the null, effect and assumed-null curves, the shaded rejection region, the histograms, the threshold chart, the running rates and the inspected observations. The page computes no statistic, p-value, decision, rate or power itself, and draws no random number.
Inputs and units
The means and the standard deviation are in the units of the measurement (any unit, the same for all three). Every input is required and is checked on its entered value, whether or not the selected test or challenge uses it.
| Input | Meaning and unit |
|---|---|
| Random seed | The start of the random sequence, a whole number from 1 to 2,147,483,646 (default 8080). The same seed always gives the same experiments. |
| Number of experiments R | A multiple of 20 from 20 to 200 (default 200). Raising it adds experiments and keeps the earlier ones unchanged. |
| Observations per experiment n | A whole number from 2 to 16 (default 10). |
| Null mean μ₀ | The mean the null hypothesis states, -1,000 to 1,000 units (default 100). |
| True mean μ₁ | The mean of the effect world, -1,000 to 1,000 units (default 105), and within 5 population SDs of the null mean. |
| Population SD σ | 0.1 to 1,000 units (default 10). The z test uses it; the two t-based tests estimate it from each sample. |
| Test | z test (known SD), t test (sample SD), or z test with the sample SD (misused) (default the z test). |
| Tails | Two-sided, Upper tail or Lower tail: where the rejection region lies (default Two-sided). |
| Significance level α | 0.001 to 0.2 (default 0.05). |
| Inspected experiment | A whole number from 1 to 200 (default 1): the experiment whose observations are shown. Above R it reads the last experiment, without a status. |
| Challenge | 0 for free exploration, or 1 to 5 to load a challenge (default 0). |
| Predicted false alarms, low end and high end | Challenge 1 only: a whole number from 0 to 200 each, blank until you predict. Blank means not yet predicted; 0 is a real prediction. |
Where each input is set:
- On the canvas only: the significance level (the marker on the threshold chart, on a log scale; an exact value can be typed in its popover), the tails (the chips), and the inspected experiment (a click on an experiment dot).
- In the input panel: the seed, the number of experiments, the observations per experiment, the two means, the population SD, the test, the challenge and challenge 1's two predicted counts.
Governing relationships
The generator. The model implements Park and Miller's minimal standard generator (MINSTD) with the multiplier 48271. Starting from s₀ = the seed, each state is
sₖ = MOD(48271 · sₖ₋₁, 2³¹ - 1), and each draw is uₖ = sₖ / (2³¹ - 1),
which lies strictly between 0 and 1. Every product stays below 2⁴⁷, so the integer sequence is exact. The stream discards the generator's first two outputs, so the first draw used is s₃, three steps from the seed; this keeps a small seed from giving a tiny first draw. Experiment r (1 to 200) always owns the 16 draws s₁₆₍ᵣ₋₁₎₊₃ to s₁₆ᵣ₊₂, whatever n and R are. Changing n adds or drops the last observations of the same experiments, and changing R adds or drops whole experiments; the stream never shifts.
Normal draws. The 16 draws of an experiment are turned into 16 standard normal values in 8 pairs by the Box-Muller transform: with ua and ub the two draws of a pair,
z₁ = √(-2 ln ua) · cos(2π ub), z₂ = √(-2 ln ua) · sin(2π ub).
The first n of the 16 are the experiment's observations, in standard units. The null-world observations are x = μ₀ + σ·z, and the effect-world observations are the same values shifted by μ₁ - μ₀.
Sample statistics. For each experiment, z̄ is the mean of its n standard values and s_z their sample standard deviation (dividing by n - 1). With δ = (μ₁ - μ₀)/σ, the effect in population SDs, and λ = δ·√n, the shift of the statistic's centre in standard errors:
- z test: T₀ = z̄·√n in the null world and T₁ = (z̄ + δ)·√n in the effect world.
- t test and misused z test: T₀ = z̄·√n / s_z and T₁ = (z̄ + δ)·√n / s_z. These equal the usual (x̄ - μ₀)/(s/√n) in each world. If every observation were equal, s_z would be 0 and both are taken as 0, which never rejects.
Decisions. The signed position of a statistic T is |T| for a two-sided test, T for an upper-tail test and -T for a lower-tail test. An experiment rejects at a critical value c when its signed position is at least c. The critical value for a level a is:
| Test | Two-sided | One-sided |
|---|---|---|
| z test, misused z test | the normal value with a/2 above it | the normal value with a above it |
| t test | the t value (n - 1 degrees of freedom) with a/2 in each tail | the t value with a above it |
The misused z test judges a t statistic against a normal critical value: that is the error the lab teaches.
p-values. For the z test and the misused z test, the p-value is 2Φ(-|T|) two-sided, Φ(-T) upper and Φ(T) lower, with Φ the standard normal distribution function; for the t test the same forms use the t distribution with n - 1 degrees of freedom. The misused test reports the normal p-value of a t statistic. The p-values are shown and binned but never used to decide: every decision uses the critical-value rule above.
Exact Type I rate. It is α for the z test and the t test. For the misused z test it is the chance that a t statistic with n - 1 degrees of freedom passes the normal critical value: 2·P(t > c) two-sided, P(t > c) one-sided.
Theory power. For the z test, with μ = |λ| for a two-sided test, λ for upper and -λ for lower:
power = Φ(μ - c) + Φ(-μ - c) two-sided, power = Φ(μ - c) one-sided.
For the two t-based tests the effect-world statistic follows a noncentral t distribution. Writing T₁ = Z / (W/√df), with Z normal with mean λ and SD 1, W following a chi distribution with df = n - 1 degrees of freedom and density f(w), and s = √df, the power is the average over W of
Φ(μ - c·W/s) + Φ(-μ - c·W/s) two-sided, Φ(μ - c·W/s) one-sided.
The model evaluates this integral with a fixed 32-point Gauss-Legendre rule on two windows of w chosen so that both the slowly varying chi density and the sharp step of the normal term are resolved; the result is accurate to about 1×10⁻⁹. The theory Type II rate is 1 - power.
Curves. On 101 points across the statistic axis the model returns the null density (normal for the z test, t for the t-based tests), the effect density (the normal curve shifted by λ for the z test, the noncentral t density for the t-based tests by a 24-point Gauss-Legendre rule, accurate to about 1×10⁻⁶ of the peak), the density the test assumes (the normal curve for the misused test, otherwise the null density) and whether each point lies in the rejection region. For the misused test, the area of the true (t) null curve beyond the normal critical value is the true Type I rate.
Wilson bands. Each simulated rate k/R gets a 95% Wilson score interval: with z the normal value with 2.5% above it (about 1.96), p̂ = k/R and d = 1 + z²/R, the centre is (p̂ + z²/(2R))/d and the half-width is z·√(p̂(1 - p̂)/R + z²/(4R²))/d. The low end is exactly 0 when k = 0 and the high end exactly 1 when k = R.
Calculation sequence
- Check the entered values in the order of the status list under Errors and warnings.
- Fix the values used: your values, or in a challenge the challenge's own values, except the input it leaves to you.
- Generate the 3,200 draws (16 for each of the 200 experiments, always all of them) and turn them into normal values by Box-Muller.
- For each experiment compute z̄, s_z, the two statistics, their decisions at your α, the outcome code and their histogram bins and p-value bins.
- Count the false alarms and detections of the first R experiments at your α and at the seven sweep levels, with the Wilson bands, the exact Type I rates and the theory power.
- Build the statistic axis, the curves, the two histograms, the running rates and the inspected experiment.
- Count the decisions that sit within rounding of a critical value, set the status and, in a challenge, score the challenge.
Outputs and interpretation
- False alarms and simulated Type I rate: the null-world experiments that reject at your α, and that count over R, with the low and high ends of its 95% Wilson band. Exact Type I rate: what that share approaches over many experiments.
- Detections and simulated power: the effect-world experiments that reject, and that count over R, with its band. Simulated Type II rate: the effect-world experiments that do not reject, over R. Theory power and theory Type II rate: the values the simulated ones approach. When the true mean equals the null mean both worlds are the same, so the "detections" are false alarms too.
- Effect δ, shift λ and degrees of freedom: δ in population SDs, λ in standard errors, and n - 1.
- Lower and upper critical values: ±c at your α; the lower one is blank for an upper-tail test and the upper one for a lower-tail test.
- Inspected experiment: the experiment shown (your choice, or R if that is smaller), its statistic and p-value in both worlds, its null-world sample mean and its sample SD in units, and its n observations in both worlds.
- Decisions within rounding of a critical value: the number of (experiment, world, level) decisions whose statistic lies within 1×10⁻⁹ (relative) of a critical value, at your α and the seven sweep levels.
- Statistic axis, low and high ends: the smaller of 0 and λ less 5, and the larger plus 5, so both centres show with 5 standard errors on each side.
- Experiment table: one row per experiment: the two statistics, their histogram bins, and the outcome code (0 neither world rejects, 1 only the null world, 2 only the effect world, 3 both). Rows past R are blank.
- Statistic histogram: 30 equal bins across the axis, the end bins also holding every statistic beyond it, with the count of each world. With the t test at n = 2 many dots pile into the end bins: the t statistic has heavy tails.
- p-value histogram: 20 bins of width 0.05 for each world. Under a true null the z and t tests give a roughly flat histogram; under an effect the p-values pile near 0; the misused test piles extra p-values near 0 even under the null.
- Threshold sweep: the seven levels 0.001, 0.005, 0.01, 0.02, 0.05, 0.1 and 0.2, and your α as an eighth row: the critical value, the simulated and exact Type I rates, the simulated and theory Type II rates, both Wilson bands and the near-threshold count. Lowering α trades fewer false alarms for more misses.
- Running rates: every 10 experiments, the share rejected so far in each world. The last row equals the rates at your α.
- Settings used: the seed, counts, means, SD, test, tails and level of the run (a challenge's own values while a challenge is selected), the model version and the formula modules.
Reading the canvas: two stacked number lines show every experiment as a dot at its statistic, null world above and effect world below, with the critical values and the shaded rejection region; a coloured dot rejected. The curves show where the statistic falls in each world, and the threshold chart shows the error rates against α on a log scale.
Challenges
Choosing a challenge from 1 to 5 loads it. Every challenge uses the z test (known SD) and 200 experiments, fixes its own seed, means, SD, tails and level, and leaves you one input. The model scores it MET (score 100) or NOT MET (score 0), with the goal, the value scored, and a short explanation or hint.
- Predict the false alarms. In 200 experiments where the null is true, enter the lowest and highest false-alarm counts of a range no wider than 10 that holds the count at least 90% of the time. Open: the two predicted counts. Scored on the exact chance P(low ≤ X ≤ high) for X binomial with 200 tries and chance α = 0.05; MET when low ≤ high, high - low ≤ 10 and that chance is at least 0.9. The value scored is blank until both counts are entered (or when low is above high), and the page keeps the results and curves hidden until both are set.
- Hold the miss rate. With 16 observations and an effect of 0.75 SD, choose the smallest significance level that keeps the theory miss rate (Type II) at or below 30%. Open: the significance level. MET when the theory Type II rate at your α is at most 0.3 and at α - 0.001 it is above 0.3 (an α of 0.001 counts as having no smaller step).
- The best threshold for equal costs. Among the seven sweep thresholds, choose the one with the smallest Type I rate plus Type II rate, both counted equally. Open: the significance level. Scored on the exact Type I rate plus the theory Type II rate; MET when your α equals the sweep level with the smallest total (a tie goes to the lower level). While this challenge is not met the threshold sweep is held back.
- Point the test the right way. The true mean is above the null mean; choose the tails that give the test the most power at α = 0.05. Open: the tails. MET when the theory power equals the largest of the two-sided, upper-tail and lower-tail powers.
- Catch a false discovery. The null is true in both worlds; find an experiment that the test rejects at α = 0.01. Open: the inspected experiment. MET when the inspected experiment's null-world statistic passes the critical value. The value shown is its null-world p-value. While this challenge is not met the outcome codes and the running rates are held back, and the page draws no dot coloured and no region shaded.
On the page the threshold lines are also hidden in challenges 2 and 3, since they would show the answer; your α's own rates stay in Results. Challenges 1 to 4 are scored on exact or theory values, so the seed does not decide them; challenge 5 is about the simulated experiments themselves.
What to take from the lab: a significance level is a long-run false-alarm rate, so about one experiment in twenty rejects a true null at α = 0.05 and any one run strays from that by chance; lowering α buys fewer false alarms with more misses; power depends on the effect, n, α and the direction of the test; and using normal critical values with an estimated SD raises the true Type I rate well above α for small samples.
Validation and status logic
The status reads the first match in the list under Errors and warnings. The NOT VALID checks apply to the entered values (the effect bound is tested only when both means and the SD are in range); the two CHECK items apply to the values used, so a challenge's own settings decide them, and they apply inside challenges too. A CHECK does not stop the calculation: every result is still shown. A challenge number that is not a whole number from 0 to 5 counts as no challenge. An unordered pair of predicted counts is a challenge result (NOT MET), not a status.
Results that look surprising are not statuses: a misused z test with a Type I rate far above α, a simulated rate away from its theory value, a power near 0 in the wrong tail, and a false discovery are what the lab teaches, and the status reads OK.
Assumptions and limitations
- The population is normal with the stated SD, and the observations are independent. Both worlds use the same draws (common random numbers), so with no effect they give identical counts by design.
- At most 200 experiments of at most 16 observations; a simulated rate from 200 experiments is rough, which the 95% bands show.
- The effect is at most 5 population SDs; at exactly 5 SDs the theory power is 1.
- The generator is deterministic and reproducible from its seed. It is a small linear congruential generator; fed to Box-Muller it leaves a faint lattice pattern in the far tails, harmless at these sizes. It is fine for teaching and wrong for cryptography or serious simulation.
- The theory power of the t-based tests is a quadrature value, accurate to about 1×10⁻⁹, not an exact closed form; the effect-density curve of those tests is a display curve accurate to about 1×10⁻⁶ of its peak.
- A statistic within about 1×10⁻⁹ of a bin edge, or a p-value within 1×10⁻⁹ of a multiple of 0.05, may fall in the neighbouring bin; it never changes a decision.
Restrictions and non-computing states
Under NOT VALID every result and every table is blank, except the model version and formula modules, the challenge goal and the not-met text of a selected challenge, the challenge result (NOT MET when a challenge is selected, otherwise No challenge selected) and a score of 0.
While a challenge is active and not met, some results are held back rather than computed blank by error: the threshold sweep in challenge 3, and the outcome codes of the experiment table and the running rates in challenge 5. In challenge 1 the value scored is blank until both predicted counts are entered. Once a challenge is met every result shows.
When the t-based tests meet an experiment whose observations are all equal, that experiment's statistic is taken as 0 and counted as not rejected, and the inspected statistic and p-values read blank if it is the inspected one.
Errors and warnings
The status messages, first match wins:
- NOT VALID: the seed must be a whole number from 1 to 2147483646
- NOT VALID: the number of experiments must be a multiple of 20 from 20 to 200
- NOT VALID: the sample size must be a whole number from 2 to 16
- NOT VALID: the null mean and the true mean must be from -1000 to 1000, and the population SD from 0.1 to 1000
- NOT VALID: the true mean must lie within 5 population SDs of the null mean : the size of μ₁ - μ₀ is more than 5σ.
- NOT VALID: choose the test from the list
- NOT VALID: choose the tails from the list
- NOT VALID: the significance level must be from 0.001 to 0.2
- NOT VALID: the inspected experiment must be a whole number from 1 to 200
- NOT VALID: the challenge number must be a whole number from 0 to 5, and each predicted count a whole number from 0 to 200
- CHECK: an experiment's observations are all equal, so its t statistic is undefined; it is counted as not rejected. Try another seed : a t-based test meets an experiment with a sample SD of 0. It cannot happen with distinct draws and is a guard.
- CHECK: a simulated statistic sits within rounding of a critical value, so that experiment's decision could change in the last digit; try another seed : the count of decisions within rounding of a critical value is above 0.
- OK : the run is computed with nothing to flag.
Most out-of-range values are refused by the page before the model sees them; the model's own checks above give the same verdict.
References
This laboratory is an educational model of one-sample hypothesis tests of a normal mean: the z test with a known standard deviation, the t test with the sample standard deviation, and the misuse of normal critical values with the sample standard deviation. Its random numbers come from the Park-Miller minimal standard (MINSTD) generator with the multiplier 48271, turned into normal values by the Box-Muller transform, so every run is exact and repeatable from its seed. The simulated false-alarm and detection rates get Wilson score intervals, and the theory power of the t-based tests is the noncentral t probability, evaluated by Gauss-Legendre quadrature.
Sources and further reading
- The one-sample t test and z test of a mean, with two-sided and one-sided critical values: NIST/SEMATECH, e-Handbook of Statistical Methods, section 7.2.2, Are the data consistent with the assumed process mean? NIST
- Type I and Type II errors, the power of a test and power for a test of a normal mean: The Pennsylvania State University, STAT 415, Introduction to Mathematical Statistics, lesson 10, Hypothesis Tests (Part II), section 10.3, Power of a Test. Penn State
- The Wilson score interval for a proportion: NIST/SEMATECH, e-Handbook of Statistical Methods, section 7.2.4.1, Confidence intervals. NIST
- Gauss-Legendre quadrature: NIST, Digital Library of Mathematical Functions, section 3.5(v), Gauss Quadrature. DLMF
- The minimal standard generator: S. K. Park and K. W. Miller, "Random number generators: good ones are hard to find", Communications of the ACM 31(10), 1988, pages 1192-1201. doi:10.1145/63039.63042
- The Box-Muller transform: G. E. P. Box and M. E. Muller, "A note on the generation of random normal deviates", The Annals of Mathematical Statistics 29(2), 1958, pages 610-611. doi:10.1214/aoms/1177706645
No table or text from any of these publications is reproduced, and none of their authors or publishers has reviewed or endorses this laboratory.
This laboratory is for learning. Its results hold only within its stated ranges; a simulated rate from at most 200 experiments is one rough run, not a property of the test, and real data need a test chosen for how they were collected.
Links were current at build time. If one moves, search the document's title rather than reporting a broken link as a defect.
Found a problem, or have an idea?
Tell us if a result looks wrong, a label is unclear, or something is missing. We read every message.