Statistics & decisions

Hypothesis Testing Simulator

Up to 200 seeded one-sample experiments of 2 to 16 normal observations, each tested twice on the same draws (a world where the null is true and one where the mean differs) by the z test, the t test or the misused z test with the sample SD, two-sided or one-sided: the false alarms and detections with their Wilson bands beside the exact Type I rate and the theory power (a closed form for z, a fixed Gauss-Legendre quadrature of the noncentral t), the same at seven fixed significance levels, the statistic and p-value histograms, the running rates, an inspected experiment, and five scored challenges.

Last updated

Calculator overview

Inputs and outputs

This summary comes from the calculator's published input and output contract.

Inputs

Significance level alpha (required)
About this input

The significance level alpha, from 0.001 to 0.2. Set with the marker on the threshold chart.

Default 0.05 Range 0.001 to 0.2
Observations per experiment n (required)
About this input

The number of observations n in each experiment, a whole number from 2 to 16.

Default 10 Range 2 to 16
Random seed (required)
About this input

The seed of the random stream, a whole number from 1 to 2147483646. The same seed always gives the same experiments.

Default 8080 Range 1 to 2147483646
True mean μ₁ (required)
About this input

The true mean mu1 of the effect world, from -1000 to 1000 units and within 5 population SDs of the null mean.

Unit units Default 105 Range -1000 to 1000
Tails (required)
About this input

Two-sided, Upper tail or Lower tail: where the rejection region lies. Set with the chip on the canvas.

Default Two-sided Allowed Two-sided, Upper tail, Lower tail
Test (required)
About this input

The test: z test (known SD), t test (sample SD), or z test with the sample SD (misused), which judges the t statistic against normal critical values.

Default z test (known SD) Allowed z test (known SD), t test (sample SD), z test with the sample SD (misused)
Predicted false alarms, low end
When omitted Blank
About this input

Challenge 1: the low end of your predicted false-alarm count, a whole number from 0 to 200. Blank until you predict; 0 is a real prediction.

Default Not set Range 0 to 200
Inspected experiment (required)
About this input

The experiment whose observations are shown, a whole number from 1 to 200; above the number of experiments it reads the last one. Set by clicking a dot.

Default 1 Range 1 to 200
Number of experiments R (required)
About this input

The number of experiments R, a multiple of 20 from 20 to 200. Raising it adds experiments and keeps the earlier ones unchanged.

Default 200 Range 20 to 200
Challenge (required)
About this input

Selects a challenge: 0 for none, 1 to 5 for a predefined challenge, which fixes the settings except the input it leaves to you.

Default 0 Range 0 to 5
Predicted false alarms, high end
When omitted Blank
About this input

Challenge 1: the high end of your predicted false-alarm count, a whole number from 0 to 200. Blank until you predict.

Default Not set Range 0 to 200
Population SD sigma (required)
About this input

The population SD sigma, from 0.1 to 1000 units. The z test uses it; the t-based tests estimate it from each sample.

Unit units Default 10 Range 0.1 to 1000
Null mean μ₀ (required)
About this input

The null mean mu0: the mean the null hypothesis states, from -1000 to 1000 units.

Unit units Default 100 Range -1000 to 1000

Outputs

Significance level used
About this output

The significance level the calculation used. Blank when the inputs are not valid.

No unit declared
Seed used
About this output

The seed the calculation used (the challenge value when a challenge is selected). Blank when the inputs are not valid.

No unit declared
Sample size used
About this output

The sample size the calculation used. Blank when the inputs are not valid.

No unit declared
Statistic axis, low end
About this output

The low end of the statistic axis: the smaller of 0 and lambda, less 5.

No unit declared
Statistic axis, high end
About this output

The high end of the statistic axis: the larger of 0 and lambda, plus 5.

No unit declared
Effect in population SDs (delta)
About this output

delta, the effect in population SDs: (true mean - null mean)/population SD. Blank when the inputs are not valid.

No unit declared
Population SD used
About this output

The population SD the calculation used. Blank when the inputs are not valid.

Unit units
Null mean used
About this output

The null mean the calculation used. Blank when the inputs are not valid.

Unit units
Formula module versions
About this output

The formula modules the model is built from, with their versions.

No unit declared
The running rates: the share of experiments rejected so far, every 10 experiments
About this output

The share of experiments rejected so far in each world, every 10 experiments (rows past the number of experiments blank); the last row equals the rates at your alpha. Withheld while challenge 5 is active and not met.

No unit declared
The p-value histogram: 20 bins of width 0.05, per world
About this output

The p-values of each world in 20 bins of width 0.05: flat under a true null for the z and t tests, piled near 0 under an effect.

No unit declared
Theory power
About this output

The theory power at your alpha: a closed form for the z test; a fixed 32-point quadrature of the noncentral t for the t-based tests (about 1E-9).

No unit declared
The curves of the statistic: the null and effect densities, the density the test assumes, and the rejection region
About this output

The statistic axis at 101 points: the null density, the effect density, the density the test assumes (the normal curve for the misused test) and whether the point is in the rejection region.

No unit declared
Exact Type I rate
About this output

The exact Type I rate: alpha for the z and t tests; for the misused test the chance that a t statistic passes the normal critical value.

No unit declared
False alarms (null-world rejections)
About this output

False alarms: the null-world experiments that reject at your alpha. Blank when the inputs are not valid.

No unit declared
Type I rate, 95% band low
About this output

The low end of the 95% Wilson band of the Type I rate (exactly 0 with no false alarm).

No unit declared
Simulated Type I rate
About this output

The simulated Type I rate: false alarms over the number of experiments. Blank when the inputs are not valid.

No unit declared
Theory Type II rate
About this output

The theory Type II rate, 1 - theory power.

No unit declared
Simulated Type II rate (misses)
About this output

The simulated Type II rate: the effect-world experiments that do not reject, over the number of experiments.

No unit declared
Tails used
About this output

The tails the calculation used. Blank when the inputs are not valid.

No unit declared
Test used
About this output

The test the calculation used. Blank when the inputs are not valid.

No unit declared
The statistic histogram: 30 bins across the axis and the experiments in each, per world
About this output

The 30 statistic bins across the axis (the end bins also hold every statistic beyond it) and the experiments of each world in each.

No unit declared
Type I rate, 95% band high
About this output

The high end of the 95% Wilson band of the Type I rate (exactly 1 when every experiment rejects).

No unit declared
True mean used
About this output

The true mean the calculation used. Blank when the inputs are not valid.

Unit units
The threshold sweep: seven fixed significance levels and yours, with the simulated and theory error rates and their 95% bands
About this output

Seven fixed significance levels and yours: the critical value, the simulated and exact Type I rates, the simulated and theory Type II rates, their 95% Wilson bands and the borderline count. Withheld while challenge 3 is active and not met.

No unit declared
Model version
About this output

The model version.

No unit declared
Degrees of freedom
About this output

n - 1, the degrees of freedom of the t distribution (shown for every test). Blank when the inputs are not valid.

No unit declared
Lower critical value
About this output

The lower critical value at your alpha (minus the critical value); blank for an upper-tail test or when the inputs are not valid.

No unit declared
Upper critical value
About this output

The upper critical value at your alpha; blank for a lower-tail test or when the inputs are not valid.

No unit declared
Detections (effect-world rejections)
About this output

Detections: the effect-world experiments that reject at your alpha. Blank when the inputs are not valid.

No unit declared
Detection rate, 95% band low
About this output

The low end of the 95% Wilson band of the detection rate.

No unit declared
Detection rate, 95% band high
About this output

The high end of the 95% Wilson band of the detection rate.

No unit declared
Challenge goal
About this output

The goal of the selected challenge; blank with no challenge.

No unit declared
Challenge feedback
About this output

The met text when the challenge is met, otherwise a hint; blank with no challenge.

No unit declared
Decisions within rounding of a critical value
About this output

The (experiment, world, threshold) triples whose statistic sits within 1E-9 relative of a critical value, at your alpha and the seven sweep thresholds; above 0 the status reads CHECK.

No unit declared
Challenge result
About this output

No challenge selected, MET or NOT MET.

No unit declared
Challenge score
About this output

100 when the selected challenge is met, else 0.

No unit declared
Challenge: the value scored
About this output

The value the selected challenge scores: the chance the predicted range holds the count (1), the theory Type II rate (2), alpha plus the theory Type II rate (3), the theory power (4) or the inspected null-world p-value (5). Blank with no challenge, before both counts are predicted, or when the inputs are not valid.

No unit declared
Simulated power (detection rate)
About this output

The simulated power: detections over the number of experiments. Blank when the inputs are not valid.

No unit declared
Inspected statistic, effect world
About this output

The inspected experiment's statistic in the effect world. Blank as the null-world statistic.

No unit declared
Inspected sample SD
About this output

The inspected experiment's sample SD (dividing by n - 1), in units; the same in both worlds.

Unit units
Inspected sample mean (null world)
About this output

The inspected experiment's sample mean in the null world, in units.

Unit units
Model Status
About this output

NOT VALID with the reason when an input cannot be used; CHECK when an experiment's statistic sits within rounding of a critical value, or a t statistic is undefined; otherwise OK. A misused test with a high false-alarm rate reads OK: it is the lesson.

No unit declared
Experiment inspected
About this output

The experiment inspected: your inspected experiment, or the number of experiments when it is larger.

No unit declared
Inspected statistic, null world
About this output

The inspected experiment's statistic in the null world. Blank when its t statistic is undefined or the inputs are not valid.

No unit declared
The experiments: each one's statistic in the null world and the effect world, their histogram bins, and the outcome (0 neither rejects, 1 a false alarm, 2 a detection, 3 both)
About this output

One row per experiment (rows past the number of experiments blank): the null-world and effect-world statistics, their histogram bins and the outcome code (0 neither world rejects, 1 only the null world, 2 only the effect world, 3 both). The outcome column is withheld while challenge 5 is active and not met.

No unit declared
Experiments used
About this output

The number of experiments the calculation used. Blank when the inputs are not valid.

No unit declared
Shift of the statistic (lambda)
About this output

lambda = delta x SQRT(n), the shift of the effect-world statistic's centre in standard errors. Blank when the inputs are not valid.

No unit declared
Inspected p-value, null world
About this output

The inspected experiment's null-world p-value (shown, never used to decide). Blank as the statistic.

No unit declared
Inspected p-value, effect world
About this output

The inspected experiment's effect-world p-value. Blank as the statistic.

No unit declared
The inspected experiment's observations in both worlds
About this output

The inspected experiment's n observations in the null world and the effect world (the same draws shifted by the true mean less the null mean).

No unit declared
This page is provided by LogicCommons for informational purposes only. Results are analysis outputs computed from the inputs you supply and are not professional advice. Verify all inputs and results independently.

LogicCommons is in beta. If a result, label, or reference looks wrong, tell us here; we read every message.

Methodology

Purpose and model boundary

This lab repeats one small experiment many times and tests each repeat, so that the two kinds of error a hypothesis test can make become counts you can see. Each of up to 200 experiments draws 2 to 16 observations from a normal population with a known standard deviation and tests the null hypothesis "the mean is μ₀". Every experiment is tested twice on the same draws: once in a null world, where the mean really is μ₀, and once in an effect world, where the mean is the true mean μ₁. A rejection in the null world is a false alarm (a Type I error); a non-rejection in the effect world is a miss (a Type II error). Three tests are offered: the z test with the known SD, the t test with the sample SD, and the misused z test, which divides by the sample SD but still uses normal critical values. Each can be two-sided or one-sided.

For the run the lab returns the false alarms and detections with 95% bands, the exact Type I rate and the theory power beside them, the same rates at seven fixed significance levels, the histograms of the statistic and of the p-value in each world, the running rates, one inspected experiment in full, and five scored challenges.

It is a learning tool for what a significance level, a Type I rate, a Type II rate and power mean, and why a test must match how the standard deviation is known. It is not a statistics tool for real decisions. Two-sample, paired and proportion tests, non-normal populations, a search over the sample size, and more than 200 experiments or 16 observations are not modelled.

The model is the calculation authority. The page sends the values you set to the model and draws only what it returns: the experiment dots in both worlds, the null, effect and assumed-null curves, the shaded rejection region, the histograms, the threshold chart, the running rates and the inspected observations. The page computes no statistic, p-value, decision, rate or power itself, and draws no random number.

Inputs and units

The means and the standard deviation are in the units of the measurement (any unit, the same for all three). Every input is required and is checked on its entered value, whether or not the selected test or challenge uses it.

Input Meaning and unit
Random seed The start of the random sequence, a whole number from 1 to 2,147,483,646 (default 8080). The same seed always gives the same experiments.
Number of experiments R A multiple of 20 from 20 to 200 (default 200). Raising it adds experiments and keeps the earlier ones unchanged.
Observations per experiment n A whole number from 2 to 16 (default 10).
Null mean μ₀ The mean the null hypothesis states, -1,000 to 1,000 units (default 100).
True mean μ₁ The mean of the effect world, -1,000 to 1,000 units (default 105), and within 5 population SDs of the null mean.
Population SD σ 0.1 to 1,000 units (default 10). The z test uses it; the two t-based tests estimate it from each sample.
Test z test (known SD), t test (sample SD), or z test with the sample SD (misused) (default the z test).
Tails Two-sided, Upper tail or Lower tail: where the rejection region lies (default Two-sided).
Significance level α 0.001 to 0.2 (default 0.05).
Inspected experiment A whole number from 1 to 200 (default 1): the experiment whose observations are shown. Above R it reads the last experiment, without a status.
Challenge 0 for free exploration, or 1 to 5 to load a challenge (default 0).
Predicted false alarms, low end and high end Challenge 1 only: a whole number from 0 to 200 each, blank until you predict. Blank means not yet predicted; 0 is a real prediction.

Where each input is set:

  • On the canvas only: the significance level (the marker on the threshold chart, on a log scale; an exact value can be typed in its popover), the tails (the chips), and the inspected experiment (a click on an experiment dot).
  • In the input panel: the seed, the number of experiments, the observations per experiment, the two means, the population SD, the test, the challenge and challenge 1's two predicted counts.

Governing relationships

The generator. The model implements Park and Miller's minimal standard generator (MINSTD) with the multiplier 48271. Starting from s₀ = the seed, each state is

sₖ = MOD(48271 · sₖ₋₁, 2³¹ - 1), and each draw is uₖ = sₖ / (2³¹ - 1),

which lies strictly between 0 and 1. Every product stays below 2⁴⁷, so the integer sequence is exact. The stream discards the generator's first two outputs, so the first draw used is s₃, three steps from the seed; this keeps a small seed from giving a tiny first draw. Experiment r (1 to 200) always owns the 16 draws s₁₆₍ᵣ₋₁₎₊₃ to s₁₆ᵣ₊₂, whatever n and R are. Changing n adds or drops the last observations of the same experiments, and changing R adds or drops whole experiments; the stream never shifts.

Normal draws. The 16 draws of an experiment are turned into 16 standard normal values in 8 pairs by the Box-Muller transform: with ua and ub the two draws of a pair,

z₁ = √(-2 ln ua) · cos(2π ub), z₂ = √(-2 ln ua) · sin(2π ub).

The first n of the 16 are the experiment's observations, in standard units. The null-world observations are x = μ₀ + σ·z, and the effect-world observations are the same values shifted by μ₁ - μ₀.

Sample statistics. For each experiment, z̄ is the mean of its n standard values and s_z their sample standard deviation (dividing by n - 1). With δ = (μ₁ - μ₀)/σ, the effect in population SDs, and λ = δ·√n, the shift of the statistic's centre in standard errors:

  • z test: T₀ = z̄·√n in the null world and T₁ = (z̄ + δ)·√n in the effect world.
  • t test and misused z test: T₀ = z̄·√n / s_z and T₁ = (z̄ + δ)·√n / s_z. These equal the usual (x̄ - μ₀)/(s/√n) in each world. If every observation were equal, s_z would be 0 and both are taken as 0, which never rejects.

Decisions. The signed position of a statistic T is |T| for a two-sided test, T for an upper-tail test and -T for a lower-tail test. An experiment rejects at a critical value c when its signed position is at least c. The critical value for a level a is:

Test Two-sided One-sided
z test, misused z test the normal value with a/2 above it the normal value with a above it
t test the t value (n - 1 degrees of freedom) with a/2 in each tail the t value with a above it

The misused z test judges a t statistic against a normal critical value: that is the error the lab teaches.

p-values. For the z test and the misused z test, the p-value is 2Φ(-|T|) two-sided, Φ(-T) upper and Φ(T) lower, with Φ the standard normal distribution function; for the t test the same forms use the t distribution with n - 1 degrees of freedom. The misused test reports the normal p-value of a t statistic. The p-values are shown and binned but never used to decide: every decision uses the critical-value rule above.

Exact Type I rate. It is α for the z test and the t test. For the misused z test it is the chance that a t statistic with n - 1 degrees of freedom passes the normal critical value: 2·P(t > c) two-sided, P(t > c) one-sided.

Theory power. For the z test, with μ = |λ| for a two-sided test, λ for upper and -λ for lower:

power = Φ(μ - c) + Φ(-μ - c) two-sided, power = Φ(μ - c) one-sided.

For the two t-based tests the effect-world statistic follows a noncentral t distribution. Writing T₁ = Z / (W/√df), with Z normal with mean λ and SD 1, W following a chi distribution with df = n - 1 degrees of freedom and density f(w), and s = √df, the power is the average over W of

Φ(μ - c·W/s) + Φ(-μ - c·W/s) two-sided, Φ(μ - c·W/s) one-sided.

The model evaluates this integral with a fixed 32-point Gauss-Legendre rule on two windows of w chosen so that both the slowly varying chi density and the sharp step of the normal term are resolved; the result is accurate to about 1×10⁻⁹. The theory Type II rate is 1 - power.

Curves. On 101 points across the statistic axis the model returns the null density (normal for the z test, t for the t-based tests), the effect density (the normal curve shifted by λ for the z test, the noncentral t density for the t-based tests by a 24-point Gauss-Legendre rule, accurate to about 1×10⁻⁶ of the peak), the density the test assumes (the normal curve for the misused test, otherwise the null density) and whether each point lies in the rejection region. For the misused test, the area of the true (t) null curve beyond the normal critical value is the true Type I rate.

Wilson bands. Each simulated rate k/R gets a 95% Wilson score interval: with z the normal value with 2.5% above it (about 1.96), p̂ = k/R and d = 1 + z²/R, the centre is (p̂ + z²/(2R))/d and the half-width is z·√(p̂(1 - p̂)/R + z²/(4R²))/d. The low end is exactly 0 when k = 0 and the high end exactly 1 when k = R.

Calculation sequence

  1. Check the entered values in the order of the status list under Errors and warnings.
  2. Fix the values used: your values, or in a challenge the challenge's own values, except the input it leaves to you.
  3. Generate the 3,200 draws (16 for each of the 200 experiments, always all of them) and turn them into normal values by Box-Muller.
  4. For each experiment compute z̄, s_z, the two statistics, their decisions at your α, the outcome code and their histogram bins and p-value bins.
  5. Count the false alarms and detections of the first R experiments at your α and at the seven sweep levels, with the Wilson bands, the exact Type I rates and the theory power.
  6. Build the statistic axis, the curves, the two histograms, the running rates and the inspected experiment.
  7. Count the decisions that sit within rounding of a critical value, set the status and, in a challenge, score the challenge.

Outputs and interpretation

  • False alarms and simulated Type I rate: the null-world experiments that reject at your α, and that count over R, with the low and high ends of its 95% Wilson band. Exact Type I rate: what that share approaches over many experiments.
  • Detections and simulated power: the effect-world experiments that reject, and that count over R, with its band. Simulated Type II rate: the effect-world experiments that do not reject, over R. Theory power and theory Type II rate: the values the simulated ones approach. When the true mean equals the null mean both worlds are the same, so the "detections" are false alarms too.
  • Effect δ, shift λ and degrees of freedom: δ in population SDs, λ in standard errors, and n - 1.
  • Lower and upper critical values: ±c at your α; the lower one is blank for an upper-tail test and the upper one for a lower-tail test.
  • Inspected experiment: the experiment shown (your choice, or R if that is smaller), its statistic and p-value in both worlds, its null-world sample mean and its sample SD in units, and its n observations in both worlds.
  • Decisions within rounding of a critical value: the number of (experiment, world, level) decisions whose statistic lies within 1×10⁻⁹ (relative) of a critical value, at your α and the seven sweep levels.
  • Statistic axis, low and high ends: the smaller of 0 and λ less 5, and the larger plus 5, so both centres show with 5 standard errors on each side.
  • Experiment table: one row per experiment: the two statistics, their histogram bins, and the outcome code (0 neither world rejects, 1 only the null world, 2 only the effect world, 3 both). Rows past R are blank.
  • Statistic histogram: 30 equal bins across the axis, the end bins also holding every statistic beyond it, with the count of each world. With the t test at n = 2 many dots pile into the end bins: the t statistic has heavy tails.
  • p-value histogram: 20 bins of width 0.05 for each world. Under a true null the z and t tests give a roughly flat histogram; under an effect the p-values pile near 0; the misused test piles extra p-values near 0 even under the null.
  • Threshold sweep: the seven levels 0.001, 0.005, 0.01, 0.02, 0.05, 0.1 and 0.2, and your α as an eighth row: the critical value, the simulated and exact Type I rates, the simulated and theory Type II rates, both Wilson bands and the near-threshold count. Lowering α trades fewer false alarms for more misses.
  • Running rates: every 10 experiments, the share rejected so far in each world. The last row equals the rates at your α.
  • Settings used: the seed, counts, means, SD, test, tails and level of the run (a challenge's own values while a challenge is selected), the model version and the formula modules.

Reading the canvas: two stacked number lines show every experiment as a dot at its statistic, null world above and effect world below, with the critical values and the shaded rejection region; a coloured dot rejected. The curves show where the statistic falls in each world, and the threshold chart shows the error rates against α on a log scale.

Challenges

Choosing a challenge from 1 to 5 loads it. Every challenge uses the z test (known SD) and 200 experiments, fixes its own seed, means, SD, tails and level, and leaves you one input. The model scores it MET (score 100) or NOT MET (score 0), with the goal, the value scored, and a short explanation or hint.

  1. Predict the false alarms. In 200 experiments where the null is true, enter the lowest and highest false-alarm counts of a range no wider than 10 that holds the count at least 90% of the time. Open: the two predicted counts. Scored on the exact chance P(low ≤ X ≤ high) for X binomial with 200 tries and chance α = 0.05; MET when low ≤ high, high - low ≤ 10 and that chance is at least 0.9. The value scored is blank until both counts are entered (or when low is above high), and the page keeps the results and curves hidden until both are set.
  2. Hold the miss rate. With 16 observations and an effect of 0.75 SD, choose the smallest significance level that keeps the theory miss rate (Type II) at or below 30%. Open: the significance level. MET when the theory Type II rate at your α is at most 0.3 and at α - 0.001 it is above 0.3 (an α of 0.001 counts as having no smaller step).
  3. The best threshold for equal costs. Among the seven sweep thresholds, choose the one with the smallest Type I rate plus Type II rate, both counted equally. Open: the significance level. Scored on the exact Type I rate plus the theory Type II rate; MET when your α equals the sweep level with the smallest total (a tie goes to the lower level). While this challenge is not met the threshold sweep is held back.
  4. Point the test the right way. The true mean is above the null mean; choose the tails that give the test the most power at α = 0.05. Open: the tails. MET when the theory power equals the largest of the two-sided, upper-tail and lower-tail powers.
  5. Catch a false discovery. The null is true in both worlds; find an experiment that the test rejects at α = 0.01. Open: the inspected experiment. MET when the inspected experiment's null-world statistic passes the critical value. The value shown is its null-world p-value. While this challenge is not met the outcome codes and the running rates are held back, and the page draws no dot coloured and no region shaded.

On the page the threshold lines are also hidden in challenges 2 and 3, since they would show the answer; your α's own rates stay in Results. Challenges 1 to 4 are scored on exact or theory values, so the seed does not decide them; challenge 5 is about the simulated experiments themselves.

What to take from the lab: a significance level is a long-run false-alarm rate, so about one experiment in twenty rejects a true null at α = 0.05 and any one run strays from that by chance; lowering α buys fewer false alarms with more misses; power depends on the effect, n, α and the direction of the test; and using normal critical values with an estimated SD raises the true Type I rate well above α for small samples.

Validation and status logic

The status reads the first match in the list under Errors and warnings. The NOT VALID checks apply to the entered values (the effect bound is tested only when both means and the SD are in range); the two CHECK items apply to the values used, so a challenge's own settings decide them, and they apply inside challenges too. A CHECK does not stop the calculation: every result is still shown. A challenge number that is not a whole number from 0 to 5 counts as no challenge. An unordered pair of predicted counts is a challenge result (NOT MET), not a status.

Results that look surprising are not statuses: a misused z test with a Type I rate far above α, a simulated rate away from its theory value, a power near 0 in the wrong tail, and a false discovery are what the lab teaches, and the status reads OK.

Assumptions and limitations

  • The population is normal with the stated SD, and the observations are independent. Both worlds use the same draws (common random numbers), so with no effect they give identical counts by design.
  • At most 200 experiments of at most 16 observations; a simulated rate from 200 experiments is rough, which the 95% bands show.
  • The effect is at most 5 population SDs; at exactly 5 SDs the theory power is 1.
  • The generator is deterministic and reproducible from its seed. It is a small linear congruential generator; fed to Box-Muller it leaves a faint lattice pattern in the far tails, harmless at these sizes. It is fine for teaching and wrong for cryptography or serious simulation.
  • The theory power of the t-based tests is a quadrature value, accurate to about 1×10⁻⁹, not an exact closed form; the effect-density curve of those tests is a display curve accurate to about 1×10⁻⁶ of its peak.
  • A statistic within about 1×10⁻⁹ of a bin edge, or a p-value within 1×10⁻⁹ of a multiple of 0.05, may fall in the neighbouring bin; it never changes a decision.

Restrictions and non-computing states

Under NOT VALID every result and every table is blank, except the model version and formula modules, the challenge goal and the not-met text of a selected challenge, the challenge result (NOT MET when a challenge is selected, otherwise No challenge selected) and a score of 0.

While a challenge is active and not met, some results are held back rather than computed blank by error: the threshold sweep in challenge 3, and the outcome codes of the experiment table and the running rates in challenge 5. In challenge 1 the value scored is blank until both predicted counts are entered. Once a challenge is met every result shows.

When the t-based tests meet an experiment whose observations are all equal, that experiment's statistic is taken as 0 and counted as not rejected, and the inspected statistic and p-values read blank if it is the inspected one.

Errors and warnings

The status messages, first match wins:

  • NOT VALID: the seed must be a whole number from 1 to 2147483646
  • NOT VALID: the number of experiments must be a multiple of 20 from 20 to 200
  • NOT VALID: the sample size must be a whole number from 2 to 16
  • NOT VALID: the null mean and the true mean must be from -1000 to 1000, and the population SD from 0.1 to 1000
  • NOT VALID: the true mean must lie within 5 population SDs of the null mean : the size of μ₁ - μ₀ is more than 5σ.
  • NOT VALID: choose the test from the list
  • NOT VALID: choose the tails from the list
  • NOT VALID: the significance level must be from 0.001 to 0.2
  • NOT VALID: the inspected experiment must be a whole number from 1 to 200
  • NOT VALID: the challenge number must be a whole number from 0 to 5, and each predicted count a whole number from 0 to 200
  • CHECK: an experiment's observations are all equal, so its t statistic is undefined; it is counted as not rejected. Try another seed : a t-based test meets an experiment with a sample SD of 0. It cannot happen with distinct draws and is a guard.
  • CHECK: a simulated statistic sits within rounding of a critical value, so that experiment's decision could change in the last digit; try another seed : the count of decisions within rounding of a critical value is above 0.
  • OK : the run is computed with nothing to flag.

Most out-of-range values are refused by the page before the model sees them; the model's own checks above give the same verdict.

References

This laboratory is an educational model of one-sample hypothesis tests of a normal mean: the z test with a known standard deviation, the t test with the sample standard deviation, and the misuse of normal critical values with the sample standard deviation. Its random numbers come from the Park-Miller minimal standard (MINSTD) generator with the multiplier 48271, turned into normal values by the Box-Muller transform, so every run is exact and repeatable from its seed. The simulated false-alarm and detection rates get Wilson score intervals, and the theory power of the t-based tests is the noncentral t probability, evaluated by Gauss-Legendre quadrature.

Sources and further reading

  • The one-sample t test and z test of a mean, with two-sided and one-sided critical values: NIST/SEMATECH, e-Handbook of Statistical Methods, section 7.2.2, Are the data consistent with the assumed process mean? NIST
  • Type I and Type II errors, the power of a test and power for a test of a normal mean: The Pennsylvania State University, STAT 415, Introduction to Mathematical Statistics, lesson 10, Hypothesis Tests (Part II), section 10.3, Power of a Test. Penn State
  • The Wilson score interval for a proportion: NIST/SEMATECH, e-Handbook of Statistical Methods, section 7.2.4.1, Confidence intervals. NIST
  • Gauss-Legendre quadrature: NIST, Digital Library of Mathematical Functions, section 3.5(v), Gauss Quadrature. DLMF
  • The minimal standard generator: S. K. Park and K. W. Miller, "Random number generators: good ones are hard to find", Communications of the ACM 31(10), 1988, pages 1192-1201. doi:10.1145/63039.63042
  • The Box-Muller transform: G. E. P. Box and M. E. Muller, "A note on the generation of random normal deviates", The Annals of Mathematical Statistics 29(2), 1958, pages 610-611. doi:10.1214/aoms/1177706645

No table or text from any of these publications is reproduced, and none of their authors or publishers has reviewed or endorses this laboratory.

This laboratory is for learning. Its results hold only within its stated ranges; a simulated rate from at most 200 experiments is one rough run, not a property of the test, and real data need a test chosen for how they were collected.

Links were current at build time. If one moves, search the document's title rather than reporting a broken link as a defect.

Results are informational and not professional advice. See our Terms of Use. Powered by SpreadsheetWeb · About · Privacy · Terms