Statistics & decisions

Factorial Experiment Design Lab

A two-level experiment on 2, 3 or 4 factors, full factorial or either half fraction, with 1 to 4 replicates: responses simulated from true effects, an optional linear drift and seeded normal noise; every effect estimated by an exact +/-1 contrast with its alias, its true (aliased) value and its drift part; an F test from pure error or Lenth's method without replicates, with one decision rule; the design cube, the run table, main-effect, interaction and half-normal plots. Five scored challenges are included. For learning the design and analysis of factorial experiments.

Last updated

Calculator overview

Inputs and outputs

This summary comes from the calculator's published input and output contract.

Inputs

True effect ABC (required)
About this input

The true effect ABC (the mean at its high sign minus the mean at its low sign), from -1000 to 1000 units; ignored when it involves a factor beyond the number chosen. Hidden during challenges 1 to 3.

Unit units Default 0 Range -1000 to 1000
True effect AB (required)
About this input

The true effect AB (the mean at its high sign minus the mean at its low sign), from -1000 to 1000 units; ignored when it involves a factor beyond the number chosen. Hidden during challenges 1 to 3.

Unit units Default 0 Range -1000 to 1000
True effect ABCD (required)
About this input

The true effect ABCD (the mean at its high sign minus the mean at its low sign), from -1000 to 1000 units; ignored when it involves a factor beyond the number chosen. Hidden during challenges 1 to 3.

Unit units Default 0 Range -1000 to 1000
True effect AC (required)
About this input

The true effect AC (the mean at its high sign minus the mean at its low sign), from -1000 to 1000 units; ignored when it involves a factor beyond the number chosen. Hidden during challenges 1 to 3.

Unit units Default 4 Range -1000 to 1000
True effect ABD (required)
About this input

The true effect ABD (the mean at its high sign minus the mean at its low sign), from -1000 to 1000 units; ignored when it involves a factor beyond the number chosen. Hidden during challenges 1 to 3.

Unit units Default 0 Range -1000 to 1000
Replicates (required)
About this input

How many times each treatment combination is run, 1 to 4. Replicates give the noise estimate for the F test; with 1 replicate the effects are tested by Lenth's method.

Default 2 Range 1 to 4
Random seed (required)
About this input

The seed of the random stream, a whole number from 1 to 2147483646; the same seed always gives the same noise and the same randomized order.

Default 2026 Range 1 to 2147483646
Run order (required)
About this input

Standard order (replicate by replicate, in the table's order) or Randomized order (shuffled by the seed). A drift fakes an effect in standard order and becomes noise in a random order.

Default Standard order Allowed Standard order, Randomized order
True effect A (required)
About this input

The true effect A (the mean at its high sign minus the mean at its low sign), from -1000 to 1000 units; ignored when it involves a factor beyond the number chosen. Hidden during challenges 1 to 3.

Unit units Default 8 Range -1000 to 1000
Significance level (required)
About this input

The significance level of the test: Strict (1%), Standard (5%) or Lenient (10%), for the F test and Lenth's method alike.

Default Standard (5%) Allowed Strict (1%), Standard (5%), Lenient (10%)
True effect C (required)
About this input

The true effect C (the mean at its high sign minus the mean at its low sign), from -1000 to 1000 units; ignored when it involves a factor beyond the number chosen. Hidden during challenges 1 to 3.

Unit units Default -5 Range -1000 to 1000
True effect BD (required)
About this input

The true effect BD (the mean at its high sign minus the mean at its low sign), from -1000 to 1000 units; ignored when it involves a factor beyond the number chosen. Hidden during challenges 1 to 3.

Unit units Default 0 Range -1000 to 1000
True effect CD (required)
About this input

The true effect CD (the mean at its high sign minus the mean at its low sign), from -1000 to 1000 units; ignored when it involves a factor beyond the number chosen. Hidden during challenges 1 to 3.

Unit units Default 0 Range -1000 to 1000
True intercept (required)
About this input

The true intercept (the overall mean) of the response, from -1000 to 1000 units. Hidden during challenges 1 to 3 (the mystery process sets its own).

Unit units Default 50 Range -1000 to 1000
True effect D (required)
About this input

The true effect D (the mean at its high sign minus the mean at its low sign), from -1000 to 1000 units; ignored when it involves a factor beyond the number chosen. Hidden during challenges 1 to 3.

Unit units Default 0 Range -1000 to 1000
True effect AD (required)
About this input

The true effect AD (the mean at its high sign minus the mean at its low sign), from -1000 to 1000 units; ignored when it involves a factor beyond the number chosen. Hidden during challenges 1 to 3.

Unit units Default 0 Range -1000 to 1000
True effect ACD (required)
About this input

The true effect ACD (the mean at its high sign minus the mean at its low sign), from -1000 to 1000 units; ignored when it involves a factor beyond the number chosen. Hidden during challenges 1 to 3.

Unit units Default 0 Range -1000 to 1000
True effect B (required)
About this input

The true effect B (the mean at its high sign minus the mean at its low sign), from -1000 to 1000 units; ignored when it involves a factor beyond the number chosen. Hidden during challenges 1 to 3.

Unit units Default 0 Range -1000 to 1000
True effect BCD (required)
About this input

The true effect BCD (the mean at its high sign minus the mean at its low sign), from -1000 to 1000 units; ignored when it involves a factor beyond the number chosen. Hidden during challenges 1 to 3.

Unit units Default 0 Range -1000 to 1000
True effect BC (required)
About this input

The true effect BC (the mean at its high sign minus the mean at its low sign), from -1000 to 1000 units; ignored when it involves a factor beyond the number chosen. Hidden during challenges 1 to 3.

Unit units Default 0 Range -1000 to 1000
Mark: AB is active (required)
About this input

Your mark for effect AB: 1 = I think it is active, 0 = not marked. Set by tapping the bar on the canvas (in a half fraction a bar marks its lead effect).

Default 0 Range 0 to 1
Mark: A is active (required)
About this input

Your mark for effect A: 1 = I think it is active, 0 = not marked. Set by tapping the bar on the canvas (in a half fraction a bar marks its lead effect).

Default 0 Range 0 to 1
Mark: ABC is active (required)
About this input

Your mark for effect ABC: 1 = I think it is active, 0 = not marked. Set by tapping the bar on the canvas (in a half fraction a bar marks its lead effect).

Default 0 Range 0 to 1
Mark: ABD is active (required)
About this input

Your mark for effect ABD: 1 = I think it is active, 0 = not marked. Set by tapping the bar on the canvas (in a half fraction a bar marks its lead effect).

Default 0 Range 0 to 1
Mark: ABCD is active (required)
About this input

Your mark for effect ABCD: 1 = I think it is active, 0 = not marked. Set by tapping the bar on the canvas (in a half fraction a bar marks its lead effect).

Default 0 Range 0 to 1
Design (required)
About this input

Full factorial (every combination), Half fraction (the half with I = +ABC or +ABCD) or Other half fraction (I = -ABC or -ABCD); a half fraction needs 3 or 4 factors (with 2 the full design is run). Chosen on the canvas.

Default Full factorial Allowed Full factorial, Half fraction, Other half fraction
Challenge number (required)
About this input

Selects a challenge: 0 for none, 1 to 5 for a predefined challenge, which fixes the settings except the inputs it leaves to you.

Default 0 Range 0 to 5
Drift per run (required)
About this input

A slow drift added to every run in turn: run i gets (i - 1) times this amount, in units per run, from -10 to 10. 0 = no drift.

Unit units/run Default 0 Range -10 to 10
Noise standard deviation (required)
About this input

The standard deviation of the run-to-run noise, in response units: 0 (no noise, no test) or 0.01 to 100.

Unit units Default 2 Range 0 to 100
Number of factors (required)
About this input

The number of factors, 2, 3 or 4 (A, B, C, D in that order): a square, a cube or two cubes on the canvas. Chosen on the canvas.

Default 3 Range 2 to 4
Mark: BD is active (required)
About this input

Your mark for effect BD: 1 = I think it is active, 0 = not marked. Set by tapping the bar on the canvas (in a half fraction a bar marks its lead effect).

Default 0 Range 0 to 1
Mark: BCD is active (required)
About this input

Your mark for effect BCD: 1 = I think it is active, 0 = not marked. Set by tapping the bar on the canvas (in a half fraction a bar marks its lead effect).

Default 0 Range 0 to 1
Mark: C is active (required)
About this input

Your mark for effect C: 1 = I think it is active, 0 = not marked. Set by tapping the bar on the canvas (in a half fraction a bar marks its lead effect).

Default 0 Range 0 to 1
Mark: D is active (required)
About this input

Your mark for effect D: 1 = I think it is active, 0 = not marked. Set by tapping the bar on the canvas (in a half fraction a bar marks its lead effect).

Default 0 Range 0 to 1
Mark: CD is active (required)
About this input

Your mark for effect CD: 1 = I think it is active, 0 = not marked. Set by tapping the bar on the canvas (in a half fraction a bar marks its lead effect).

Default 0 Range 0 to 1
Mark: ACD is active (required)
About this input

Your mark for effect ACD: 1 = I think it is active, 0 = not marked. Set by tapping the bar on the canvas (in a half fraction a bar marks its lead effect).

Default 0 Range 0 to 1
Mark: AC is active (required)
About this input

Your mark for effect AC: 1 = I think it is active, 0 = not marked. Set by tapping the bar on the canvas (in a half fraction a bar marks its lead effect).

Default 0 Range 0 to 1
Mark: AD is active (required)
About this input

Your mark for effect AD: 1 = I think it is active, 0 = not marked. Set by tapping the bar on the canvas (in a half fraction a bar marks its lead effect).

Default 0 Range 0 to 1
Mark: BC is active (required)
About this input

Your mark for effect BC: 1 = I think it is active, 0 = not marked. Set by tapping the bar on the canvas (in a half fraction a bar marks its lead effect).

Default 0 Range 0 to 1
Mark: B is active (required)
About this input

Your mark for effect B: 1 = I think it is active, 0 = not marked. Set by tapping the bar on the canvas (in a half fraction a bar marks its lead effect).

Default 0 Range 0 to 1

Outputs

Formula module versions
About this output

The formula modules the model is built from, with their versions.

No unit declared
Model version
About this output

The model version.

No unit declared
Noise standard deviation used
About this output

The noise standard deviation used, in units. Blank when the inputs are not valid.

Unit units
Marks agreeing with the test
About this output

How many of the shown effects you marked the way the test decided (marked and active, or unmarked and not active). Blank with no noise, when the inputs are not valid, and while mystery challenge 1 or 2 is not met.

No unit declared
Your marks
About this output

How many of the shown effects you have marked as active. Blank when the inputs are not valid.

No unit declared
Simultaneous margin
About this output

Lenth's simultaneous margin of error, a stricter second line for screening many effects at once, in units. Blank with replicates, with no noise and when the inputs are not valid.

Unit units
Lenth pseudo standard error
About this output

Lenth's pseudo standard error, 1.5 times the median of the absolute estimates kept after trimming at 2.5 s0, in units. Blank with replicates, with no noise and when the inputs are not valid.

Unit units
Main effects: the mean response at the low and the high level of each factor
About this output

For each factor used: the mean response at its low and at its high level.

No unit declared
Model Status
About this output

NOT VALID with the reason when an input cannot be used; CHECK when a two-factor half fraction falls back to the full design, when Lenth's method has only three effects, when the noise estimate is 0, or when an effect lies within a billionth of the significance line; otherwise OK.

No unit declared
Mean square error
About this output

The pure-error mean square: the squared deviations of the runs from their treatment means over the error degrees of freedom. Blank without replicates, with no noise and when the inputs are not valid.

Unit square units
Random seed used
About this output

The seed used. Blank when the inputs are not valid.

No unit declared
Test used
About this output

The test used: F test from the replicates (pure error), Lenth's method (no replicates), or no test when there is no noise. Blank when the inputs are not valid.

No unit declared
Significance level used
About this output

The significance level used. Blank when the inputs are not valid.

No unit declared
Critical value
About this output

The critical t value of the test at the chosen level (with the error degrees of freedom, or Lenth's m/3). Blank with no noise and when the inputs are not valid.

No unit declared
True intercept used
About this output

The true intercept used, in units. Blank while a mystery challenge (1 to 3) is not met and when the inputs are not valid.

Unit units
Treatment combinations run
About this output

The treatment combinations the design runs (2^k, or half of them in a fraction). Blank when the inputs are not valid.

No unit declared
R squared
About this output

The share of the response variation the full treatment model explains. Blank without replicates, with no noise and when the inputs are not valid.

No unit declared
Replicates used
About this output

The replicates used. Blank when the inputs are not valid.

No unit declared
Run order used
About this output

The run order used. Blank when the inputs are not valid.

No unit declared
The runs in run order: treatment, factor levels (+1 high, -1 low), replicate, response and residual
About this output

The runs in run order (rows 1 to N): the run number, the treatment combination, the level of each factor (+1 high, -1 low; blank for a factor not used), the replicate, the response and its residual from the treatment mean (blank without replicates).

No unit declared
Runs
About this output

N, the runs in the design: the treatment combinations run times the replicates. Blank when the inputs are not valid.

No unit declared
Interactions: the mean response at each pair of levels of two factors
About this output

For each pair of factors used: the mean response at (low, low), (low, high), (high, low) and (high, high).

No unit declared
Challenge result
About this output

No challenge selected, MET or NOT MET.

No unit declared
Challenge score
About this output

The challenge score: 100 when met, else 0.

No unit declared
The design cube: one row per treatment combination, whether the design runs it, its mean response and its runs
About this output

One row per treatment combination of the chosen factors: its label, 1 when the design runs it (a filled corner) or 0 (hollow), its mean response (blank when not run) and its number of runs.

No unit declared
Resolution
About this output

Full for a full factorial, III for the half of three factors, IV for the half of four factors. Blank when the inputs are not valid.

No unit declared
Defining relation
About this output

None (full factorial), or I = ABC, I = -ABC, I = ABCD or I = -ABCD for a half fraction. Blank when the inputs are not valid.

No unit declared
Effects on the line
About this output

How many effects lie within a billionth of the significance line (or of Lenth's trimming cut); above 0 the status reads CHECK. Blank with no noise, when the inputs are not valid, and while mystery challenge 1 or 2 is not met.

No unit declared
Active effects
About this output

How many of the shown effects reach the significance line. Blank with no noise, when the inputs are not valid, and while mystery challenge 1 or 2 is not met.

No unit declared
Challenge feedback
About this output

Feedback on the selected challenge: the met text or the not-met hint (blank with none).

No unit declared
Challenge: the value scored
About this output

The value the selected challenge scores: how many of the scored effects you marked the way the truth is (1 to 3), the runs (4) or the error of the D estimate (5); blank with none and when the inputs are not valid.

No unit declared
Challenge goal
About this output

The goal of the selected challenge (blank with none).

No unit declared
Design used
About this output

The design actually run: Full factorial when a half fraction of two factors is asked for. Blank when the inputs are not valid.

No unit declared
Standard error of an effect (estimated)
About this output

The standard error of an effect estimated from the replicates, 2 SQRT(MSE / N), in units. Blank without replicates, with no noise and when the inputs are not valid.

Unit units
Error degrees of freedom
About this output

The pure-error degrees of freedom, the treatments run times (replicates - 1). Blank without replicates, with no noise and when the inputs are not valid.

No unit declared
Factors used
About this output

The number of factors used (a challenge sets its own). Blank when the inputs are not valid.

No unit declared
The half-normal plot: the absolute estimates in rising order against half-normal quantiles, with the noise reference line
About this output

The shown effects in rising order of absolute estimate: the rank, the half-normal plotting quantile, the effect, its absolute estimate and the noise reference line (the scale times the quantile). Blank with no noise.

No unit declared
Grand mean
About this output

The mean response over all runs, in units. Blank when the inputs are not valid.

Unit units
Effects estimated
About this output

The effects the design can estimate (one less than the combinations run). Blank when the inputs are not valid.

No unit declared
Drift per run used
About this output

The drift per run used, in units per run. Blank when the inputs are not valid.

Unit units/run
Standard error of an effect (true)
About this output

The design's true standard error of an effect, 2 sigma / SQRT(N), in units. Blank when the inputs are not valid.

Unit units
Significance line
About this output

The significance line: the critical value times the scale (the estimated standard error, or Lenth's pseudo standard error), in units. An effect is active when its absolute estimate reaches it. Blank with no noise and when the inputs are not valid.

Unit units
The estimated effects: alias, estimate, true value, drift part, sum of squares, test ratio, p-value, the test's verdict and your mark
About this output

One row per estimated effect: its label, what it estimates (C = AB in a fraction), the estimate (high mean minus low mean), the true value it estimates (blank while a mystery challenge is not met), the part the drift added, the sum of squares, the test ratio, the p-value (F test only), whether the test finds it active (blank while mystery challenge 1 or 2 is not met), and your mark.

No unit declared
This page is provided by LogicCommons for informational purposes only. Results are analysis outputs computed from the inputs you supply and are not professional advice. Verify all inputs and results independently.

LogicCommons is in beta. If a result, label, or reference looks wrong, tell us here; we read every message.

Methodology

Purpose and model boundary

This lab simulates a small two-level experiment and analyses it the way a designed experiment is analysed. The experiment has 2, 3 or 4 factors (A, B, C and D), each run at a low and a high level. The design is the full factorial (every combination of levels) or, with 3 or 4 factors, one of the two half fractions, and each combination is run 1 to 4 times. The responses come from a true process that you set (an overall mean and up to 15 true effects), plus an optional slow drift in run order and seeded normal noise. For each run the lab returns every effect the design can estimate, what that estimate really measures (in a half fraction, two effects at once), the part of it the drift added, a significance test with one decision rule, and the design cube, the run table, main-effect, interaction and half-normal plots. Five scored challenges use the same model.

It is a lab for learning how the design shapes what an experiment can reveal: why replicates sharpen every estimate, why a half fraction cannot tell an effect from its alias, how a drift in standard order fakes an effect, and how a random run order turns that drift into noise. It is not a tool for planning or analysing a real experiment. Each result comes from one simulated data set, so a verdict of active or not active is the outcome of one test on one draw, not the truth about a process. More than two levels, centre points and curvature, blocking, quarter or smaller fractions and more than four factors are not modelled.

The model is the calculation authority. The page sends the values you set to the model and draws only what it returns: the square, cube or two cubes with their corner means, the run player, the effect bars with the significance line and your marks, the main-effect, interaction and half-normal plots, and the residuals in run order. The page computes no response, estimate, significance line or verdict itself.

Inputs and units

The response, the true values and the effects are in response units (a generic measured quantity, written "units"). An effect is the mean response at the factor's high level minus the mean at its low level. Every input is required and is checked on its entered value, even when the chosen design does not use it.

Input Meaning and unit
Number of factors k, 2, 3 or 4 (A, B, C, D in that order): a square, a cube or two cubes. Default 3.
Design Full factorial (every combination), Half fraction (the half with I = +ABC or I = +ABCD) or Other half fraction (I = -ABC or I = -ABCD). A half fraction needs 3 or 4 factors; with 2 the full design is run. Default Full factorial.
Replicates r, how many times each combination is run, a whole number from 1 to 4. Default 2. With 1 replicate the effects are tested by Lenth's method.
Noise standard deviation σ, the standard deviation of the run-to-run noise: 0 (no noise, no test) or 0.01 to 100 units. Default 2.
Random seed The starting number of the random stream, a whole number from 1 to 2,147,483,646. The same seed always gives the same noise and the same randomized order. Default 2026.
True overall mean β₀, -1,000 to 1,000 units. Default 50.
True effects A, B, C, D, AB, AC, AD, BC, BD, CD, ABC, ABD, ACD, BCD, ABCD Each from -1,000 to 1,000 units. Defaults: A 8, C -5, AC 4, every other 0. An effect that involves a factor beyond the number chosen is checked but ignored.
Significance level Strict (1%), Standard (5%) or Lenient (10%), for the F test and Lenth's method alike. Default Standard (5%).
Drift per run -10 to 10 units per run: run i in run order gets (i - 1) times this amount added. Default 0 (no drift).
Run order Standard order (replicate by replicate, in the table's order) or Randomized order (shuffled by the seed). Default Standard order.
Marks: A is active, ... , ABCD is active Your mark for each of the 15 effects, 0 (not marked) or 1 (I think it is active). Default 0.
Challenge number 0 for free exploration, or 1 to 5 to load a challenge. Default 0.

Where each input is set:

  • On the canvas only: the number of factors (the factor chips), the design (the design chips) and the 15 marks, set by tapping an effect bar (in a half fraction a bar marks its lead effect). None of these is in the input panel.
  • In the input panel: the replicates, the noise standard deviation, the seed, the significance level, the challenge number, the drift per run and the run order, and the true overall mean and the 15 true effects in the True effects table. The table hides the effects that involve a factor beyond the number chosen, and in challenges 1 to 3 it hides the true values altogether.
  • The three learning levels show more as they go: Explore shows the factors, replicates, noise, seed and true values; Experiment adds the design, the significance level, the marks and the challenges; Advanced adds the drift and the run order, Lenth's values, R squared, the values used and the half-normal plot. A level changes only what is shown, never a result.

Governing relationships

Coding and the design. Each factor is coded -1 at its low level and +1 at its high level. The 2^k treatment combinations are listed in standard (Yates) order, (1), a, b, ab, c, ac, bc, abc, d, ad, ... , abcd, where a lower-case letter means that factor is high. The sign of effect j in combination t, signⱼ(t), is the product of the codes of the factors in that effect: the sign of AC in combination ab is (+1)(-1) = -1. The lab stores the full 16 by 15 table of signs.

  • A full factorial runs all T = 2^k combinations.
  • A half fraction of 3 factors runs the 4 combinations whose ABC sign is +1 (Half fraction: a, b, c, abc) or -1 (Other half fraction: (1), ab, ac, bc). With 4 factors the word is ABCD and the half has 8 combinations.
  • With r replicates the design has N = T × r runs, at most 64.

Aliasing. In a half fraction every column of signs equals another column times the sign of the defining word, so each estimate measures two effects at once. With I = ABC the pairs are A = BC, B = AC and C = AB; with I = ABCD they are A = BCD, B = ACD, C = ABD, D = ABC, AB = CD, AC = BD and AD = BC. With the other half the partner enters with a minus sign (C = -AB). The bar is labelled with the lower-order effect of the pair, its lead. The resolution of the design is the length of the shortest word in its defining relation: III for the half of three factors (main effects aliased with two-factor interactions), IV for the half of four factors (main effects clear of two-factor interactions, which are aliased in pairs), and Full for a full factorial.

The true process. Combination t has the true mean

μₜ = β₀ + Σⱼ signⱼ(t) · eⱼ / 2,

the sum over the effects that involve only the first k factors, with eⱼ the true effect. Run i, in run order, of combination t gives the response

yᵢ = μₜ + d · (i - 1) + σ · zᵢ,

with d the drift per run and zᵢ a standard normal draw.

The generator and the noise. The model implements Park and Miller's minimal standard generator (MINSTD) with the multiplier 48271. Starting from s₀ = the seed, each state is

sₙ = MOD(48271 · sₙ₋₁, 2³¹ - 1),

and the model computes the states s₁ to s₁₃₀. The first two states are a warm-up and are never used. Uniform draw n, for n = 1 to 64, is uₙ = sₙ₊₂ / (2³¹ - 1), which lies strictly between 0 and 1. Every product stays below 2⁴⁷, so the integer sequence is exact. The Box-Muller transform turns each pair of uniforms into two independent standard normal draws:

R = √(-2 ln u₂ₖ₋₁), z₂ₖ₋₁ = R · cos(2π u₂ₖ), z₂ₖ = R · sin(2π u₂ₖ), for k = 1 to 32.

The 64 noise values and 64 sort keys sit in fixed run slots, slot s holding replicate 1 to 4 of combination 1 to 16 (s = (replicate - 1) × 16 + combination). Slot s always uses noise zₛ and the sort key s₆₆₊ₛ, whatever the number of factors, the design and the replicates. Adding a replicate therefore never changes the noise of the runs already there, and a half fraction's runs carry exactly the noise of the same runs in the full design.

Run order. In standard order the runs go replicate by replicate, each replicate in standard order. In randomized order the position of each run is the rank of its sort key among the runs used. The keys are distinct generator states, so the ranks never tie and the order is a permutation of 1 to N.

Effect estimates. With Sₜ the sum of the responses of combination t, each effect is estimated by its ±1 contrast:

estimateⱼ = 2 · Σₜ signⱼ(t) · Sₜ / N,

which is the mean response where the effect's sign is +1 minus the mean where it is -1. This is twice the regression coefficient in the ±1 coding. The same contrast applied to the drift added to each run gives the estimate's drift part; with no drift it is 0. The true value of an estimate is what it measures without noise or drift: eⱼ in a full design, and eⱼ plus or minus its alias partner's true effect in a half fraction. The grand mean is the mean of all N responses. Each effect's sum of squares is N · estimate² / 4.

The precision of an effect. The design's true standard error of an effect is 2σ / √N, so it falls with the number of runs, whether the runs come from replicates or from more factors.

The F test from pure error (2 or more replicates). The residual of a run is its response minus its combination's mean. With SSE the sum of squared residuals and T(r - 1) error degrees of freedom:

MSE = SSE / (T(r - 1)), estimated standard error = 2 √(MSE / N),

and the test ratio of an effect is |estimate| divided by the estimated standard error. The critical value is Student's t at 1 - α/2 with the error degrees of freedom, so the test is the F test with 1 and T(r - 1) degrees of freedom (F equals t squared). The p-value is the upper tail of that F distribution at F = sum of squares / MSE; it is shown, and never used in a decision. R squared is (SST - SSE) / SST, with SST the sum of squared deviations of all responses from the grand mean: the share of the variation the full treatment model explains.

Lenth's method (1 replicate). With m the number of effects estimated and the absolute estimates |c|:

s₀ = 1.5 × median of |c|, PSE = 1.5 × median of the |c| below 2.5 s₀,

where PSE is the pseudo standard error (an even number of kept values takes the mean of the middle two). The critical value is Student's t at 1 - α/2 with m/3 degrees of freedom (1, 7/3 or 5), so the margin of error is ME = t × PSE. The simultaneous margin SME uses Student's t at γ = (1 + (1 - α)^(1/m)) / 2 with the same degrees of freedom; it is a stricter second line for screening many effects at once and never decides a verdict. The test ratio of an effect is |estimate| / PSE, and there is no p-value. The critical values for both tests are stored constants computed at these exact degrees of freedom.

One decision rule. The significance line is the critical value times the scale (the estimated standard error, or Lenth's PSE). An effect is active when its absolute estimate is at or above the line. An effect within 1 part in a billion of the line, or in Lenth's method of the trimming cut 2.5 s₀, is counted as on the line.

Half-normal plot. The m absolute estimates are sorted in rising order, and the i-th is plotted against the half-normal quantile Φ⁻¹(0.5 + 0.5 (i - 0.5) / m), with Φ the standard normal distribution function. The reference line is the scale times the quantile: effects that are only noise scatter about it, and real effects stand off it.

Main-effect and interaction plots. For each factor, the mean response over the runs at its low level and at its high level; for each pair of factors, the mean response at each of the four pairs of levels. Every level holds N/2 runs and every pair of levels N/4, in every design.

Calculation sequence

The model works through the relationships above in this order:

  1. It checks the entered values in the order of the list under Errors and warnings.
  2. It fixes the values used: your inputs, or in a challenge the challenge's own values for every input the challenge does not leave to you. When the inputs are not valid it computes on the shipped default, and every result is blank.
  3. It fixes the design: the combinations run, the effects that can be estimated, their aliases, the defining relation and the resolution.
  4. It computes each combination's true mean.
  5. It generates the seeded stream, the normal noise and the sort keys, places the runs in standard or randomized order, and computes every response with its drift.
  6. It sums the responses by combination and computes the grand mean, every estimate, its true value, its drift part and its sum of squares.
  7. With noise, it tests the effects: the F test when there are replicates, Lenth's method when there are none. It draws the significance line, sets each verdict and builds the half-normal plot.
  8. It computes the main-effect and interaction plots, compares your marks with the verdicts, sets the status and, in a challenge, scores the challenge.

Outputs and interpretation

  • Active effects: how many of the shown effects reach the significance line.
  • Significance line: the critical value times the scale, in units. A bar at or beyond it is active.
  • Standard error of an effect (true): 2σ / √N, in units, the design's precision.
  • Standard error of an effect (estimated): 2 √(MSE / N), in units, from the replicates.
  • Mean square error and error degrees of freedom: the pure-error variance, in square units, and T(r - 1).
  • Test used and critical value: F test from the replicates (pure error), Lenth's method (no replicates), or No test when there is no noise; and the critical t value at the chosen level.
  • Lenth pseudo standard error and simultaneous margin: in units, for an unreplicated design.
  • R squared: the share of the variation the full treatment model explains, with replicates.
  • Grand mean: the mean response over all runs, in units. In a half fraction it also carries half of the defining word's true effect, with that word's sign.
  • Effects estimated, resolution and defining relation: one less than the combinations run; Full, III or IV; None (full factorial) or I = ABC, I = -ABC, I = ABCD or I = -ABCD.
  • Effects on the line: how many effects sit within a billionth of the line or of Lenth's trimming cut.
  • Your marks and marks agreeing with the test: how many effects you marked, and how many of the shown effects you marked the way the test decided.
  • Settings used: the runs N, the combinations run, and the factors, design, replicates, noise, seed, true overall mean, significance level, drift and run order actually used (in a challenge, the challenge's values). The design used reads Full factorial when a half fraction of two factors was asked for.
  • Estimated effects table: for each effect, what it estimates (C = AB in a half fraction), the estimate, the true value it measures, the drift part, the sum of squares, the test ratio, the p-value (F test only), whether the test finds it active, and your mark.
  • Run table: the runs in run order, with the combination, each factor's level (+1 or -1), the replicate, the response and the residual from the combination's mean (blank without replicates).
  • Design cube, main effects, interactions and half-normal plot: each combination with whether the design runs it, its mean response and its count of runs; and the plot values described above.
  • Model version and formula module versions.

Reading the canvas: the square, cube or two cubes (D low and D high) show the combinations, filled when the design runs them, with the mean response at each corner, and the player steps through the runs in run order. The effect bars show each estimate against the significance line; a bar you have tapped is marked. In a half fraction a bar carries two names, because the design cannot separate them. The main-effect lines are steeper for a larger effect, and interaction lines that are not parallel show an interaction. On the half-normal plot, effects far above the reference line are the candidates for real effects.

What to take from the lab: an estimate is a difference of two means, so its precision depends only on the number of runs and the noise. Not significant does not mean no effect, and a test of many effects at 5% flags some noise as active by chance. A half fraction halves the cost and pays for it with aliases, and the design's resolution says which effects stay apart. Randomizing the run order does not remove a drift; it spreads it as noise across every effect instead of piling it onto one.

Challenges

Choosing a challenge number from 1 to 5 loads that challenge. The challenge fixes its own process and settings and leaves you only the inputs it names; your marks are always yours. The model scores each one MET (score 100) or NOT MET (score 0), with the goal, the value scored, and a short explanation once met or a hint while not met. An effect counts as truly active when the challenge's true value for it is not 0.

  1. Mystery: three factors. A three-factor process hides its true effects. Run the full design, add replicates if you like, and mark exactly the main effects (A, B, C) that are really active. Open: the replicates, the seed and the significance level. Scored on how many of A, B and C you marked the way the truth is; MET at 3.
  2. Mystery: an 8-run budget. A four-factor process, but you may use at most 8 runs. Choose a design that fits, then mark exactly the main effects (A, B, C, D) that are really active. Open: the design, the seed and the significance level; one replicate. Scored on how many of A, B, C and D you marked the way the truth is; MET at 4 with no more than 8 runs.
  3. The aliasing trap. The half fraction shows a large bar labelled C = AB. Find out which of C and AB is really active, mark it, and leave the other unmarked. Open: the design, the seed and the significance level. Scored on how many of C and AB you marked the way the truth is; MET at 2.
  4. Precision by design. Make the effect standard error no more than half the noise standard deviation, using the fewest runs. Open: the factors, the design, the replicates, the seed and the significance level. Scored on the runs; MET at exactly 16 runs, the fewest for which 2σ / √N is at most σ / 2.
  5. Drift. A slow drift adds 0.5 to every run in turn. Choose a run order (and seed) so the estimate of D lands within 1 of its true value, 3. Open: the run order, the seed and the significance level. Scored on the error of the D estimate, |estimate of D - 3|; MET at 1 or less.

Challenges 1 to 4 are scored on whole-number counts and do not depend on the noise: the seed cannot decide them, and the marks are judged against the hidden truth, not against the test. Challenge 5 is scored on the estimate for the chosen seed. In standard order D changes only once, halfway through the runs, so the drift adds exactly 4 to the D estimate for every seed and the challenge cannot be met; in randomized order most seeds meet it, but not every one.

Challenges 1 to 3 hold back the truth until met. While one of them is NOT MET, the true value column of the effects table, the true overall mean used and the value scored are blank, and the page hides the true-effect fields. In challenges 1 and 2 the test's verdict is held back too: the Active column, the active-effect count, the effects on the line and the marks agreeing with the test are blank. The estimates, test ratios, p-values, the significance line and the half-normal plot stay visible as the evidence. Everything shows once the challenge is met.

Validation and status logic

The status reads the first match in the list under Errors and warnings. The NOT VALID checks apply to the entered values, inside a challenge too, and include the true effects of factors the design does not use. The CHECK items apply to the values used, so a challenge's own settings decide them. A CHECK does not stop the calculation: every result is still shown. A challenge number that is not a whole number from 0 to 5 counts as no challenge selected.

Assumptions and limitations

  • Two levels a factor and the full model of a two-level design, linear in the ±1 coding with every interaction. No centre points, curvature or blocking.
  • 2 to 4 factors; the full factorial and the two half fractions only, no quarter fractions; 1 to 4 replicates, at most 64 runs.
  • The noise is normal, independent and of one standard deviation. The drift is linear in run position.
  • The generator is deterministic and reproducible. It is a small linear congruential generator: its draws fall on a lattice, so the normal noise never exceeds about 6.56 standard deviations, and seeds that differ by a small factor can give related streams. Seed 2,147,483,646 mirrors seed 1 (every draw is 1 - u). It is fine for teaching and wrong for serious simulation.
  • The F test's significance level is the false-alarm rate for each effect: a screen of 15 effects at 5% with nothing active expects about 0.75 false alarms.
  • Lenth's method is conservative at these sizes (about 3% false alarms at a nominal 5% for 15 effects, about 2.3% for 7) and can miss a real effect in a small unreplicated design. With only three effects it has too little to go on.
  • Randomization removes the drift's bias on average, not in every run: one random order can still leave a bias of about one standard error of the drift part.
  • An effect within 1 part in a billion of the significance line, or of Lenth's trimming cut, could go either way; the status says so.
  • The true values of the challenges are part of the downloadable model, so the hidden truth is a learning aid, not a secret.

Restrictions and non-computing states

Every input is required and is checked on its entered value, even when the design does not use it.

Under NOT VALID every result and every table row is blank, except the model and module versions, the challenge result (NOT MET when a challenge is selected, otherwise No challenge selected), a score of 0, and the goal and the hint of a selected challenge.

With no noise there is no test: the test used reads "No test: there is no noise, so every estimate is exact for these data", and the test columns, the significance line, the counts, Lenth's values, the error values, R squared and the half-normal plot are blank. Every estimate then equals its true value plus its drift part. Without replicates the pure-error values are blank; with replicates Lenth's values are blank.

In challenges 1 to 3, while not met, the truth (and in challenges 1 and 2 the verdict) is held back as described under Challenges.

Errors and warnings

The status messages, first match wins:

  • NOT VALID: the number of factors must be 2, 3 or 4
  • NOT VALID: choose a design from the list
  • NOT VALID: the number of replicates must be a whole number from 1 to 4
  • NOT VALID: the noise standard deviation must be 0 or a number from 0.01 to 100 : a nonzero value below 0.01 is refused too.
  • NOT VALID: the seed must be a whole number from 1 to 2147483646
  • NOT VALID: the true intercept must be a number from -1000 to 1000 : the true overall mean.
  • NOT VALID: the true effect A must be a number from -1000 to 1000 : names the first effect out of range, in the order A, B, C, D, AB, AC, AD, BC, BD, CD, ABC, ABD, ACD, BCD, ABCD, whether or not the design uses it.
  • NOT VALID: choose a significance level from the list
  • NOT VALID: the drift per run must be a number from -10 to 10, and the run order one of the listed choices
  • NOT VALID: the prediction for A must be 0 or 1 : names the first mark that is not 0 or 1, in the same order.
  • NOT VALID: the challenge number must be a whole number from 0 to 5
  • CHECK: a half fraction needs 3 or 4 factors, so the full 2 x 2 design is run instead : a half fraction was chosen with two factors.
  • CHECK: with no replicates and only three effects, Lenth's method has too little to go on and its line is very wide; add replicates or a factor : the half fraction of three factors, or two factors, without replicates.
  • CHECK: the noise estimate is 0, so no effect can be tested : the scale is not positive; with a noise of at least 0.01 this does not occur in practice.
  • CHECK: an effect lies on the significance line (within 1 part in a billion), so its verdict could go either way : at least one effect is on the line or on Lenth's trimming cut.
  • OK : the run is computed with nothing to flag.

References

This laboratory is an educational model of two-level factorial experiments. It codes each factor -1 and +1, runs the full factorial or a half fraction defined by I = ±ABC or I = ±ABCD, and estimates every effect as the mean response at the high sign minus the mean at the low sign, with the aliases and the resolution of a half fraction. With replicates it tests each effect against a standard error of 2σ/√N estimated from pure error, an F test; without replicates it uses Lenth's pseudo standard error, and it draws a half-normal plot of the effects. The noise comes from the Park-Miller minimal standard (MINSTD) generator with the multiplier 48271, turned into normal draws by the Box-Muller transform, so every run is exact and repeatable from its seed.

Sources and further reading

  • Two-level full factorial designs, 2^k runs and the -1 and +1 coding: NIST/SEMATECH, e-Handbook of Statistical Methods, section 5.3.3.3, Full factorial designs. NIST
  • Confounding in a half fraction, the defining relation I = 123 and an estimate that measures the sum of two aliased effects: NIST/SEMATECH, e-Handbook of Statistical Methods, section 5.3.3.4.3, Confounding (also called aliasing). NIST
  • Design resolution as the length of the shortest word in the defining relation, and what resolutions III and IV mean for main effects and two-factor interactions: NIST/SEMATECH, e-Handbook of Statistical Methods, section 5.3.3.4.4, Fractional factorial design specifications and design resolution. NIST
  • The standard deviation of an effect, 2σ/√n, and a t cutoff with σ estimated from replication: NIST/SEMATECH, e-Handbook of Statistical Methods, section 5.5.9.7.1, Statistical significance. NIST
  • The pseudo standard error, the margin of error and the simultaneous margin of error for unreplicated factorials: R. V. Lenth, "Quick and easy analysis of unreplicated factorials", Technometrics 31(4), 1989, pages 469-473. doi:10.1080/00401706.1989.10488595
  • The random numbers: S. K. Park and K. W. Miller, "Random number generators: good ones are hard to find", Communications of the ACM 31(10), 1988, pages 1192-1201 (doi:10.1145/63039.63042), with the multiplier 48271 from the authors' reply in the journal's "Technical correspondence", Communications of the ACM 36(7), 1993 (doi:10.1145/159544.376068); and the normal draws: G. E. P. Box and M. E. Muller, "A note on the generation of random normal deviates", The Annals of Mathematical Statistics 29(2), 1958, pages 610-611 (doi:10.1214/aoms/1177706645).

No table or text from any of these publications is reproduced, and none of their authors or publishers has reviewed or endorses this laboratory.

This laboratory is for learning. Its results hold only within its stated ranges; each verdict comes from one simulated data set, so it shows how a design behaves, not which factors matter in a real process.

Links were current at build time. If one moves, search the document's title rather than reporting a broken link as a defect.

Results are informational and not professional advice. See our Terms of Use. Powered by SpreadsheetWeb · About · Privacy · Terms