Testing, further

Testing covers #[test], unit, integration and doc tests, should_panic, Result tests and #[ignore]. This page is for the next stage, when you have many tests. Everything shown with output ran in the image. A section that needs a crate or tool the image lacks says “needs network” and pins the version. Anything I named and did not run says so.

RememberGive every test its own data, its own folder and its own clock. Tests that share nothing can run in any order, or at the same time.

Project layout

timekeep/
├── src/
│   ├── cache.rs                a cache with an injected fetch function
│   ├── lib.rs                  duration parsing, averages, and its unit tests
│   ├── session.rs              a session with an injected clock
│   └── store.rs                notes saved as files
├── tests/
│   ├── common/
│   │   └── mod.rs              helpers shared by the files in tests/
│   ├── isolation.rs
│   ├── property.rs             a property test with no crate
│   └── store.rs
├── Cargo.lock
└── Cargo.toml                  no dependencies: std only

Rustsrc/lib.rsCopy

//! Small helpers for a time-tracking tool.

pub mod cache;
pub mod session;
pub mod store;

/// Why a duration like `"1h30m"` could not be read.
#[derive(Debug, PartialEq)]
pub enum ParseError {
    Empty,
    NoNumber,
    MissingUnit,
    UnknownUnit(char),
    TooBig,
}

// Private on purpose: unit tests inside this file can call it, the tests/ folder cannot.
fn unit_seconds(unit: char) -> Option<u64> {
    match unit {
        's' => Some(1),
        'm' => Some(60),
        'h' => Some(3600),
        _ => None,
    }
}

/// Reads text like `"1h30m"` or `"90s"` and returns the number of seconds.
///
/// ```
/// assert_eq!(timekeep::parse_duration("1h30m"), Ok(5400));
/// ```
///
/// A number without a unit is an error. The `?` in a doc test needs a function that returns
/// a `Result`; hidden lines (they start with `# `) supply it:
///
/// ```
/// # fn main() -> Result<(), timekeep::ParseError> {
/// let seconds = timekeep::parse_duration("2m")?;
/// assert_eq!(seconds, 120);
/// # Ok(())
/// # }
/// ```
///
/// ```
/// assert!(timekeep::parse_duration("15").is_err());
/// ```
pub fn parse_duration(text: &str) -> Result<u64, ParseError> {
    if text.is_empty() {
        return Err(ParseError::Empty);
    }
    let mut total: u64 = 0;
    let mut digits = String::new();
    for c in text.chars() {
        if c.is_ascii_digit() {
            digits.push(c);
            continue;
        }
        let per = unit_seconds(c).ok_or(ParseError::UnknownUnit(c))?;
        if digits.is_empty() {
            return Err(ParseError::NoNumber);
        }
        let n: u64 = digits.parse().map_err(|_| ParseError::TooBig)?;
        total = n
            .checked_mul(per)
            .and_then(|s| total.checked_add(s))
            .ok_or(ParseError::TooBig)?;
        digits.clear();
    }
    if digits.is_empty() {
        Ok(total)
    } else {
        Err(ParseError::MissingUnit)
    }
}

/// The average of some samples, or `None` when there are none.
///
/// This version adds up in `i32`, so a few large samples overflow. See `average`.
pub fn average_naive(samples: &[i32]) -> Option<i32> {
    if samples.is_empty() {
        return None;
    }
    Some(samples.iter().sum::<i32>() / samples.len() as i32)
}

/// The average of some samples, or `None` when there are none. Adds up in `i64`.
///
/// ```
/// assert_eq!(timekeep::average(&[i32::MAX, i32::MAX]), Some(i32::MAX));
/// assert_eq!(timekeep::average(&[]), None);
/// ```
pub fn average(samples: &[i32]) -> Option<i32> {
    if samples.is_empty() {
        return None;
    }
    let sum: i64 = samples.iter().map(|&s| i64::from(s)).sum();
    Some((sum / samples.len() as i64) as i32)
}

#[cfg(test)]
mod tests {
    use super::*;

    // Table-driven: one row per case. The loop collects every failure, so one run lists them all.
    #[test]
    fn parse_duration_table() {
        let cases: &[(&str, Result<u64, ParseError>)] = &[
            ("90s", Ok(90)),
            ("2m", Ok(120)),
            ("1h30m", Ok(5400)),
            ("1h1m1s", Ok(3661)),
            ("", Err(ParseError::Empty)),
            ("15", Err(ParseError::MissingUnit)),
            ("m", Err(ParseError::NoNumber)),
            ("5x", Err(ParseError::UnknownUnit('x'))),
            ("99999999999999999999h", Err(ParseError::TooBig)),
        ];
        let mut failures = Vec::new();
        for (input, expected) in cases {
            let got = parse_duration(input);
            if &got != expected {
                failures.push(format!("{input:?}: expected {expected:?}, got {got:?}"));
            }
        }
        assert!(
            failures.is_empty(),
            "{} case(s) failed:\n{}",
            failures.len(),
            failures.join("\n")
        );
    }

    // A unit test can call a private function: `use super::*` brings it in.
    #[test]
    fn unit_seconds_knows_three_units() {
        assert_eq!(unit_seconds('m'), Some(60));
        assert_eq!(unit_seconds('d'), None);
    }

    // Example-based tests with small numbers do not see the overflow.
    #[test]
    fn naive_average_works_for_small_numbers() {
        assert_eq!(average_naive(&[1, 2, 3]), Some(2));
        assert_eq!(average_naive(&[]), None);
    }
}

$ cargo test -- --test-threads=1

   Compiling timekeep v0.1.0 (/home/you/timekeep)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.14s
     Running unittests src/lib.rs (target/debug/deps/timekeep-d0809f4455427576)

running 6 tests
test cache::tests::fetches_each_key_once ... ok
test session::tests::expires_after_the_ttl ... ok
test session::tests::ttl_comes_from_the_lookup ... ok
test tests::naive_average_works_for_small_numbers ... ok
test tests::parse_duration_table ... ok
test tests::unit_seconds_knows_three_units ... ok

test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/isolation.rs (target/debug/deps/isolation-60e75b487f21e158)

running 2 tests
test creates_alice ... ok
test creates_bob ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/property.rs (target/debug/deps/property-02921c9eca5bf4d1)

running 2 tests
test fixed_average_holds_for_ten_thousand_cases ... ok
test naive_average_has_a_counterexample ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/store.rs (target/debug/deps/store-93a9b1bc14669be9)

running 4 tests
test lists_notes_in_sorted_order ... ok
test missing_note_is_not_found ... ok
test saves_and_loads_a_note ... ok
test temp_dir_is_removed_even_when_the_test_body_panics ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests timekeep

running 5 tests
test src/lib.rs - average (line 88) ... ok
test src/lib.rs - parse_duration (line 29) ... ok
test src/lib.rs - parse_duration (line 36) ... ok
test src/lib.rs - parse_duration (line 44) ... ok
test src/session.rs - session::Session (line 27) - compile ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s


running 1 test
test src/session.rs - session::Session (line 36) - compile fail ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

all doctests ran in 0.14s; merged doctests compilation took 0.12s

$ ls /tmp | grep -c '^timekeep-' # temp folders left behind

0

This output changes from run to run. This is one real run.

Table-driven tests

A table-driven test is one loop over a list of cases. parse_duration_table above has nine rows of (input, expected result). It collects a message for every wrong row and asserts once at the end, because an assert inside the loop stops at the first wrong row (Try it 1 breaks two). The harness prints one line per #[test] function, so the loop is one line. rstest and test-case generate one test per case; I did not run either.

Helpers, fixtures and the tests/ folder

Each file in tests/ is its own crate, so shared helpers go in tests/common/mod.rs and each file says mod common;. A tests/common.rs would be run as a test crate of its own and print running 0 tests (the Book).

Rust

tests/common/mod.rstests/store.rs

Copy

//! Helpers shared by the files in tests/. Each file in tests/ is its own crate, so a file
//! that wants these says `mod common;`. The folder form (common/mod.rs) keeps Cargo from
//! treating this file as a test crate of its own.

use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::{env, fs, process};

/// A fresh folder in the system temp dir that is deleted when the value is dropped.
pub struct TempDir {
    path: PathBuf,
}

impl TempDir {
    pub fn new(label: &str) -> TempDir {
        // The process id keeps two test runs apart, the counter keeps two tests apart.
        static NEXT: AtomicUsize = AtomicUsize::new(0);
        let n = NEXT.fetch_add(1, Ordering::SeqCst);
        let path = env::temp_dir().join(format!("timekeep-{label}-{}-{n}", process::id()));
        fs::create_dir_all(&path).expect("create the temp dir");
        TempDir { path }
    }

    pub fn path(&self) -> &Path {
        &self.path
    }
}

impl Drop for TempDir {
    fn drop(&mut self) {
        let _ = fs::remove_dir_all(&self.path);
    }
}

/// A fixture: a folder that already holds two notes.
pub fn dir_with_two_notes() -> TempDir {
    let dir = TempDir::new("two-notes");
    timekeep::store::save_note(dir.path(), "standup", "9:30").unwrap();
    timekeep::store::save_note(dir.path(), "review", "14:00").unwrap();
    dir
}

TempDir is a guard: its Drop deletes the folder (Drop). The name holds the process id and a counter, so two tests do not share a folder. The last test panics inside catch_unwind and checks the folder is gone anyway. The tempfile crate does this job; I did not run it.

A unit test sits inside the module it tests, so use super::* gives it private functions such as unit_seconds (Modules). A file in tests/ is an outside user:

tests/private.rsprivate.rsCopy

// tests/private.rs: an integration test is an outside user of the crate.
#[test]
fn unit_seconds_from_outside() {
    assert_eq!(timekeep::unit_seconds('m'), Some(60));
}

$ cargo test --test private

error[E0603]: function `unit_seconds` is private
  --> tests/private.rs:4:26
   |
 4 |     assert_eq!(timekeep::unit_seconds('m'), Some(60));
   |                          ^^^^^^^^^^^^ private function
   |
note: the function `unit_seconds` is defined here
  --> src/lib.rs:18:1
   |
18 | fn unit_seconds(unit: char) -> Option<u64> {
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Make the function pub, or test it through a public function.

Tests share a process

The tests of one file run as threads of one process, in parallel by default (the Book). Anything global is shared: statics, environment variables, a file name. Here two tests each take “the first id” from one process-wide counter:

Rusttests/ids.rsCopy

use std::sync::atomic::{AtomicUsize, Ordering};

// One counter for the whole test program: every test in this file shares it.
static NEXT_ID: AtomicUsize = AtomicUsize::new(1);

fn new_id() -> usize {
    NEXT_ID.fetch_add(1, Ordering::SeqCst)
}

#[test]
fn creates_alice() {
    assert_eq!(new_id(), 1);
}

#[test]
fn creates_bob() {
    assert_eq!(new_id(), 1);
}

$ cargo test --test ids -- --test-threads=1

running 2 tests
test creates_alice ... ok
test creates_bob ... FAILED

failures:

---- creates_bob stdout ----

thread 'creates_bob' (57) panicked at tests/ids.rs:17:5:
assertion `left == right` failed
  left: 2
 right: 1
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

failures:
    creates_bob

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

error: test failed, to rerun pass `--test ids`

$ cargo test --test ids --no-run # build once, so the loop below only runs the tests

$ for i in $(seq 100); do cargo test --test ids; done # count which test failed, 100 runs, 8 cores

      5 test creates_alice ... FAILED
     95 test creates_bob ... FAILED

This output changes from run to run. This is one real run.

With one thread, creates_bob ran second and failed. With the default thread count (8 cores here) the loser is whichever thread reaches the counter second; the counts are one batch of 100 runs and change from batch to batch. --test-threads=1 makes a failure repeatable but does not fix the cause. The fix is tests/isolation.rs: each test builds its own Ids (Threads, Shared State).

Rusttests/isolation.rsCopy

//! Each test below builds its own state, so they can run in any order or at the same time.

use std::sync::atomic::{AtomicUsize, Ordering};

/// Hands out ids starting at 1. The counter belongs to the value, not to the process.
struct Ids(AtomicUsize);

impl Ids {
    fn new() -> Ids {
        Ids(AtomicUsize::new(1))
    }
    fn next(&self) -> usize {
        self.0.fetch_add(1, Ordering::SeqCst)
    }
}

#[test]
fn creates_alice() {
    let ids = Ids::new();
    assert_eq!(ids.next(), 1);
}

#[test]
fn creates_bob() {
    let ids = Ids::new();
    assert_eq!(ids.next(), 1);
}

Other flaky tests have the same cure, a value the test owns: a fake clock, a TempDir, a seed.

The environment is shared too. In edition 2024, std::env::set_var is unsafe: the docs say that in a multi-threaded program on systems other than Windows “the only sound option is to not use set_var or remove_var at all” (docs, why), and a test program is multi-threaded by default:

tests/env.rsenv.rsCopy

// tests/env.rs: changing the environment from a test.
#[test]
fn reads_the_ttl_from_the_environment() {
    std::env::set_var("SESSION_TTL", "45");
    assert_eq!(std::env::var("SESSION_TTL").unwrap(), "45");
}

$ cargo test --test env

error[E0133]: call to unsafe function `set_var` is unsafe and requires unsafe block
 --> tests/env.rs:4:5
  |
4 |     std::env::set_var("SESSION_TTL", "45");
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ call to unsafe function
  |
  = note: consult the function's documentation for information on how to avoid undefined behavior

So pass the setting in: ttl_from in src/session.rs takes a lookup function. Its test gives it a HashMap; a program would give it |key| std::env::var(key).ok().

Fakes: a trait or a closure

To fake a dependency without a library, pass in the part that varies. Session asks a Clock for the time; the test’s clock is a Cell<u64> the test sets, so nothing sleeps. Cache takes a fetch closure that records what it was asked for.

Rust

src/session.rssrc/cache.rs

Copy

//! A session that expires, with the clock passed in so a test can control time.

use std::time::{SystemTime, UNIX_EPOCH};

/// Where "now" comes from.
pub trait Clock {
    /// Seconds since some fixed point.
    fn now(&self) -> u64;
}

/// The real clock: seconds since the Unix epoch.
pub struct SystemClock;

impl Clock for SystemClock {
    fn now(&self) -> u64 {
        SystemTime::now()
            .duration_since(UNIX_EPOCH)
            .map(|d| d.as_secs())
            .unwrap_or(0)
    }
}

/// A session that ends `ttl` seconds after it starts.
///
/// This example reads the real clock, so it is only compiled, not run:
///
/// ```no_run
/// use timekeep::session::{Session, SystemClock};
///
/// let session = Session::new(SystemClock, 3600);
/// assert!(!session.expired());
/// ```
///
/// A number is not a clock, and the compiler says so. `compile_fail` makes that the test:
///
/// ```compile_fail
/// let _ = timekeep::session::Session::new(5, 60);
/// ```
pub struct Session<C: Clock> {
    clock: C,
    started: u64,
    ttl: u64,
}

impl<C: Clock> Session<C> {
    pub fn new(clock: C, ttl: u64) -> Self {
        let started = clock.now();
        Session {
            clock,
            started,
            ttl,
        }
    }

    pub fn expired(&self) -> bool {
        self.clock.now() >= self.started + self.ttl
    }
}

/// How long a session lasts, in seconds: `SESSION_TTL` if the lookup finds it and it parses,
/// otherwise 300. The lookup is a parameter, so a test does not touch the real environment.
pub fn ttl_from(get: impl Fn(&str) -> Option<String>) -> u64 {
    get("SESSION_TTL")
        .and_then(|text| text.parse().ok())
        .unwrap_or(300)
}

#[cfg(test)]
mod tests {
    use super::*;
    use std::cell::Cell;
    use std::collections::HashMap;

    // The test's clock is a `Cell` the test can set. Implementing the trait for `&Cell<u64>`
    // lets the session borrow it while the test keeps changing the time.
    impl Clock for &Cell<u64> {
        fn now(&self) -> u64 {
            self.get()
        }
    }

    #[test]
    fn expires_after_the_ttl() {
        let now = Cell::new(1_000);
        let session = Session::new(&now, 60);
        assert!(!session.expired());
        now.set(1_059);
        assert!(!session.expired());
        now.set(1_060);
        assert!(session.expired());
    }

    #[test]
    fn ttl_comes_from_the_lookup() {
        let env: HashMap<&str, &str> = HashMap::from([("SESSION_TTL", "45")]);
        assert_eq!(ttl_from(|k| env.get(k).map(|v| v.to_string())), 45);
        assert_eq!(ttl_from(|_| None), 300);
        assert_eq!(ttl_from(|_| Some("soon".to_string())), 300);
    }
}

Use a closure for one function, a trait for several methods (Traits, Closures, Generics). The cost is code that is generic only so a test can replace a part. mockall generates fakes from traits; I did not run it.

Doc tests for a library

cargo test runs the /// examples of a library. This crate uses three things beyond a plain block (rustdoc book). Lines that start with # are compiled but hidden from the rendered docs; the second parse_duration example uses them to get a main that returns a Result, so it can use ?. no_run compiles an example without running it, here for the real clock. compile_fail passes only if the example does not compile, and the rustdoc book warns that code failing today may compile later.

terminal

$ cargo test --doc -- --test-threads=1
   Compiling timekeep v0.1.0 (/home/you/timekeep)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.02s
   Doc-tests timekeep

running 5 tests
test src/lib.rs - average (line 88) ... ok
test src/lib.rs - parse_duration (line 29) ... ok
test src/lib.rs - parse_duration (line 36) ... ok
test src/lib.rs - parse_duration (line 44) ... ok
test src/session.rs - session::Session (line 27) - compile ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s


running 1 test
test src/session.rs - session::Session (line 36) - compile fail ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

all doctests ran in 0.14s; merged doctests compilation took 0.12s

This output changes from run to run. This is one real run.

Property tests: a loop, a seed and a shrink

An example test checks the inputs you thought of. A property test states something that should hold for any input and tries many generated ones. The property: an average lies between the smallest and largest sample. average_naive adds up in an i32, so the example test with [1, 2, 3] passes but large samples overflow. Iterator::sum panics on overflow when overflow checks are on (docs); the report below shows attempt to add with overflow, so they were on in this test build. I did not run a release build.

tests/property.rs needs no crate: a seeded generator, 1,000 cases, and shrink, which keeps dropping or halving samples while the input still fails. The same seed gives the same inputs.

Rusttests/property.rsCopy

//! A property test with no crate: a seeded random generator, a loop, and shrinking by hand.

use std::panic::{self, catch_unwind};
use timekeep::{average, average_naive};

/// A tiny linear congruential generator. The same seed gives the same numbers on every machine.
struct Lcg(u64);

impl Lcg {
    fn next_u32(&mut self) -> u32 {
        self.0 = self
            .0
            .wrapping_mul(6364136223846793005)
            .wrapping_add(1442695040888963407);
        (self.0 >> 32) as u32
    }

    /// 1 to 6 samples, each any `i32`.
    fn samples(&mut self) -> Vec<i32> {
        let len = 1 + self.next_u32() % 6;
        (0..len).map(|_| self.next_u32() as i32).collect()
    }
}

type Average = fn(&[i32]) -> Option<i32>;

/// The same seed gives the same inputs, so a failure can be replayed.
const SEED: u64 = 42;

/// The property: an average lies between the smallest and the largest sample.
/// A panic counts as a failure.
fn holds(f: Average, samples: &[i32]) -> bool {
    match catch_unwind(|| f(samples)) {
        Ok(Some(avg)) => {
            avg >= *samples.iter().min().unwrap() && avg <= *samples.iter().max().unwrap()
        }
        _ => false,
    }
}

/// Run `cases` random inputs; return the first that breaks the property.
fn find_failure(f: Average, seed: u64, cases: u32) -> Option<(u32, Vec<i32>)> {
    let mut rng = Lcg(seed);
    (0..cases).find_map(|n| {
        let samples = rng.samples();
        (!holds(f, &samples)).then_some((n, samples))
    })
}

/// Shrinking by hand: keep making the input smaller while it still fails.
/// Two moves: drop one sample, or halve one sample.
fn shrink(f: Average, mut samples: Vec<i32>) -> Vec<i32> {
    loop {
        let mut candidates = Vec::new();
        for i in 0..samples.len() {
            if samples.len() > 1 {
                let mut shorter = samples.clone();
                shorter.remove(i);
                candidates.push(shorter);
            }
            if samples[i] != 0 {
                let mut halved = samples.clone();
                halved[i] /= 2;
                candidates.push(halved);
            }
        }
        match candidates.into_iter().find(|c| !holds(f, c)) {
            Some(smaller) => samples = smaller,
            None => return samples,
        }
    }
}

#[test]
fn naive_average_has_a_counterexample() {
    // The overflow panics are expected, so keep their messages out of the output. A panic hook
    // is global to the process: a test running at the same time would lose its message too.
    let default_hook = panic::take_hook();
    panic::set_hook(Box::new(|_| {}));
    let (n, found) = find_failure(average_naive, SEED, 1000).expect("a failing input");
    let small = shrink(average_naive, found.clone());
    let still_fails = !holds(average_naive, &small);
    panic::set_hook(default_hook);
    println!("seed {SEED}, case {n}: {found:?}");
    println!("shrunk to: {small:?}");
    assert!(still_fails);
    // Minimal for these moves: dropping any sample, or halving any sample, makes it pass.
    assert!(small.len() >= 2);
}

#[test]
fn fixed_average_holds_for_ten_thousand_cases() {
    assert_eq!(find_failure(average, SEED, 10_000), None);
}

$ cargo test --test property -- --nocapture --test-threads=1

running 2 tests
test fixed_average_holds_for_ten_thousand_cases ... ok
test naive_average_has_a_counterexample ... seed 42, case 0: [968358053, 1773127077, -1587428289, -1373754708]
shrunk to: [-793714144, -1373754708]
ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Seed 42 failed at case 0 and four samples shrank to two. Try it 3 uses seed 7 and ends at three. Each is a dead end for the two moves shrink has, not necessarily the smallest input.

The proptest crate (1.12.0, needs network) generates, runs and shrinks for you, over a copy of the two functions. I fixed the seed and turned off the regression file so this page repeats; by default proptest saves a failing case under proptest-regressions and replays it (failure persistence), and runs 256 cases (Config::cases).

Rust

tests/prop.rsCargo.toml

Copy

use avg::{average, average_naive};
use proptest::prelude::*;
use proptest::test_runner::{Config, RngSeed, TestError, TestRunner};

// A fixed seed and no regression file make this page's run repeatable. A real project
// would leave both out (see the page).
fn config() -> ProptestConfig {
    ProptestConfig {
        failure_persistence: None,
        rng_seed: RngSeed::Fixed(42),
        ..ProptestConfig::default()
    }
}

proptest! {
    #![proptest_config(config())]

    // 256 random vectors (the default), 1 to 7 samples each, any i32.
    #[test]
    fn average_is_between_min_and_max(samples in prop::collection::vec(any::<i32>(), 1..8)) {
        let avg = average(&samples).unwrap();
        prop_assert!(avg >= *samples.iter().min().unwrap());
        prop_assert!(avg <= *samples.iter().max().unwrap());
    }
}

// The same property for the naive version, run by hand so that the test can pass while
// it prints what proptest found.
#[test]
fn naive_average_is_caught_and_shrunk() {
    // proptest catches the overflow panics itself; the hook only hides their messages.
    let default_hook = std::panic::take_hook();
    std::panic::set_hook(Box::new(|_| {}));
    let mut runner = TestRunner::new(Config { ..config() });
    let result = runner.run(&prop::collection::vec(any::<i32>(), 1..8), |samples| {
        let avg = average_naive(&samples).unwrap();
        prop_assert!(avg >= *samples.iter().min().unwrap());
        Ok(())
    });
    std::panic::set_hook(default_hook);
    match result {
        Err(TestError::Fail(reason, minimal)) => {
            println!("proptest says: {reason}");
            println!("minimal failing input: {minimal:?}");
        }
        other => panic!("expected a failure, got {other:?}"),
    }
}

$ cargo test --test prop -- --nocapture --test-threads=1

running 2 tests
test average_is_between_min_and_max ... ok
test naive_average_is_caught_and_shrunk ... proptest says: attempt to add with overflow
minimal failing input: [468979578, 1678504070]
ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

A failing property fails the test, and the report gives the minimal failing input. Its two samples add up to 2,147,483,648, one more than i32::MAX.

$ cargo test --test prop_naive # the last 19 lines; before them come the panic messages from every shrinking try

thread 'naive_average_is_between_min_and_max' (1777) panicked at tests/prop_naive.rs:5:1:
Test failed: attempt to add with overflow.
minimal failing input: samples = [
    468979578,
    1678504070,
]
	successes: 0
	local rejects: 0
	global rejects: 0

failures:
    naive_average_is_between_min_and_max

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

error: test failed, to rerun pass `--test prop_naive`

Fuzzing gives the software pseudo-random data as input (Fuzz Book). cargo-fuzz requires the nightly compiler (setup); I did not run it.

Snapshot tests with insta

A snapshot test compares output with a file saved earlier, which helps when the output is long, like this report. With insta 1.49.0 (needs network) the first run has no snapshot, so it stores a .snap.new file and fails. INSTA_UPDATE=always accepts what the code prints now. After a change the test fails with a diff of old against new. cargo insta review is the interactive way to accept (docs); I did not run it. Read a snapshot diff like a code diff: always accepts a wrong output as readily as a right one. The same docs say insta does not write new snapshot files in CI, which they say is enabled by the CI environment variable; I did not run that.

Rust

tests/snapshot.rssrc/lib.rs

Copy

use report::report;

#[test]
fn weekly_report() {
    let text = report(&[("website", 12.5), ("api", 7.5), ("docs", 3.0)]);
    insta::assert_snapshot!(text);
}

$ cargo test --test snapshot

running 1 test
stored new snapshot /home/you/timekeep/tests/snapshots/snapshot__weekly_report.snap.new
test weekly_report ... FAILED

failures:

---- weekly_report stdout ----
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Snapshot Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Snapshot file: tests/snapshots/snapshot__weekly_report.snap
Snapshot: weekly_report
Source: tests/snapshot.rs:6
────────────────────────────────────────────────────────────────────────────────
Expression: text
────────────────────────────────────────────────────────────────────────────────
+new results
────────────┬───────────────────────────────────────────────────────────────────
          1 │+website   12.5h
          2 │+api        7.5h
          3 │+docs       3.0h
          4 │+total     23.0h
────────────┴───────────────────────────────────────────────────────────────────
To update snapshots run `cargo insta review` or set `INSTA_UPDATE=always`
Stopped on the first failure. Run `cargo insta test` to run all snapshots.

thread 'weekly_report' (1855) panicked at /opt/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/insta-1.49.0/src/runtime.rs:734:13:
snapshot assertion for 'weekly_report' failed in line 6
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

failures:
    weekly_report

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

error: test failed, to rerun pass `--test snapshot`

$ INSTA_UPDATE=always cargo test --test snapshot

running 1 test
updated snapshot /home/you/timekeep/tests/snapshots/snapshot__weekly_report.snap
test weekly_report ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

$ cat tests/snapshots/snapshot__weekly_report.snap

---
source: tests/snapshot.rs
expression: text
---
website   12.5h
api        7.5h
docs       3.0h
total     23.0h

$ cargo test --test snapshot

running 1 test
test weekly_report ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

$ cargo test --test snapshot # after changing the report's "h" to " hours"

running 1 test
stored new snapshot /home/you/timekeep/tests/snapshots/snapshot__weekly_report.snap.new
test weekly_report ... FAILED

failures:

---- weekly_report stdout ----
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Snapshot Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Snapshot file: tests/snapshots/snapshot__weekly_report.snap
Snapshot: weekly_report
Source: tests/snapshot.rs:6
────────────────────────────────────────────────────────────────────────────────
Expression: text
────────────────────────────────────────────────────────────────────────────────
-old snapshot
+new results
────────────┬───────────────────────────────────────────────────────────────────
    1       │-website   12.5h
    2       │-api        7.5h
    3       │-docs       3.0h
    4       │-total     23.0h
          1 │+website   12.5 hours
          2 │+api        7.5 hours
          3 │+docs       3.0 hours
          4 │+total     23.0 hours
────────────┴───────────────────────────────────────────────────────────────────
To update snapshots run `cargo insta review` or set `INSTA_UPDATE=always`
Stopped on the first failure. Run `cargo insta test` to run all snapshots.

thread 'weekly_report' (1982) panicked at /opt/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/insta-1.49.0/src/runtime.rs:734:13:
snapshot assertion for 'weekly_report' failed in line 6
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

failures:
    weekly_report

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

error: test failed, to rerun pass `--test snapshot`

This output changes from run to run. This is one real run.

Coverage

cargo-llvm-cov is not in the image, so this session installs the llvm-tools-preview component and then the tool at 0.9.1 (needs network). It runs the tests and reports the lines nothing ran:

terminal

$ rustup component add llvm-tools-preview   # needs network
cargo-aarch64-unknown-linux-gnu
clippy-aarch64-unknown-linux-gnu
llvm-tools-aarch64-unknown-linux-gnu
rust-std-aarch64-unknown-linux-gnu
rustc-aarch64-unknown-linux-gnu
rustfmt-aarch64-unknown-linux-gnu
$ cargo install cargo-llvm-cov --locked --version 0.9.1   # needs network
cargo-llvm-cov 0.9.1
$ cargo llvm-cov report --show-missing-lines   # after cargo llvm-cov --no-report; the table cut to its first columns
info: cargo-llvm-cov currently setting cfg(coverage); you can opt-out it by passing --no-cfg-coverage
Filename                      Regions    Missed Regions     Cover   
--------------------------------------------------------------------
cache.rs                           39                 0   100.00%   
lib.rs                            123                 8    93.50%   
session.rs                         69                 8    88.41%   
store.rs                           44                 6    86.36%   
--------------------------------------------------------------------
TOTAL                             275                22    92.00%   
Uncovered Lines:
/home/you/timekeep/src/lib.rs: 94, 122, 128-129
/home/you/timekeep/src/session.rs: 15-20
/home/you/timekeep/src/store.rs: 24

This output changes from run to run. This is one real run.

The uncovered lines: lib.rs line 94 is the None return in average, reached only by a doc test, and the tool’s README says doc-test coverage needs nightly. Lines 122 and 128-129 are the table test’s failure branch. session.rs lines 15-20 are SystemClock::now: with a fake clock no test calls the real one. store.rs line 24 is a continue for a file with no name. Coverage says which lines ran, not whether a test checked the result.

Benchmarks

#[bench] belongs to the test crate, which the Unstable Book lists as an unstable feature. On stable it does not build:

$ cargo bench

error[E0554]: `#![feature]` may not be used on the stable release channel
 --> src/lib.rs:1:1
  |
1 | #![feature(test)]
  | ^^^^^^^^^^^^^^^^^

For more information about this error, try `rustc --explain E0554`.
error: could not compile `bench-builtin` (lib test) due to 1 previous error

The criterion crate (0.8.2, needs network) runs on stable; its bench target sets harness = false. std::hint::black_box asks the compiler to assume the worst about the value; its docs say this is best-effort. The output is one run on this VM, 8 cores, and varies; this page claims nothing from it. Performance covers measuring honestly.

Rust

benches/parse.rsCargo.toml

Copy

use bench_demo::parse_seconds;
use criterion::{Criterion, criterion_group, criterion_main};
use std::hint::black_box;

fn parse(c: &mut Criterion) {
    // black_box hides the input from the optimiser, so it cannot precompute the answer.
    c.bench_function("parse 1h30m15s", |b| {
        b.iter(|| parse_seconds(black_box("1h30m15s")))
    });
}

criterion_group!(benches, parse);
criterion_main!(benches);

$ cargo bench # one run on this VM, 8 cores (colima), varies

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Gnuplot not found, using plotters backend
Benchmarking parse 1h30m15s
Benchmarking parse 1h30m15s: Warming up for 3.0000 s
Benchmarking parse 1h30m15s: Collecting 100 samples in estimated 5.0000 s (1.2B iterations)
Benchmarking parse 1h30m15s: Analyzing
parse 1h30m15s          time:   [4.0515 ns 4.0811 ns 4.1185 ns]
Found 9 outliers among 100 measurements (9.00%)
  4 (4.00%) high mild
  5 (5.00%) high severe

This output changes from run to run. This is one real run.

Try it yourself

1. Break two rows. Change two expected values in the table. One run lists both wrong rows, with the input, what you expected and what the code returned:

   $ cargo test --lib parse_duration_table # after changing two expected values in the table

   running 1 test
   test tests::parse_duration_table ... FAILED
   
   failures:
   
   ---- tests::parse_duration_table stdout ----
   
   thread 'tests::parse_duration_table' (495) panicked at src/lib.rs:125:9:
   2 case(s) failed:
   "2m": expected Ok(121), got Ok(120)
   "5x": expected Err(UnknownUnit('y')), got Err(UnknownUnit('x'))
   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
   
   failures:
       tests::parse_duration_table
   
   test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 5 filtered out; finished in 0.00s
   
   error: test failed, to rerun pass `--lib`

2. Find the boundary. In expires_after_the_ttl, change now.set(1_059) to now.set(1_060). The assert!(!session.expired()) fails, because expired uses >=: at started + ttl the session is over, and nothing slept:

   $ cargo test --lib expires_after_the_ttl # after changing now.set(1_059) to now.set(1_060)

   running 1 test
   test session::tests::expires_after_the_ttl ... FAILED
   
   failures:
   
   ---- session::tests::expires_after_the_ttl stdout ----
   
   thread 'session::tests::expires_after_the_ttl' (615) panicked at src/session.rs:88:9:
   assertion failed: !session.expired()
   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
   
   failures:
       session::tests::expires_after_the_ttl
   
   test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 5 filtered out; finished in 0.00s
   
   error: test failed, to rerun pass `--lib`

3. Change the seed. Set SEED in tests/property.rs to 7. The same bug turns up with another input:

   terminal

   $ cargo test --test property -- --nocapture --test-threads=1   # after changing the seed 42 to 7
   running 2 tests
   test fixed_average_holds_for_ten_thousand_cases ... ok
   test naive_average_has_a_counterexample ... seed 7, case 0: [-190440833, -401253790, 1171437346, 1144091090, 594514439]
   shrunk to: [585718673, 1144091090, 594514439]
   ok
   
   test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
   

The same job in your language

Language	Table-driven	Property tests, fuzzing	Fakes	Run here
Rust	a loop (above); rstest	proptest; cargo-fuzz	traits, closures; mockall	loop, proptest
Go	t.Run subtests	built-in fuzzing	testify	table test
Python	unittest subTest	Hypothesis	unittest.mock	subTest, Hypothesis
TypeScript	test() in a loop (node:test)	fast-check	node:test mocking	loop, fast-check
Java	JUnit @ParameterizedTest	jqwik	Mockito	no
Kotlin	JUnit (parameterized tests, not run)	Kotest	MockK	no
C#	xUnit [Theory]	FsCheck	NSubstitute	no
C	a loop (frameworks: Unity, CMocka)	none named here	CMocka	no
C++	GoogleTest parameterized tests	RapidCheck	GoogleMock	no

The image has no test runner or library for Java, C#, C or C++, and for Kotlin only the kotlin.test API without JUnit, so those cells only name a library. Below are the runs for Go, Python and Node; the timings vary:

Table-driven tests in Go, Python and TypeScript

Go, Python, TypeScript

go/parse_test.gopy/test_parse.pyts/parse.test.ts

Copy

package parse

import "testing"

func TestSeconds(t *testing.T) {
  cases := []struct {
    name    string
    input   string
    want    uint64
    wantErr bool
  }{
    {"seconds", "90s", 90, false},
    {"minutes", "2m", 120, false},
    {"hours and minutes", "1h30m", 5400, false},
    {"empty", "", 0, true},
    {"no unit", "15", 0, true},
    {"unknown unit", "5x", 0, true},
  }
  for _, tc := range cases {
    t.Run(tc.name, func(t *testing.T) {
      got, err := Seconds(tc.input)
      if (err != nil) != tc.wantErr || got != tc.want {
        t.Errorf("Seconds(%q) = %d, %v; want %d, error %v", tc.input, got, err, tc.want, tc.wantErr)
      }
    })
  }
}

$ go test -v

=== RUN   TestSeconds
=== RUN   TestSeconds/seconds
=== RUN   TestSeconds/minutes
=== RUN   TestSeconds/hours_and_minutes
=== RUN   TestSeconds/empty
=== RUN   TestSeconds/no_unit
=== RUN   TestSeconds/unknown_unit
--- PASS: TestSeconds (0.00s)
    --- PASS: TestSeconds/seconds (0.00s)
    --- PASS: TestSeconds/minutes (0.00s)
    --- PASS: TestSeconds/hours_and_minutes (0.00s)
    --- PASS: TestSeconds/empty (0.00s)
    --- PASS: TestSeconds/no_unit (0.00s)
    --- PASS: TestSeconds/unknown_unit (0.00s)
PASS
ok  	example.com/parse	0.001s

$ python3 -m unittest -v

test_invalid (test_parse.SecondsTable.test_invalid) ... ok
test_valid (test_parse.SecondsTable.test_valid) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

$ node --test parse.test.ts

✔ parses 90s (0.371546ms)
✔ parses 2m (0.061584ms)
✔ parses 1h30m (0.041376ms)
✔ parses 1h1m1s (0.054959ms)
✔ rejects "" (0.095501ms)
✔ rejects "15" (0.037083ms)
✔ rejects "m" (0.043042ms)
✔ rejects "5x" (0.053625ms)
ℹ tests 8
ℹ suites 0
ℹ pass 8
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 59.552081

This output changes from run to run. This is one real run.

The same bug as the Rust one, for two languages whose numbers do not overflow by themselves: Python wraps the sum to 32 bits by hand, TypeScript with | 0. The first step installs pinned Hypothesis and fast-check (needs network).

Property tests in Python and TypeScript

Python, TypeScript

py/prop.pyts/prop.ts

Copy

from hypothesis import given, settings, strategies as st


def wrap32(n):
    # Python ints never overflow, so wrap by hand to mimic a 32-bit int (Java's int, C's int32_t).
    return (n + 2**31) % 2**32 - 2**31


def average_naive(xs):
    total = 0
    for x in xs:
        total = wrap32(total + x)
    return int(total / len(xs))


# derandomize=True makes the run repeatable; database=None keeps it from writing .hypothesis/.
@settings(derandomize=True, database=None)
@given(st.lists(st.integers(-(2**31), 2**31 - 1), min_size=1, max_size=7))
def check_between_min_and_max(xs):
    assert min(xs) <= average_naive(xs) <= max(xs)


check_between_min_and_max()

$ bash setup.sh # needs network: installs hypothesis (pip) and fast-check (npm), pinned

installed

$ python3 prop.py 2>&1 | grep -A2 'Failing test case' # Python, hypothesis

Failing test case: check_between_min_and_max(
    xs=[1, 2_147_483_647],
)

$ node prop.ts 2>&1 | grep -E 'Property failed|Counterexample|Shrunk' # TypeScript, fast-check

Error: Property failed after 1 tests
Counterexample: [[-5,-2147483644]]
Shrunk 5 time(s)
  [cause]: Error: Property failed by returning false

Read more

• The Rust Book: test organization and controlling how tests run.
• The proptest book, the insta docs, the Rust Fuzz Book, cargo-llvm-cov and the Criterion.rs book.
• In this course: Testing, Threads, Modules, Performance.

Versions this page is about: proptest 1.12.0, insta 1.49.0, criterion 0.8.2 and cargo-llvm-cov 0.9.1, as pinned in the Cargo.lock files and the session commands.