Docs / Guides / Running the test suite
Running the test suitev0.5.0
The v0.5.0 suite is 158 ztest cases across 21 suites, all passing, and it runs entirely on QEMU — no hardware, no debugger, no flashing. (v0.1.0 shipped with 61 tests in 10 suites; Phase 2 added the pipeline, scheduler, FFT, and processor suites; Phase 3 grew the IPC suite from a placeholder to 10 real cases; Phase 4 added 25 store/OTA/hot-swap tests; Phase 5 adds 25 more across five new suites — layer preemption, the drain gate, health, DMA ingest, and the profiler.) The tests live in two twister apps: tests/unit/ (synaptic.unit, 133 tests) and — since Phase 4 — tests/unit_store/ (synaptic.store, 25 tests), split because the store tests' 18 KB RAM-emulated flash fixture does not fit beside the main suites in QEMU's 64 KB. Since Phase 5 all tests/unit suites share one 8 KB arena (test_common.c) re-initialized per suite — the price of fitting 18 suites in 64 KB.
Run it with twister
From the west workspace root, with your Zephyr environment active — pointing twister at tests/ picks up both apps:
west twister -T synaptic-os/tests -p qemu_cortex_m3
The testcase.yaml allows two platforms — qemu_cortex_m3 and frdm_mcxn947/mcxn947/cpu0 — so the identical suites can also run on the board. The test prj.conf shrinks the arena to fit QEMU's 64 KB RAM: CONFIG_SYNAPTIC_TENSOR_ARENA_SIZE=4096, CONFIG_SYNAPTIC_SCRATCH_POOL_SIZE=1024.
The 21 suites
| Suite | File | Tests | Covers |
|---|---|---|---|
syn_mem_suite | test_mem.c | 18 | The tensor arena allocator: init, tensor allocation, regions, reset behavior, exhaustion. |
syn_process_suite | test_process.c | 15 | New in Phase 2 — the built-in pre/post-processors: resize, normalize, quantize, MFCC, softmax, argmax, top-k, NMS, dequantize. |
syn_dsp_suite | test_dsp_hal.c | 11 | DSP HAL operations — normalize, softmax, argmax, FFT, matrix multiply — including argument validation. |
syn_store_suite | test_model_store.c | 11 | New in Phase 4 (in tests/unit_store/) — the flash-backed model store: install + CRC-gated load, reboot survival, corrupt payload refused on load (-EILSEQ), a corrupted newest registry copy falling back to the older generation, ping-pong wear accounting (wear sum == generation, copies within one of each other, survives reboot), update + rollback semantics, power loss during registry commit (torn copy rejected, previous generation adopted, no phantom staged slot), power loss during payload write (old model intact and loadable), oversize install rejected (-EFBIG), and operation timing. |
syn_layer_preempt_suite | test_layer_preempt.c | 10 | New in Phase 5 — layer-granular execution: end-to-end determinism of the synthetic layered format, bad-input rejection, preempt + bit-exact resume with a < 1 ms context-save assertion, non-preemptible jobs running through, EDF ordering across four jobs, the deadline-miss counter, cancel-of-suspended with slot reuse, and the DAG format: bit-exactness against an all-buffers-live reference executor, a ≥ 30% planned-peak reduction (55% on the test DAG) with sub-millisecond planning, and DAG preempt + resume with the skip activation surviving the parked context. |
syn_ipc_suite | test_ipc.c | 10 | New in Phase 3 — the IPC layer, single-core testable by design: 20-byte message ABI pinning, shared-region layout offsets, memory-map tiling, empty/full -EAGAIN semantics, FIFO order, a 10,000-message no-loss/no-corruption sweep, index wraparound at UINT32_MAX, and a ring-cost benchmark (25 cycles per push). |
syn_model_suite | test_model_registry.c | 9 | Model registry: register, list, load/unload, capacity limits. |
syn_dsp_fft_suite | test_dsp_fft.c | 9 | New in Phase 2 — the radix-2 FFT: known spectra, length/power-of-two validation, in-place operation, and the Q15 matrix multiply. |
syn_ota_suite | test_model_ota.c | 9 | New in Phase 4 (in tests/unit_store/) — the OTA engine: a full cycle in 4 KB chunks with byte-exact flash verification, prime-size (997-byte) chunks across page boundaries, bad magic / payload CRC / size / name mismatch each rejected with ERROR-state recovery, state-machine guards, power loss mid-transfer (old model intact, update re-runnable), and a staged update surviving reboot to activate from IDLE. |
syn_pipeline_suite | test_infer_pipeline.c | 8 | New in Phase 2 — pipeline construction: pool limits, add-time ordering rules, single-model enforcement, build validation. |
syn_sched_suite | test_scheduler.c | 7 | New in Phase 2 — the job scheduler: priority dispatch, wait/timeout, cancel semantics, result retrieval, run_sync. |
syn_npu_suite | test_npu_hal.c | 6 | NPU HAL lifecycle against the stub: init, load, set-input, invoke, get-output, state. |
syn_dma_suite | test_dma_ingest.c | 5 | New in Phase 5 — the DMA HAL (stub backend) and ingest pump: one-shot async copy with zero remaining, invalid-argument rejection, circular mode running ≥ 3 iterations then stopping, an 8-frame pump run with byte-exact frames, buffer alternation, and in-place consumption asserted, and pump config validation. |
syn_dsp_verify_suite | test_dsp_verify.c | 5 | Numerical verification: softmax output sums to 1.0 with known inputs, argmax correctness, with printable output. |
syn_swap_suite | test_model_swap.c | 5 | New in Phase 4 (in tests/unit_store/) — hot-swap under a live scheduler: swap + timing, invalid arguments, a CRC-corrupt new model refused with clean residency state, swap blocking until a deliberately slow in-flight job completes with its result intact, and a job queued during the swap executing only after the new model is resident (postprocess probe). |
syn_init_suite | test_init.c | 4 | Runtime lifecycle: syn_init() / syn_shutdown(), re-init behavior. |
syn_prof_suite | test_prof.c | 4 | New in Phase 5 — the profiler through a real inference: stage marks populate syn_prof_get_last(), the disable gate suppresses capture, NULL-argument rejection, and the layer-trace entry points. Lifted syn_prof.c line coverage from 17% to 98%. |
syn_drain_gate_suite | test_drain_gate.c | 3 | New in Phase 5 — the single-server drain gate behind OTA's serving pause: idle drain with paused admission, a drain waiting out a worker thread and staying paused, and a timeout while the gate is held followed by a clean drain. |
syn_health_suite | test_health.c | 3 | New in Phase 5 — the health monitor (QEMU-testable via the fault callback): stale-episode detection, recovery, and a second episode; idle sources skipped and re-armed when busy; introspection counters. The hardware watchdog reset and CPU1 heartbeat recovery are board-verified instead — QEMU has no watchdog device. |
syn_mem_bench_suite | test_mem_bench.c | 3 | Allocation-throughput benchmarks: varying tensor sizes, cycle counts, throughput stats. |
syn_mem_regions_suite | test_mem_regions.c | 3 | Region semantics: persistent allocations survive an arena reset, scratch isolation. |
Total: 158 tests, 100% pass on qemu_cortex_m3. No placeholder suites remain: syn_ipc_suite, a single placeholder case through Phase 2, gained its real tests in Phase 3 — the SPSC ring logic is deliberately testable single-core (both roles driven from one core over the identical object code the board runs cross-core). The Phase 4 store/OTA/swap suites use the same seam discipline: a RAM-backed flash port with NOR semantics (program requires erased cells, sector/page granularity enforced) and write fault injection, so every power-loss and corruption scenario runs deterministically — the logic under test is byte-for-byte the code the board runs, only the flash backend differs. What QEMU cannot verify — MPU faults, CPU1 release, MAILBOX signaling, cross-core latency, real flash timing, true reboot persistence — is verified on the FRDM-MCXN947 instead. The scheduler suite, a placeholder in v0.1.0, gained its real tests in Phase 2.
Why this works without hardware
The QEMU build links the stub HAL backends (src/hal/stub/). The NPU stub is fully deterministic: it hashes the input bytes (a simple sum), picks sum % 10 as the winning class, and writes confidence 127 to that slot. Same input, same output, every run, on every machine — which is what makes assertion-based CI on an "NPU" possible with no silicon attached. The DSP stub provides bit-exact pure-software normalize/softmax/argmax, and as of Phase 2 the FFT and Q15 matrix multiply run as shared software kernels (src/hal/common/syn_dsp_soft.c) used by the stub and as the fallback of the PowerQuad driver — they are the reference implementation the hardware paths are validated against (on the board, syn dsp bench cross-checks PowerQuad output against them: 976 ppm of peak max error for the FFT, 1 LSB for the matmul).
syn_mem_bench_suite, syn_mem_regions_suite, and syn_dsp_verify_suite intentionally print detailed, human-readable output (throughput numbers, region maps, softmax sums) — they double as reproducible evidence for the published Phase 1 results.
Coverage
New in Phase 5: tools/syn_coverage.sh builds both test apps with CONFIG_SYNAPTIC_COVERAGE=y, runs them under QEMU, harvests the embedded gcov console dumps, and merges the runs with gcovr into coverage-report/. The instrumentation is deliberately library-only: Zephyr's tree-wide coverage overflows QEMU's 64 KB RAM by roughly 128 KB, so the build blanks the toolchain's global coverage flags and instruments only the src/core files named by CONFIG_SYNAPTIC_COVERAGE_FILES. First measurement at v0.5.0: 83.7% lines, 91.5% functions over the QEMU-buildable subset of src/core (1,882 lines). Scope stated honestly: syn_boot, syn_ipc, syn_mpu, syn_infer_remote, and syn_shell only build on the board and are exercised in the hardware sessions instead, and syn_model.c sits at 63% because its flash-backed load internals run in the store app, where it is not in the instrumented set (no RAM headroom).
Integration tests
tests/integration/ holds two on-target suites, both still placeholders:
test_npu_inference.c— end-to-end NPU inference on the FRDM-MCXN947 (requires on-target execution).test_dual_core.c— dual-core IPC and inference (requires the dual-core target; Phase 3).
They exist so the CI wiring, suite names, and target requirements are settled before the features they exercise are implemented.