Docs / Concepts / Inference pipelines & scheduler

Inference pipelines & schedulerv0.2.0

In SynapticOS the fundamental scheduling unit is the inference job, not the thread. The syn_infer subsystem defines pipelines as first-class objects and a scheduler that understands priorities, deadlines, and — its core differentiator — preemption at NPU layer boundaries. As of Phase 2 (v0.2.0), the pipeline engine and the priority scheduler are implemented in src/core/syn_infer.c; this page describes how they actually behave, and is explicit about what is and is not implemented yet.

Pipelines as first-class objects

A pipeline is a named, opaque object built from stages: zero or more pre-processing functions, a model, and zero or more post-processing functions. You construct it once, build it, and then submit jobs against it repeatedly:

/* Pipeline construction */
syn_pipeline_t *syn_pipeline_create(const char *name);
int  syn_pipeline_add_preprocess(syn_pipeline_t *pipe,
                                 syn_preprocess_fn_t fn, void *config);
int  syn_pipeline_add_model(syn_pipeline_t *pipe,
                            syn_model_handle_t model);
int  syn_pipeline_add_postprocess(syn_pipeline_t *pipe,
                                  syn_postprocess_fn_t fn, void *config);
int  syn_pipeline_build(syn_pipeline_t *pipe);   /* Validates the graph */
void syn_pipeline_destroy(syn_pipeline_t *pipe);

Making the pipeline an object rather than a call sequence is what lets the runtime see the whole preprocess → model → postprocess graph up front at syn_pipeline_build() time and plan tensor placement for it. The header also declares a library of built-in stages — syn_preprocess_image_resize, image_normalize, quantize_int8, and audio_mfcc on the way in; syn_postprocess_softmax, argmax, top_k, nms, and dequantize on the way out — all implemented in Phase 2, configured through the structures in syn_process.h. A pipeline holds up to CONFIG_SYNAPTIC_MAX_PIPELINE_STAGES stages (default 8, range 4–16).

The Phase 2 engine enforces these semantics:

• Static pool. Pipelines come from a fixed pool of 4 — syn_pipeline_create() claims a slot, syn_pipeline_destroy() releases it. No heap involved.
• Canonical ordering, enforced at add-time. A pipeline is pre-processors, then exactly one model, then post-processors. Adding a pre-processor after the model stage, or a second model, fails immediately rather than at build time.
• Validated build. syn_pipeline_build() checks the chain (a model stage is required) and computes a worst-case memory estimate for the pipeline's intermediate tensors.
• Stage buffer convention. Each stage's output is an ephemeral-arena tensor. Built-in stages get exact output-buffer capacities computed from their configs; a custom stage gets a worst-case fallback of MAX(4 × input_size, 64) bytes (the 4× covers uint8 → float32 expansion), and the stage feeding the model is always sized to hold a full model input. Stage functions receive the capacity in out->size and must set the final geometry (shape, ndim, dtype, size) before returning.
• Result lifetime. Job results live in the ephemeral arena: they stay valid until syn_mem_reset_ephemeral() is called. Consume or copy results, then reset — that is what keeps per-frame memory flat (see the face_detection sample).

Job priorities and deadlines

Every submitted job carries scheduling parameters:

typedef enum {
    SYN_PRIORITY_BEST_EFFORT = 0,   /* No deadline, runs when idle     */
    SYN_PRIORITY_NORMAL      = 1,   /* Standard priority               */
    SYN_PRIORITY_REALTIME    = 2,   /* Hard deadline, preempts others  */
} syn_priority_t;

typedef struct {
    syn_priority_t  priority;
    uint32_t        deadline_us;    /* 0 = no deadline                 */
    bool            preemptible;    /* Can be paused between layers    */
    syn_infer_cb_t  callback;       /* Completion callback             */
    void           *user_data;
} syn_infer_params_t;

/* Job submission */
syn_job_id_t syn_infer_submit(syn_pipeline_t *pipe,
                              const syn_tensor_t *input,
                              const syn_infer_params_t *params);
int  syn_infer_wait(syn_job_id_t job, uint32_t timeout_ms);
int  syn_infer_cancel(syn_job_id_t job);
int  syn_infer_get_result(syn_job_id_t job, syn_tensor_t *output);

Submission is asynchronous — it returns a job id (SYN_JOB_INVALID on failure) that you can wait on, cancel, or collect results from; the completion callback covers fire-and-forget use. The Phase 2 scheduler is a dedicated thread over a fixed job table of CONFIG_SYNAPTIC_MAX_CONCURRENT_JOBS entries (default 2, range 1–4, adjustable at runtime with syn_infer_set_max_concurrent()). It dispatches strictly by priority — REALTIME before NORMAL before BEST_EFFORT, FIFO within the same class — and signals completion through a per-job semaphore. The concrete semantics:

• syn_infer_wait() returns 0 on completion, the job's error code on failure, and -EAGAIN if the timeout elapses first.
• syn_infer_cancel() succeeds on a queued job, returns -EBUSY for a job already running (a dispatched pipeline runs to completion), and -EALREADY for a finished one.
• syn_infer_get_result() hands back the output tensor and frees the job slot; the data lives in the ephemeral arena until syn_mem_reset_ephemeral().
• syn_infer_run_sync() is the blocking one-shot: it takes a model handle directly (no pipeline construction), submits at the given priority, and copies the result into a caller-provided buffer. The hello_inference sample and the syn infer run shell command use it.
• deadline_us and preemptible are recorded with the job but not acted on yet — deadline-aware dispatch is future work (see below).

Layer-boundary preemption

This is the core differentiator of SynapticOS. A general RTOS scheduler preempts threads at arbitrary instructions; an NPU cannot be preempted that way — once a layer is dispatched, it runs to completion. But the Neutron NPU processes one layer at a time, and between layers, control returns to the CPU. The SynapticOS scheduler intercepts exactly that return point, so it operates at layer granularity. Between NPU layer invocations it can:

1. Preempt a low-priority job to run a high-priority one.
2. Time-slice between equal-priority jobs at layer boundaries.
3. Deadline-check — if a realtime job risks missing its deadline, it takes over immediately.

Job A (normal):    [Layer 0][Layer 1]...........[Layer 2][Layer 3][Done]
Job B (realtime):                    [Layer 0][Layer 1][Layer 2][Done]
                   ▲                 ▲                          ▲
                   A starts          B preempts A               A resumes

A keyword-spotting model can therefore interrupt a long-running vision model mid-inference and respond within one layer's latency, instead of waiting for the whole vision inference to finish. Jobs opt in via the preemptible flag; the scheduler also understands model latency profiles, which is what makes the deadline check possible. Because the scheduler owns the decision points, it composes directly with the NPU state machine and the ephemeral arena reset that runs between jobs.

Honest status: what Phase 2 shipped

What exists in v0.2.0, and what does not:

• Implemented (Phase 2, v0.2.0): the pipeline engine (static pool, add-time ordering checks, validated build with a worst-case memory estimate), the priority job scheduler (dedicated thread, fixed job table, REALTIME > NORMAL > BEST_EFFORT with FIFO within a class, per-job completion semaphores, cancel semantics), syn_infer_run_sync(), all nine built-in pre/post-processors, and profiling marks wired into the execution path. All of it runs against the frozen v0.1.0 syn_infer.h contract — code written against the Phase 1 header needed no changes.
• Tested on QEMU and verified on the board: the 99-test suite (including syn_pipeline_suite and syn_sched_suite) passes at 100% on qemu_cortex_m3 against the stub backends. On QEMU (icount timing, stub NPU), syn_infer_run_sync() measures 1361 µs wall versus 781 µs for the Phase 1 direct-HAL path. On the FRDM-MCXN947 (2026-07-12, stub NPU) the same call measures 1130 µs wall versus the 1038 µs Phase 1 direct-HAL capture — the whole scheduler path (build the transient pipeline, submit, dispatch, wait, copy out, destroy) costs about 92 µs on hardware, and inside the profile the dispatch overhead is ~1 µs (total 1069 µs vs 1068 µs NPU).
• Not implemented yet: layer-boundary preemption and deadline-aware dispatch (the parameters are recorded, not enforced), TFLite model execution, and the real Neutron NPU backend — the model stage still runs on the deterministic stub / software path.