Docs / Concepts / The tensor memory model

The tensor memory modelv0.1.0

SRAM is the scarcest resource on a microcontroller. SynapticOS manages it with a tensor-shape-aware arena allocator — syn_mem — optimized for the allocation pattern of neural network inference: large contiguous buffers, allocated per layer, reused aggressively, never fragmented.

Why a heap fails tensor workloads

A general-purpose heap solves a problem inference does not have: arbitrary allocations with arbitrary lifetimes. Inference allocations follow a rigid, repeating pattern — model weights live for the whole session, activations live for exactly one inference, and per-layer intermediates live for one layer. Feeding that pattern into a heap buys two failure modes:

• Fragmentation. Interleaving long-lived weight buffers with short-lived activation buffers punches holes in the heap. After enough inferences, a large contiguous activation buffer can fail to allocate even though total free memory is sufficient — fatal on a device with 512 KB of SRAM and no swap.
• Non-determinism. Heap allocation time depends on the current free-list state, so the same inference takes a different amount of time on iteration 1 and iteration 10,000. A runtime that schedules jobs against microsecond deadlines cannot budget around an allocator with unpredictable latency.

The arena sidesteps both by exploiting the pattern instead of fighting it: allocation is a pointer bump, and "free" is a pointer reset.

Arena layout: persistent, ephemeral, scratch

The arena is one fixed SRAM region split into three parts, mirroring the three tensor lifetimes:

[base ... persistent → ... ephemeral → ...      | scratch pool ...]
 └─ weights, biases    └─ activations             └─ per-layer
    (grow one way)        (grow after persistent)    intermediates
                                                     (top of arena)

• Persistent (SYN_MEM_PERSISTENT) grows upward from the arena base and holds data that survives across inference calls — weights and biases.
• Ephemeral (SYN_MEM_EPHEMERAL) grows immediately after the persistent region and holds activations, freed wholesale after each inference.
• Scratch is an isolated pool at the top of the arena for per-layer intermediate buffers, acquired with syn_mem_scratch_acquire(). Keeping it separate means scratch churn can never collide with tensor allocations.

With the default Kconfig values — CONFIG_SYNAPTIC_TENSOR_ARENA_SIZE=131072 and CONFIG_SYNAPTIC_SCRATCH_POOL_SIZE=16384 — a 128 KB arena yields 112 KB of tensor space plus a 16 KB scratch pool. The arena is configurable from 4 KB to 512 KB and scratch from 1 KB to 64 KB.

16-byte DMA alignment

Every allocation is aligned to a 16-byte boundary for NPU DMA compatibility. Each syn_mem_tensor_alloc() call places the syn_tensor_t descriptor at an aligned offset and starts the data buffer at the next 16-byte boundary after it, so tensor->data is always directly usable as a DMA target or accelerator input with no realignment copies.

O(1) allocation, measured

Allocation is a bump: align the current offset up to 16 bytes, check it fits, advance the pointer. There is no free list to walk and no metadata to merge, so cost is constant regardless of tensor size. The benchmark suite in tests/unit/test_mem_bench.c confirms this on QEMU Cortex-M3: about 154 cycles per allocation, and the per-allocation cost stays flat as tensor size varies across 4-, 16-, 32-, and 64-byte tensors. The same suite also measures ephemeral reset speed, which is likewise constant time — it writes two offsets and bumps a counter, no matter how many tensors it reclaims.

The allocator additionally tracks a high-water mark (arena_peak), allocation count, and reset count, all readable at runtime.

Reset semantics

Freeing follows the lifetime classes, not individual pointers:

• syn_mem_tensor_free() and syn_mem_scratch_release() are deliberate no-ops — a bump allocator cannot reclaim mid-region holes, and it never needs to.
• syn_mem_reset_ephemeral(), called between inference jobs, reclaims the entire ephemeral region and the scratch pool in one shot by resetting their offsets to zero.
• The persistent region survives every ephemeral reset — it holds a loaded model's weights for as long as the model stays resident. Reclaiming it means re-initializing the whole arena with syn_mem_init(), a step tied to the model lifecycle rather than the per-inference loop.

The allocation API

The complete surface, from include/synaptic/syn_mem.h:

/** Memory lifetime classification */
typedef enum {
    SYN_MEM_PERSISTENT,   /**< Lives across inference calls (weights, biases) */
    SYN_MEM_EPHEMERAL,    /**< Freed after each inference (activations)       */
    SYN_MEM_SHARED,       /**< In shared IPC region (input/output tensors)    */
} syn_mem_lifetime_t;

/* Arena management */
int  syn_mem_init(void *arena_base, size_t arena_size);
void syn_mem_reset_ephemeral(void);

/* Tensor allocation */
syn_tensor_t *syn_mem_tensor_alloc(const uint32_t *shape, uint8_t ndim,
                                   syn_npu_dtype_t dtype,
                                   syn_mem_lifetime_t lifetime);
void syn_mem_tensor_free(syn_tensor_t *tensor);

/* Scratch pool */
void *syn_mem_scratch_acquire(size_t size);
void  syn_mem_scratch_release(void *ptr);

/* Statistics */
int  syn_mem_get_stats(syn_mem_stats_t *stats);
void syn_mem_print_stats(void);

Tensors are described by shape (up to 4D: batch, height, width, channels) and dtype; the allocator computes the byte size from both. syn_mem_get_stats() exposes usage, peak, and counters — the same numbers the syn mem stats shell command prints.