TinyML: Running Neural Networks on Microcontrollers with TensorFlow Lite

Getting a neural network to run on a microcontroller with 256KB of SRAM feels wrong the first time you do it. On a server you throw memory and compute at the problem; on an STM32 or nRF52, every tensor allocation is a negotiation. I have shipped three TinyML products in the last two years — a keyword spotting remote, a vibration-based motor monitor, and a battery-powered people counter — and each one taught me that TensorFlow Lite for Microcontrollers (TFLM) is less a framework and more a set of brutal constraints you learn to design around. This article walks through the workflow I use to take a model from Python training to reliable, low-power inference on Cortex-M hardware, with the memory tricks, quantization details, and debugging techniques that only become obvious once you are staring at a hard fault.

Why TensorFlow Lite Micro Architecture Fits Inside 256KB SRAM

In my experience, the reason TFLM persists while other runtimes come and go is its refusal to do anything dynamically. There is no malloc during inference, no file system, no dynamic loading of operators unless you explicitly ask for it. Everything is resolved at compile time and executed against a single, statically allocated memory arena you provide. That design matches how embedded firmware actually works.

A standard TensorFlow Lite model is a FlatBuffer — a serialized graph with operators, tensors, and weights. On a microcontroller, you do not parse this at runtime the way you would on Android. You convert it to a C array with xxd -i and compile it directly into flash. The TFLM interpreter then walks that FlatBuffer, binds only the operators you need via an AllOpsResolver or better, a MicroMutableOpResolver, and executes the graph in place.

The Interpreter Without Dynamic Allocation

The core object is tflite::MicroInterpreter. Unlike the standard TFLite interpreter, it takes four things: the model, an operator resolver, a pointer to your arena, and an error reporter. It never calls new or malloc after AllocateTensors(). This is critical for certification and for devices that run for months without a reboot. If your arena is too small, allocation fails deterministically at startup, not randomly after 14 days in the field.

I have found that teams new to TinyML try to use AllOpsResolver because it is convenient. Do not. On a Cortex-M4 build, that pulls in every kernel and can add 80-120KB of flash. For a production build, I explicitly register only what my model uses:

#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"

// Only include 5 ops = ~18KB flash saved vs AllOpsResolver
tflite::MicroMutableOpResolver<5> resolver;
resolver.AddConv2D();
resolver.AddDepthwiseConv2D();
resolver.AddFullyConnected();
resolver.AddSoftmax();
resolver.AddQuantize();

The FlatBuffer Model as a C Array

After training and conversion, your .tflite file is converted to a source file. The workflow I use keeps the model versioned and reproducible:

# Convert trained model to C array for compilation
xxd -i model_quant.tflite > model_data.cc

// model_data.cc now contains:
// unsigned char model_quant_tflite[] = { 0x1c, 0x00, ... };
// unsigned int model_quant_tflite_len = 38472;

This array lives in flash (often with __attribute__((aligned(16))) on Cortex-M33). You pass it to GetModel() on boot. No SD card, no loading from filesystem. For OTA updates, I store two model slots in external flash and swap the pointer after CRC verification — the interpreter does not care where the bytes come from as long as they remain readable during inference.

From Float32 to Int8: Quantization Workflow That Actually Preserves Accuracy

Quantization is non-negotiable. A float32 keyword spotting model that is 420KB will never fit, and even if it did, the Cortex-M4F FPU is too slow for real-time audio. Int8 quantization reduces model size by ~4x and lets you use the CMSIS-NN accelerated kernels, which give a 3-4x speedup via SIMD SMLAD instructions. The catch is accuracy loss if you treat it as an afterthought.

For a deeper treatment of the theory, I often point colleagues to Edge AI Model Optimization: Pruning, Quantization and Knowledge Distillation, but here is the practical workflow that has worked for me on audio and accelerometer models.

Post-Training Quantization vs Quantization-Aware Training

For simple models like a 2-layer fully connected anomaly detector, full-integer post-training quantization (PTQ) with a representative dataset is enough. For anything with depthwise convolutions or residual connections — like a DS-CNN for keyword spotting — I have seen PTQ drop accuracy from 94% to 81%. Quantization-aware training (QAT) recovers most of that, typically to 92-93% in my tests, by simulating int8 behavior during the last 20-30% of training epochs.

My rule: start with PTQ. If validation accuracy drops more than 2.5% absolute, switch to QAT. Do not waste time tuning PTQ calibrations endlessly; QAT is faster in the long run.

Representative Dataset Pitfalls and Calibration

The representative dataset is where most TinyML projects silently fail. It must cover your real sensor distribution, not just the training set mean. In one vibration monitoring project, we calibrated using lab data at 25°C and saw 11% accuracy loss on factory floor data with higher amplitude. The fix was adding field recordings to the calibration set.

import tensorflow as tf

def representative_dataset():
    # Use 500-1000 real samples, not synthetic
    # Must include edge cases: quiet, loud, noisy, clipped
    for audio_sample in calibration_samples:  # shape [1, 49, 10, 1]
        yield [audio_sample.astype("float32")]

converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8
tflite_quant = converter.convert()
open("model_quant.tflite", "wb").write(tflite_quant)

Two details I check every time: first, ensure inference_input_type and output_type are both int8 — otherwise you keep float I/O and lose the performance benefit and need extra dequantization code on the MCU. Second, validate the quantized model on-device, not just in Python. The TFLM kernels can produce slightly different results than the Python simulated quantized kernels, especially for softmax and tanh.

Static Memory Planning: Sizing the Tensor Arena on Cortex-M Targets

The tensor arena is a single contiguous byte array you allocate for all intermediate tensors, scratch buffers, and operator state. Too small and AllocateTensors() returns kTfLiteError; too large and you starve your application stack, Bluetooth buffers, or sensor DMA. I've found that guessing leads to painful refactoring later, so I measure.

Calculating Arena Size with the Recording Allocator

TFLM includes a RecordingMicroAllocator that reports exactly how much arena was used. I enable it in a debug build, run one inference with worst-case input, and log the high-water mark over UART.

#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/recording_micro_allocator.h"

constexpr int kArenaSize = 60 * 1024;  // start large for measurement
uint8_t tensor_arena[kArenaSize] __attribute__((aligned(16)));

tflite::RecordingMicroAllocator* allocator =
    tflite::RecordingMicroAllocator::Create(arena_buffer, kArenaSize);
tflite::MicroInterpreter interpreter(model, resolver, allocator, error_reporter);

TfLiteStatus status = interpreter.AllocateTensors();
// After allocation, query actual usage
size_t used_bytes = allocator->GetUsedBytes();
size_t head_used = allocator->GetHeadUsedBytes();
size_t tail_used = allocator->GetTailUsedBytes();

printf("Arena used: %u bytes (head %u, tail %u)\n",
       used_bytes, head_used, tail_used);
// Then set final arena to used_bytes + 1-2KB safety margin

In my keyword spotting model, the recording allocator reported 27,840 bytes used. I shipped with 32KB. For a 96x96 grayscale person detection model on STM32H7, it was 188KB — which immediately told me I needed to use external PSRAM or switch to a smaller input resolution.

Placing the Arena in the Right Memory Bank

On Cortex-M7 and M33, memory placement matters. DTCM and ITCM are zero-wait-state, while SRAM1 may have wait states. I place the arena in DTCM when possible for a 10-15% latency reduction. On Zephyr, I use the __dtcm_bss_section attribute; on FreeRTOS-based bare metal, I modify the linker script. The Zephyr Project Documentation and FreeRTOS Documentation both have excellent sections on memory regions and MPU configuration that I reference whenever I adjust the linker script — getting the alignment wrong will hard fault on first inference.

One more tip from a field bug: if you use an RTOS, the arena must not be on the task stack. I have seen a 40KB arena overflow a 4KB default task stack and corrupt the heap. Allocate it globally or in a dedicated static section.

From Sensor DMA to Inference: Building a Low-Latency Preprocessing Pipeline

The model is only half the system. The pipeline that fills its input tensor determines both accuracy and power consumption. You are not feeding JPEGs from disk; you are streaming PDM microphone data via DMA or accelerometer samples via SPI at 100Hz while the MCU sleeps between interrupts.

This is where many TinyML demos cheat by doing preprocessing in Python and ignoring the cost on device. On device, a 30ms MFCC calculation can cost more cycles than the inference itself.

Audio Frontend: MFCC vs Log-Mel on Device

For keyword spotting, I use a 16kHz PDM mic with a DMA double buffer. Every 32ms, I have 512 new samples. I compute a 30ms window with 10ms stride, then 40 mel filterbanks and 10 MFCCs. I avoid floating-point MFCC libraries; I use a fixed-point implementation derived from CMSIS-DSP.

Crucially, the preprocessing must exactly match training. I once spent a week chasing a 9% accuracy drop that turned out to be a Hann vs Hamming window mismatch between Python's librosa and the embedded code. Now I export the preprocessing parameters (window coefficients, mel matrix, normalization mean/variance) as a header file generated from Python.

For IMU data, the pipeline is simpler but still sensitive. I sample the accelerometer at 100Hz, keep a 2-second sliding window (200 samples x 3 axes), and normalize per-axis using the same mean/variance as training. If you are combining sensors, the approach in Sensor Fusion Algorithms: Combining IMU, GPS and Magnetometer Data is relevant — I often run a complementary filter before windowing to remove gravity from raw accel data so the model sees linear acceleration only.

Double Buffering and Zero-Copy Input

I avoid copying data into the input tensor. TFLM lets you get a direct pointer:

// Get int8 input tensor pointer - write directly, no memcpy
TfLiteTensor* input = interpreter.input(0);
int8_t* input_buffer = input->data.int8;

// Preprocessing writes directly into input_buffer
// input->dims = [1, 49, 10, 1] for my KWS model
preprocess_mfcc(dma_audio_buffer, input_buffer, 
                input->params.scale, input->params.zero_point);

// Quantize on the fly: float -> int8
// value_int8 = round(float_value / scale) + zero_point
for (int i = 0; i < input->bytes; i++) {
    float normalized = (mfcc_float[i] - kMean) / kStd;
    input_buffer[i] = (int8_t)__SSAT(
        (int32_t)roundf(normalized / input->params.scale) + input->params.zero_point,
        8);
}

The DMA interrupt fills one buffer while the main loop preprocesses the other. Inference runs every 250ms, not every sample, to save power. Between inferences, I put the MCU in STOP2 mode and gate the microphone's clock.

Profiling Inference Latency and Power Draw on nRF52840, ESP32-S3 and STM32H743

Numbers on a datasheet mean little until you measure with your model and your clock config. I profile three metrics for every build: latency per inference (measured with DWT_CYCCNT), peak stack/arena usage, and average current with a Power Profiler Kit II. Below is data from a DS-CNN-S keyword spotting model (196KB int8, 49x10 input) that I measured last quarter.

The nRF52840 is the most efficient per inference in energy (6.8mA * 0.184s = 1.25 mJ) but too slow for sub-100ms wake-word response. The STM32H743 is fastest but burns power that kills a coin cell in days. For my battery-powered sensor, I chose the nRF52840 and optimized: I reduced the model to 10ms stride with 30ms window overlap, cut latency to 118ms with CMSIS-NN, and duty-cycled the mic to 12% — waking the MCU only when energy threshold exceeded -45dB.

If you need to evaluate alternative runtimes for larger ARM edge devices, ONNX Runtime for Edge: Cross-Framework Model Deployment on ARM covers where TFLM stops and a full ONNX Runtime makes more sense — typically above Cortex-A or when you need dynamic model loading.

Clock Gating and Sleep Between Inferences

The biggest power win is not a faster inference but fewer inferences. I use a two-stage pipeline: a low-power threshold detector (energy or simple high-pass) runs continuously at 2-3 uA, and only when it triggers do I power the full MFCC + inference chain. In MQTT Specification based deployments where the device also reports to the cloud, I batch inferences and sync via AWS IoT Greengrass: Edge Computing, Local Processing and Cloud Sync patterns — keeping inference local and only sending events, not raw audio, which also solves privacy concerns.

Debugging Model Failures When Serial Logs Are Your Only Lifeline

When a model works in Python but fails on device, the error reporting is minimal by design to save flash. You get a hard fault, or AllocateTensors() failed, or silent wrong outputs. I have built a small debugging routine I run on every new board.

Operator Not Supported and Kernel Fallout

TFLM supports a subset of TFLite ops, and even supported ops have constraints. For example, CONV_2D with dilation >1 was not supported in the stable branch I used last year, and SVDF (used in older KWS models) requires specific quantization params. When you see Didn't find op for builtin opcode 'TRANSPOSE_CONV' over serial, you need to either replace the op in training or add the kernel from the TFLM git main branch. I keep a fork with patches and pin to a commit hash — never track TFLM head directly in production.

Enable the debug error reporter early:

#include "tensorflow/lite/micro/micro_error_reporter.h"
#include "tensorflow/lite/micro/micro_interpreter.h"

tflite::MicroErrorReporter error_reporter;
tflite::MicroInterpreter interpreter(model, resolver, tensor_arena,
                                     kArenaSize, &error_reporter);

// This will print which op failed and why over serial
TfLiteStatus allocate_status = interpreter.AllocateTensors();
if (allocate_status != kTfLiteStatusOk) {
    TF_LITE_REPORT_ERROR(&error_reporter, "AllocateTensors failed");
    // Check serial: often "Failed to allocate memory for tensor 7"
    while(1);
}

// After Invoke, check output sanity
TfLiteStatus invoke_status = interpreter.Invoke();
TfLiteTensor* output = interpreter.output(0);
for (int i = 0; i < output->dims->data[1]; i++) {
    // Dequantize for logging
    float prob = (output->data.int8[i] - output->params.zero_point) *
                 output->params.scale;
    printf("class %d: %.3f\n", i, prob);
}

Numerical Divergence Between Host and Device

I log the first inference's input and output tensors both in Python (using the TFLite interpreter) and on device, and compare byte-for-byte. They should match exactly for int8. If they diverge by more than 1 LSB, I suspect a CMSIS-NN bug or an incorrect scale/zero-point handling in preprocessing. One concrete case: the CMSIS-NN quantized softmax had an overflow for large logits in a version from late 2023 — updating the kernel fixed a 15% accuracy drop we only caught because we compared outputs.

My final check before shipping is a 24-hour soak: run inference every second on device, log outputs, and compare distribution to Python validation set. If the on-device confusion matrix drifts, it is almost always a preprocessing mismatch — sample rate, windowing, or byte order — not the model itself.

Frequently Asked Questions

Can I run TensorFlow Lite Micro without an RTOS?

Yes, and I often do. TFLM has no dependency on FreeRTOS, Zephyr, or any OS. It needs only a C++17 compiler and a way to allocate the arena. I have deployed it on bare-metal Cortex-M0+ with a simple super-loop. An RTOS helps when you need to manage sensor DMA, BLE, and inference as separate tasks with priorities — for example, giving the audio DMA interrupt higher priority than inference — but it adds context switch overhead and memory. If you use an RTOS, allocate the arena statically and ensure the inference task has enough stack (at least 2KB plus arena not on stack).

How do I choose between 8-bit and 16-bit quantization for TinyML?

Int8 is the default for TFLM because CMSIS-NN accelerates it and most models fit. Use int16 (or int16x8 with int16 activations and int8 weights) when your model is sensitive to quantization noise — I have needed it for models with very small dynamic range, like ECG anomaly detection where amplitude differences are subtle. Int16 doubles your activation memory and is slower on M4, so I only switch after confirming int8 accuracy loss is unacceptable and cannot be recovered with QAT. Test both using the TFLite converter's 16x8 flag and measure arena size and latency before deciding.

Why does my model work in Python but return all zeros on the microcontroller?

The most common cause I have seen is a mismatch in input quantization. Your Python preprocessing may be feeding float32 in [0,1] while the quantized model expects int8 in [-128,127] with a specific scale and zero-point. If you write float values directly into an int8 tensor, they truncate to zero. Always quantize using value_int8 = round(float_value / scale) + zero_point with the scale/zero-point from input->params. The second common cause is an undersized arena where AllocateTensors() silently fails if you ignore the return code — always check the status and log with MicroErrorReporter.

What is a realistic model size limit for a 256KB SRAM device?

For a device like the nRF52840 with 256KB SRAM, I budget roughly: 32-64KB for arena, 20KB for stack/heap/BLE, leaving ~170KB for model flash plus application code. In practice, that means a 150-200KB quantized model is the comfortable maximum. You can push to 300KB if your arena is small (simple model) and you optimize flash with MicroMutableOpResolver. For larger models like MobileNetV1 0.25, you need a part with 512KB+ SRAM such as ESP32-S3 or STM32H7, or you must apply pruning and knowledge distillation first.