How much accuracy is lost when quantizing a neural network from float32 to int8?

Post-training quantization from float32 to int8 typically results in less than 1-2% accuracy degradation for well-designed models. The exact impact depends on model architecture, dataset characteristics, and quantization strategy. Quantization-aware training (QAT), where quantization effects are simulated during training, often recovers most of this loss and can achieve accuracy within 0.5% of the float32 baseline. Models with batch normalization and ReLU activations quantize more cleanly than those using sigmoid or tanh. Calibration dataset quality directly affects the per-tensor or per-channel scale factors that determine quantized accuracy.

How do you update TinyML models on deployed devices in the field?

Over-the-air (OTA) model updates for TinyML devices require a dual-bank flash architecture or external flash storage to hold the new model while the current one remains operational. The update process involves: downloading the new quantized model binary (typically 50-500 KB) over BLE, Wi-Fi, or LoRaWAN in chunks with CRC verification, writing it to the secondary flash bank, validating the model header and tensor arena requirements against available SRAM, running a brief self-test inference with known inputs, and atomically swapping the model pointer. Bootloader-level rollback protection ensures the device reverts to the previous model if the new one fails validation or produces anomalous outputs during a burn-in period.

Edge AI and TinyML Deployment on Microcontrollers

Q: What is TinyML and how does it differ from standard edge AI?

TinyML refers to machine learning inference on microcontrollers with less than 1 MB of RAM and operating at milliwatt-level power budgets. Unlike standard edge AI running on application processors (Cortex-A series, GPUs) with Linux and full ML frameworks, TinyML targets bare-metal or RTOS environments on Cortex-M class MCUs. Models must be quantized to 8-bit integers, operations limited to those supported by the inference engine (typically TensorFlow Lite Micro or CMSIS-NN), and total model size including weights must fit in flash memory (256 KB to 2 MB typical). The key advantage is always-on inference at microwatt power levels, enabling battery-powered devices to run ML models for years without recharging.

Q: Which microcontrollers are best suited for TinyML inference?

The best microcontrollers for TinyML balance compute capability, memory, power efficiency, and DSP/ML accelerator availability. Top choices include: STM32H747 (Cortex-M7 at 480 MHz, 1 MB RAM, with DSP instructions), Nordic nRF5340 (dual Cortex-M33, 512 KB RAM, BLE 5.3 for wireless ML sensor nodes), Espressif ESP32-S3 (dual-core Xtensa with vector instructions, 512 KB SRAM, integrated Wi-Fi/BLE), and the Arm Ethos-U55 NPU paired with Cortex-M55 for dedicated ML acceleration delivering up to 128 GOPS/W. For the most constrained applications, the RP2040 (dual Cortex-M0+, 264 KB SRAM) can run small models at remarkably low cost.

Q: Can convolutional neural networks run on a Cortex-M4 microcontroller?

Yes, small CNNs can run on Cortex-M4 MCUs, though with significant constraints. A typical Cortex-M4 at 168 MHz with 256 KB SRAM can execute a MobileNetV2 variant with 128x128 input and alpha=0.25 in approximately 200-400ms per inference. The CMSIS-NN library provides optimized kernels that exploit the M4's single-cycle MAC DSP instructions, achieving 2-4x speedup over naive implementations. The key limitations are memory bandwidth (no cache on most M4 parts) and SRAM for activation buffers. Layer-by-layer execution with memory planning (as TFLite Micro's arena allocator provides) keeps peak SRAM usage within bounds.

By Budi Santoso· Oct 1, 2026· 14 min read

Running machine learning inference directly on microcontrollers eliminates cloud dependency, reduces latency to microseconds, preserves data privacy, and enables always-on intelligent sensing at milliwatt power budgets. TinyML has matured from academic curiosity to production reality, with frameworks like TensorFlow Lite for Microcontrollers (TFLM) and the CMSIS-NN kernel library making deployment practical on sub-dollar MCUs. This guide covers the complete pipeline from model design through quantization to bare-metal deployment.

The TinyML Hardware Landscape

Selecting the right microcontroller for TinyML requires balancing four constraints: flash memory (for model weights), SRAM (for activation tensors during inference), compute throughput (determines inference latency), and power consumption (determines battery life). The landscape has evolved dramatically, with dedicated neural processing units (NPUs) now appearing on MCU-class silicon.

MCU Comparison for ML Workloads

MCU	Core	Clock	Flash	SRAM	ML Accel	Power (Active)
STM32H747	Cortex-M7+M4	480 MHz	2 MB	1 MB	DSP	280 mW
nRF5340	Cortex-M33	128 MHz	1 MB	512 KB	DSP	58 mW
ESP32-S3	Xtensa LX7 x2	240 MHz	16 MB (ext)	512 KB	Vector	240 mW
MAX78002	Cortex-M4+RISC-V	100 MHz	512 KB	512 KB	CNN Accel	1.2 mW (CNN)
RP2040	Cortex-M0+ x2	133 MHz	16 MB (ext)	264 KB	None	93 mW
Cortex-M55+U55	M55+Ethos-U55	400 MHz	4 MB	2 MB	NPU 128 MAC	150 mW

The Maxim MAX78002 deserves special attention: its dedicated CNN accelerator achieves 1.2 mW during inference by processing convolutional layers in specialized SRAM-based compute arrays, delivering 50x better energy efficiency than software execution on a Cortex-M4 at equivalent throughput.

Model Quantization Pipeline

Quantization converts floating-point model weights and activations to lower-precision integers (typically int8), reducing model size by 4x and enabling efficient execution on integer-only MCU cores. The quantization pipeline is the critical bridge between training (float32 on GPU) and deployment (int8 on MCU).

Post-Training Quantization with Calibration

Post-training quantization (PTQ) requires a representative calibration dataset to determine the dynamic range of each tensor. The calibration process computes per-tensor (or per-channel for weights) scale and zero-point values that map the float32 range to int8:

# quantize_model.py - Full integer quantization pipeline
import tensorflow as tf
import numpy as np

def representative_dataset_gen():
    """Yield calibration samples matching training distribution."""
    # Load 200-500 representative samples
    cal_data = np.load('calibration_data.npy')
    for i in range(min(300, len(cal_data))):
        sample = cal_data[i:i+1].astype(np.float32)
        yield [sample]

# Load trained Keras model
model = tf.keras.models.load_model('anomaly_detector.h5')

# Configure full integer quantization
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset_gen
converter.target_spec.supported_ops = [
    tf.lite.OpsSet.TFLITE_BUILTINS_INT8
]
converter.inference_input_output_type = tf.int8

# Convert and save
tflite_model = converter.convert()

with open('anomaly_detector_int8.tflite', 'wb') as f:
    f.write(tflite_model)

print(f"Model size: {len(tflite_model)} bytes")
print(f"  = {len(tflite_model)/1024:.1f} KB")

Converting to C Array for Firmware Embedding

The quantized TFLite model must be embedded in firmware as a C byte array stored in flash. The xxd utility or a custom script generates the header file:

/* model_data.h - Generated from anomaly_detector_int8.tflite */
#ifndef MODEL_DATA_H
#define MODEL_DATA_H

#include <stdint.h>

/* Model: anomaly_detector_int8 (87,432 bytes) */
alignas(16) const uint8_t g_model_data[] = {
    0x20, 0x00, 0x00, 0x00, 0x54, 0x46, 0x4C, 0x33,
    /* ... 87,424 more bytes ... */
};

const unsigned int g_model_data_len = sizeof(g_model_data);

#endif /* MODEL_DATA_H */

TensorFlow Lite Micro Inference Engine

TFLM provides the runtime interpreter for executing quantized models on microcontrollers. Unlike its mobile counterpart, TFLM uses no dynamic memory allocation after initialization, making it deterministic and suitable for real-time embedded systems. The interpreter operates within a pre-allocated tensor arena that holds all intermediate activation buffers.

Complete Inference Pipeline on STM32

The following implementation demonstrates a complete TinyML inference pipeline for vibration-based anomaly detection on an STM32H743, including sensor data preprocessing, model invocation, and result interpretation:

/* tinyml_inference.c - TFLite Micro anomaly detector on STM32H7 */
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "tensorflow/lite/micro/system_setup.h"
#include "tensorflow/lite/schema/schema_generated.h"
#include "model_data.h"

/* Tensor arena - must be large enough for all intermediate tensors.
 * Use the TFLM arena size estimation tool or binary search. */
#define TENSOR_ARENA_SIZE  (48 * 1024)
static uint8_t tensor_arena[TENSOR_ARENA_SIZE]
    __attribute__((aligned(16)));

/* Feature extraction constants */
#define NUM_FEATURES    64    /* FFT bins as model input */
#define ANOMALY_THRESH  0.75f /* Detection threshold */

typedef struct {
    tflite::MicroInterpreter *interpreter;
    TfLiteTensor *input;
    TfLiteTensor *output;
    bool initialized;
    float input_scale;
    int32_t input_zero_point;
    float output_scale;
    int32_t output_zero_point;
} tinyml_ctx_t;

static tinyml_ctx_t ctx;

/* Register only the operators this model uses */
static tflite::MicroMutableOpResolver<8> resolver;

int tinyml_init(void) {
    tflite::InitializeTarget();

    const tflite::Model *model = tflite::GetModel(g_model_data);
    if (model->version() != TFLITE_SCHEMA_VERSION) {
        return -1;  /* Schema version mismatch */
    }

    /* Register only needed ops to minimize code size */
    resolver.AddFullyConnected();
    resolver.AddConv2D();
    resolver.AddDepthwiseConv2D();
    resolver.AddReshape();
    resolver.AddSoftmax();
    resolver.AddMaxPool2D();
    resolver.AddRelu();
    resolver.AddQuantize();

    static tflite::MicroInterpreter interp(
        model, resolver, tensor_arena,
        TENSOR_ARENA_SIZE);

    if (interp.AllocateTensors() != kTfLiteOk) {
        return -2;  /* Arena too small */
    }

    ctx.interpreter = &interp;
    ctx.input = interp.input(0);
    ctx.output = interp.output(0);

    /* Cache quantization parameters for efficient conversion */
    ctx.input_scale = ctx.input->params.scale;
    ctx.input_zero_point = ctx.input->params.zero_point;
    ctx.output_scale = ctx.output->params.scale;
    ctx.output_zero_point = ctx.output->params.zero_point;

    ctx.initialized = true;

    /* Report actual arena usage */
    size_t used = interp.arena_used_bytes();
    /* Log: "Arena: used %zu / %d bytes", used, TENSOR_ARENA_SIZE */

    return 0;
}

/* Quantize float input to int8 using model's quantization params */
static inline int8_t quantize_input(float value) {
    int32_t q = (int32_t)roundf(value / ctx.input_scale)
                + ctx.input_zero_point;
    if (q < -128) q = -128;
    if (q >  127) q = 127;
    return (int8_t)q;
}

/* Dequantize int8 output back to float */
static inline float dequantize_output(int8_t value) {
    return (value - ctx.output_zero_point) * ctx.output_scale;
}

/* Run anomaly detection inference on FFT features */
float tinyml_detect_anomaly(const float *fft_features,
                             int num_features) {
    if (!ctx.initialized || num_features != NUM_FEATURES)
        return -1.0f;

    /* Quantize input features to int8 */
    int8_t *input_data = ctx.input->data.int8;
    for (int i = 0; i < num_features; i++) {
        input_data[i] = quantize_input(fft_features[i]);
    }

    /* Run inference - deterministic, no allocation */
    uint32_t start_cycles = DWT->CYCCNT;

    if (ctx.interpreter->Invoke() != kTfLiteOk) {
        return -1.0f;
    }

    uint32_t elapsed = DWT->CYCCNT - start_cycles;
    /* At 480 MHz: elapsed / 480000 = inference time in ms */

    /* Dequantize output - anomaly score [0.0, 1.0] */
    float anomaly_score = dequantize_output(ctx.output->data.int8[0]);

    return anomaly_score;
}

/* Main inference loop */
void tinyml_task(void *arg) {
    float fft_features[NUM_FEATURES];

    while (1) {
        /* Wait for new vibration data block from DAQ */
        if (xSemaphoreTake(vib_data_ready, portMAX_DELAY)) {
            /* Extract spectral features from vibration data */
            extract_fft_features(vib_buffer, VIB_BLOCK_SIZE,
                                 fft_features, NUM_FEATURES);

            float score = tinyml_detect_anomaly(fft_features,
                                                 NUM_FEATURES);

            if (score > ANOMALY_THRESH) {
                /* Trigger alarm: bearing defect detected */
                alarm_raise(ALARM_VIBRATION_ANOMALY, score);
            }
        }
    }
}

CMSIS-NN Optimized Kernels

The ARM CMSIS-NN library provides hand-optimized neural network kernels that exploit Cortex-M DSP instructions (SIMD, saturating arithmetic) for 2-5x speedup over generic C implementations. TFLM automatically uses CMSIS-NN kernels when available, but understanding the optimization strategy helps in model architecture decisions.

Performance Impact of CMSIS-NN

Operation	Generic C (cycles)	CMSIS-NN (cycles)	Speedup	Key Optimization
Conv2D 3x3, 32ch	1,240,000	310,000	4.0x	im2col + SIMD MAC
Depthwise Conv2D	820,000	205,000	4.0x	Specialized kernel
Fully Connected	520,000	130,000	4.0x	SMLAD (dual 16-bit MAC)
Max Pool 2x2	95,000	48,000	2.0x	SIMD comparison
ReLU	32,000	8,000	4.0x	SIMD saturation
Softmax	45,000	22,000	2.0x	LUT + interpolation

The SMLAD instruction (Signed Multiply Accumulate Dual) is the workhorse of CMSIS-NN fully connected layers. It performs two 16-bit multiplications and accumulates both results in a single cycle, effectively doubling throughput for int8 matrix operations when packed into int16 pairs.

Model Architecture Design for MCU Constraints

Not all neural network architectures deploy efficiently on microcontrollers. The constraints of limited SRAM (for activation buffers), flash (for weights), and compute (no hardware multiply-accumulate beyond DSP) heavily influence architecture choices.

Design Principles for MCU-Friendly Models

Depthwise separable convolutions: Replace standard Conv2D with Depthwise + Pointwise convolution, reducing parameters by 8-9x with minimal accuracy loss (MobileNet pattern)
Small input resolution: Use 96x96 or 64x64 inputs instead of 224x224. Quadratic reduction in compute and activation memory
Narrow channel counts: Start with 8-16 filters in the first layer, scaling to 64-128 maximum. Each doubling quadruples activation memory
Global average pooling: Replace flatten + dense with global average pooling before the final classifier. Eliminates the largest fully connected layer
ReLU6 activations: Quantize more cleanly than ReLU (bounded output range) and are free on MCUs with saturation instructions

Memory Planning and Arena Sizing

The tensor arena must be large enough to hold the two largest consecutive layers' activations simultaneously (the current layer's output and the next layer's input). For a model with layer output sizes of 16 KB, 32 KB, 48 KB, 24 KB, 8 KB, the minimum arena size is determined by the peak pair (32 + 48 = 80 KB), not the sum of all layers.

/* memory_planner.h - Estimate tensor arena requirements */

/*
 * For a sequential model, minimum arena ≈ max(out[i] + out[i+1])
 * where out[i] = output_height * output_width * channels * sizeof(int8_t)
 *
 * Example: 3-layer CNN on 64x64 input
 *
 * Layer 1: Conv2D(8, 3x3, stride=2) -> 32x32x8 = 8,192 bytes
 * Layer 2: DWConv(3x3) + PWConv(16) -> 32x32x16 = 16,384 bytes
 * Layer 3: DWConv(3x3, stride=2) + PWConv(32) -> 16x16x32 = 8,192 bytes
 *
 * Peak pair: Layer1 + Layer2 = 24,576 bytes
 * Add scratch buffers (~10%): 27,033 bytes
 * Add model overhead (~2 KB): 29,033 bytes
 * Round up with margin: 32,768 bytes (32 KB arena)
 */
#define TENSOR_ARENA_SIZE  (32 * 1024)

Real-World Deployment Patterns

Pattern 1: Keyword Spotting on Battery-Powered Devices

Always-on keyword detection requires ultra-low-power audio preprocessing with ML inference triggered only when audio energy exceeds a threshold. The audio frontend extracts 40-dimensional Mel-Frequency Cepstral Coefficients (MFCCs) over 30ms windows with 20ms stride, feeding a 1D CNN or DS-CNN classifier:

/* keyword_spotter.c - Ultra-low-power keyword detection */
#include "arm_math.h"
#include "mfcc.h"

#define MFCC_COEFFS     40
#define NUM_FRAMES      49    /* ~1 second of audio */
#define AUDIO_SAMPLE_RATE 16000
#define FRAME_LEN       480   /* 30ms window */
#define FRAME_STRIDE    320   /* 20ms stride */
#define ENERGY_THRESHOLD 500.0f

static float mfcc_buffer[NUM_FRAMES][MFCC_COEFFS];
static int frame_idx = 0;

/* Voice Activity Detector - gate ML inference */
static bool vad_check(const int16_t *audio, int len) {
    float energy = 0.0f;
    for (int i = 0; i < len; i++) {
        float s = (float)audio[i];
        energy += s * s;
    }
    energy /= len;
    return (energy > ENERGY_THRESHOLD);
}

void audio_frame_callback(const int16_t *frame, int len) {
    if (!vad_check(frame, len)) {
        frame_idx = 0;  /* Reset on silence */
        return;
    }

    /* Extract MFCCs for this frame */
    compute_mfcc(frame, len, mfcc_buffer[frame_idx], MFCC_COEFFS);
    frame_idx++;

    if (frame_idx >= NUM_FRAMES) {
        /* Full 1-second window captured - run inference */
        float *flat = (float *)mfcc_buffer;
        int8_t result = tinyml_classify_keyword(flat,
            NUM_FRAMES * MFCC_COEFFS);

        if (result > 0) {
            /* Keyword detected with class = result */
            wake_application(result);
        }
        frame_idx = 0;
    }
}

Pattern 2: Anomaly Detection with Autoencoder

For industrial vibration monitoring, a compact autoencoder trained on normal operating data detects anomalies through reconstruction error. The encoder-decoder architecture compresses 64 FFT bins to an 8-dimensional latent space and reconstructs the input. High reconstruction error indicates deviation from learned normal patterns. This approach complements the Zephyr RTOS task scheduling for real-time sensor processing.

Pattern 3: Predictive Maintenance with Time-Series Classification

Classifying equipment health states (normal, degrading, critical) from time-series sensor windows uses a 1D CNN architecture that processes sequences of sensor readings. The model runs inference every 10 seconds on the latest 5-second window of accelerometer data, requiring minimal memory while providing continuous health assessment.

Over-the-Air Model Updates

Deploying a TinyML model is not a one-time event. Models must be updated as operating conditions change, new failure modes are discovered, or retraining on field data improves accuracy. The OTA update mechanism must be robust against interrupted transfers, power loss during flash writes, and incompatible model versions.

Dual-Bank Flash Update Strategy

/* ota_model_update.c - Safe OTA model swap with rollback */
#include "flash_driver.h"
#include "model_data.h"

#define MODEL_BANK_A  0x08100000  /* Primary model location */
#define MODEL_BANK_B  0x08180000  /* Secondary / update target */
#define MODEL_MAX_SIZE (512 * 1024)

typedef struct {
    uint32_t magic;        /* 0x4D4C4D44 = "MLMD" */
    uint32_t version;
    uint32_t size;
    uint32_t crc32;
    uint32_t arena_required;
    uint32_t input_size;
    uint32_t output_size;
    uint8_t  model_data[];
} model_header_t;

int ota_validate_model(const model_header_t *hdr) {
    if (hdr->magic != 0x4D4C4D44) return -1;
    if (hdr->size > MODEL_MAX_SIZE) return -2;
    if (hdr->arena_required > TENSOR_ARENA_SIZE) return -3;

    uint32_t computed_crc = crc32_compute(hdr->model_data, hdr->size);
    if (computed_crc != hdr->crc32) return -4;

    /* Verify TFLite schema version */
    const tflite::Model *model = tflite::GetModel(hdr->model_data);
    if (!model || model->version() != TFLITE_SCHEMA_VERSION)
        return -5;

    return 0;  /* Model valid */
}

int ota_apply_model(void) {
    const model_header_t *new_model =
        (const model_header_t *)MODEL_BANK_B;

    if (ota_validate_model(new_model) != 0)
        return -1;

    /* Run self-test inference with known input/output pair */
    if (selftest_inference(new_model->model_data) != 0) {
        /* Model produces incorrect output - reject */
        return -2;
    }

    /* Atomic swap: update model pointer in persistent config */
    config_set_active_model_bank(MODEL_BANK_B);

    /* Reinitialize interpreter with new model */
    return tinyml_init_from(new_model->model_data,
                            new_model->size);
}

Power Optimization for Always-On Inference

The power budget determines whether a TinyML device runs for days or years on a single battery. Effective power management combines hardware duty cycling, inference scheduling, and wake-on-event architectures. For wireless sensor networks feeding TinyML nodes, Zigbee 3.0 mesh networks provide energy-efficient local communication, while Wi-Fi HaLow offers longer range for outdoor deployments with comparable power efficiency.

Power Profile Breakdown

Deep sleep (STOP2 mode): 2-5 uA with RTC and SRAM retention
Sensor acquisition: 1-5 mA for 10-50 ms (accelerometer + ADC)
Feature extraction (FFT): 30-60 mA for 2-5 ms (DSP-accelerated)
ML inference: 30-80 mA for 5-50 ms (model-dependent)
Radio transmission: 10-40 mA for 50-500 ms (if anomaly detected)

With a 15-second duty cycle and inference-only-on-event strategy, average current consumption drops to 15-30 uA, yielding 2-4 years of operation on a 1000 mAh coin cell battery. The WebAssembly runtime approach offers an alternative deployment model for edge devices with more compute headroom, enabling portable model containers that update without firmware reflashing.

Frequently Asked Questions

What is TinyML and how does it differ from standard edge AI?

TinyML targets microcontrollers with less than 1 MB of RAM at milliwatt power budgets, running on bare-metal or RTOS environments on Cortex-M class MCUs. Unlike edge AI on application processors with Linux, TinyML models must be quantized to int8, limited to supported operations, and fit entirely in flash memory. The key advantage is always-on inference at microwatt power levels enabling years of battery operation.

How much accuracy is lost when quantizing from float32 to int8?

Post-training quantization typically costs less than 1-2% accuracy. Quantization-aware training recovers most of this, achieving within 0.5% of the float32 baseline. Models with batch normalization and ReLU quantize more cleanly than those using sigmoid or tanh. Calibration dataset quality directly affects the scale factors that determine quantized accuracy.

Which microcontrollers are best suited for TinyML inference?

Top choices include the STM32H747 (Cortex-M7, 1 MB RAM, DSP), Nordic nRF5340 (Cortex-M33, BLE 5.3), ESP32-S3 (vector instructions, Wi-Fi/BLE), and MAX78002 (dedicated CNN accelerator at 1.2 mW). For maximum ML efficiency, the Arm Ethos-U55 NPU paired with Cortex-M55 delivers up to 128 GOPS/W. The RP2040 handles small models at remarkably low cost.

Can convolutional neural networks run on a Cortex-M4 microcontroller?

Yes, small CNNs run on Cortex-M4 MCUs with constraints. A typical M4 at 168 MHz with 256 KB SRAM can execute a MobileNetV2 variant with 128x128 input in 200-400ms per inference. The CMSIS-NN library provides optimized kernels exploiting single-cycle MAC DSP instructions for 2-4x speedup. Memory bandwidth and SRAM for activation buffers are the key limitations.

How do you update TinyML models on deployed devices?

OTA model updates require dual-bank flash or external storage to hold the new model while the current one runs. The process downloads the quantized model (50-500 KB) in verified chunks, writes to the secondary bank, validates the header and arena requirements, runs a self-test inference, and atomically swaps the model pointer. Bootloader rollback protection reverts to the previous model if validation fails.