VAD-Gated KWS on RISC-V Part 4: Full System Results

Introduction

This is Part 4 of a series on VAD-gated Keyword Spotting on the ESP32-C3 Lyra. Part 1 introduced the system architecture, Part 2 detailed the four-parameter VAD, and Part 3 compared it to other VADs

System Architecture

Audio is captured via the ESP32-C3’s ADC1 CH0 and passed to the hybrid VAD, which extracts a four-feature vector. The four features are Energy, ZCR, Golay, and Hadamard. This four-feature vector is then used as input for a TFLite Micro neural network (4→8→1). When the NN classifies the frame as SPEECH, the KWS engine wakes and runs TensorFlow Lite for Microcontrollers inference, returning a Yes / No / Unknown response. During SILENCE, the KWS engine remains idle.

KWS Neural Network

The VAD gate is a fully integer, three-layer feedforward network trained on 994 labeled frames collected directly from the Lyra V2.0 microphone. The minimal architecture consists of four inputs (one per feature), a single hidden layer of eight neurons, and a binary output.

Parameter	Value
Architecture	4 → 8 → 1
Training frames	994 labeled
Model size	1,868 bytes
Runtime	TFLite Micro
Quantization	Integer-only

The four features fed to the network span complementary signal dimensions. The amplitude-domain structure is measured by Energy and the Golay feature, oscillation rate is captured by ZCR, and the FWHT-derived Hadamard L1 norm measures spectral complexity. Their observed ranges on the Lyra V2.0 are summarized below.

Feature	Silence	Speech
Hadamard (h)	5,800 – 9,800	10,000 – 176,000+
Energy (e)	63,000 – 150,000	83,000 – 2,200,000+
ZCR	0	0 – 73
Golay (g)	0 – 9	0 – 229

The separation between the silence and speech ranges confirms that each feature contributes an effective discriminative signal prior to the NN classification stage.

KWS Accuracy

The network achieved 96.98% test accuracy on the 994-frame labeled dataset, with inference running entirely on-device via TFLite Micro. At 1,868 bytes, the model fits comfortably within the ESP32-C3’s SRAM budget alongside the VAD feature extraction buffers.

Compute Performance

CPU utilization was measured on the ESP32-C3 Lyra V2.0 using esp_timer_get_time() to time each inference, reported as a duty-cycle percentage over a 5-second window.

Mode	Avg Inference	CPU Usage
Always-on KWS	~15.29 ms	1.64%
VAD only	~0.80 ms	0.08%

The VAD gate consumes approximately 20× less CPU per inference than the KWS engine and runs at 0.08% duty cycle during silence, compared to 1.64% for always-on KWS.

Full System Performance

The VAD-gated system operates as a two-state machine, with VAD mode during silence, and KWS mode upon speech detection. KWS mode has a 10-second inactivity timeout before returning to VAD. In a representative 1-minute scenario — 50 seconds of silence and 10 seconds of active speech — the weighted average CPU utilization is:

Terminal

(50 × 0.08% + 10 × 1.64%) / 60 = 0.34% average CPU

This yields a 4.8× reduction in average CPU utilization relative to always-on KWS at 1.64%, corresponding directly to a reduction in active power draw over time.

Future Directions

The pipeline demonstrates that an open, edge-side VAD layer can be inserted before cloud-dependent speech recognition with minimal resource cost. Existing voice assistants rely on proprietary cloud pipelines with no support for on-device voice authentication or arbitrary remote device access. The ESP32-C3 implementation establishes a local wake-word stage that reduces both latency and cloud dependency. It also serves as a foundation for voice-authenticated remote command execution. A system of this type could include locally run VAD and wake-word detection, as well as authenticated commands routed to the cloud and executed on a remote device.

Conclusion

The VAD-gated KWS pipeline on the ESP32-C3 Lyra V2.0 demonstrates that speech-aware inference is achievable at near-zero compute cost during silence. A 1,868-byte TFLite Micro model classifies four integer features with 96.98% accuracy, gating the KWS engine to an average of 0.34% CPU — a 4.8× reduction over always-on operation. The result confirms the viability of multi-stage, integer-only speech pipelines on RISC-V class microcontrollers without floating-point hardware or DSP extensions.

References

1. J. L. Walsh, “A closed set of normal orthogonal functions,” American Journal of Mathematics, vol. 45, no. 1, pp. 5–24, Jan. 1923.
2. J. Hadamard, “Résolution d’une question relative aux déterminants,” Bulletin des Sciences Mathématiques, ser. 2, vol. 17, pp. 240–246, 1893.
3. M. J. E. Golay, “Complementary series,” IRE Transactions on Information Theory, vol. 7, no. 2, pp. 82–87, Apr. 1961.
4. L. R. Rabiner and M. R. Sambur, “An algorithm for determining the endpoints of isolated utterances,” The Bell System Technical Journal, vol. 54, no. 2, pp. 297–315, Feb. 1975.

VAD-Gated KWS on ESP-32 Part 3: VAD Analysis

This is Part 3 of a four-part series on building a VAD-gated keyword spotting system on the ESP32-C3 Lyra. Part 1 provides an overview of the project, and Part 2 covers the four-feature voice activity detector (VAD) in detail. This post compares the VAD used to the other VAD approaches available for small microcontrollers(MCUs). Part 4 will discuss implementation on RISC-V architecure and summarize the results.

The Landscape

Currently VADs for MCUs today falls into four categories: bare energy thresholds, classical multi-feature detectors, ported statistical classifiers, and tiny neural networks. This project uses a multi-feature detector, which combines the Hadamard L1, energy, ZCR, and Golay features.

Energy Thresholding

The default VAD common in Arduino sketches, as well as the new ESP32-P4, implements its low-power hardware VAD using an energy threshold plus a passband-energy check. These energy—focused VADs are used for their low computational cost. They are cost efficient because they compute only one multiplication per frame, and don’t use any FFTs or AI inference. However, these VADs are lacking in accuracy, as they are susceptible to activation by high-energy background noise.

Classical Multi-Feature Detectors

To increase the accuracy of lightweight models, many VADs use low-power detectors in parallel. The VAD used in this project is one such example, as the VAD classifies a signal as speech if two of four detection thresholds are met simultaneously. This VAD is effective as it keeps compute costs low while achieving a high accuracy. It does this by using features that can operate solely on integer arithmetic, with no need for floating-point multiplication or division. On the C3’s RISC-V core, this keeps the VAD’s power consumption low enough to run as always-on.

WebRTC VAD: The Ported Standard

The most widely ported VAD on small systems is the WebRTC VAD, originally from Google’s real-time communication stack. It splits the signal into six sub-bands between 80 Hz and 4 kHz and classifies their energies with a Gaussian mixture model. This VAD operates smoothly on a microcontroller as it runs in fixed-point, and the frame buffers stay under 3 KB.

The main shortcoming of WebRTC, however, is false positives in noise. The cost of mistaking noise for speech is much greater in a power-gated KWS pipeline than it was in WebRTC’s original application of phone calls. This is because in our application every false trigger wakes the keyword spotter awhich in turn uses much more power. Espressif now positions its own neural VADNet as a replacement for WebRTC VAD for this reason.

Tiny Neural VADs

Most of the current VAD research is in this class of AI-based detectors, with a focus on reducing the models’ size. Many of these detectors are computationally heavy because they are spectrogram-based: each one first computes a spectrogram — requiring an FFT and a mel filterbank — before running neural network inference. Some examples of tiny neural VADs include:

• MarbleNet-class CNNs: (~88k parameters) achieve high accuracy, but expect a mel-spectrogram front end and tens of kilobytes of weights
• NAS-VAD: use neural architecture search to reach state-of-the-art accuracy at 151k parameters, however it is costly and spectrogram-based
• AtomicVAD: (2025) an end-to-end model that drops to roughly 0.3k parameters, a 99.7% cut compared to MarbleNet, with feature extraction learned inside the network rather than a separate spectrogram stage
• Espressif VADNet: trained on ~15,000 hours of speech, ships inside the ESP-SR audio front end, and achieves better noise rejection than WebRTC VAD.

While these models are accurate, they do not fit the ESP32-C3 Lyra cleanly. ESP-SR’s neural models assume a chip with PSRAM and SIMD, and Espressif had to ship a cut-down wake model (WakeNet9s) to make the family compatible with the C3. The four-feature VAD used in this project avoid the FFT and mel-filterbank computations that these AI-based detectors depends on, which is what keeps it light enough to run always-on.

The strongest alternative to our approach is AtomicVAD, an AI-based end-to-end neural network of roughly 0.3k parameters that learns its own feature extraction. It is similar to the 4-feature VAD in that it skips the spectrogram front end and runs on a bare microcontroller.

A major Drawback to AtomicVAD however, is that it requires a much larger sample size than our four-parameter VAD. AtomicVAD requires 10080 samples (630ms) of input per decision, while the 4-feature VAD requires only 256 samples(16ms). This is because AtomicVAD is a learned model that needs a training pipeline and a labelled dataset. Its accuracy is entirely based on the data it receives. Alternatively, our VAD is a deterministic four-feature majority vote. It involves no model training and instead uses pure integer arithmetic alonside manual tuning to known thresholds.

Both options are suitable for this application. The four-feature design was chosen because it is transparent and auditable. Every decision made can be traced to a specific feature and threshold rather than to learned weights, which make it the better fit for studying how a VAD actually behaves on RISC-V hardware.

Overall, we use the 4-feature VAD because it is fully transparent, training-free, and runs in pure integer arithmetic. In Part 4, we will assemble the complete system, gating the keyword spotter with this VAD on RISC-V and reporting full accuracy and CPU results.

VAD-Gated KWS on RISC-V Part 2 - VAD Structure

This is Part 2 of a series on Voice Activity Detection (VAD) gated Keyword Spotting (KWS) on the ESP32-C3 Lyra. Part 1 provides an overview of the system and the power reduction results achieved. this post examines the VAD in detail.

Four parameters are used simultaneously to create the entire four-feature VAD. The resulting VAD achieves high accuracy as it uses each of the four parameters to prevent noise from being falsely identified as speech.

1. Short-Time Energy (E)

Short-time energy E measures how loud a frame is by squaring each integer ADC sample. The squares are then summed across the frame. The squaring is done to seperate speech from noise in a non-linear manner. A frame with 4× the amplitude of noise yields 16× the energy, sharpening the contrast. Additionally, squaring removes the negative sign from any negative AC values, so that their amplitudes can be treated the same as their positive counterparts.

c

int64_t e = 0; for (int i = 0; i < n; i++) { e += (int64_t)frame[i] * frame[i]; }

This is the classic Short-Time Energy (STE) feature used throughout DSP-based VAD.

2. Zero Crossing Rate (ZCR)

ZCR counts how many times the waveform changes sign within a single frame. It separates broad classes of sound, as spoken vowels produce a moderate ZCR, unvoiced consonants produce a high ZCR, and silence produces a low ZCR. The frequency of a waveform is directly proportional to its ZCR.

c

int32_t zcr = 0; for (int i = 1; i < n; i++) { if ((frame[i] >= 0) != (frame[i-1] >= 0)) zcr++; }

ZCR is ambiguous on its own — noise can oscillate as rapidly as an unvoiced consonant — so it is interpreted alongside energy E. The possible combinations of ‘E’ and ZCR yield the following:

Condition	Interpretation
High ZCR + low E	Background noise
High ZCR + high E	Speech
Low ZCR + low E	Silence

This ZCR/energy combination is the well-established Rabiner & Sambur (1975) endpoint-detection approach.

3. Golay Correlation

For this feature, each frame is correlated against an approximated Golay reference sequence (Golay, 1961). These sequences are built from period-8 alternating blocks {1,1,1,1,-1,-1,-1,-1, ...}. Every audio sample is multiplied by the corresponding reference value. The resulting feature is the absolute value of the sum of these products. It measures how strongly the frame aligns with the reference pattern.

In practice the output separates cleanly into three regions:

Correlation sum	Interpretation
0–10	Silence (all samples near zero)
10–50	Noise (positive and negative samples cancel)
50–300+	Speech (structured signal produces a large sum)

c

// ±1 reference: period-8 alternating blocks { 1,1,1,1,-1,-1,-1,-1, ... } int64_t corr = 0; for (int i = 0; i < n; i++) { corr += (int64_t)frame[i] * golay_ref[i]; } if (corr < 0) corr = -corr; // absolute value

Restricting the reference to ±1 reduces the inner product to solely additions and subtractions. This lack of multiplication makes this feature deterministic and keeps computational costs low. The low compute cost allows for real-time execution on the ESP32-C3.

4. Fast Walsh-Hadamard Transform (FWHT)

The FWHT transforms the recorded audio samples into coefficients using only additions and subtractions. The L1 norm of the coefficients is then used to collapse them into a single value. Its butterfly structure is identical to the FFT except that the weights are restricted to ±1 — so there are no multiplications and no twiddle factors, making it integer-only and well suited to the ESP32-C3.

c

// Butterfly transform — additions only for (int len = 1; len < n; len <<= 1) { for (int i = 0; i < n; i += len << 1) { for (int j = 0; j < len; j++) { int32_t a = buf[i+j], b = buf[i+j+len]; buf[i+j] = a + b; // sum buf[i+j+len] = a - b; // diff } } } // L1 norm of coefficients acc += abs32_sat(buf[i]);

The L1 norm separates into three regions:

L1 norm	Interpretation
Low	Silence (all samples near zero)
Moderate	Noise (random, broadband, energy spread evenly)
High	Speech (structured signal, large coefficients)

The FWHT is an attractive feature for this application because of its low-power, multiplication-free pattern.

To identify speech, the VAD examines the values for the four. A frame is classified as speech if at least two of the four features exceed their respective thresholds.

c

bool vad_decide(const vad_features_t *f) { int score = 0; if (f->h >= THRESH_H) score++; // Hadamard L1 ≥ 10,000 if (f->e >= THRESH_E) score++; // Energy ≥ 200,000 if (f->g >= THRESH_G) score++; // Golay ≥ 10 if (f->zcr >= THRESH_ZCR) score++; // ZCR ≥ 2 return score >= 2; }

This majority-voting scheme provides robustness against false positives. For example, a transient noise spike may push one feature over threshold, but is unlikely to trigger two simultaneously. The thresholds were derived empirically from observed data on the ESP32-C3 Lyra V2.0 with direct microphone input.

Implementation on the ESP32-C3

All four VAD features are computed using integer arithmetic. This removes the high computational cost of floating-point arithmetic. For example, division is replaced throughout with right-shifts. This is present when energy uses a >> 16 to stay within int32 range, and the Golay correlation sum uses a >> 10 before threshold comparison. These operations keep the VAD entirely within the ESP32-C3’s integer pipeline.

Each frame consists of 256 samples taken directly from the microphone input buffer. The four features are computed sequentially per frame:

c

int32_t h = hadamard_compute_feature(g_audio_output_buffer, vad_workbuf, 256, HADAMARD_FEATURE_L1_SUM); int32_t e = vad_feature_energy(g_audio_output_buffer, 256); int32_t zcr = vad_feature_zcr(g_audio_output_buffer, 256); int32_t g_feat = golay_compute_feature(g_audio_output_buffer, 256);

The KWS model requires a 30 KB statically allocated tensor arena. The VAD, by contrast, requires only a 256-element int32 working buffer (~1 KB) for the Hadamard transform, which removes the need for dynamic allocation.

The system operates as a two-state machine: STATE_VAD and STATE_KWS. The VAD runs continuously in STATE_VAD. On a positive speech decision, the system transitions to STATE_KWS and begins running KWS inference. If no keyword is detected within 10 seconds, the system returns to STATE_VAD.

Standalone Power and Compute Cost

CPU utilization was measured on the ESP32-C3 Lyra V2.0 using esp_timer_get_time() to time each inference, reported as a duty-cycle percentage over a 5-second window.

Mode	Avg Inference	CPU Usage
Always-on KWS	~15.29 ms	1.64%
VAD only	~0.80 ms	0.08%

The VAD consumes approximately 20× less CPU per inference and runs at 0.08% CPU — roughly 20× lower duty cycle than always-on KWS at 1.64%.

In a realistic 1-minute usage scenario — 50 seconds of silence (VAD mode) and 10 seconds of active speech (KWS mode) — the VAD-gated system averages 0.34% CPU, compared to 1.64% for always-on KWS. This yields a 4.8× reduction in average CPU utilization, which directly corresponds to a reduction in active power draw over time.

In conclusion, the lightweight, integer-only VAD on the ESP32-C3 Lyra yielded a 4.8× reduction in CPU utilization over always-on KWS. Part 3 benchmarks it against alternative approaches, and Part 4 presents the full KWS on RISC-V results.

VAD KWS on RISC-V Part 1: Overview and Executive Summary

This post is Part 1 of 4 in a series covering Voice Activity Detection (VAD) gated Keyword Spotting (KWS) on the ESP32-C3 Lyra, a RISC-V based microcontroller. Precursor work solely focused on KWS on the Lyra can be viewed at Keyword Spotting on ESP32-C3-Lyra V2 Using ESP-IDF.

This is the subject of a project I completed for two of my M.S. courses, Digital Signal Processing and AI-accelerated computing, both taught by Dr. Adly Fam. The goal of this project is to reduce the power consumption of always-on KWS by gating the KWS model behind a lightweight VAD. Rather than running the full KWS model continuously, the VAD listens first — only waking the KWS model when speech is detected.

The VAD: Four-Parameter Approach

The VAD used in this project is based on four acoustic parameters:

1. Energy
2. Zero Crossing Rate (ZCR)
3. The Golay Feature
4. The Fast Walsh Hadamard Transform (FWHT)

Power Consumption Results

By running the VAD instead of the full KWS model during silence, the system achieves approximately a 4.8× reduction in average power consumption over time on the ESP32-C3.

This concludes part 1. Part 2 will explore the VAD's structure, Part 3 will discuss other existing methods for VAD, and Part 4 will summarize the results from testing the full system.