VAD-Gated KWS on RISC-V Part 4: Full System Results

Introduction

This is Part 4 of a series on VAD-gated Keyword Spotting on the ESP32-C3 Lyra. Part 1 introduced the system architecture, Part 2 detailed the four-parameter VAD, and Part 3 compared it to other VADs

System Architecture

Audio is captured via the ESP32-C3’s ADC1 CH0 and passed to the hybrid VAD, which extracts a four-feature vector. The four features are Energy, ZCR, Golay, and Hadamard. This four-feature vector is then used as input for a TFLite Micro neural network (4→8→1). When the NN classifies the frame as SPEECH, the KWS engine wakes and runs TensorFlow Lite for Microcontrollers inference, returning a Yes / No / Unknown response. During SILENCE, the KWS engine remains idle.

KWS Neural Network

The VAD gate is a fully integer, three-layer feedforward network trained on 994 labeled frames collected directly from the Lyra V2.0 microphone. The minimal architecture consists of four inputs (one per feature), a single hidden layer of eight neurons, and a binary output.

Parameter	Value
Architecture	4 → 8 → 1
Training frames	994 labeled
Model size	1,868 bytes
Runtime	TFLite Micro
Quantization	Integer-only

The four features fed to the network span complementary signal dimensions. The amplitude-domain structure is measured by Energy and the Golay feature, oscillation rate is captured by ZCR, and the FWHT-derived Hadamard L1 norm measures spectral complexity. Their observed ranges on the Lyra V2.0 are summarized below.

Feature	Silence	Speech
Hadamard (h)	5,800 – 9,800	10,000 – 176,000+
Energy (e)	63,000 – 150,000	83,000 – 2,200,000+
ZCR	0	0 – 73
Golay (g)	0 – 9	0 – 229

The separation between the silence and speech ranges confirms that each feature contributes an effective discriminative signal prior to the NN classification stage.

KWS Accuracy

The network achieved 96.98% test accuracy on the 994-frame labeled dataset, with inference running entirely on-device via TFLite Micro. At 1,868 bytes, the model fits comfortably within the ESP32-C3’s SRAM budget alongside the VAD feature extraction buffers.

Compute Performance

CPU utilization was measured on the ESP32-C3 Lyra V2.0 using esp_timer_get_time() to time each inference, reported as a duty-cycle percentage over a 5-second window.

Mode	Avg Inference	CPU Usage
Always-on KWS	~15.29 ms	1.64%
VAD only	~0.80 ms	0.08%

The VAD gate consumes approximately 20× less CPU per inference than the KWS engine and runs at 0.08% duty cycle during silence, compared to 1.64% for always-on KWS.

Full System Performance

The VAD-gated system operates as a two-state machine, with VAD mode during silence, and KWS mode upon speech detection. KWS mode has a 10-second inactivity timeout before returning to VAD. In a representative 1-minute scenario — 50 seconds of silence and 10 seconds of active speech — the weighted average CPU utilization is:

Terminal

(50 × 0.08% + 10 × 1.64%) / 60 = 0.34% average CPU

This yields a 4.8× reduction in average CPU utilization relative to always-on KWS at 1.64%, corresponding directly to a reduction in active power draw over time.

Future Directions

The pipeline demonstrates that an open, edge-side VAD layer can be inserted before cloud-dependent speech recognition with minimal resource cost. Existing voice assistants rely on proprietary cloud pipelines with no support for on-device voice authentication or arbitrary remote device access. The ESP32-C3 implementation establishes a local wake-word stage that reduces both latency and cloud dependency. It also serves as a foundation for voice-authenticated remote command execution. A system of this type could include locally run VAD and wake-word detection, as well as authenticated commands routed to the cloud and executed on a remote device.

Conclusion

The VAD-gated KWS pipeline on the ESP32-C3 Lyra V2.0 demonstrates that speech-aware inference is achievable at near-zero compute cost during silence. A 1,868-byte TFLite Micro model classifies four integer features with 96.98% accuracy, gating the KWS engine to an average of 0.34% CPU — a 4.8× reduction over always-on operation. The result confirms the viability of multi-stage, integer-only speech pipelines on RISC-V class microcontrollers without floating-point hardware or DSP extensions.

References

1. J. L. Walsh, “A closed set of normal orthogonal functions,” American Journal of Mathematics, vol. 45, no. 1, pp. 5–24, Jan. 1923.
2. J. Hadamard, “Résolution d’une question relative aux déterminants,” Bulletin des Sciences Mathématiques, ser. 2, vol. 17, pp. 240–246, 1893.
3. M. J. E. Golay, “Complementary series,” IRE Transactions on Information Theory, vol. 7, no. 2, pp. 82–87, Apr. 1961.
4. L. R. Rabiner and M. R. Sambur, “An algorithm for determining the endpoints of isolated utterances,” The Bell System Technical Journal, vol. 54, no. 2, pp. 297–315, Feb. 1975.