International Journal of Computer Science and Artificial Intelligence
DOI: 10.64823/ijcsa.2601004
Speech emotion recognition aims to identify affective states from vocal signals. Traditional approaches rely on cloud computing, which introduces latency and raises privacy concerns. In contrast, this work explores deploying SER models directly on microcontrollers using TinyML, enabling local, secure, and fast inference.
Let x(t) denote the discrete-time audio signal. The objective is to learn a function fθ(x(t)) that maps the signal to one of six emotion classes y1, ..., y6, where θ are the learnable parameters of a convolutional neural network (CNN).
Prior work in speech emotion recognition has predominantly used spectrograms and Mel-frequency cepstral coefficients (MFCCs) as input features, coupled with deep learning architectures such as convolutional or recurrent neural networks. However, most of these approaches target high-resource environments, with limited focus on deployment to resource-constrained edge devices. This study builds on recent advances in embedded artificial intelligence, particularly platforms such as Edge Impulse and TensorFlow Lite for Microcontrollers, to enable real-time, on-device emotion recognition in a low-power, memory-limited setting.
This section describes the dataset preparation, feature extraction pipeline, and model architecture used to build the SER system.
An 80–20 split was used to divide the dataset into training and test sets. A Noise class was additionally created using non-verbal environmental sounds to improve model robustness under real-world conditions.
Table 1. Class distribution and duration of the training dataset.
Class | Total samples | Training samples | Test samples | Total duration |
|---|---|---|---|---|
Angry | 400 | 322 | 77 | 9m 51s |
Happy | 400 | 321 | 79 | 10m 33s |
Disgust | 400 | 320 | 80 | 13m 4s |
Fear | 400 | 320 | 79 | 8m 50s |
Neutral | 400 | 320 | 80 | 10m 57s |
Noise | 720 | 578 | 142 | 9m 47s |
To illustrate the variability of speech patterns across emotional states, representative waveforms were plotted for each of the six target classes. These time-domain plots show the amplitude envelope of 1-second clips sampled at 16 kHz.
Fig. 1. Representative waveform for the Noise class.
Fig. 2. Representative waveform for the Neutral class.
Fig. 3. Representative waveform for the Angry class.
Fig. 4. Representative waveform for the Disgust class.
Fig. 5. Representative waveform for the Fear class.
Fig. 6. Representative waveform for the Happy class.
The input audio x(t) is segmented into frames of length T = 32 ms with a stride S = 32 ms. Each frame is transformed using the Short-Time Fourier Transform (STFT) followed by a Mel-filterbank operation, as shown in Eq. (1) and Eq. (2).
X(k, n) = (n+m)w(m)e | (1) |
MFE(n, b) = log | (2) |
where w(m) is the Hamming window, Hb(k) is the b-th triangular Mel filter, ε = 10−6 is added for numerical stability, n is the frame index, and b ∈ {1, ..., 32} indexes the 32 filterbanks. Each 1-second clip yields a 32 × 31 time-frequency matrix of Mel energy features.
Fig. 7 shows the weighting profile applied to FFT bins by the 32 Mel filters, and Fig. 8 shows the resulting Mel energy spectrogram generated from a single 1-second audio sample.
Fig. 7. Weighting profile applied to FFT bins by the 32 Mel filters.
Fig. 8. Mel energy spectrogram generated from a single 1-second audio sample.
The CNN is defined as a function fθ mapping a 32 × 31 input to a 6-dimensional output. The layers are: a Conv2D layer with weights W1 ∈ R3×3×1×8 with ReLU activation; a MaxPool2D layer with stride 2; a second Conv2D layer with weights W2 ∈ R3×3×8×16, repeated up to 64 filters; and a final flatten and dense layer computing a softmax output.
Fig. 9. CNN architecture used for emotion classification.
The loss function is categorical cross-entropy, as shown in Eq. (3). Training uses the Adam optimizer with learning rate η = 0.005, batch size B = 32, and E = 80 epochs.
(3) |
Fig. 10. Confusion matrix on the validation set.
Let TPi, FPi, and FNi denote the true positive, false positive, and false negative counts for class i. Precision, recall, and F1-score are computed as shown in Eq. (4) and Eq. (5).
(4) | |
(5) |
Average precision, recall, and F1-score across all six classes exceed 98%.
To evaluate model convergence and potential overfitting, accuracy and loss were tracked across 80 training epochs. The model converged quickly, reaching over 95% validation accuracy within 20 epochs. Loss curves also stabilized early, with validation loss remaining consistently low across later epochs, suggesting minimal overfitting.
Fig. 11. Progression of training and validation accuracy and loss across 80 epochs.
Despite the significant DSP time, the system operates comfortably under a 1-second window, making it viable for real-time inference in low-latency applications.
To better understand class separability in the feature space, Edge Impulse's Data Explorer tool was used to project the high-dimensional Mel-filterbank energy feature vectors into a 2D space using Principal Component Analysis (PCA). Fig. 12 shows the distribution of training data by class in this projected space. Clear clustering indicates that the extracted features encode emotional categories effectively, facilitating accurate classification.
Fig. 12. PCA projection of training data by class in feature space.
The quantized TensorFlow Lite (int8) model was deployed to an ESP32-PICO-D4 microcontroller (dual-core, 240 MHz, 520 KB SRAM, 4 MB Flash) interfaced with an SPM1423 digital PDM microphone for real-time audio capture. Audio signals were streamed through the ESP32's I²S peripheral configured in PDM receive mode, processed by onboard Mel-filterbank DSP blocks, and classified by the CNN.
Fig. 13. ESP32-based deployment hardware with PDM microphone.
The I²S-PDM integration required configuring the I²S driver with the following parameters:
This configuration enables streaming inference with minimal latency and CPU load. Using the EON compiler, the quantized int8 model was compiled into a binary for the ESP32, and inference is executed entirely offline: audio is acquired via the PDM I2S microphone and passed through the DSP pipeline in real time to produce a classification output.
This experiment validates that Mel-filterbank energy features and CNN-based classifiers can be efficiently deployed on microcontrollers with limited memory and computation. Despite hardware constraints, the quantized model generalizes well. Limitations include dataset size, noise robustness, and absence of speaker variation. The training and validation metrics in Section 4.1.1 confirm that the model fits well with minimal overfitting under the given dataset. However, the drop in real-world accuracy suggests that overfitting may occur at the feature level due to limited speaker and noise diversity.
Although the proposed SER model achieved a validation accuracy of 99.5% under controlled conditions, its performance in real-world scenarios showed noticeable degradation. In field testing with naturally spoken utterances recorded in less controlled environments, the model often misclassified emotions, particularly among acoustically similar categories such as Angry and Disgust. In several cases, only one or two out of five utterances per class were correctly identified, indicating a significant drop in generalization performance. This discrepancy is likely due to a combination of the following factors:
These findings emphasize the importance of incorporating robust data augmentation techniques, collecting diverse real-world samples, and applying adaptive noise filtering to enhance the model's resilience and reliability in deployment environments.
A high-accuracy Speech Emotion Recognition (SER) model was implemented on the ESP32 using Mel-filterbank energy features and a quantized 2D Convolutional Neural Network. Achieving near real-time performance and a compact model size, this TinyML-based solution enables efficient, privacy-preserving, and fully offline emotion recognition on ultra-low-power embedded devices.
None.
This research received no external funding.
The author declares no conflict of interest.
The modified Toronto Emotional Speech Set (TESS) used in this study is available on Kaggle (see Reference 6). Trained model files and Edge Impulse project data are available from the author upon reasonable request.
Claude (Anthropic) was used to reformat this manuscript into the journal's required template and to check consistency of headings, figure/table numbering, and reference formatting. It was not used to design the experiment, train the model, or generate results. All technical content, analysis, and conclusions are the author's own and were reviewed and verified by the author.
Conceptualization, methodology, software, formal analysis, investigation, and writing — original draft, M.M.A.H. Ahnaf. The author has read and agreed to the published version of the manuscript.