DOI: 10.17587/prin.17.613-625
Intelligent Speech Interaction System with Real-Time Speech Modification and Recognition
V. A. Nechaev, Lecturer-Researcher, nechaev@gapps.ispu.ru,
S. V. Kosyakov, D. Sc. (Eng.), Professor, Head of Department, ksv@ispu.ru,
Ivanovo State Power Engineering University, Ivanovo, 153003, Russian Federation
Corresponding author: Vladimir A. Nechaev, Lecturer-Researcher, Ivanovo State Power Engineering University, Ivanovo, 153003, Russian Federation, E-mail: nechaev@gapps.ispu.ru
Received on February 01, 2026
Accepted on May 05, 2026
The paper presents an intelligent speech interaction system designed for real-time voice communication scenarios, primarily in call centers. When conventional systems are used, both conversational efficiency and the quality of downstream automatic processing deteriorate due to multilingual and mixed-language speech, foreign accents, background noise, and domain-specific terminology. The proposed approach integrates, within a unified architecture, a multilingual domain-oriented automatic speech recognition (ASR) system and a speech modification system. Parallel streams are employed — alongside the direct (unprocessed) audio signal, a modified stream is generated, which can be used both to improve intelligibility for the interlocutor and to increase recognition robustness. At the same time, both versions of the speech data are preserved for subsequent analysis and continuous service quality improvement.
The study formalizes a correction-level indicator that reflects the depth of speech modification — from accent conversion to pronunciation correction and full speech-to-speech translation. It is shown that increasing the correction level requires a longer context window for the corresponding speech transformation and generation models and, consequently, leads to higher end-to-end real-time latency. This makes it possible to justify the choice of the speech modification strategy under the constraints of interactive dialogue, such as the user's proficiency in the intermediary language and the maximum permissible real-time delays. A practical threshold of approximately 30o ms is emphasized, beyond which latency becomes perceptible to users and may cause discomfort during communication.
Experiments were conducted on real call center data comprising Russian speech and mixed Russian-English speech that includes foreign accents, background noise, and domain-specific terminology. The results show that combining a multilingual, domain-oriented ASR system with a speech modification system provides the best recognition quality for both Russian-only and mixed Russian-English data. For practical deployment, the proposed intelligent speech interaction system was implemented for GPU-based inference. The average recognition latency under concurrent load was 90-100 ms, while the total latency, including the accent conversion module, reached up to 350 ms. At the same time, the real-time delay between the client and the operator in the modified channel remains within 250 ms, confirming the practical applicability of the proposed approach.
Keywords: speech processing, speech recognition, accent conversion, voice conversion, speech generation, machine learning, machine translation, neural network, call center
pp. 613—625
For citation:
Nechaev V. A., Kosyakov S. V. Intelligent Speech Interaction System with Real-Time Speech Modification and Recognition, Programmnaya Ingeneria, 2026, vol. 17, no. 11, pp. 613—625. DOI: 10.17587/prin.613-625.
References:
- Shneiderman B. Human-centered artificial intelligence: Reliable, safe & trustworthy, International Journal of Human— Computer Interaction, 2020, vol. 36, no. 6, pp. 495—504. DOI: 10.1080/10447 318.2020.1741118.
- Nicolescu L., Tudorache M. T. Human-computer interaction in customer service: the experience with AI chatbots—a systematic literature review, Electronics, 2022, vol. 11, no. 10, article 1579. DOI: 10.3390/electronics11101579.
- Jin M., Serai P., Wu J. et al. Voice-preserving zero-shot multiple accent conversion, ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, IEEE, 2023, pp. 1—5. DOI: 10.1109/icassp49357.2023.10094737.
- Nechaev V. A., Kosyakov S. V. Accent conversion method with real-time voice cloning based on a nonautoregressive neural network model, Russian Technological Journal, 2025, vol. 13, no. 3, pp. 7—20. DOI: 10.32362/2500-316X-2025-13-3-7-20 (in Russian).
- SEAMLESS Communication Team. Joint speech and text machine translation for up to 100 languages, Nature, 2025, vol. 637, no. 8046, pp. 587—593. DOI: 10.1038/s41586-024-08359-z.
- Diwan A., Srinivasan A., Harwath D., Choi E. Textless Speech-to-Speech Translation With Limited Parallel Data, Findings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 16208—16224. DOI: 10.18653/v1/2024.findings-emnlp.951.
- Gulati A., Qin J., Chiu C. et al. Conformer: Convolution-augmented transformer for speech recognition, Interspeech 2020, 2020, pp. 5036—5040. DOI: 10.21437/Interspeech.2020-3015.
- Radford A., Kim J. W., Xu T. et al. Robust speech recognition via large-scale weak supervision, International conference on machine learning, PMLR, 2023, pp. 28492—28518. DOI: 10.48550/arXiv.2212.04356.
- Li K., Li J., Ye G. et al. Towards code-switching ASR for end-to-end CTC models, ICASSP 2019—2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, IEEE, 2019, pp. 6076 — 6080. DOI: 10.1109/icassp.2019.8683223.
- Liu L., Gu Y., Gourav A. et al. Domain-aware neural language models for speech recognition, ICASSP 2021—2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada, IEEE, 2021, pp. 7373—7377. DOI: 10.1109/icassp39728.2021.9414800.
- Nechaev V. A., Kosyakov S. V. Development of automatic speech recognition model for energy facilities, ISPU Bulletin, 2023, no. 4, pp. 94—100. DOI: 10.17588/2072-2672.2023.4.094-100 (in Russian).
- Balakshin P. V., Petrov G. Y. Implementation Aspects of Speech Recognition Applications in Customer-Service Call Centers, Nauchno-tehnicheskiy vestnik informacionnih tehnologii, mehaniki i optiki, 2012, no. 1 (77), pp. 71—76 (in Russian).
- Romanova D. R., Podkolzin V. V. Sistema golosovogo obsluzhivaniya dlya optimizacii raboty koll-centra, Prikladnaya matema-tika: sovremennye problemy matematiki, informatiki i modelirovaniya, Sb. trudov konf., Krasnodar, 22—25 April 2020, FGBU "Rossijskoe energeticheskoe agentstvo" Minenergo Rossii, Krasnodarskij CNTI-filial FGBU "REA" Minenergo Rossii, 2020, pp. 274—279 (in Russian).
- Wang L., Huang N., Hong Y. et al. Voice-based AI in call center customer service: A natural field experiment, Production and Operations Management, 2023, vol. 32, no. 4, pp. 1002—1018. DOI: 10.1111/poms.13953.
- Valizada A., Akhundova N., Rustamov S. Development of speech recognition systems in emergency call centers, Symmetry, 2021, vol. 13, no. 4, article 634. DOI: 10.3390/sym13040634.
- Thuestad J. A., Grutle 0. Speech-to-text models to transcribe emergency calls: dis, The University of Bergen, 2023, 53 p.
- Verma T., Shree A., Modi A. ASR for Low Resource and Multilingual Noisy Code-Mixed Speech, Proceedings of Interspeech 2023, Dublin, Ireland, 2023, pp. 3242—3246. DOI: 10.21437/Interspeech.2023-757.
- Biadsy F., Weiss R. J., Moreno P. J. et al. Parrotron: An End-to-End Speech-to-Speech Conversion Model and Its Applications to Hearing-Impaired Speech and Speech Separation, Proceedings of Interspeech 2019, Graz, Austria, 2019, pp. 4115—4119. DOI: 10.21437/interspeech.2019-1789.
- Baese-Berk M. M., Morrill T. H. Speaking rate consistency in native and non-native speakers of English, The Journal of the Acoustical Society of America, 2015, vol. 138, no. 3, pp. EL223— EL228. DOI: 10.1121/1.4929622.
- Zhao G., Gutierrez-Osuna R. Using phonetic posterior-gram based frame pairing for segmental accent conversion, IEEE/ ACM Transactions on Audio, Speech, and Language Processing, 2019, vol. 27, no. 10, pp. 1649—1660. DOI: 10.1109/taslp.2019.2926754.
- Wadhawan A. Simultaneous speech to speech translation: dis, University of Pennsylvania, 2023, 46 p.
- Xu M., Cai D., Yin W. et al. Resource-efficient algorithms and systems of foundation models: A survey, ACM Computing Surveys, 2025, vol. 57, no. 5, article 110. DOI: 10.1145/3706418.
- Matzinger T., Pleyer M., Zywiczynski P. Pause Length and Differences in Cognitive State Attribution in Native and Non-Native Speakers, Languages, 2023, vol. 8, no. 1, article 26. DOI: 10.3390/languages8010026.
- Liu M., Zhang C., Xing H. et al. Scalable data annotation pipeline for high-quality large speech datasets development, arXiv preprint, 2021, 13 p. DOI: 10.48550/arXiv.2109.01164.
- Rastorgueva E., Lavrukhin V., Ginsburg B. NeMo Forced Aligner and its application to word alignment for subtitle generation, Proc. of Interspeech 2023, Dublin, Ireland, 2023, pp. 5257—5258.
- Hu K., Li B., Sainath T. N. et al. Mixture-of-Expert Conformer for Streaming Multilingual ASR, Proc. of Interspeech 2023, Dublin, Ireland, 2023, pp. 3327—3331. DOI: 10.21437/interspeech.2023-480.
- Bakhturina E., Zhang Y., Ginsburg B. Shallow Fusion of Weighted Finite-State Transducer and Language Model for Text Normalization, Proc. of Interspeech 2022, 2022, pp. 491—495. DOI: 10.21437/interspeech.2022-11074.
- Karpov A. A., Kipyatkova I. S. Methodology for estimation of automatic speech recognition system performance, Izvestiya vysshih uchebnyh zavedeni, Priborostroenie, 2012, vol. 55, no. 11, pp. 38—43 (in Russian).