FineSpeaker: Reliable Speaker Representation Learning at Fine Temporal Resolution
Yongkang Yin, Wenhao Yang, Lichun Fan, Jian Luan, Hang Su, Yuexian Zou
原文摘要
Speaker representation learning is fundamental to speech processing tasks such as speaker verification and diarization. Although long-context speaker encoders can produce stable and discriminative embeddings, short-duration verification and diarization require finer-grained temporal representations. Short temporal contexts limit the available speaker evidence and make speaker identity difficult to disentangle from phonetic and acoustic variability. This creates a temporal resolution-representation reliability trade-off: speaker representations should maintain local within-speaker consistency, between-speaker discriminability and remain sensitive to speaker changes at fine temporal resolution. Accordingly, we propose a framework for reliable speaker representation learning at fine temporal resolution. FineSpeaker first constructs resolution-controllable frame-wise speaker representations through mean pooling over an adjustable n-frame window, enabling flexible control of representation granularity. It then introduces a speaker encoder with local temporal modeling that exploits short-range speaker context to improve the stability and discriminability of short-duration and frame-wise representations. Furthermore, we develop a generative disentanglement strategy for optimizing frame-level speaker representations, combining VAE-based probabilistic modeling with flow matching to characterize speaker-related variation and suppress non-speaker interference. Experiments demonstrate competitive full-utterance performance, improved fine-grained speaker verification on VoxCeleb and CN-Celeb, and strong zero-shot diarization results, highlighting the benefits of balancing temporal resolution and representation reliability.
cs.SD