← 研踪 PaperTrailarXiv:2605.14736 · v3
IsoNet: Face-Guided Speech Extraction for Compact Microphone Arrays
首次提交 2026-05-14 20:00
版本更新 2026-10-29 12:59
雷达观测 2026-10-30 11:25
摘要
Listening to one chosen talker in a crowded room is effortless for people but hard for small devices, whose microphone arrays are too compact for classical beamforming to separate nearby voices. We present IsoNet, a select-to-listen system: the user selects a face in the camera view, and a 4-microphone array with a 9.4 cm aperture returns that person's speech. IsoNet fuses multi-channel complex spectra, GCC-PHAT spatial cues, and a face embedding at the bottleneck of a U-Net mask estimator, trained with auxiliary direction-of-arrival supervision and an SNR curriculum on simulated reverberant VoxCeleb mixtures. On a hard test set spanning -1 to 10 dB SNR, IsoNet reaches 9.31 dB SI-SDR, a 4.85 dB improvement over the mixture, and raises PESQ from 1.40 to 2.13 and STOI from 0.72 to 0.84. The gains are largest where the task is hardest (5.71 dB SI-SDR improvement at -1 to 1 dB SNR), and IsoNet outperforms oracle-steered delay-and-sum and MVDR beamformers by 9.7 and 10.9 dB SI-SDR. Ablations show that visual and spatial cues are complementary, and a 4.9M-parameter audio-spatial variant stays within 0.16 dB of the full model while running 26 times faster than real time.
作者备注：8 pages
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v3 · 2026-10-29 12:59 · 最新观测版本
IsoNet: Face-Guided Speech Extraction for Compact Microphone Arrays
Listening to one chosen talker in a crowded room is effortless for people but hard for small devices, whose microphone arrays are too compact for classical beamforming to separate nearby voices. We present IsoNet, a select-to-listen system: the user selects a face in the camera view, and a 4-microphone array with a 9.4 cm aperture returns that person's speech. IsoNet fuses multi-channel complex spectra, GCC-PHAT spatial cues, and a face embedding at the bottleneck of a U-Net mask estimator, trained with auxiliary direction-of-arrival supervision and an SNR curriculum on simulated reverberant VoxCeleb mixtures. On a hard test set spanning -1 to 10 dB SNR, IsoNet reaches 9.31 dB SI-SDR, a 4.85 dB improvement over the mixture, and raises PESQ from 1.40 to 2.13 and STOI from 0.72 to 0.84. The gains are largest where the task is hardest (5.71 dB SI-SDR improvement at -1 to 1 dB SNR), and IsoNet outperforms oracle-steered delay-and-sum and MVDR beamformers by 9.7 and 10.9 dB SI-SDR. Ablations show that visual and spatial cues are complementary, and a 4.9M-parameter audio-spatial variant stays within 0.16 dB of the full model while running 26 times faster than real time.
查看此版本解读 →查看此版本原文 ↗