fix: guard neighbour lookups in replace_blank (IndexError on trailing space)#318
fix: guard neighbour lookups in replace_blank (IndexError on trailing space)#318koriyoshi2041 wants to merge 1 commit into
Conversation
replace_blank keeps a space only when both neighbours are ASCII non-space characters, but it indexed text[i + 1] / text[i - 1] without bounds checks. A trailing space raised IndexError, and a leading space read text[-1] (wrapping to the last character) and was wrongly kept. Restrict the check to interior positions so edge spaces are dropped and no out-of-range access occurs; interior spacing is unchanged.
a710128
left a comment
There was a problem hiding this comment.
Thanks for the clean fix! 🙏
I pulled the branch down and reproduced both bugs against main:
replace_blank("hello ")→IndexError: string index out of range(trailing space,text[i + 1])replace_blank(" hello")→' hello'— the leading space is wrongly kept becausetext[i - 1]wraps around to the last character
The 0 < i < len(text) - 1 guard is the right minimal fix: edge spaces aren't between two words, so dropping them is correct, and interior spacing is unchanged.
I also ran your test file locally (had to pip install -e . first, which pulls torch — that's likely why it didn't run on your machine), and all 3 tests pass. I spot-checked a few extra cases too, all behaving as expected:
"a b"→"ab","中 a"→"中a","a 中"→"a中",""/" "/" "all safe
LGTM, approving. 🚀
Optional (non-blocking): a couple more assertions like "a b" → "ab" and CJK/ASCII mixed cases would lock down the semantics nicely, but the current coverage is already solid.
|
Thank you for catching this and for the clear write-up — your root-cause analysis (trailing-space Just a heads up: this overlaps with #320, which targets the same bug. Both PRs even add a So we'll close this one as a duplicate, but your report and fix were genuinely valuable — thanks for taking the time to dig in and explain it so clearly. 🙏 |
|
Thanks for checking this and for the detailed note. #320 covers the same boundary-space bug with broader isolated tests, so closing this one to keep the queue clean. |
replace_blank(src/voxcpm/utils/text_normalize.py) crashes on text ending in a space, and mishandles text starting with a space.The intent is to keep a space only when it sits between two ASCII word characters, but the neighbour lookups aren't bounds-checked:
text[i + 1]raiseIndexError: string index out of range.replace_blank("hello ")crashes. This is reachable fromTextNormalizer.normalize()for anyzhtext that ends in a space.text[i - 1]readtext[-1](the last character), so the decision wraps around and a leading space can be wrongly kept.Fix: only keep the space at interior positions (
0 < i < len(text) - 1), so edge spaces are dropped (they aren't between two words) and no out-of-range access happens. Interior spacing (e.g."a b"kept,"中 文"→"中文") is unchanged.Added
tests/test_text_normalize.pycovering trailing/leading/interior spaces and a CJK-adjacent space.Note: I verified the function's behaviour directly (it's pure string logic), but couldn't run the full test locally because importing the package pulls in torch/torchaudio and a native
torchaudiolib fails to load on my machine — CI should exercise the added test normally.