ximgproc: fix heap buffer overflow in EdgeDrawing::detectLines() - #4205
Open
purehol wants to merge 1 commit into
Open
ximgproc: fix heap buffer overflow in EdgeDrawing::detectLines()#4205purehol wants to merge 1 commit into
purehol wants to merge 1 commit into
Conversation
detectLines() sized its line-fitting scratch buffers x/y as (width+height)*8 and then copied every pixel of each edge segment into them. An edge segment is a 1-pixel-wide chain that can wind through the whole image, so its length is bounded by width*height, not by the image perimeter. A segment longer than the buffer overflowed it (heap corruption; AddressSanitizer reports a WRITE past the end at the fill loop). Size the buffers to the longest segment instead, matching the reference implementation ED_Lib (CihanTopal/ED_Lib@2778deb). Adds a regression test using a single long serpentine segment (far larger than the (width+height)*8 buffer); it triggers the overflow under AddressSanitizer before this change and completes cleanly after it.
purehol
marked this pull request as ready for review
September 2, 2026 15:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
detectLines()allocated its line-fitting scratch buffersx/yas(width + height) * 8doubles, then copied every pixel of each edge segment intothem. An edge segment is a 1-pixel-wide chain that can wind through the whole
image, so its length is bounded by
width * height, not by the image perimeter.A segment longer than the buffer overflows it at the fill loop — heap corruption;
AddressSanitizer reports a WRITE past the end at edge_drawing.cpp:1409.
Fix
Size the buffers to the longest segment, mirroring the reference implementation
ED_Lib (CihanTopal/ED_Lib@2778deb). The
(size_t)cast avoids int overflow onlarge images.
Verification
Added
TEST_F(ximgproc_ED, detectLinesLongWindingSegment): a single serpentinesegment (~17k px, far larger than the (width+height)*8 buffer). It asserts the
oversized-segment precondition, then calls
detectLines(). Under AddressSanitizerthis triggers a heap-buffer-overflow before the change and is clean after it; the
existing
opencv_test_ximgproc*ED*tests still pass.This may be the cause of #3429 (a detectLines segfault whose reproducibility
depends on heap layout, consistent with this overflow). Ordinary photographs do
not produce a segment large enough to overflow, so it could not be verified
against that reporter's specific input; the reporters have been asked to confirm.