Skip to content

ximgproc: fix heap buffer overflow in EdgeDrawing::detectLines() - #4205

Open
purehol wants to merge 1 commit into
opencv:4.xfrom
purehol:fix-edgedrawing-detectlines-overflow
Open

ximgproc: fix heap buffer overflow in EdgeDrawing::detectLines()#4205
purehol wants to merge 1 commit into
opencv:4.xfrom
purehol:fix-edgedrawing-detectlines-overflow

Conversation

@purehol

@purehol purehol commented Sep 2, 2026

Copy link
Copy Markdown

Root cause

detectLines() allocated its line-fitting scratch buffers x/y as
(width + height) * 8 doubles, then copied every pixel of each edge segment into
them. An edge segment is a 1-pixel-wide chain that can wind through the whole
image, so its length is bounded by width * height, not by the image perimeter.
A segment longer than the buffer overflows it at the fill loop — heap corruption;
AddressSanitizer reports a WRITE past the end at edge_drawing.cpp:1409.

Fix

Size the buffers to the longest segment, mirroring the reference implementation
ED_Lib (CihanTopal/ED_Lib@2778deb). The (size_t) cast avoids int overflow on
large images.

Verification

Added TEST_F(ximgproc_ED, detectLinesLongWindingSegment): a single serpentine
segment (~17k px, far larger than the (width+height)*8 buffer). It asserts the
oversized-segment precondition, then calls detectLines(). Under AddressSanitizer
this triggers a heap-buffer-overflow before the change and is clean after it; the
existing opencv_test_ximgproc *ED* tests still pass.

This may be the cause of #3429 (a detectLines segfault whose reproducibility
depends on heap layout, consistent with this overflow). Ordinary photographs do
not produce a segment large enough to overflow, so it could not be verified
against that reporter's specific input; the reporters have been asked to confirm.

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
  • The PR is proposed to the proper branch
  • There is a reference to the original bug report and related work
  • There is accuracy test, performance test and test data in the repository, if applicable
  • The feature is well documented and sample code can be built with the project CMake

detectLines() sized its line-fitting scratch buffers x/y as (width+height)*8 and
then copied every pixel of each edge segment into them. An edge segment is a
1-pixel-wide chain that can wind through the whole image, so its length is
bounded by width*height, not by the image perimeter. A segment longer than the
buffer overflowed it (heap corruption; AddressSanitizer reports a WRITE past the
end at the fill loop). Size the buffers to the longest segment instead, matching
the reference implementation ED_Lib (CihanTopal/ED_Lib@2778deb).

Adds a regression test using a single long serpentine segment (far larger than
the (width+height)*8 buffer); it triggers the overflow under AddressSanitizer
before this change and completes cleanly after it.
@purehol purehol mentioned this pull request Sep 2, 2026
4 tasks
@purehol
purehol marked this pull request as ready for review September 2, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant