[BUG] Malformed COCO RLE counts can overflow the vendored pycocotools decode buffer

[starsalt0124]

Describe the bug

The vendored COCO mask implementation under detection/retinaface/rcnn/pycocotools accepts compressed RLE objects whose decoded run lengths do not match the declared mask size.

_mask.decode(...) allocates the destination mask from the untrusted size field:

h, w, n = Rs._R[0].h, Rs._R[0].w, Rs._n
masks = Masks(h, w, n)
rleDecode(<RLE*>Rs._R, masks._mask, n)

But rleFrString(...) decodes the untrusted counts string into R->cnts without checking that the run-length total equals h * w. rleDecode(...) then trusts R->cnts[j] and writes that many bytes to the destination pointer:

for( k=0; k<R[i].cnts[j]; k++ ) *(M++)=v;

As a result, a malformed compressed RLE can make the native decoder write past the heap allocation.

Affected paths:

• detection/retinaface/rcnn/pycocotools/mask.py::decode
• detection/retinaface/rcnn/pycocotools/coco.py::annToMask
• detection/retinaface/rcnn/pycocotools/_mask.pyx::_frString
• detection/retinaface/rcnn/pycocotools/_mask.pyx::decode
• detection/retinaface/rcnn/pycocotools/maskApi.c::rleFrString
• detection/retinaface/rcnn/pycocotools/maskApi.c::rleDecode

The attacker-controlled input can be a COCO annotation/result file, or any caller-provided RLE dictionary passed to rcnn.pycocotools.mask.decode(...).

To Reproduce

I reproduced this with an isolated ASan build of the vendored _mask.pyx, maskApi.c, and maskApi.h.

Minimal PoC:

import _mask

_mask.decode([{"size": [1, 1], "counts": b"02"}])

This declares a 1-byte mask (size=[1, 1]) while counts=b"02" decodes to run lengths [0, 2], causing a 2-byte write into a 1-byte destination.

Build and run:

python setup.py build_ext --inplace
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so.8 \
ASAN_OPTIONS=detect_leaks=0:abort_on_error=1:symbolize=1 \
python poc_decode_overflow.py

Observed ASan output:

ERROR: AddressSanitizer: heap-buffer-overflow
WRITE of size 1
#0 rleDecode .../maskApi.c:46
#1 __pyx_pf_5_mask_6decode .../_mask.c
0x... is located 0 bytes after 1-byte region
allocated by Masks.__cinit__

Expected behavior

Malformed RLE input should be rejected with a Python exception before native decoding writes to the destination buffer.

Recommended checks:

• Validate that h, w, and n are positive.
• Check that h * w * n cannot overflow size_t.
• After rleFrString(...), verify that each decoded RLE's run-length total equals the declared h * w.
• Reject decoded negative/delta values that wrap into huge uint counts.
• Check malloc results before use.
• Consider changing rleDecode(...) to take the output capacity and fail if a run would write past it.

Environment

Local verification environment:

Repo commit: 658b034e7fc0f4b08a01e11347b6118d8d04c76b
OS: Ubuntu 24.04.3 LTS under WSL2
Python: 3.13.9
NumPy: 2.5.0
Cython: 3.2.5
GCC/G++: 13.3.0
ASan runtime: /usr/lib/x86_64-linux-gnu/libasan.so.8

The RetinaFace Makefile builds the vendored extension with:

cd detection/retinaface
make

which includes:

cd rcnn/pycocotools/; python setup.py build_ext --inplace

Additional context

The cross-language path is:

COCO annotation/result RLE
  -> ann["segmentation"]["counts"] and ann["segmentation"]["size"]
  -> COCO.annToMask(...)
  -> maskUtils.decode(rle)
  -> pycocotools.mask.decode(...)
  -> Cython extension _mask.decode(...)
  -> _frString(...) / rleFrString(...)
  -> R->cnts
  -> rleDecode(...)
  -> heap out-of-bounds write

This is at least a reliable process crash/DoS for code that decodes untrusted COCO RLE masks, and it is native heap memory corruption rather than a normal Python-level parsing error.