Skip to content

bound mean-cut partition scan in crawl_decision_tree#120

Open
jmestwa-coder wants to merge 1 commit into
dloebl:mainfrom
jmestwa-coder:mean-cut-partition-bound
Open

bound mean-cut partition scan in crawl_decision_tree#120
jmestwa-coder wants to merge 1 commit into
dloebl:mainfrom
jmestwa-coder:mean-cut-partition-bound

Conversation

@jmestwa-coder

Copy link
Copy Markdown

mean-cut color quantization in cgif_rgb can split a node on the wrong side of its range:

  • both partition scans assumed the float mean sits strictly between the node's min and max color along cutDim
  • a frequency-weighted mean rounds to the channel extreme for skewed inputs, so a scan runs past idxMax/idxMin and creates a child with an empty range
  • get_mean then divides a zero frequency sum, and the NaN mean is cast to uint8_t in get_palette_from_decision_tree (undefined behavior; the over-scan can also read past pFrequDense)

Bounded both scans to the node range and kept the split non-empty.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant