Skip to content

Refactor opengrep runner to stream JSON output - #26

Closed
lolgab wants to merge 1 commit into
mainfrom
streaming
Closed

Refactor opengrep runner to stream JSON output#26
lolgab wants to merge 1 commit into
mainfrom
streaming

Conversation

@lolgab

@lolgab lolgab commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Switched command execution from cmd.Output() buffering to pipe-based streaming decode with json.Decoder, reducing peak memory usage during large scans. Added bounded stderr tail buffering and updated tests to cover streaming execution and buffer truncation behavior.

Switched command execution from cmd.Output() buffering to pipe-based
streaming decode with json.Decoder, reducing peak memory usage during
large scans. Added bounded stderr tail buffering and updated tests to
cover streaming execution and buffer truncation behavior.
@codacy-production

Copy link
Copy Markdown

Not up to standards ⛔

🔴 Issues 1 medium

Alerts:
⚠ 1 issue (≤ 0 issues of at least minor severity)

Results:
1 new issue

Category Results
Complexity 1 medium

View in Codacy

AI Reviewer: first review requested successfully. AI can make mistakes. Always validate suggestions.

Run reviewer

TIP This summary will be updated as you push new changes.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors command execution to stream and parse output directly from stdout, reducing memory usage, and introduces a size-limited buffer for stderr. The review feedback highlights three key improvements: checking the parsing error before the stderr copy error to avoid masking the root cause, copying bytes to a temporary slice in limitedBuffer.Write to avoid fragile overlapping writes, and adding a defensive nil check in isBenignStreamClose to prevent potential nil pointer dereferences.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread internal/tool/command.go
Comment on lines +131 to +139
if stderrCopyErr != nil && !isBenignStreamClose(stderrCopyErr) {
return nil, stderrTail.String(), stderrCopyErr
}
if parseErr != nil {
return nil, stderrTail.String(), parseErr
}
cmdOutputString := string(cmdOutput)
return &cmdOutputString, nil, nil
if waitErr != nil {
return nil, stderrTail.String(), waitErr
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Checking stderrCopyErr before parseErr can mask the actual parsing error. If parseCommandOutput fails (e.g., due to invalid JSON), the process is killed, which often causes io.Copy on the stderr pipe to fail with a non-benign error (like a closed pipe). If this happens, the secondary stderrCopyErr will be returned instead of the root cause parseErr, making debugging extremely difficult. We should check and return parseErr first.

Suggested change
if stderrCopyErr != nil && !isBenignStreamClose(stderrCopyErr) {
return nil, stderrTail.String(), stderrCopyErr
}
if parseErr != nil {
return nil, stderrTail.String(), parseErr
}
cmdOutputString := string(cmdOutput)
return &cmdOutputString, nil, nil
if waitErr != nil {
return nil, stderrTail.String(), waitErr
}
if parseErr != nil {
return nil, stderrTail.String(), parseErr
}
if stderrCopyErr != nil && !isBenignStreamClose(stderrCopyErr) {
return nil, stderrTail.String(), stderrCopyErr
}
if waitErr != nil {
return nil, stderrTail.String(), waitErr
}

Comment thread internal/tool/command.go
Comment on lines +183 to +188
if l.buf.Len()+len(p) > l.max {
drop := l.buf.Len() + len(p) - l.max
current := l.buf.Bytes()
l.buf.Reset()
_, _ = l.buf.Write(current[drop:])
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current implementation of limitedBuffer.Write performs an overlapping write on the same underlying array of bytes.Buffer after calling Reset(). While Go's copy is overlap-safe, relying on this behavior alongside bytes.Buffer's internal implementation details (e.g., that Reset does not reallocate or clear the slice) is fragile and non-idiomatic. Copying the remaining bytes to a temporary slice first is much safer and more maintainable.

Suggested change
if l.buf.Len()+len(p) > l.max {
drop := l.buf.Len() + len(p) - l.max
current := l.buf.Bytes()
l.buf.Reset()
_, _ = l.buf.Write(current[drop:])
}
if l.buf.Len()+len(p) > l.max {
drop := l.buf.Len() + len(p) - l.max
temp := make([]byte, l.buf.Len()-drop)
copy(temp, l.buf.Bytes()[drop:])
l.buf.Reset()
_, _ = l.buf.Write(temp)
}

Comment thread internal/tool/command.go
Comment on lines +197 to +199
func isBenignStreamClose(err error) bool {
return errors.Is(err, io.EOF) || errors.Is(err, os.ErrClosed) || strings.Contains(err.Error(), "file already closed")
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If err is nil, calling err.Error() will cause a nil pointer dereference panic. Although the current call sites guard against nil, adding a defensive nil check at the beginning of isBenignStreamClose makes this helper function robust and safe for future reuse.

func isBenignStreamClose(err error) bool {
	if err == nil {
		return false
	}
	return errors.Is(err, io.EOF) || errors.Is(err, os.ErrClosed) || strings.Contains(err.Error(), "file already closed")
}

@lolgab lolgab closed this Jun 29, 2026
@lolgab
lolgab deleted the streaming branch June 29, 2026 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant