fix(mysql): add lock_wait_timeout to DSN to stop statement pile-up - #423
Open
jabbrwcky wants to merge 3 commits into
Open
fix(mysql): add lock_wait_timeout to DSN to stop statement pile-up#423jabbrwcky wants to merge 3 commits into
jabbrwcky wants to merge 3 commits into
Conversation
jabbrwcky
force-pushed
the
fix/mysql-lock-wait-timeout
branch
from
July 22, 2026 08:13
3a907a7 to
d44b68f
Compare
MySQL/Percona could become unresponsive under a growing backlog of pending ALTER USER (and other) statements issued by provider-sql. Root cause: go-sql-driver/mysql cancels a context by closing the TCP socket without sending KILL QUERY (intentional, all versions incl. the latest v1.10.0). With MySQL's default lock_wait_timeout of 1 year, a statement blocked on a metadata/ACL lock (e.g. behind a backup or long transaction) stays pending server-side even after the 60s reconcile deadline makes the provider give up. Every reconcile retry issues another blocked statement; account-management statements additionally serialize on a single global ACL lock, so they pile up until max_connections is exhausted or ACL contention wedges the server. Append lock_wait_timeout=30 and a dial timeout=10s to the MySQL DSN so blocked statements fail fast server-side and release instead of accumulating. This covers every MySQL controller (User, Grant, Database; cluster and namespaced) since they share this client. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Jens Hausherr <jens.hausherr@pflege.de>
Records why go-sql-driver/mysql cancelling a context by closing the socket (rather than issuing KILL QUERY) matters operationally, the full incident chain it can cause, the lock_wait_timeout mitigation, what it does not cover, and follow-up ideas. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Jens Hausherr <jens.hausherr@pflege.de>
Signed-off-by: Jens Hausherr <jens.hausherr@pflege.de>
jabbrwcky
force-pushed
the
fix/mysql-lock-wait-timeout
branch
from
July 23, 2026 12:58
a3c8bc3 to
b067c97
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of your changes
Under a backlog of pending
ALTER USER/GRANT/DDL statements, MySQL/Percona can become unresponsive.go-sql-driver/mysqlcancels a context by closing the TCP socket without sendingKILL QUERY(intentional; all versions incl. latest v1.10.0). With MySQL's defaultlock_wait_timeoutof 1 year, a statement blocked on a metadata/ACL lock (e.g. behind a backup or long transaction) stays pending server-side even after the 60s reconcile deadline. Every retry adds another; account-management statements serialize on a single global ACL lock, so they pile up untilmax_connectionsis exhausted or the server wedges.This PR appends
lock_wait_timeout=30and dialtimeout=10sto the MySQL DSN so blocked statements fail fast server-side and release instead of accumulating. Applies to all MySQL controllers (User, Grant, Database; cluster + namespaced) via the shared client.Not covered / follow-ups:
KILL QUERYis the only remedy.sql.Open) — separate concern, see Provider should use connection pooling or single transaction to minimize load on database when reconciling #195.lock_wait_timeoutconfigurable viaProviderConfig.See
docs/mysql-driver-context-cancellation.mdfor the full analysis.Fixes #422
I have:
make reviewableto ensure this PR is ready for review.How has this code been tested
TestDSNParsesWithLockWaitTimeoutvalidating the DSN parses via the driver and carries the params.go build ./...,go vet, and full MySQL controller test suites pass.makeandmake reviewablepass.