Upgrade aliyun-datahub-sdk-go to v1.2.0 and switch to Producer API - #223
Merged
Conversation
The new version fixes a nil pointer dereference panic in topicSchemaCacheImpl.getVersionIdBySchema when schema cache initialization fails (e.g. transient GetTopic error), which crashed the whole process during feature log reporting.
Switch from raw PutRecordsByShard with hand-written shard management to the recommended Producer API: - Producer.Init() fails fast on init error instead of leaving a broken schema cache behind - shard selection, sealed shard recovery and retryable error handling (network/limit exceeded) are managed by producer - fetch latest topic schema via producer.GetSchema() with the schema cached at Init as fallback - log SetValueByName errors instead of silently dropping them - remove loopListShards goroutine and unsynchronized d.shards / d.active fields which caused data races Sync log fallback on send failure keeps unchanged.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
symphony09
approved these changes
Jul 29, 2026
Init() has two network failure points(producer.Init and GetSchema), when it fails the caller may still hold a Datahub whose producer and syncLog are nil. Calling SendMessage on such instance panicked on the nil producer, and the sync log fallback panicked again on nil syncLog. Feature log is sent from a bare goroutine, so this panic could not be recovered by the upper layer and crashed the whole process. The old implementation degraded safely by returning on empty shard list, so restore that behavior: - guard producer/syncLog with nil checks and degrade with an error log - extract writeSyncLog to share the guarded sync log fallback - do not register a half initialized instance in feature consistency job service, so the existing nil check can take effect Add a regression test that reproduces the panic without the guards.
…ahub" This reverts commit e65a0a7.
This reverts commit 1d06c82.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
In production we hit a nil pointer dereference panic inside the DataHub SDK v1.1.4 during feature log reporting, which crashed the whole process:
Root cause: in v1.1.4, when the lazy schema cache initialization fails (e.g. a transient
GetTopicerror), the broken cache (topicResult == nil) is still registered andgetVersionIdBySchemadereferencestsc.topicResultwithout a nil check. The refresh window is also locked for 5 minutes, so every write during that window panics.Changes
1. Upgrade SDK v1.1.4 -> v1.2.0
v1.2.0 fixes the root cause: schema cache getters return errors instead of panicking, a cache that failed to initialize is no longer registered, and failed refreshes retry after 1s instead of 5min.
2. Refactor datasource/datahub to the recommended Producer API
Switch from raw
PutRecordsByShardwith hand-written shard management toProducer:Producer.Init()fails fast on init error instead of leaving a broken schema cache behindproducer.GetSchema()on each send, with the schema cached atInitas fallback, so schema evolution no longer breaks writesSetValueByNameerrors instead of silently dropping fieldsloopListShardsgoroutine and the unsynchronizedd.shards/d.activefields which caused data racesThe sync-log fallback on send failure keeps unchanged, and all public interfaces (
NewDatahub/Init/GetDatahub/SendMessage/Shards/Stop/StopLoopListShards) keep their signatures, so all callers (feature_log, callback service, feature consistency job, embedding service, debug service) are unaffected.Behavior changes
Init()now fails fast (previously the service started but writes were dropped with an error log).compressorTypeconfig still applies to the admin client used for topic creation.Verification
go build ./...,go vetandgo test ./datasource/datahub/...passgofmtclean