Add predefined field whitelist validation and parallel processing for dump#9
Merged
Conversation
… dump Additionally, implemented concurrent query processing in the `dump` command by introducing workers and rate limiting, significantly improving data extraction performance. Key changes: - Created official field whitelists (ValidFieldsAll and ValidFieldsNext) in fields.go. - Added validation logic (ValidateFieldsAll, ValidateFieldsNext) and integrated it into the search, dump, and random commands. - Enhanced the `dump` command with a worker pool for concurrency and thread-safe data writing mechanics. - Included comprehensive unit test coverage for the validation and concurrency logic in fields_test.go and dump_test.go. - Updated README.md and README_ZH.md to natively reference the official FOFA API documentation URLs for valid fields.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add predefined field whitelist validation.
Additionally, implemented concurrent query processing in the
dumpcommand by introducing workers and rate limiting, significantly improving data extraction performance.Key changes:
dumpcommand with a worker pool for concurrency and thread-safe data writing mechanics.