Refactor, extend robots and machines lists, and add datacenter/browser access detection - #20
Open
slint wants to merge 9 commits into
Open
Refactor, extend robots and machines lists, and add datacenter/browser access detection#20slint wants to merge 9 commits into
slint wants to merge 9 commits into
Conversation
Restructure the flat module functions into a Classifier built by a fluent ClassifierBuilder, so lists can be extended without editing the package. Lists are added as presets, callables that add sources to the builder, applied with builder.use(preset); counter_preset is the generic COUNTER baseline. A deployment can add its own patterns with builder.robots(file_patterns(path), ignore_case=True). _PatternSet supports both case-sensitive and case-insensitive matching. The module-level is_robot / is_machine / is_robot_or_machine remain, backed by a default classifier built from counter_preset, so existing callers are unchanged.
Add extended_preset, which builds on the COUNTER baseline using the maintained crawler-user-agents dataset (added as a dependency, so updates arrive through the package). Rather than its single mixed regex, the preset reads the raw entries and splits them by tag: http-library and browser-automation tools become machines, every other tag becomes a robot, matched case-sensitively as the dataset intends. This catches the modern crawlers (CamelCase bots, AI crawlers, link-preview bots) the frozen, case-sensitive baseline lists miss. Also add a curated machine_extra.txt of non-browser tools and CLIs the dataset does not cover (HTTP clients and libraries, download managers, scientific and bioinformatics tools, reference managers, runtimes), gathered from research-data repository access logs and matched case-insensitively. These are reusable beyond any one instance; deployment-specific patterns still go through the builder.
Add network-origin detection to the classifier, to catch automation that fakes a real browser user agent but runs from cloud/hosting infrastructure. is_datacenter classifies an ASN against datacenter_asn.txt (generated by scripts/update-asn-list.py from the union of brianhama/bad-asn-list and PeeringDB Content networks) minus an allow list (Apple iCloud Private Relay). is_browser exposes a conservative browser-UA heuristic. extended_preset also adds the datacenter ASN lists. is_datacenter_ip resolves an IP to its ASN through a resolver supplied to the builder; maxminddb_resolver (the asn extra) builds one over a GeoLite2-ASN mmdb with an in-memory cache. The library reads no config and ships no geo database; only ASN numbers are vendored.
The README only described the package purpose. Add usage for the module-level functions, the composable ClassifierBuilder with the counter and extended presets, how a deployment adds its own lists through the builder, and the datacenter ASN detection (is_datacenter / is_datacenter_ip / maxminddb_resolver and the asn extra). Refresh the data-sources section to cover machine_extra.txt, the crawler-user-agents dependency, and the datacenter ASN list and its sources.
The COUNTER robot list and the Make-Data-Count machine list overlap almost entirely (34 of the 35 machine patterns are also in the robot list), so wget, curl, python and friends matched both is_robot and is_machine. With filter_robots running before flag_machines downstream, that traffic was excluded as robots instead of counted as machine access. COUNTER CoP for Research Data 10.5 requires the robot blacklist to leave out the general-purpose agents researchers use and to stay a subset of the COUNTER list. is_robot now returns false when the machine list also matches, so a user agent in both lists is a machine. is_robot_or_machine is unchanged. Closes inveniosoftware#15
update-asn-list.py now unions a third source into datacenter_asn.txt: the hosting, cloud and cdn ASN categories from O-X-L/open-bot-list. The isp, education, proxy and vpn categories are skipped because they carry human traffic. This adds ~970 ASNs (3158 -> 4129) that bad-asn-list and PeeringDB Content did not cover. open-bot-list data is BSD-3-Clause (OXL IT Services), credited in the generated file header. The script is run by maintainers and the library only ever loads the committed list, so there is no runtime dependency on the upstream repo.
scripts/update-lists.py now also fetches ai-robots-txt/ai.robots.txt and writes the bot names to counter_robots/data/ai_robots.txt (regex-escaped, one per line). extended_preset loads them as case-insensitive robot patterns, so AI crawlers the COUNTER baseline and crawler-user-agents miss (Crawlspace, QuillBot, Andibot, bedrockbot, ...) get flagged. ai.robots.txt is MIT.
open-bot-list files 13 ASNs under both a datacenter (hosting/cloud/cdn) and its isp category; 11 reached our list only through that union (China Telecom/Unicom, Rostelecom, SK Broadband, VNPT, ...). The generator now subtracts open-bot-list's own isp/education set from its datacenter contribution (4129 -> 4118 ASNs). Separately, bad-asn-list flags some networks PeeringDB confirms are eyeball or academic (Airtel Nigeria, University of Lagos, NigComSat, Only, two universities); these go in datacenter_asn_allow.txt. PeeringDB Content self-declarations stay.
open-bot-list flags these five as isp and PeeringDB does not declare them Content, so they likely carry real browser users (China Telecom, Phase3 Telecom, IPTELECOM ASIA, TEL Communications, Powerhouse Management). Excluding a real user's events is worse than counting the occasional bot, so they go on the allow list. The 31 isp-labeled ASNs that PeeringDB-Content self-declares as hosting stay excluded: an operator registers eyeball customers under separate ISP-typed ASNs, so a declared-Content ASN does not carry that operator's real users.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix: machine takes precedence over robot
The COUNTER robot list and the Make-Data-Count machine list overlap almost
entirely (34 of the 35 machine patterns are also in the robot list), so wget,
curl, python and friends matched both is_robot and is_machine. With filter_robots
running before flag_machines downstream, that traffic was excluded as robots
instead of counted as machine access.
COUNTER CoP for Research Data 10.5 requires the robot blacklist to leave out the
general-purpose agents researchers use and to stay a subset of the COUNTER list.
is_robot now returns false when the machine list also matches, so a user agent in
both lists is a machine. is_robot_or_machine is unchanged.
refactor: composable, class-based Classifier API
Restructure the flat module functions into a Classifier built by a fluent
ClassifierBuilder, so lists can be extended without editing the package. Lists are
added as presets, callables that add sources to the builder, applied with
builder.use(preset); counter_preset is the generic COUNTER baseline. A deployment
can add its own patterns with builder.robots(file_patterns(path), ignore_case=True).
_PatternSet supports both case-sensitive and case-insensitive matching.
The module-level is_robot / is_machine / is_robot_or_machine remain, backed by a
default classifier built from counter_preset, so existing callers are unchanged.
feat: extended preset from the crawler-user-agents dataset
Add extended_preset, which builds on the COUNTER baseline using the maintained
crawler-user-agents dataset (added as a dependency, so updates arrive through the
package). Rather than its single mixed regex, the preset reads the raw entries and
splits them by tag: http-library and browser-automation tools become machines,
every other tag becomes a robot, matched case-sensitively as the dataset intends.
This catches the modern crawlers (CamelCase bots, AI crawlers, link-preview bots)
the frozen, case-sensitive baseline lists miss.
Also add a small curated machine_extra.txt of non-browser tools and CLIs the
dataset does not cover (geospatial, astronomy, bioinformatics and workflow tools,
reference managers, download managers), matched case-insensitively. These were
observed accessing research-data repositories and are reusable beyond any one
instance; deployment-specific patterns still go through the builder.
feat: datacenter/hosting ASN classification
Add network-origin detection to the classifier, to catch automation that fakes a
real browser user agent but runs from cloud/hosting infrastructure. is_datacenter
classifies an ASN against datacenter_asn.txt (generated by scripts/update-asn-list.py
from the union of brianhama/bad-asn-list and PeeringDB Content networks) minus an
allow list (Apple iCloud Private Relay). is_browser exposes a conservative
browser-UA heuristic. extended_preset also adds the datacenter ASN lists.
is_datacenter_ip resolves an IP to its ASN through a resolver supplied to the
builder; maxminddb_resolver (the asn extra) builds one over a GeoLite2-ASN mmdb with
an in-memory cache. The library reads no config and ships no geo database; only ASN
numbers are vendored.