Skip to content

Refactor, extend robots and machines lists, and add datacenter/browser access detection - #20

Open
slint wants to merge 9 commits into
inveniosoftware:masterfrom
slint:refactor
Open

Refactor, extend robots and machines lists, and add datacenter/browser access detection#20
slint wants to merge 9 commits into
inveniosoftware:masterfrom
slint:refactor

Conversation

@slint

@slint slint commented Jun 11, 2026

Copy link
Copy Markdown
Member

AI disclosure: did some iterations/exploration with Claude on the Classifier "builder" API to arrive at what is now committed. Also used some smaller Claude subagents to batch-classify and split some of the more generic UAs in machine_extra.txt from the original set dumped from the Zenodo web access logs.

  • fix: machine takes precedence over robot
    The COUNTER robot list and the Make-Data-Count machine list overlap almost
    entirely (34 of the 35 machine patterns are also in the robot list), so wget,
    curl, python and friends matched both is_robot and is_machine. With filter_robots
    running before flag_machines downstream, that traffic was excluded as robots
    instead of counted as machine access.

    COUNTER CoP for Research Data 10.5 requires the robot blacklist to leave out the
    general-purpose agents researchers use and to stay a subset of the COUNTER list.
    is_robot now returns false when the machine list also matches, so a user agent in
    both lists is a machine. is_robot_or_machine is unchanged.

  • refactor: composable, class-based Classifier API
    Restructure the flat module functions into a Classifier built by a fluent
    ClassifierBuilder, so lists can be extended without editing the package. Lists are
    added as presets, callables that add sources to the builder, applied with
    builder.use(preset); counter_preset is the generic COUNTER baseline. A deployment
    can add its own patterns with builder.robots(file_patterns(path), ignore_case=True).
    _PatternSet supports both case-sensitive and case-insensitive matching.

    The module-level is_robot / is_machine / is_robot_or_machine remain, backed by a
    default classifier built from counter_preset, so existing callers are unchanged.

  • feat: extended preset from the crawler-user-agents dataset
    Add extended_preset, which builds on the COUNTER baseline using the maintained
    crawler-user-agents dataset (added as a dependency, so updates arrive through the
    package). Rather than its single mixed regex, the preset reads the raw entries and
    splits them by tag: http-library and browser-automation tools become machines,
    every other tag becomes a robot, matched case-sensitively as the dataset intends.
    This catches the modern crawlers (CamelCase bots, AI crawlers, link-preview bots)
    the frozen, case-sensitive baseline lists miss.

    Also add a small curated machine_extra.txt of non-browser tools and CLIs the
    dataset does not cover (geospatial, astronomy, bioinformatics and workflow tools,
    reference managers, download managers), matched case-insensitively. These were
    observed accessing research-data repositories and are reusable beyond any one
    instance; deployment-specific patterns still go through the builder.

  • feat: datacenter/hosting ASN classification
    Add network-origin detection to the classifier, to catch automation that fakes a
    real browser user agent but runs from cloud/hosting infrastructure. is_datacenter
    classifies an ASN against datacenter_asn.txt (generated by scripts/update-asn-list.py
    from the union of brianhama/bad-asn-list and PeeringDB Content networks) minus an
    allow list (Apple iCloud Private Relay). is_browser exposes a conservative
    browser-UA heuristic. extended_preset also adds the datacenter ASN lists.

    is_datacenter_ip resolves an IP to its ASN through a resolver supplied to the
    builder; maxminddb_resolver (the asn extra) builds one over a GeoLite2-ASN mmdb with
    an in-memory cache. The library reads no config and ships no geo database; only ASN
    numbers are vendored.

Restructure the flat module functions into a Classifier built by a fluent
ClassifierBuilder, so lists can be extended without editing the package. Lists are
added as presets, callables that add sources to the builder, applied with
builder.use(preset); counter_preset is the generic COUNTER baseline. A deployment
can add its own patterns with builder.robots(file_patterns(path), ignore_case=True).
_PatternSet supports both case-sensitive and case-insensitive matching.

The module-level is_robot / is_machine / is_robot_or_machine remain, backed by a
default classifier built from counter_preset, so existing callers are unchanged.
@slint slint changed the title refactor Refactor, extend robots and machines lists, and add datacenter/browser access detection Jun 11, 2026
Add extended_preset, which builds on the COUNTER baseline using the maintained
crawler-user-agents dataset (added as a dependency, so updates arrive through the
package). Rather than its single mixed regex, the preset reads the raw entries and
splits them by tag: http-library and browser-automation tools become machines,
every other tag becomes a robot, matched case-sensitively as the dataset intends.
This catches the modern crawlers (CamelCase bots, AI crawlers, link-preview bots)
the frozen, case-sensitive baseline lists miss.

Also add a curated machine_extra.txt of non-browser tools and CLIs the dataset does
not cover (HTTP clients and libraries, download managers, scientific and
bioinformatics tools, reference managers, runtimes), gathered from research-data
repository access logs and matched case-insensitively. These are reusable beyond any
one instance; deployment-specific patterns still go through the builder.
slint added 7 commits June 11, 2026 21:52
Add network-origin detection to the classifier, to catch automation that fakes a
real browser user agent but runs from cloud/hosting infrastructure. is_datacenter
classifies an ASN against datacenter_asn.txt (generated by scripts/update-asn-list.py
from the union of brianhama/bad-asn-list and PeeringDB Content networks) minus an
allow list (Apple iCloud Private Relay). is_browser exposes a conservative
browser-UA heuristic. extended_preset also adds the datacenter ASN lists.

is_datacenter_ip resolves an IP to its ASN through a resolver supplied to the
builder; maxminddb_resolver (the asn extra) builds one over a GeoLite2-ASN mmdb with
an in-memory cache. The library reads no config and ships no geo database; only ASN
numbers are vendored.
The README only described the package purpose. Add usage for the module-level
functions, the composable ClassifierBuilder with the counter and extended presets,
how a deployment adds its own lists through the builder, and the datacenter ASN
detection (is_datacenter / is_datacenter_ip / maxminddb_resolver and the asn
extra). Refresh the data-sources section to cover machine_extra.txt, the
crawler-user-agents dependency, and the datacenter ASN list and its sources.
The COUNTER robot list and the Make-Data-Count machine list overlap almost
entirely (34 of the 35 machine patterns are also in the robot list), so wget,
curl, python and friends matched both is_robot and is_machine. With filter_robots
running before flag_machines downstream, that traffic was excluded as robots
instead of counted as machine access.

COUNTER CoP for Research Data 10.5 requires the robot blacklist to leave out the
general-purpose agents researchers use and to stay a subset of the COUNTER list.
is_robot now returns false when the machine list also matches, so a user agent in
both lists is a machine. is_robot_or_machine is unchanged.

Closes inveniosoftware#15
update-asn-list.py now unions a third source into datacenter_asn.txt: the
hosting, cloud and cdn ASN categories from O-X-L/open-bot-list. The isp,
education, proxy and vpn categories are skipped because they carry human
traffic. This adds ~970 ASNs (3158 -> 4129) that bad-asn-list and PeeringDB
Content did not cover.

open-bot-list data is BSD-3-Clause (OXL IT Services), credited in the generated
file header. The script is run by maintainers and the library only ever loads
the committed list, so there is no runtime dependency on the upstream repo.
scripts/update-lists.py now also fetches ai-robots-txt/ai.robots.txt and writes
the bot names to counter_robots/data/ai_robots.txt (regex-escaped, one per line).
extended_preset loads them as case-insensitive robot patterns, so AI crawlers the
COUNTER baseline and crawler-user-agents miss (Crawlspace, QuillBot, Andibot,
bedrockbot, ...) get flagged. ai.robots.txt is MIT.
open-bot-list files 13 ASNs under both a datacenter (hosting/cloud/cdn) and its
isp category; 11 reached our list only through that union (China Telecom/Unicom,
Rostelecom, SK Broadband, VNPT, ...). The generator now subtracts open-bot-list's
own isp/education set from its datacenter contribution (4129 -> 4118 ASNs).

Separately, bad-asn-list flags some networks PeeringDB confirms are eyeball or
academic (Airtel Nigeria, University of Lagos, NigComSat, Only, two universities);
these go in datacenter_asn_allow.txt. PeeringDB Content self-declarations stay.
open-bot-list flags these five as isp and PeeringDB does not declare them
Content, so they likely carry real browser users (China Telecom, Phase3 Telecom,
IPTELECOM ASIA, TEL Communications, Powerhouse Management). Excluding a real
user's events is worse than counting the occasional bot, so they go on the allow
list.

The 31 isp-labeled ASNs that PeeringDB-Content self-declares as hosting stay
excluded: an operator registers eyeball customers under separate ISP-typed ASNs,
so a declared-Content ASN does not carry that operator's real users.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant