Skip to content

Migrate Gentoo importer to advisory V2 #2090

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
ziadhany wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Migrate Gentoo importer to advisory V2 #2090

ziadhany wants to merge 8 commits into from

Conversation

@ziadhany
Copy link
Collaborator

@ziadhany ziadhany commented Dec 30, 2025 •
edited
Loading

@ziadhany ziadhany changed the title Add initial migration to Gentoo importer v2 Migrate Gentoo importer to advisory V2 Dec 30, 2025
@ziadhany ziadhany marked this pull request as ready for review January 1, 2026 13:58
@ziadhany
Copy link
Collaborator Author

ziadhany commented Jan 1, 2026 •
edited
Loading

gentoo importer V2 logs:

/home/ziad-hany/PycharmProjects/vulnerablecode/venv/bin/python /home/ziad-hany/PycharmProjects/vulnerablecode/manage.py import localhost:8000 --all 
INFO 2026-02-02 17:54:38.166692 UTC Pipeline [GentooImporterPipeline] starting
INFO 2026-02-02 17:54:38.166802 UTC Step [clone] starting
INFO 2026-02-02 17:54:38.166832 UTC Cloning `git+https://anongit.gentoo.org/git/data/glsa.git`
Importing data using gentoo_importer_v2
INFO 2026-02-02 17:54:43.419509 UTC Step [clone] completed in 5 seconds
INFO 2026-02-02 17:54:43.419620 UTC Step [collect_and_store_advisories] starting
INFO 2026-02-02 17:54:43.447538 UTC Collecting 3,814 advisories
INFO 2026-02-02 17:54:47.695981 UTC Progress: 10% (382/3814) ETA: 38 seconds
INFO 2026-02-02 17:54:51.021923 UTC InvalidVersion constraints version: 1.3* error:'1.3*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:54:51.826717 UTC Progress: 20% (763/3814) ETA: 34 seconds
INFO 2026-02-02 17:54:56.169353 UTC Progress: 30% (1145/3814) ETA: 30 seconds
INFO 2026-02-02 17:54:58.417666 UTC InvalidVersion constraints version: 3.24.48:3 error:'3.24.48:3' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:54:58.417815 UTC InvalidVersion constraints version: 3.24.48:3 error:'3.24.48:3' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:00.338608 UTC Progress: 40% (1526/3814) ETA: 25 seconds
INFO 2026-02-02 17:55:04.620972 UTC Progress: 50% (1907/3814) ETA: 21 seconds
INFO 2026-02-02 17:55:06.363401 UTC InvalidVersion constraints version: 7.4* error:'7.4*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:06.363539 UTC InvalidVersion constraints version: 7.3* error:'7.3*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:08.923560 UTC Progress: 60% (2289/3814) ETA: 17 seconds
INFO 2026-02-02 17:55:09.268841 UTC InvalidVersion constraints version: 7.4* error:'7.4*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:11.946939 UTC InvalidVersion constraints version: 6.9.3:6 error:'6.9.3:6' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:11.947075 UTC InvalidVersion constraints version: 6.9.3:6 error:'6.9.3:6' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:12.803704 UTC InvalidVersion constraints version: 7.4* error:'7.4*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:12.803886 UTC InvalidVersion constraints version: 7.3* error:'7.3*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:13.023839 UTC Progress: 70% (2670/3814) ETA: 13 seconds
INFO 2026-02-02 17:55:14.632720 UTC InvalidVersion constraints version: 7.4* error:'7.4*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:14.632854 UTC InvalidVersion constraints version: 7.3* error:'7.3*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:16.977535 UTC Progress: 80% (3052/3814) ETA: 8 seconds
INFO 2026-02-02 17:55:21.247673 UTC Progress: 90% (3433/3814) ETA: 4 seconds
INFO 2026-02-02 17:55:25.279534 UTC Progress: 100% (3814/3814)
INFO 2026-02-02 17:55:25.291572 UTC Successfully collected 3,814 advisories
INFO 2026-02-02 17:55:25.291680 UTC Step [collect_and_store_advisories] completed in 42 seconds
INFO 2026-02-02 17:55:25.291715 UTC Step [clean_downloads] starting
INFO 2026-02-02 17:55:25.291739 UTC Removing cloned repository
INFO 2026-02-02 17:55:25.328524 UTC Step [clean_downloads] completed in 0 seconds
INFO 2026-02-02 17:55:25.328685 UTC Pipeline completed in 47 seconds

Process finished with exit code 0

from vulnerabilities.models import AdvisoryV2
from django.db.models import Count
duplicates = (
    AdvisoryV2.objects
    .values('avid')
    .annotate(count=Count('id'))
    .filter(count__gt=1)
)
len(duplicates)
Out[2]: 0
AdvisoryV2.objects.count()
Out[3]: 3814

gentoo importer V1 logs:

Importing data using vulnerabilities.importers.gentoo.GentooImporter
Invalid safe_version 3.24.48:3 - error: '3.24.48:3' is not a valid <class 'univers.versions.GentooVersion'>
Invalid safe_version 6.9.3:6 - error: '6.9.3:6' is not a valid <class 'univers.versions.GentooVersion'>
Successfully imported data using vulnerabilities.importers.gentoo.GentooImporter

@ziadhany
Copy link
Collaborator Author

ziadhany commented Jan 1, 2026

Related issue:

@ziadhany ziadhany force-pushed the gentoo-migration branch 2 times, most recently from 76f65a8 to 0bed9dd Compare January 12, 2026 23:46
@ziadhany ziadhany mentioned this pull request Jan 13, 2026
Open
@ziadhany ziadhany requested a review from TG1999 January 13, 2026 23:31
@ziadhany
Add initial migration to Gentoo importer v2
1f3ad4d 
Signed-off-by: ziad hany <[email protected]>
@ziadhany
Add a test for gentoo importer v2
f3bec2a 
Signed-off-by: ziad hany <[email protected]>
@ziadhany
Fix Gentoo importer v1
ea7434f 
Update the Gentoo get_safe_and_affected_versions function in advisory v2

Signed-off-by: ziad hany <[email protected]>
@ziadhany
Fix code style
f416c5d 
Signed-off-by: ziad hany <[email protected]>
@ziadhany
Avoid duplicate affected_packages
002300d 
Signed-off-by: ziad hany <[email protected]>
@ziadhany
Avoid skipping some versions in Gentoo importer
5ef6840 
Signed-off-by: ziad hany <[email protected]>
@ziadhany
Update gentoo importer to include rle, rgt, and rge versions.
a5e7bd4 
Signed-off-by: ziad hany <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@TG1999 TG1999 Awaiting requested review from TG1999

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ziadhany