Skip to content

chore(infra): migrate from Elasticsearch to OpenSearch#216

Open
rerowep wants to merge 1 commit into
rero:stagingfrom
rerowep:wep-os
Open

chore(infra): migrate from Elasticsearch to OpenSearch#216
rerowep wants to merge 1 commit into
rero:stagingfrom
rerowep:wep-os

Conversation

@rerowep

@rerowep rerowep commented May 19, 2026

Copy link
Copy Markdown
Contributor
  • replace docker/elasticsearch/Dockerfile with OPENSEARCH_PLUGINS=analysis-icu env var — no custom image needed
  • rename docker-compose service es → os; update all links and INVENIO_SEARCH_HOSTS in docker-services.yml and docker-compose.full.yml
  • add OPENSEARCH_INITIAL_ADMIN_PASSWORD to satisfy OpenSearch 2.12+ entrypoint check even when plugins.security.disabled=true
  • switch rero-invenio-base to git branch wep-opensearch to remove transitive elasticsearch-py 7.x dependency and unblock urllib3 upgrade to 2.7
  • drop five now-resolved urllib3 CVE exceptions from scripts/test (CVE-2025-50181, CVE-2025-66418, CVE-2025-66471, CVE-2026-21441, CVE-2026-44431); update pytest CVE comment to name correct blocker (pytest-invenio 4.0.1 pins pytest<9.0.0)
  • replace "Elasticsearch"/"elasticsearch" with "Search"/"search" in all docstrings and comments throughout rero_mef/ and tests/
  • update CLAUDE.md stack description: Elasticsearch → OpenSearch

Summary by CodeRabbit

  • New Features

    • Added OpenSearch 2.x mapping support for agents, concepts, and places.
    • Added a periodic monitoring task to collect and index OpenSearch cluster/node stats.
  • Updates

    • Migrated search backend integration to OpenSearch.
    • Updated UI/CLI/messages and docs to use “Search/OpenSearch” terminology.
    • Improved search-related error handling and monitoring outputs.

Review Change Stack

@coderabbitai

coderabbitai Bot commented May 19, 2026

Copy link
Copy Markdown

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Replaces Elasticsearch client/DSL imports and exceptions with OpenSearch equivalents, updates mapping/docstring terminology, adds os-v2 mapping initializers, and implements a Celery task with tests to index OpenSearch cluster/node statistics.

Changes

Elasticsearch to OpenSearch Migration

Layer / File(s) Summary
Backend client imports & exceptions
rero_mef/api.py, rero_mef/cli.py, rero_mef/ext.py, rero_mef/views.py, rero_mef/monitoring/api.py, tests/ui/*
Replaces elasticsearch imports with opensearchpy equivalents and updates exception handling, bulk helper imports, and test mocks.
Query DSL imports and integration
rero_mef/agents/viaf/api.py, rero_mef/api_mef.py, rero_mef/filter.py, rero_mef/query.py, tests/*
Switches Q imports from elasticsearch_dsl.query to opensearch_dsl.query in modules and tests that build queries.
OpenSearch 2 mapping modules (os-v2)
rero_mef/*/mappings/os-v2/__init__.py
Adds or updates os-v2 package initializers and docstrings to describe OpenSearch 2 mapping definitions for agents, concepts, and places.
Mapping & core docstring terminology
rero_mef/*/mappings/*, rero_mef/agents/__init__.py, rero_mef/concepts/__init__.py, rero_mef/places/__init__.py, rero_mef/agents/api.py
Systematic docstring updates replacing "Elasticsearch" with "Search"/"OpenSearch" across mapping packages, indexer module docstrings, and small API docstrings.
Monitoring task
rero_mef/monitoring/tasks.py
Adds index_os_stats Celery task with helpers _last_snapshot and _rate that query OpenSearch cluster/node stats, compute per-second rates, and index per-node documents into daily os-monitor-YYYY.MM.dd indices.
Monitoring tests and mocks
tests/ui/test_monitoring.py
New unit tests that mock an OpenSearch client to validate snapshot retrieval, rate computations, indexing behavior, and health status mapping.
Tests and fixtures updates
tests/api/*, tests/ui/*, tests/conftest.py
Update test imports to opensearchpy/opensearch_dsl and adjust fixture/docstring wording to reference search/OpenSearch where applicable.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

  • rero/rero-mef#205: Related changes touching all-mef alias/search exception handling paths in rero_mef/views.py and rero_mef/ext.py.

Suggested reviewers

  • Garfield-fr
  • PascalRepond
  • jma

Poem

🐰 From elastic roots we hop to OpenSearch ground,

imports swapped and docstrings newly crowned,
a task that watches nodes by day,
mappings rebadged and tests at play,
the rabbit hums — search metrics safe and sound!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The pull request title accurately summarizes the main infrastructure change: migrating the project from Elasticsearch to OpenSearch across the codebase.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coveralls

coveralls commented May 19, 2026

Copy link
Copy Markdown
Collaborator

Coverage Status

coverage: 87.403% (+0.08%) from 87.325% — rerowep:wep-os into rero:staging

@rerowep rerowep force-pushed the wep-os branch 2 times, most recently from 90c7f04 to 7ae371f Compare May 19, 2026 14:17

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rero_mef/monitoring/tasks.py`:
- Around line 43-44: Replace the silent handler "except Exception: return None"
in the snapshot lookup (the shown except block in rero_mef/monitoring/tasks.py)
with a guarded catch that captures the exception as e and logs it (e.g.,
logger.exception or logger.error with traceback) before returning/handling the
fallback; specifically change the bare except to "except Exception as e:" and
call the module's logger (or monitoring logger) to record the error and context
(search query, client info, function name) so OpenSearch search/client failures
are visible.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: d1b7e221-99f9-4270-aa53-9e98d690c854

📥 Commits

Reviewing files that changed from the base of the PR and between 90c7f04 and 7ae371f.

⛔ Files ignored due to path filters (26)
  • CLAUDE.md is excluded by none and included by none
  • INSTALL.md is excluded by none and included by none
  • docker-compose.full.yml is excluded by none and included by none
  • docker-compose.yml is excluded by none and included by none
  • docker-services.yml is excluded by none and included by none
  • docker/dashboards/mef-dashboard.ndjson is excluded by none and included by none
  • docker/elasticsearch/Dockerfile is excluded by none and included by none
  • docker/wait-for-services.sh is excluded by none and included by none
  • pyproject.toml is excluded by none and included by none
  • rero_mef/agents/gnd/mappings/os-v2/agents_gnd/gnd-agent-v0.0.1.json is excluded by none and included by none
  • rero_mef/agents/idref/mappings/os-v2/agents_idref/idref-agent-v0.0.1.json is excluded by none and included by none
  • rero_mef/agents/mef/mappings/os-v2/mef/mef-v0.0.1.json is excluded by none and included by none
  • rero_mef/agents/rero/mappings/os-v2/agents_rero/rero-agent-v0.0.1.json is excluded by none and included by none
  • rero_mef/agents/viaf/mappings/os-v2/viaf/viaf-v0.0.1.json is excluded by none and included by none
  • rero_mef/concepts/gnd/mappings/os-v2/concepts_gnd/gnd-concept-v0.0.1.json is excluded by none and included by none
  • rero_mef/concepts/idref/mappings/os-v2/concepts_idref/idref-concept-v0.0.1.json is excluded by none and included by none
  • rero_mef/concepts/mef/mappings/os-v2/concepts_mef/mef-concept-v0.0.1.json is excluded by none and included by none
  • rero_mef/concepts/rero/mappings/os-v2/concepts_rero/rero-concept-v0.0.1.json is excluded by none and included by none
  • rero_mef/config.py is excluded by !rero_mef/config.py and included by rero_mef/**/*.py
  • rero_mef/places/gnd/mappings/os-v2/places_gnd/gnd-place-v0.0.1.json is excluded by none and included by none
  • rero_mef/places/idref/mappings/os-v2/places_idref/idref-place-v0.0.1.json is excluded by none and included by none
  • rero_mef/places/mef/mappings/os-v2/places_mef/mef-place-v0.0.1.json is excluded by none and included by none
  • scripts/setup is excluded by none and included by none
  • scripts/setup-dashboards is excluded by none and included by none
  • scripts/test is excluded by none and included by none
  • uv.lock is excluded by !**/*.lock and included by none
📒 Files selected for processing (48)
  • rero_mef/agents/__init__.py
  • rero_mef/agents/api.py
  • rero_mef/agents/gnd/mappings/__init__.py
  • rero_mef/agents/gnd/mappings/os-v2/__init__.py
  • rero_mef/agents/idref/mappings/__init__.py
  • rero_mef/agents/idref/mappings/os-v2/__init__.py
  • rero_mef/agents/idref/mappings/v7/__init__.py
  • rero_mef/agents/mef/mappings/__init__.py
  • rero_mef/agents/mef/mappings/os-v2/__init__.py
  • rero_mef/agents/rero/mappings/__init__.py
  • rero_mef/agents/rero/mappings/os-v2/__init__.py
  • rero_mef/agents/viaf/api.py
  • rero_mef/agents/viaf/mappings/__init__.py
  • rero_mef/agents/viaf/mappings/os-v2/__init__.py
  • rero_mef/api.py
  • rero_mef/api_mef.py
  • rero_mef/cli.py
  • rero_mef/concepts/__init__.py
  • rero_mef/concepts/gnd/mappings/__init__.py
  • rero_mef/concepts/gnd/mappings/os-v2/__init__.py
  • rero_mef/concepts/idref/mappings/__init__.py
  • rero_mef/concepts/idref/mappings/os-v2/__init__.py
  • rero_mef/concepts/mef/mappings/__init__.py
  • rero_mef/concepts/mef/mappings/os-v2/__init__.py
  • rero_mef/concepts/rero/mappings/__init__.py
  • rero_mef/concepts/rero/mappings/os-v2/__init__.py
  • rero_mef/concepts/rero/mappings/v7/__init__.py
  • rero_mef/ext.py
  • rero_mef/filter.py
  • rero_mef/monitoring/__init__.py
  • rero_mef/monitoring/api.py
  • rero_mef/monitoring/cli.py
  • rero_mef/monitoring/tasks.py
  • rero_mef/monitoring/views.py
  • rero_mef/places/__init__.py
  • rero_mef/places/gnd/mappings/__init__.py
  • rero_mef/places/gnd/mappings/os-v2/__init__.py
  • rero_mef/places/idref/mappings/__init__.py
  • rero_mef/places/idref/mappings/os-v2/__init__.py
  • rero_mef/places/mef/mappings/__init__.py
  • rero_mef/places/mef/mappings/os-v2/__init__.py
  • rero_mef/query.py
  • rero_mef/tasks.py
  • rero_mef/views.py
  • tests/api/test_all_mef_rest.py
  • tests/api/test_entity_serializers.py
  • tests/conftest.py
  • tests/ui/test_ext.py
💤 Files with no reviewable changes (2)
  • rero_mef/agents/idref/mappings/v7/init.py
  • rero_mef/concepts/rero/mappings/v7/init.py
✅ Files skipped from review due to trivial changes (32)
  • rero_mef/concepts/gnd/mappings/os-v2/init.py
  • rero_mef/concepts/gnd/mappings/init.py
  • rero_mef/agents/init.py
  • rero_mef/places/idref/mappings/os-v2/init.py
  • rero_mef/agents/idref/mappings/os-v2/init.py
  • rero_mef/monitoring/init.py
  • rero_mef/agents/gnd/mappings/os-v2/init.py
  • rero_mef/concepts/rero/mappings/os-v2/init.py
  • rero_mef/places/mef/mappings/os-v2/init.py
  • rero_mef/agents/rero/mappings/init.py
  • rero_mef/places/idref/mappings/init.py
  • rero_mef/concepts/init.py
  • rero_mef/concepts/mef/mappings/os-v2/init.py
  • rero_mef/monitoring/cli.py
  • rero_mef/places/gnd/mappings/os-v2/init.py
  • rero_mef/concepts/idref/mappings/os-v2/init.py
  • rero_mef/agents/mef/mappings/init.py
  • rero_mef/concepts/mef/mappings/init.py
  • rero_mef/agents/api.py
  • tests/conftest.py
  • rero_mef/agents/idref/mappings/init.py
  • rero_mef/concepts/idref/mappings/init.py
  • rero_mef/agents/viaf/mappings/init.py
  • rero_mef/places/init.py
  • rero_mef/places/gnd/mappings/init.py
  • rero_mef/agents/mef/mappings/os-v2/init.py
  • rero_mef/query.py
  • rero_mef/places/mef/mappings/init.py
  • rero_mef/concepts/rero/mappings/init.py
  • rero_mef/agents/gnd/mappings/init.py
  • rero_mef/agents/viaf/mappings/os-v2/init.py
  • rero_mef/agents/rero/mappings/os-v2/init.py
🚧 Files skipped from review as they are similar to previous changes (12)
  • tests/api/test_entity_serializers.py
  • rero_mef/tasks.py
  • rero_mef/api_mef.py
  • tests/ui/test_ext.py
  • tests/api/test_all_mef_rest.py
  • rero_mef/filter.py
  • rero_mef/ext.py
  • rero_mef/monitoring/views.py
  • rero_mef/monitoring/api.py
  • rero_mef/views.py
  • rero_mef/cli.py
  • rero_mef/api.py

Comment thread rero_mef/monitoring/tasks.py Outdated
@rerowep rerowep force-pushed the wep-os branch 3 times, most recently from e679982 to 070deda Compare May 20, 2026 08:10

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
rero_mef/ext.py (1)

85-93: ⚠️ Potential issue | 🔴 Critical | ⚡ Quick win

Fix invalid Python 3 exception syntax

Line 87 uses except RuntimeError, AttributeError: which is Python 2 syntax and invalid in Python 3. This will cause a SyntaxError and prevent the module from being imported.

Proposed fix
-            except RuntimeError, AttributeError:
+            except (RuntimeError, AttributeError):
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rero_mef/ext.py` around lines 85 - 93, The except clause uses Python 2 syntax
and must be updated: in the try/except around assigning current_search_client
(the block that logs "Skipping %s alias setup" using alias_name and app.logger),
replace the invalid "except RuntimeError, AttributeError:" with a Python
3-compatible form such as "except (RuntimeError, AttributeError):" (or separate
except blocks) so the module can import without a SyntaxError.
🧹 Nitpick comments (1)
tests/ui/test_monitoring.py (1)

303-309: ⚡ Quick win

Move function-local imports to module scope

Line 305 and Line 385 add deferred imports without a circular-import justification. Please move them to top-level imports to keep import behavior predictable and consistent.

Proposed diff
+import logging
+from datetime import UTC, datetime, timedelta
 from unittest.mock import MagicMock, patch
 
 import pytest
 from click.testing import CliRunner
@@
 def test_last_snapshot_returns_none_and_logs_on_exception(caplog):
     """Returns None and logs the exception when the search call fails."""
-    import logging
 
     client = _make_client(raise_on_search=True)
     with caplog.at_level(logging.ERROR, logger="rero_mef.monitoring.tasks"):
         result = _last_snapshot(client, "node1")
@@
 def test_index_os_stats_rates_computed_from_prev_snapshot():
     """Rates are non-zero when a previous snapshot provides a baseline."""
-    from datetime import UTC, datetime, timedelta
-
     prev_ts = (datetime.now(UTC) - timedelta(seconds=60)).strftime(
         "%Y-%m-%dT%H:%M:%S.000Z"
     )

As per coding guidelines: "**/*.py: Organize imports in order: standard library → third-party → local, sorted within groups, with all imports at the top of the file" and "**/*.py: Only use deferred (inside-function) imports when they genuinely break circular dependencies, and document why with a comment."

Also applies to: 385-389

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/ui/test_monitoring.py` around lines 303 - 309, This test file has
function-local imports (e.g., the "import logging" inside test_last_snapshot and
another deferred import around lines 385-389) that should be moved to
module-level; move these imports to the top of tests/ui/test_monitoring.py,
organize them into the standard library → third-party → local groups and sort
them within each group, and remove the in-function import statements from
functions like test_last_snapshot and the other test at ~385; if any import
truly breaks a circular dependency, keep it deferred but add a short comment
explaining why and reference the specific symbol(s) that require the deferred
import.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@rero_mef/ext.py`:
- Around line 85-93: The except clause uses Python 2 syntax and must be updated:
in the try/except around assigning current_search_client (the block that logs
"Skipping %s alias setup" using alias_name and app.logger), replace the invalid
"except RuntimeError, AttributeError:" with a Python 3-compatible form such as
"except (RuntimeError, AttributeError):" (or separate except blocks) so the
module can import without a SyntaxError.

---

Nitpick comments:
In `@tests/ui/test_monitoring.py`:
- Around line 303-309: This test file has function-local imports (e.g., the
"import logging" inside test_last_snapshot and another deferred import around
lines 385-389) that should be moved to module-level; move these imports to the
top of tests/ui/test_monitoring.py, organize them into the standard library →
third-party → local groups and sort them within each group, and remove the
in-function import statements from functions like test_last_snapshot and the
other test at ~385; if any import truly breaks a circular dependency, keep it
deferred but add a short comment explaining why and reference the specific
symbol(s) that require the deferred import.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 7480e9f3-d322-4ba0-95af-46fc6cbffc54

📥 Commits

Reviewing files that changed from the base of the PR and between e679982 and 070deda.

⛔ Files ignored due to path filters (26)
  • CLAUDE.md is excluded by none and included by none
  • INSTALL.md is excluded by none and included by none
  • docker-compose.full.yml is excluded by none and included by none
  • docker-compose.yml is excluded by none and included by none
  • docker-services.yml is excluded by none and included by none
  • docker/dashboards/mef-dashboard.ndjson is excluded by none and included by none
  • docker/elasticsearch/Dockerfile is excluded by none and included by none
  • docker/wait-for-services.sh is excluded by none and included by none
  • pyproject.toml is excluded by none and included by none
  • rero_mef/agents/gnd/mappings/os-v2/agents_gnd/gnd-agent-v0.0.1.json is excluded by none and included by none
  • rero_mef/agents/idref/mappings/os-v2/agents_idref/idref-agent-v0.0.1.json is excluded by none and included by none
  • rero_mef/agents/mef/mappings/os-v2/mef/mef-v0.0.1.json is excluded by none and included by none
  • rero_mef/agents/rero/mappings/os-v2/agents_rero/rero-agent-v0.0.1.json is excluded by none and included by none
  • rero_mef/agents/viaf/mappings/os-v2/viaf/viaf-v0.0.1.json is excluded by none and included by none
  • rero_mef/concepts/gnd/mappings/os-v2/concepts_gnd/gnd-concept-v0.0.1.json is excluded by none and included by none
  • rero_mef/concepts/idref/mappings/os-v2/concepts_idref/idref-concept-v0.0.1.json is excluded by none and included by none
  • rero_mef/concepts/mef/mappings/os-v2/concepts_mef/mef-concept-v0.0.1.json is excluded by none and included by none
  • rero_mef/concepts/rero/mappings/os-v2/concepts_rero/rero-concept-v0.0.1.json is excluded by none and included by none
  • rero_mef/config.py is excluded by !rero_mef/config.py and included by rero_mef/**/*.py
  • rero_mef/places/gnd/mappings/os-v2/places_gnd/gnd-place-v0.0.1.json is excluded by none and included by none
  • rero_mef/places/idref/mappings/os-v2/places_idref/idref-place-v0.0.1.json is excluded by none and included by none
  • rero_mef/places/mef/mappings/os-v2/places_mef/mef-place-v0.0.1.json is excluded by none and included by none
  • scripts/setup is excluded by none and included by none
  • scripts/setup-dashboards is excluded by none and included by none
  • scripts/test is excluded by none and included by none
  • uv.lock is excluded by !**/*.lock and included by none
📒 Files selected for processing (49)
  • rero_mef/agents/__init__.py
  • rero_mef/agents/api.py
  • rero_mef/agents/gnd/mappings/__init__.py
  • rero_mef/agents/gnd/mappings/os-v2/__init__.py
  • rero_mef/agents/idref/mappings/__init__.py
  • rero_mef/agents/idref/mappings/os-v2/__init__.py
  • rero_mef/agents/idref/mappings/v7/__init__.py
  • rero_mef/agents/mef/mappings/__init__.py
  • rero_mef/agents/mef/mappings/os-v2/__init__.py
  • rero_mef/agents/rero/mappings/__init__.py
  • rero_mef/agents/rero/mappings/os-v2/__init__.py
  • rero_mef/agents/viaf/api.py
  • rero_mef/agents/viaf/mappings/__init__.py
  • rero_mef/agents/viaf/mappings/os-v2/__init__.py
  • rero_mef/api.py
  • rero_mef/api_mef.py
  • rero_mef/cli.py
  • rero_mef/concepts/__init__.py
  • rero_mef/concepts/gnd/mappings/__init__.py
  • rero_mef/concepts/gnd/mappings/os-v2/__init__.py
  • rero_mef/concepts/idref/mappings/__init__.py
  • rero_mef/concepts/idref/mappings/os-v2/__init__.py
  • rero_mef/concepts/mef/mappings/__init__.py
  • rero_mef/concepts/mef/mappings/os-v2/__init__.py
  • rero_mef/concepts/rero/mappings/__init__.py
  • rero_mef/concepts/rero/mappings/os-v2/__init__.py
  • rero_mef/concepts/rero/mappings/v7/__init__.py
  • rero_mef/ext.py
  • rero_mef/filter.py
  • rero_mef/monitoring/__init__.py
  • rero_mef/monitoring/api.py
  • rero_mef/monitoring/cli.py
  • rero_mef/monitoring/tasks.py
  • rero_mef/monitoring/views.py
  • rero_mef/places/__init__.py
  • rero_mef/places/gnd/mappings/__init__.py
  • rero_mef/places/gnd/mappings/os-v2/__init__.py
  • rero_mef/places/idref/mappings/__init__.py
  • rero_mef/places/idref/mappings/os-v2/__init__.py
  • rero_mef/places/mef/mappings/__init__.py
  • rero_mef/places/mef/mappings/os-v2/__init__.py
  • rero_mef/query.py
  • rero_mef/tasks.py
  • rero_mef/views.py
  • tests/api/test_all_mef_rest.py
  • tests/api/test_entity_serializers.py
  • tests/conftest.py
  • tests/ui/test_ext.py
  • tests/ui/test_monitoring.py
💤 Files with no reviewable changes (2)
  • rero_mef/agents/idref/mappings/v7/init.py
  • rero_mef/concepts/rero/mappings/v7/init.py
✅ Files skipped from review due to trivial changes (29)
  • rero_mef/monitoring/init.py
  • rero_mef/concepts/mef/mappings/os-v2/init.py
  • rero_mef/agents/gnd/mappings/init.py
  • rero_mef/places/gnd/mappings/init.py
  • rero_mef/agents/viaf/mappings/init.py
  • rero_mef/places/mef/mappings/os-v2/init.py
  • rero_mef/concepts/gnd/mappings/init.py
  • rero_mef/places/mef/mappings/init.py
  • rero_mef/agents/idref/mappings/os-v2/init.py
  • rero_mef/concepts/idref/mappings/os-v2/init.py
  • rero_mef/agents/rero/mappings/init.py
  • rero_mef/agents/mef/mappings/os-v2/init.py
  • rero_mef/places/init.py
  • rero_mef/tasks.py
  • rero_mef/concepts/gnd/mappings/os-v2/init.py
  • tests/api/test_entity_serializers.py
  • rero_mef/concepts/rero/mappings/os-v2/init.py
  • rero_mef/agents/api.py
  • rero_mef/concepts/mef/mappings/init.py
  • rero_mef/agents/rero/mappings/os-v2/init.py
  • rero_mef/agents/mef/mappings/init.py
  • rero_mef/agents/idref/mappings/init.py
  • rero_mef/places/idref/mappings/os-v2/init.py
  • rero_mef/agents/init.py
  • rero_mef/places/idref/mappings/init.py
  • rero_mef/agents/gnd/mappings/os-v2/init.py
  • rero_mef/agents/viaf/mappings/os-v2/init.py
  • rero_mef/places/gnd/mappings/os-v2/init.py
  • rero_mef/concepts/idref/mappings/init.py

@rerowep rerowep force-pushed the wep-os branch 2 times, most recently from d09ef6e to 05d0559 Compare May 20, 2026 09:08
@rerowep rerowep marked this pull request as ready for review May 20, 2026 11:29
Comment thread CLAUDE.md
rero-mef is the Python/Flask backend for the RERO Metadata Enrichment Framework (MEF). It provides OAI harvesting, record processing, and metadata enrichment for library data pipelines. The project is built on Invenio, with PostgreSQL, Elasticsearch, Celery, and Redis for backend services. All development and task running is managed via `uv` and `poethepoet`.

**Stack:** Python 3.12, Flask (Invenio), PostgreSQL, Elasticsearch 7, Celery, Redis
**Stack:** Python 3.12, Flask (Invenio), PostgreSQL, OpenSearch 2, Celery, Redis

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python 3.14

Comment on lines +49 to +52
// Suppress the Dart Sass "legacy JS API" module warning — it fires for
// every SCSS file and is not actionable until upstream (bootstrap-sass,
// invenio-theme) migrates to the modern API.
ignoreWarnings: [/legacy JS API/],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the link to OpenSearch?

{
test: /\.(scss|css)$/,
use: [MiniCssExtractPlugin.loader, "css-loader", "sass-loader"],
use: [

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No obvious link to OpenSearch?

Comment thread pyproject.toml
"sqlitedict>=2.1.0",
"rero-invenio-base>=0.3.1",
# "rero-invenio-base>=0.3.1",
"rero-invenio-base @ git+https://github.com/rerowep/rero-invenio-base@wep-opensearch",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Publish a rero-invenio-base release and update this before integrating.

Comment thread docker-services.yml
- "6379:6379"
db:
image: postgres:17
image: postgres:18-alpine

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should always be in sync with what is currently used in the production servers. Should we migrate to postgres18 in prod on the next release?

* replace docker/elasticsearch/Dockerfile with OPENSEARCH_PLUGINS=analysis-icu
  env var — no custom image needed
* rename docker-compose service es → os; update all links and
  INVENIO_SEARCH_HOSTS in docker-services.yml and docker-compose.full.yml
* add OPENSEARCH_INITIAL_ADMIN_PASSWORD to satisfy OpenSearch 2.12+ entrypoint
  check even when plugins.security.disabled=true
* switch rero-invenio-base to git branch wep-opensearch to remove transitive
  elasticsearch-py 7.x dependency and unblock urllib3 upgrade to 2.7
* drop five now-resolved urllib3 CVE exceptions from scripts/test
  (CVE-2025-50181, CVE-2025-66418, CVE-2025-66471, CVE-2026-21441,
  CVE-2026-44431); update pytest CVE comment to name correct blocker
  (pytest-invenio 4.0.1 pins pytest<9.0.0)
* replace "Elasticsearch"/"elasticsearch" with "Search"/"search" in all
  docstrings and comments throughout rero_mef/ and tests/
* update CLAUDE.md stack description: Elasticsearch → OpenSearch

Co-Authored-by: Peter Weber <peter.weber@rero.ch>
@jma jma removed their request for review June 9, 2026 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants