fix(superset): bridge DataModel <-> Dashboard lineage through the Chart node (1.12.9)#8
Open
joaopamaral wants to merge 7 commits into
Open
fix(superset): bridge DataModel <-> Dashboard lineage through the Chart node (1.12.9)#8joaopamaral wants to merge 7 commits into
joaopamaral wants to merge 7 commits into
Conversation
…rt node OpenMetadata's lineage graph traverses through explicit AddLineageRequest edges. For Superset the current behaviour was: - `Dashboard.dataModels` was never set, so the structural link from the dashboard to its datamodels was missing in the dashboard's Data Models panel. - `DashboardServiceSource.yield_datamodel_dashboard_lineage()` emitted a direct `DataModel -> Dashboard` lineage edge, so the lineage graph showed datamodels connected to the dashboard but skipped the chart entirely. Net result in production: the dashboard's lineage view showed `Table -> DataModel -> Dashboard` and the chart was invisible in the graph, even though it was a member of the dashboard. This change rewires the relationship so the chart is the bridge node: 1. `SupersetSourceMixin.yield_dashboard_lineage_details` now also emits `DataModel -> Chart` and `Chart -> Dashboard` edges for each chart on the dashboard. Combined with the existing `Table -> DataModel` edge the full chain in the lineage view becomes `Table -> DataModel -> Chart -> Dashboard`. 2. `SupersetSourceMixin.yield_datamodel_dashboard_lineage` is overridden to yield nothing, suppressing the base class' direct `DataModel -> Dashboard` edge so the datamodel doesn't render alongside the chart as two parallel paths into the dashboard. 3. `db_source.yield_dashboard` and `api_source.yield_dashboard` now send `Dashboard.dataModels=[]` (an empty list, not absent) on every PUT so any datamodel entries persisted from prior runs are deleted by the server. The relationship is fully represented by the lineage chain. Two helpers — `_get_chart_entity` and `_get_dashboard_entity` — resolve the Chart and Dashboard entities by FQN via `metadata.get_by_name`. If either isn't yet visible on the server (first run, before the entity POST has been committed), the bridge edges are skipped without crashing and the next run picks them up. Tests ----- Adds `tests/unit/topology/dashboard/test_superset_chart_lineage.py` with five Mockito-style unit tests: - `test_db_source_yields_empty_data_models` / `test_api_source_yields_empty_data_models`: Both sources produce `CreateDashboardRequest` with `dataModels=[]`. - `test_override_yields_no_edges`: the `yield_datamodel_dashboard_lineage` override produces zero edges. - `test_chart_bridge_edges_emitted`: when the chart and dashboard entities resolve, `yield_dashboard_lineage_details` emits both `DataModel -> Chart` and `Chart -> Dashboard` edges. - `test_no_chart_entity_skips_bridge_edges`: when the chart isn't yet resolvable on the server, both bridge edges are skipped gracefully. Three of the five tests fail on the unpatched code, exercising each of the three behaviour changes above. Manually verified in production (Superset metadata-DB connection mode) against a real Meltano dashboard: after this fix the dashboard's lineage graph renders `Table -> DataModel -> Chart -> Dashboard` with the chart visible as the bridge node, and the API confirms `Dashboard.dataModels` is cleared while all `DataModel -> Chart` and `Chart -> Dashboard` edges are present. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…exing in test Two review comments from gitar-bot on open-metadata#28240: - mixin.yield_dashboard_lineage_details was calling self._get_dashboard_entity(dashboard_details) inside the per-chart loop, doing N identical metadata.get_by_name lookups for a dashboard with N charts. Hoist the call out of the loop and reuse the result. - test_superset_chart_lineage.py was wiring SupersetSourceMixin.__mro__[1]._get_add_lineage_request onto a mock, which silently breaks if a new intermediate base class is inserted. Reference DashboardServiceSource._get_add_lineage_request directly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
add_lineage PUTs the edge first, then re-fetches the source entity
lineage. When that follow-up GET 404s (e.g. a dangling reference to a
deleted chart/datamodel), add_lineage returns None even though the edge
was written. write_lineage assumed a dict and crashed on
created_lineage.get('error') with 'NoneType' object has no attribute
'get'. Treat None as a soft failure: log a warning and return an empty
Either so the bulk sink continues.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ata#27309) Backport of open-metadata#27309 (Fix open-metadata#27308) onto 1.12.9. Adds first-class chart cleanup to dashboard ingestion so charts removed from the source (and no longer referenced by any dashboard in the scan) are soft-deleted, mirror- ing markDeletedDataModels/markDeletedDashboards. - new markDeletedCharts config (default true) on dashboard pipeline schema + generated python model - chart_source_state, register_record_chart(), mark_charts_as_deleted() on the base DashboardServiceSource; wired into post_process - connectors register charts via register_record_chart in yield_dashboard_chart This removes the source of the dangling DataModel->Chart lineage edges that triggered the write_lineage None crash: when a datamodel is deleted but its chart survived, the orphaned chart kept a dead upstream edge. Conflicts resolved for 1.12.9 (powerbi/sigma kept local connector shape); the powerbi/sigma/tableau/qliksense test additions from open-metadata#27309 were left on the 1.13 line as they depend on newer test scaffolding. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rd fallback A view ingested in one catalog can reference a source table that lives under a different database name in another service -- e.g. a view in trino_views_hiyu (catalog hiyu) using a table ingested as trino_hiyu.iceberg.<schema>.<table>. get_table_entities_from_query searched with the view's database (hiyu) for every service (incl. crossDatabaseServiceNames), so the table under database 'iceberg' never matched and no lineage edge was created. Add a final database=None fallback (after the exact and schema fallbacks) so the FQN search wildcards the database (<service>.*.<schema>.<table>) and resolves the table across the configured cross-database services. Scoped to service_names, so it only reaches the current service plus the explicitly-allowed crossDatabaseServiceNames. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Backports upstream open-metadata/OpenMetadata#28240 onto our
release-1.12.9line.Superset lineage now bridges
DataModel <-> Dashboardthrough the Chart node instead of emitting a directDataModel -> Dashboardedge:mixin.py: suppress the directDataModel -> Dashboardedge (yield_datamodel_dashboard_lineageemits nothing); emitDataModel -> ChartandChart -> Dashboardper chart. Adds_get_chart_entity/_get_dashboard_entityhelpers resolving entities by FQN viametadata.get_by_name.db_source.py/api_source.py: sendDashboard.dataModels=[]on every PUT to clear prior server-side entries.tests/unit/topology/dashboard/test_superset_chart_lineage.py.Why
Our 1.12.9 Superset/Looker ingestion produced
dashboardDataModel -> chartlineage edges whose target 404s, hitting the client's unguardedcreated_lineage.get(...)and failing the workflow. Routing lineage through the Chart node yields the correct graph shape and avoids the broken direct edge.Scope
Ingestion-only (4 files). No server schema/API changes -> runs against our 1.12.7 server (same major.minor;
match_versionspasses).Base
release-1.12.9(cut from upstream1.12.9-releasetag). Diff is exactly the open-metadata#28240 change.Testing
test_superset_chart_lineage.py).🤖 Generated with Claude Code