feat: Added support for OpenLineage integration by ntkathole · Pull Request #5884 · feast-dev/feast

ntkathole · 2026-01-21T15:19:15Z

What this PR does / why we need it:

Added support for OpenLineage integration

Fixes #5882

franciscojavierarceo · 2026-01-21T15:25:52Z

docs/reference/openlineage.md

+
+## Lineage Visualization
+
+Use [Marquez](https://marquezproject.ai/) to visualize your Feast lineage:


oh this is a weird name for openlineage's visualization but this is the name of it. leaving a comment for future

sdk/python/feast/openlineage/emitter.py

sdk/python/feast/feature_store.py

devin-ai-integration

Devin Review found 1 potential issue.

View issue and 4 additional flags in Devin Review.

sdk/python/feast/openlineage/mappers.py

devin-ai-integration

Devin Review found 3 new potential issues.

View issues and 10 additional flags in Devin Review.

sdk/python/feast/openlineage/emitter.py

sdk/python/feast/feature_store.py

sdk/python/feast/openlineage/emitter.py

devin-ai-integration

Devin Review found 1 new potential issue.

View issue and 16 additional flags in Devin Review.

sdk/python/feast/openlineage/emitter.py

devin-ai-integration

Devin Review found 1 new potential issue.

View issue and 15 additional flags in Devin Review.

sdk/python/feast/feature_store.py

devin-ai-integration

Devin Review found 3 new potential issues.

View issues and 15 additional flags in Devin Review.

sdk/python/feast/feature_store.py

devin-ai-integration

Devin Review found 1 new potential issue.

View issue and 15 additional flags in Devin Review.

devin-ai-integration · 2026-01-27T13:07:46Z

sdk/python/feast/feature_store.py

            if progress_ctx:
                progress_ctx.cleanup()

+        # Emit OpenLineage events for applied objects
+        self._emit_openlineage_apply_diffs(registry_diff)
+
+    def _emit_openlineage_apply_diffs(self, registry_diff: RegistryDiff):
+        """Emit OpenLineage events for objects applied via diffs."""
+        if self._openlineage_emitter is None:
+            return
+
+        # Collect all objects that were added or updated
+        objects: List[Any] = []
+        for feast_object_diff in registry_diff.feast_object_diffs:
+            if feast_object_diff.new_feast_object is not None:
+                objects.append(feast_object_diff.new_feast_object)
+
+        if objects:
+            self._emit_openlineage_apply(objects)
+


🟡 OpenLineage apply events emitted even when _apply_diffs fails

FeatureStore._apply_diffs() emits OpenLineage apply events unconditionally after the try/finally block.

Actual: If infra_diff.update(...), apply_diff_to_registry(...), or update_infra(...) raises, the exception propagates but _emit_openlineage_apply_diffs(registry_diff) still runs, emitting lineage for objects that were not successfully applied.

Expected: Apply lineage should only be emitted after a successful apply/commit, or at least be gated on success.

Click to expand

In sdk/python/feast/feature_store.py:859-905, the OpenLineage emission is outside the try and will run even on failure:

try: infra_diff.update(...) apply_diff_to_registry(...) self._registry.update_infra(..., commit=True) finally: ... # always runs, even if the try-block raised self._emit_openlineage_apply_diffs(registry_diff)

Impact: Produces incorrect lineage (false positives) in OpenLineage/Marquez, which can mislead debugging, auditing, and governance workflows.

(Refers to lines 859-905)

Recommendation: Move _emit_openlineage_apply_diffs(...) into the success path (after commit) or guard it with a success flag set only when the apply/commit completes without exception.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-01-27T19:26:37Z

sdk/python/feast/repo_config.py

+    def to_openlineage_config(self):
+        """Convert to feast.openlineage.OpenLineageConfig."""
+        from feast.openlineage.config import OpenLineageConfig as OLConfig
+
+        return OLConfig(
+            enabled=self.enabled,
+            transport_type=self.transport_type,
+            transport_url=self.transport_url,
+            transport_endpoint=self.transport_endpoint,
+            api_key=self.api_key,
+            namespace=self.namespace,
+            producer=self.producer,
+            emit_on_apply=self.emit_on_apply,
+            emit_on_materialize=self.emit_on_materialize,
+        )


🚩 additional_config field lost during OpenLineageConfig conversion

The OpenLineageConfig dataclass (in openlineage/config.py) defines an additional_config: Dict[str, Any] field for transport-specific settings like Kafka bootstrap servers. However, the OpenLineageConfig pydantic model (in repo_config.py) doesn't define this field, and to_openlineage_config() doesn't pass it during conversion.

This means users cannot specify additional_config in feature_store.yaml, and any programmatically-set additional_config would be lost during conversion. If additional_config is only meant for programmatic use (not YAML config), this is fine. Otherwise, the pydantic model should include this field.

See sdk/python/feast/repo_config.py:236-250 (conversion) and sdk/python/feast/openlineage/config.py:51 (field definition).

Was this helpful? React with 👍 or 👎 to provide feedback.

franciscojavierarceo · 2026-01-27T21:28:46Z

sdk/python/feast/feature_store.py

                progress_ctx.cleanup()

+        # Emit OpenLineage events for applied objects
+        self._emit_openlineage_apply_diffs(registry_diff)


we should only emit if it's successful, right? here we could have an exception and still emit a lineage event

yes, and it's not emitting if apply failed. _emit_openlineage_apply_diffs() only executes if the entire try block completes successfully without raising an exception. I think devin is confused with try/finally code block here.

devin-ai-integration

Devin Review found 1 new potential issue.

View issue and 18 additional flags in Devin Review.

sdk/python/feast/openlineage/mappers.py

devin-ai-integration

Devin Review found 1 new potential issue.

View issue and 21 additional flags in Devin Review.

sdk/python/feast/openlineage/emitter.py

devin-ai-integration

Devin Review found 1 new potential issue.

View issue and 21 additional flags in Devin Review.

sdk/python/feast/feature_store.py

docs/reference/openlineage.md

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>

Co-authored-by: Francisco Javier Arceo <arceofrancisco@gmail.com>

* feat: Added support for OpenLineage integration Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> * feat: Added openlineage in requirements Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> * feat: Keep event type as complete instead of other Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> * feat: Added blog post for OpenLineage integration Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> * Update docs/reference/openlineage.md Co-authored-by: Francisco Javier Arceo <arceofrancisco@gmail.com> --------- Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Francisco Javier Arceo <arceofrancisco@gmail.com> Signed-off-by: yassinnouh21 <yassinnouh21@gmail.com>

# [0.60.0](v0.59.0...v0.60.0) (2026-02-17) ### Bug Fixes * Added a flag to correctly download the go binaries ([0f77135](0f77135)) * Adds mapping of date Trino's type into string Feast's type ([531e839](531e839)) * **ci:** Use uv run for pytest in master_only benchmark step ([#5957](#5957)) ([5096010](5096010)) * Disable materialized odfvs for historical retrieval ([#5880](#5880)) ([739d28a](739d28a)) * Fix linting and formatting issues ([#5907](#5907)) ([42ca14a](42ca14a)) * Make timestamp field handling compatible with Athena V3 ([#5936](#5936)) ([e2bad34](e2bad34)) * Support pgvector under non-default schema ([#5970](#5970)) ([c636cd4](c636cd4)) * unit tests not running on main branch ([#5909](#5909)) ([62fe664](62fe664)) * Update java dep which blocking release ([#5903](#5903)) ([a5b8186](a5b8186)) * Update the dockerfile with golang 1.24.12. ([#5918](#5918)) ([be1b522](be1b522)) * Use context.Background() in client constructors ([#5897](#5897)) ([984f93a](984f93a)) ### Features * Add blog post for PyTorch ecosystem announcement ([#5906](#5906)) ([d2eb629](d2eb629)) * Add blog post on Feast dbt integration ([#5915](#5915)) ([b3c8138](b3c8138)) * Add DynamoDB in-place list update support for array-based features ([#5916](#5916)) ([aa5973f](aa5973f)) * Add HTTP connection pooling for remote online store client ([#5895](#5895)) ([e022bf8](e022bf8)) * Add integration tests for dbt import ([#5899](#5899)) ([a444692](a444692)) * Add lazy initialization and feature service caching ([#5924](#5924)) ([b37b7d0](b37b7d0)) * Add multiple entity support to dbt integration ([#5901](#5901)) ([05a4fb5](05a4fb5)), closes [#5872](#5872) * Add PostgreSQL online store support for Go feature server ([#5963](#5963)) ([b8c6f3d](b8c6f3d)) * Add publish docker image of Go feature server. ([#5923](#5923)) ([759d8c6](759d8c6)) * Add Set as feature type ([#5888](#5888)) ([52458fc](52458fc)) * Added online server worker config support in operator ([#5926](#5926)) ([193c72a](193c72a)) * Added support for OpenLineage integration ([#5884](#5884)) ([df70d8d](df70d8d)) * Adjust ray offline store to support abfs(s) ADLS Azure Storage ([#5911](#5911)) ([d6c0b2d](d6c0b2d)) * Batch_engine config injection in feature_store.yaml through operator ([#5938](#5938)) ([455d56c](455d56c)) * Consolidate Python packaging - remove setup.py/setup.cfg, standardize on pyproject.toml and uv ([16696b8](16696b8)) * **go:** Add MySQL registry store support for Go feature server ([#5933](#5933)) ([19f9bb8](19f9bb8)) * Improve local dev experience with file-aware hooks and auto parallelization ([#5956](#5956)) ([839b79e](839b79e)) * Modernize precommit hooks and optimize test performance ([#5929](#5929)) ([ea7d4fa](ea7d4fa)) * Optimize container infrastructure for production ([#5881](#5881)) ([5ebdac8](5ebdac8)) * Optimize DynamoDB online store for improved latency ([#5889](#5889)) ([fcc8274](fcc8274))

# [0.60.0](feast-dev/feast@v0.59.0...v0.60.0) (2026-02-17) ### Bug Fixes * Added a flag to correctly download the go binaries ([0f77135](feast-dev@0f77135)) * Adds mapping of date Trino's type into string Feast's type ([531e839](feast-dev@531e839)) * **ci:** Use uv run for pytest in master_only benchmark step ([feast-dev#5957](feast-dev#5957)) ([5096010](feast-dev@5096010)) * Disable materialized odfvs for historical retrieval ([feast-dev#5880](feast-dev#5880)) ([739d28a](feast-dev@739d28a)) * Fix linting and formatting issues ([feast-dev#5907](feast-dev#5907)) ([42ca14a](feast-dev@42ca14a)) * Make timestamp field handling compatible with Athena V3 ([feast-dev#5936](feast-dev#5936)) ([e2bad34](feast-dev@e2bad34)) * Support pgvector under non-default schema ([feast-dev#5970](feast-dev#5970)) ([c636cd4](feast-dev@c636cd4)) * unit tests not running on main branch ([feast-dev#5909](feast-dev#5909)) ([62fe664](feast-dev@62fe664)) * Update java dep which blocking release ([feast-dev#5903](feast-dev#5903)) ([a5b8186](feast-dev@a5b8186)) * Update the dockerfile with golang 1.24.12. ([feast-dev#5918](feast-dev#5918)) ([be1b522](feast-dev@be1b522)) * Use context.Background() in client constructors ([feast-dev#5897](feast-dev#5897)) ([984f93a](feast-dev@984f93a)) ### Features * Add blog post for PyTorch ecosystem announcement ([feast-dev#5906](feast-dev#5906)) ([d2eb629](feast-dev@d2eb629)) * Add blog post on Feast dbt integration ([feast-dev#5915](feast-dev#5915)) ([b3c8138](feast-dev@b3c8138)) * Add DynamoDB in-place list update support for array-based features ([feast-dev#5916](feast-dev#5916)) ([aa5973f](feast-dev@aa5973f)) * Add HTTP connection pooling for remote online store client ([feast-dev#5895](feast-dev#5895)) ([e022bf8](feast-dev@e022bf8)) * Add integration tests for dbt import ([feast-dev#5899](feast-dev#5899)) ([a444692](feast-dev@a444692)) * Add lazy initialization and feature service caching ([feast-dev#5924](feast-dev#5924)) ([b37b7d0](feast-dev@b37b7d0)) * Add multiple entity support to dbt integration ([feast-dev#5901](feast-dev#5901)) ([05a4fb5](feast-dev@05a4fb5)), closes [feast-dev#5872](feast-dev#5872) * Add PostgreSQL online store support for Go feature server ([feast-dev#5963](feast-dev#5963)) ([b8c6f3d](feast-dev@b8c6f3d)) * Add publish docker image of Go feature server. ([feast-dev#5923](feast-dev#5923)) ([759d8c6](feast-dev@759d8c6)) * Add Set as feature type ([feast-dev#5888](feast-dev#5888)) ([52458fc](feast-dev@52458fc)) * Added online server worker config support in operator ([feast-dev#5926](feast-dev#5926)) ([193c72a](feast-dev@193c72a)) * Added support for OpenLineage integration ([feast-dev#5884](feast-dev#5884)) ([df70d8d](feast-dev@df70d8d)) * Adjust ray offline store to support abfs(s) ADLS Azure Storage ([feast-dev#5911](feast-dev#5911)) ([d6c0b2d](feast-dev@d6c0b2d)) * Batch_engine config injection in feature_store.yaml through operator ([feast-dev#5938](feast-dev#5938)) ([455d56c](feast-dev@455d56c)) * Consolidate Python packaging - remove setup.py/setup.cfg, standardize on pyproject.toml and uv ([16696b8](feast-dev@16696b8)) * **go:** Add MySQL registry store support for Go feature server ([feast-dev#5933](feast-dev#5933)) ([19f9bb8](feast-dev@19f9bb8)) * Improve local dev experience with file-aware hooks and auto parallelization ([feast-dev#5956](feast-dev#5956)) ([839b79e](feast-dev@839b79e)) * Modernize precommit hooks and optimize test performance ([feast-dev#5929](feast-dev#5929)) ([ea7d4fa](feast-dev@ea7d4fa)) * Optimize container infrastructure for production ([feast-dev#5881](feast-dev#5881)) ([5ebdac8](feast-dev@5ebdac8)) * Optimize DynamoDB online store for improved latency ([feast-dev#5889](feast-dev#5889)) ([fcc8274](feast-dev@fcc8274)) Signed-off-by: soojin <soojin@dable.io>

ntkathole self-assigned this Jan 21, 2026

ntkathole requested a review from a team as a code owner January 21, 2026 15:19

ntkathole marked this pull request as draft January 21, 2026 15:19

franciscojavierarceo reviewed Jan 21, 2026

View reviewed changes

sdk/python/feast/openlineage/emitter.py Outdated Show resolved Hide resolved

franciscojavierarceo reviewed Jan 21, 2026

View reviewed changes

sdk/python/feast/openlineage/emitter.py Outdated Show resolved Hide resolved

franciscojavierarceo reviewed Jan 21, 2026

View reviewed changes

sdk/python/feast/openlineage/emitter.py Outdated Show resolved Hide resolved

franciscojavierarceo reviewed Jan 21, 2026

View reviewed changes

sdk/python/feast/openlineage/emitter.py Show resolved Hide resolved

franciscojavierarceo reviewed Jan 21, 2026

View reviewed changes

sdk/python/feast/feature_store.py Outdated Show resolved Hide resolved

ntkathole force-pushed the openlineage branch 3 times, most recently from 4eef1b9 to 73dafb6 Compare January 24, 2026 10:07

ntkathole marked this pull request as ready for review January 24, 2026 10:09

ntkathole added the ok-to-test label Jan 24, 2026

ntkathole force-pushed the openlineage branch 3 times, most recently from d87c190 to b743c1e Compare January 26, 2026 15:42

devin-ai-integration bot reviewed Jan 26, 2026

View reviewed changes

sdk/python/feast/openlineage/mappers.py Outdated Show resolved Hide resolved

ntkathole force-pushed the openlineage branch from b743c1e to e1ddd6b Compare January 26, 2026 15:54

devin-ai-integration bot reviewed Jan 26, 2026

View reviewed changes

sdk/python/feast/openlineage/emitter.py Outdated Show resolved Hide resolved

sdk/python/feast/feature_store.py Show resolved Hide resolved

sdk/python/feast/openlineage/emitter.py Outdated Show resolved Hide resolved

devin-ai-integration bot reviewed Jan 26, 2026

View reviewed changes

sdk/python/feast/openlineage/emitter.py Outdated Show resolved Hide resolved

ntkathole force-pushed the openlineage branch from f5640a8 to ce3ec64 Compare January 26, 2026 16:28

devin-ai-integration bot reviewed Jan 26, 2026

View reviewed changes

sdk/python/feast/feature_store.py Show resolved Hide resolved

ntkathole force-pushed the openlineage branch 2 times, most recently from b51e87a to 40a8559 Compare January 26, 2026 16:43

devin-ai-integration bot reviewed Jan 26, 2026

View reviewed changes

sdk/python/feast/feature_store.py Show resolved Hide resolved

sdk/python/feast/feature_store.py Show resolved Hide resolved

sdk/python/feast/feature_store.py Show resolved Hide resolved

ntkathole force-pushed the openlineage branch 2 times, most recently from e350360 to 29194d7 Compare January 27, 2026 13:03

devin-ai-integration bot reviewed Jan 27, 2026

View reviewed changes

franciscojavierarceo reviewed Jan 27, 2026

View reviewed changes

devin-ai-integration bot reviewed Jan 28, 2026

View reviewed changes

sdk/python/feast/openlineage/mappers.py Outdated Show resolved Hide resolved

ntkathole force-pushed the openlineage branch from 348187d to e8b524b Compare January 29, 2026 05:51

devin-ai-integration bot reviewed Jan 29, 2026

View reviewed changes

sdk/python/feast/openlineage/emitter.py Show resolved Hide resolved

ntkathole force-pushed the openlineage branch from e8b524b to b7fd6b5 Compare January 29, 2026 05:59

devin-ai-integration bot reviewed Jan 29, 2026

View reviewed changes

sdk/python/feast/feature_store.py Show resolved Hide resolved

franciscojavierarceo approved these changes Jan 29, 2026

View reviewed changes

docs/reference/openlineage.md Outdated Show resolved Hide resolved

ntkathole and others added 5 commits January 29, 2026 20:15

feat: Added support for OpenLineage integration

9e0fc93

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>

feat: Added openlineage in requirements

54f03f2

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>

feat: Keep event type as complete instead of other

0a7799b

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>

feat: Added blog post for OpenLineage integration

5812bce

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>

Update docs/reference/openlineage.md

01d69a1

Co-authored-by: Francisco Javier Arceo <arceofrancisco@gmail.com>

ntkathole force-pushed the openlineage branch from 026da68 to 01d69a1 Compare January 29, 2026 14:45

ntkathole merged commit df70d8d into master Jan 29, 2026
19 of 21 checks passed


		## Lineage Visualization

		Use [Marquez](https://marquezproject.ai/) to visualize your Feast lineage:

Comments

Conversation

ntkathole commented Jan 21, 2026 • edited by devin-ai-integration bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Uh oh!

franciscojavierarceo Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

devin-ai-integration bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

devin-ai-integration bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

devin-ai-integration bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

devin-ai-integration bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

devin-ai-integration bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

devin-ai-integration bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

franciscojavierarceo Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ntkathole Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

devin-ai-integration bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

devin-ai-integration bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ntkathole commented Jan 21, 2026 •

edited by devin-ai-integration bot

Loading

franciscojavierarceo Jan 27, 2026 •

edited

Loading