Page History

...

• Meeting recording:
  • zoom link
  • Passcode: +UsC8emL
• Meeting notes:
  • agenda: 

    • Update on OpenLineage latest release (0.2.1)

      • dbt integration demo

    • OpenLineage 0.3 scope discussion

      • Facet versioning mechanism (Issue #153)

      • OpenLineage Proxy Backend (Issue #152)

      • OpenLineage implementer test data and validation

      • Kafka client

    • Roadmap

      • Iceberg integration

    • Open discussion

  • Slides: https://docs.google.com/presentation/d/1fI0u8aE0iX9vG4GGrnQYAEcsJM9z7Rlv/edit#slide=id.p1 

  • Discussions:

    • added to the agenda a Discussion of Iceberg requirements for OpenLineage.

  • Demo of dbt:

    • really easy to try

    • when running from airflow, we can use the wrapper 'dbt-ol run' instead of 'dbt run'

  • Presentation of Proxy Backend design:

    • summary of discussions in Egeria

      • Egeria is less interested in instances (runs) and will keep track of OpenLineage events separately as Operational lineage

      • Two ways to use Egeria with OpenLineage

        • receives HTTP events and forwards to Kafka

        • A consumer receives the Kafka events in Egeria

    • Proxy Backend in OpenLineage:

      • direct HTTP endpoint implementation in Egeria

    • Depending on the user they might pick one or the other and we'll document

  • Use a direct OpenLineage endpoint (like Marquez)

    • Deploy the Proxy Backend to write to a queue (ex: Kafka)

    • Follow up items:

...