Page History

...

• Attendees: 
  • TSC:

    • Mandy Chessell: Egeria Lead. Integrating OpenLineage in Egeria

    • Michael Collado: Datakin, OpenLineage

    • Maciej Obuchowski: GetInData. OpenLineage integrations
    • Willy Lulciuc: Marquez co-creator.
    • Ryan Blue: Tabular, Iceberg. Interested in collecting lineage across iceberg user with OpenLineage
  • And:
    • Venkatesh Tadinada: BMC workflow automation looking to integrate with Marquez
    • Minkyu Park: Datakin. learning about OpenLineage
    • Arthur Wiedmer: Apple, lineage for Siri and AI ML. Interested in implementing Marquez and OpenLineage

• Meeting recording:
• zoom link
Passcode: +UsC8emL

• Meeting notes:
  • agenda: 

    • Update on OpenLineage latest release (0.2.1)

      • dbt integration demo

    • OpenLineage 0.3 scope discussion

      • Facet versioning mechanism (Issue #153)

      • OpenLineage Proxy Backend (Issue #152)

      • OpenLineage implementer test data and validation

      • Kafka client

    • Roadmap

      • Iceberg integration

    • Open discussion

  • Slides 

  • Discussions:

    • added to the agenda a Discussion of Iceberg requirements for OpenLineage.

  • Demo of dbt:

    • really easy to try

    • when running from airflow, we can use the wrapper 'dbt-ol run' instead of 'dbt run'

  • Presentation of Proxy Backend design:

    • summary of discussions in Egeria

      • Egeria is less interested in instances (runs) and will keep track of OpenLineage events separately as Operational lineage

      • Two ways to use Egeria with OpenLineage

        • receives HTTP events and forwards to Kafka

        • A consumer receives the Kafka events in Egeria

    • Proxy Backend in OpenLineage:

      • direct HTTP endpoint implementation in Egeria

    • Depending on the user they might pick one or the other and we'll document

  • Use a direct OpenLineage endpoint (like Marquez)

    • Deploy the Proxy Backend to write to a queue (ex: Kafka)

    • Follow up items:

...