Page History

The OpenLineage Technical Steering Committee meetings are Monthly on the Second Wednesday 9:00am to 10:00am US Pacific and the link to join the meeting is https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09

All are welcome.

Aug 11th 2021

• Attendees: 
  • TSC:
• Meeting recording:
• Meeting notes:
  • Agenda:
    • Coming in OpenLineage 0.1
      • OpenLineage spec versioning
      • Clients
      • Marquez integrations imported in OpenLineage
        • Apache Airflow:
          • BigQuery 
          • Postgres
          • Snowflake
          • Redshift
          • Great Expectations
        • Apache Spark
        • dbt
    • OpenLineage 0.2 scope discussion
      
      • Facet versioning mechanism
      • OpenLineage Proxy Backend (Issue #152)
      • Kafka client
    • Roadmap
    • Open discussion

July 14th 2021

• Attendees: 
  • TSC:
    • Julien Le Dem
    • Mandy Chessel
    • Michael Collado
    • Willy Lulciuc
• Meeting recording:
  • zoom link
  • Passcode: =!oZ?&0A
• Meeting notes
  • Agenda:
    • Finalize the OpenLineage Mission Statement
    • Review OpenLineage 0.1 scope
    • Roadmap
    • Open discussion 
    • Slides: https://docs.google.com/presentation/d/1fD_TBUykuAbOqm51Idn7GeGqDnuhSd7f/edit#slide=id.ge4b57c6942_0_46
  • Notes: 

    Mission statement:

    • https://github.com/OpenLineage/OpenLineage/issues/84

    • Overall consensus on the statement.

    • TODO: vote by commenting on the ticket

    Spec versioning mechanism:

    • The goal is to commit to compatible changes once 0.1 is published

    • We need a follow up to separate core facet versioning

    
    => TODO: create a separate github ticket.

    • The lineage event should have a field that identifies what version of the spec it was produced with

    • => TODO: create a github issue for this

    • TODO: Add issue to document version number semantics (SCHEMAVER)

    Extend Event State notion:

    • where do we capture more precise state transitions like RESTART?

    • Discussion should happen here: https://github.com/OpenLineage/OpenLineage/issues/9

    OpenLineage 0.1:

    • finalize a few spec details for 0.1 : a few items left to discuss.

    • In particular job naming

    • parent job model

    • Importing Marquez integrations in OpenLineage

    Open Discussion:

    • connecting the consumer and producer

    Do we have a mutual third party or the client know where to send?

    • • TODO: ticket to track distribution mechanism

      • options:

    • proxy similar to OpenTelemetry Collector.

    • Send to Kafka: https://github.com/OpenLineage/OpenLineage/issues/70

    • • • Would we need a consumption client to make it easy for consumers to get events from Kafka for example?

        • OpenLineage provides client libraries to serialize/deserialize events as well as sending them.

      • proxy similar to OpenTelemetry Collector.

      • Send to Kafka: https://github.com/OpenLineage/OpenLineage/issues/70

      • We can have documentation on how to send to backends that are not Marquez using HTTP and existing gateway mechanism to queues.

    • Do we have a mutual third party or the client know where to send?

    • Source code location finalization

    • job naming convention

    • parent job

    • • you don't always have a nested execution

      • can call a parent

    • parent job

    • You can have a job calling another one.

    • always distinguish a job and its run

    • need a separate notion for job dependencies

    • need to capture event driven: TODO: create ticket.

    
    TODO(Julien): update job naming ticket to have the discussion.

...