Page History

The OpenLineage Technical Steering Committee meetings are Monthly on the Second Thursday from 10:00am to 11:00am on the Second Wednesday from 9:30am to 10:30am US Pacific. Here's the meeting info.

All are welcome.

Next meeting: May 8, 2024 (9:30am PT)

April 10, 2024 (9:30am PT)

Attendees:

• TSC:
  • Julien Le Dem, OpenLineage project lead, LF AI & Data
  • Michael Robinson, Community Manager, Astronomer
  • Harel Shein, Lineage at Datadog
• And:

...

• • Sheeri Cabral, Product Manager, ETL, Collibra
  • Eric Veleker, Partnerships, Atlan
  • Jens Pfau, Engineering Manager, Lineage, Google
  • David Twaddell, Architect, HSBC

    
    

Agenda:

• Announcements
• Recent release highlights
• Discussion items
  
  • supporting job-to-job dependencies in the spec
  • improving naming conventions
• Open discussion

Meeting:

• Slides

March 13, 2024 (9:30am PT)

Tentative agenda:

• Announcements
• Recent releasesrelease 1.9.1 highlights
• Scala 2.13 support in Spark overview by @Damien Hawes
• Circuit breaker in Spark & Flink, built-in lineage in Spark @Paweł Leszczyński
• Discussion items
• Open discussion

...

• Recent releases
• Announcements
• Coming soon: simplified job hierarchy in the Spark integration
• Discussion items
• Open discussion

Meeting:

Notes:

Summary

    1. We have added a new communication resource, a LinkedIn company page.
    2. We announced a new committer, Damien Hawes, from Booking.com, who has made significant contributions to the project.
    3. Astronomer and Collibra are co-sponsoring a data lineage meetup on March 19th at the Microsoft New England Conference Center.
    4. Members have talks upcoming at Kafka Summit and Data Council.
    5. We discussed upcoming improvments to job hierarchy in Spark and how this can help answer questions about job scheduling and dependencies.
    6. Damien shared his contributions to the Apache Spark integration, specifically addressing versioning conflicts with Scala.
    7. Eric provided a general update on the interest in and adoption of OpenLineage, particularly in the enterprise space.
    8. Atlan is considering releasing a DAG (Directed Acyclic Graph) instead of a plugin to help users with configuration and troubleshooting.
    9. The next monthly call will be held at a different "location," and participants were encouraged to look out for the updated Zoom link.

...

• Recent releases
• Announcements
• Discussion items
• Open discussion

Meeting:

Notes:

Summary

    1. We closed their first ever annual ecosystem survey and the results will be published soon.
    2. There is a meetup coming up on January 31st in London, which will be our first in London. It will be an in-person event.
    3. We have a talk at the Kafka Summit in London in March, with key contributors speaking.
    4. We recently released version 1.7.0, with important compatibility notice for the Airflow integration.
    5. There was a discussion about possible improvements to job hierarchy semantics in the Spark integration.
    6. Julien updated the registry proposal and it is close to being implemented.
    7. Eric (Atlan) shared that there is growing demand and adoption of OpenLineage, and organizations are pressing forward due to the perceived business value.
    8. Eric mentioned the need for better documentation and support for different versions and integrations.
    9. Jens suggested expanding the integration matrix to include more dimensions, such as types of data sources and facets.

...

• Recent releases
• Announcements
• Proposal updates
• Open discussion

Meeting:

Notes:

Summary

    1. Harel Shein provided announcements about upcoming meetups and shared metrics on community growth.
    2. Harel Shein discussed the release of version 1.6.2, highlighting new features and bug fixes.
    3. Harel Shein shared metrics on contributors and commits, showing an increase in both.
    4. Jens Pfau presented two proposals for column-level lineage, focusing on transformation types and descriptions.
    5. Mandy Chessell suggested including the name of the masking function as an additional property for masking transformations.
    6. Harel Shein expressed appreciation for the proposals and encouraged community members to review and provide feedback.
    7. Eric Veleker expressed gratitude for the momentum and adoption of open lineage, thanking the community for their hard work.
    8. Harel Shein echoed Eric's sentiments and acknowledged the project's growth and industry standard status.
    9. Harel Shein thanked all contributors and adopters for their contributions to the community.

...