...
- TSC:
- Mike Collado, Staff Software Engineer, Astronomer
- Julien Le Dem, OpenLineage Project lead
- Maciej Obuchowski, Software Engineer, GetInData, OpenLineage contributor
- And:
- Petr Hajek, Software Engineer, MANTA
- Harel Shein, Director of Engineering, Astronomer
- Minkyu Park, Senior Software Engineer, Astronomer
- Michael Robinson, Software Engineer, Dev. Rel., Astronomer
- Howard Yoo, Staff Product Manager, Astronomer
- Tomasz Nazarewicz, Software Engineer, GetInData
- Sheeri Cabral: Technical Product Manager, Lineage, Collibra
- Hanna Moazam, Cloud Solution Architect, Microsoft
Tentative agendaAgenda:
- Recent release 0.15.1
- Project roadmap review
- Column-level lineage workshop using Jupyter + Spark
...
| Widget Connector | ||
|---|---|---|
|
Notes:
- Announcements:
- We recently removed support for Airflow 1.x
- Ross gave a talk on OpenLineage at ApacheCon in New Orleans last week
- Upcoming opportunities to give talks about OpenLineage:
- Data Teams Summit (January 2023)
- Subsurface Live (January 2023)
- Data Council Austin (March 2023)
- Giving a talk on data lineage soon? Ping Michael R. on Slack to let us know.
- Recent release 0.15.1
Added
- Airflow: improve development experience #1101 @JDarDagran
- Documentation: update issue templates for proposal & add new integration template #1116 @rossturk
- Spark: add description for URL parameters in readme, change overwriteName to appName #1130 @tnazarew
Changed
- Airflow: lazy load BigQuery client #1119 @mobuchowski
Fixed
- Spark: fix column lineage #1069 @pawel-big-lebowski
- Spark: set log level of Init OpenLineageContext to DEBUG #1064 **new contributor @varuntestaz**
- Java client: update version of SnakeYAML #1090 **new contributor Lukáš AKA @TheSpeedding**
- CI: build macos release package on medium resource class #1131 @mobuchowski
- Project roadmap review
- Improved understanding of Airflow
- Track DAG runs
- Native lineage in operators
- Increased adoption of OpenLineage consumers
- Collaborate with data catalogs
- Coverage by event producers
- Increased support for Snowflake access history
- Data quality frameworks
- Start thinking about data consumption integrations
- Continue experimenting with a Flink integration
- Increased support of column level lineage
September 8, 2022 (10am PT)
...