...
- Announcements [Julien]
- Ecosystem Survey still needs responses: https://bit.ly/ecosystem_survey
- OpenLineage graduated from the LF AI on 7/27
- The 3rd issue of our monthly newsletter shipped on 7/31. Sign up here: https://bit.ly/OL_news
- Upcoming meetups:
- 8/30 in S.F. at Astronomer
- 9/18 in Toronto at Airflow Summit
- Marquez meetup on 10/5 in S.F.
- LF AI Update [Michael R.]
- Topics covered by Julien in presentation to LF AI TAC for graduation included trends in adoption
- Recent releases [Michael R.]
1.0.0: Added
- Airflow: convert lineage from legacy File definition #2006@mobuchowski
Removed
- Spec: remove facet ref from core #1997@JDarDagran
Changed
- Airflow: change log level to DEBUG when extractor isn't found #2012@kaxil
- Airflow: make sure we cannot fail in thread despite direct execution #2010@mobuchowski
https://github.com/OpenLineage/OpenLineage/releases/tag/1.0.0
https://github.com/OpenLineage/OpenLineage/compare/0.30.1...1.0.0
0.30.1: Added
- Flink: support Iceberg sinks #1960@pawel-big-lebowski
- Spark: column-level lineage for merge into on delta tables #1958@pawel-big-lebowski
- Spark: column-level lineage for merge into on Iceberg tables #1971@pawel-big-lebowski
- Spark: add support for Iceberg REST catalog #1963@juancappi
- Airflow: add possibility to force direct-execution based on environment variable #1934@mobuchowski
- SQL: add support for Apple Silicon to openlineage-sql-java#1981@davidjgoss
- Spec: add facet deletion #1975@julienledem
- Client: add a file transport #1891@Alexkuva
Changed
- Airflow: do not run plugin if OpenLineage provider is installed #1999@JDarDagran
- Python: rename config to config_class#1998@mobuchowski
https://github.com/OpenLineage/OpenLineage/releases/tag/0.30.1
https://github.com/OpenLineage/OpenLineage/compare/0.29.2...0.30.1
- Update on the OpenLineage Airflow Provider [Maciej]
- Pypi package version 1.0.1 available at: https://pypi.org/project/apache-airflow-providers-openlineage/1.0.1/
- installable with
pip install apache-airflow-providers-openlineage==1.0.1
- installable with
- Development progresses in the Airflow repo
- What's there already:
- Operator coverage:
- A lot of SQL-related operators, especially based on SQLExecuteQueryOperator
- Some GCP ones: BigQueryInsertJobOperator, GCStoGCSOperator
- Some Sagemaker-related operators
- FTP, SFTP operators
- Basic support for Python and Bash operators
- Changed:
- Airflow: do not run plugin if OpenLineage provider is installed #1999@JDarDagran
- Python: rename config to config_class #1998 @mobuchowski
- Operator coverage:
- Next steps
- Operator coverage:
- Popular operators around BigQuery: BigQueryUpsertTableOperator…
- Transport operators, like MySQLToSnowflakeOperator, GCSToBigQueryOperator
- S3 support, like S3CopyObjectOperator
- Add support for XCom-native operators like BigQueryGetDataOperator
- This list is not a promise
- "Core" changes
- Add interfaces around OpenLineage-implementing operators - making implementation more native
- XCom dataset support - this relates to XCom operators mentioned above
- Hook-level lineage support
- Operator coverage:
- Pypi package version 1.0.1 available at: https://pypi.org/project/apache-airflow-providers-openlineage/1.0.1/
July 13, 2023 (8am PT)
Attendees:
...