Page History

...

1. Announcements [Julien]
   1. Ecosystem Survey still needs responses: https://bit.ly/ecosystem_survey
   2. OpenLineage graduated from the LF AI on 7/27
   3. The 3rd issue of our monthly newsletter shipped on 7/31. Sign up here: https://bit.ly/OL_news
   4. Upcoming meetups:
      
      1. 8/30 in S.F. at Astronomer
      2. 9/18 in Toronto at Airflow Summit
      3. Marquez meetup on 10/5 in S.F.
2. LF AI Update [Michael R.]
   1. Topics covered by Julien in presentation to LF AI TAC for graduation included trends in adoption
3. Recent releases [Michael R.]

   1. 1.0.0: Added

      • Airflow: convert lineage from legacy File definition #2006@mobuchowski

      Removed

      • Spec: remove facet ref from core #1997@JDarDagran

      Changed

      • Airflow: change log level to DEBUG when extractor isn't found #2012@kaxil
      • Airflow: make sure we cannot fail in thread despite direct execution #2010@mobuchowski

      • https://github.com/OpenLineage/OpenLineage/releases/tag/1.0.0

        https://github.com/OpenLineage/OpenLineage/compare/0.30.1...1.0.0

   2. 0.30.1: Added

      • Flink: support Iceberg sinks #1960@pawel-big-lebowski
      • Spark: column-level lineage for merge into on delta tables #1958@pawel-big-lebowski
      • Spark: column-level lineage for merge into on Iceberg tables #1971@pawel-big-lebowski
      • Spark: add support for Iceberg REST catalog #1963@juancappi
      • Airflow: add possibility to force direct-execution based on environment variable #1934@mobuchowski
      • SQL: add support for Apple Silicon to openlineage-sql-java#1981@davidjgoss
      • Spec: add facet deletion #1975@julienledem
      • Client: add a file transport #1891@Alexkuva

      Changed

      • Airflow: do not run plugin if OpenLineage provider is installed #1999@JDarDagran
      • Python: rename config to config_class#1998@mobuchowski

      • https://github.com/OpenLineage/OpenLineage/releases/tag/0.30.1

        https://github.com/OpenLineage/OpenLineage/compare/0.29.2...0.30.1

4. Update on the OpenLineage Airflow Provider [Maciej]
   1. Pypi package version 1.0.1 available at: https://pypi.org/project/apache-airflow-providers-openlineage/1.0.1/
      1. installable with pip install apache-airflow-providers-openlineage==1.0.1 
   2. Development progresses in the Airflow repo
   3. What's there already:
      1. Operator coverage:
         1. A lot of SQL-related operators, especially based on SQLExecuteQueryOperator
         2. Some GCP ones: BigQueryInsertJobOperator, GCStoGCSOperator 
         3. Some Sagemaker-related operators
         4. FTP, SFTP operators
         5. Basic support for Python and Bash operators
      2. Changed:
         1. Airflow: do not run plugin if OpenLineage provider is installed #1999@JDarDagran
         2. Python: rename config to config_class #1998 @mobuchowski
   4. Next steps
      1. Operator coverage:
         1. Popular operators around BigQuery: BigQueryUpsertTableOperator…
         2. Transport operators, like MySQLToSnowflakeOperator, GCSToBigQueryOperator
         3. S3 support, like S3CopyObjectOperator
         4. Add support for XCom-native operators like BigQueryGetDataOperator
         5. This list is not a promise
      2. "Core" changes
         1. Add interfaces around OpenLineage-implementing operators - making implementation more native
         2. XCom dataset support - this relates to XCom operators mentioned above
         3. Hook-level lineage support
5. OpenLineage 1.0 with Static Lineage Update
   1. Putting things together for 1.0 release
      1. Important features and PRs
         1. • Proposal: add static lineage deletion #1839@julienledem
            • Emit job and dataset runless metadata #1880@pawel-big-lebowski
            • Marquez: Ability to decode static metadata events #2495@pawel-big-lebowski
            • Add facet deletion #1975@julienledem
            • Spec: remove facet ref from core #1997@JDarDagran
            Clarify docs on `RunEvent` lifecycle (link)

July 13, 2023 (8am PT)

Attendees:

...