Page History

...

Notes:

• Recent releases [Michael R.]
  • 0.18.0

    • Added

      • Airflow: support SQLExecuteQueryOperator #1379 @JDarDagran
      • Airflow: introduce a new extractor for SFTPOperator #1263 @sekikn
      • Airflow: add Sagemaker extractors #1136 @fhoda
      • Airflow: add S3 extractor for Airflow operators #1166 @fhoda
      • Spec: add spec file for ExternalQueryRunFacet #1262 @howardyoo
      • Docs: add a TSC doc #1303 @merobi-hub

      Bug fixes and more details: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md

  •  0.17.0

    • Added

      • Spark: support latest Spark 3.3.1 #1183 @pawel-big-lebowski
      • Spark: add Kinesis Transport and support config Kinesis in Spark integration #1200 @yogyang
      • Spark: disable specified facets #1271 @pawel-big-lebowski
      • Python: add facets implementation to Python client #1233 @pawel-big-lebowski
      • SQL: add Rust parser interface #1172 @StarostaGit @mobuchowski
      • Proxy: add helm chart for the proxy backed #1068 @wslulciuc
      • Spec: include possible facets usage in spec #1249 @pawel-big-lebowski
      • Website: publish YML version of spec to website #1300 @rossturk
      • Docs: update language on nominating new committers #1270 @rossturk

      Changed

      • Website: publish spec into new website repo location #1295 @rossturk
      • Airflow: change how pip installs packages in tox environments #1302 @JDarDagran

      Bug fixes and more details: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md

• Rust implementation of the SQL integration [Piotr]
  • About me: dev with GetInData
  • Goal of project: to make adding more language support in the future easier to add
  • Separated into components: separate backend package for integration with language bindings with new Java interface
  • Components
    • openlineage_sql: main implementation with table + column lineage extraction
    • openlineage_sql_python: Python bindings, uses the pyo3  create, produces a Python wheel
    • openlineage_sql_java: Java bindings, using JNI, produces a jar
  • Changes
    • switch to a visitor pattern to traverse the AST
    • introduce Context Frames (like scopes) to resolve aliases, implicit contexts and shadowing
    • column lineage is a synthesized attribute over the tree – easy to compute with a visitor
  • Demo
  • Shout outs
    • Maciej Obuchowski (@mobuchowski)
    • Will Johnson (@wjohnson)
    • Hannah Moazam (@hmoazam)

November 10, 2022 (10am PT)

...