Page History

...

• Announcements [Julien]
  • OpenLineage earned the OSSF Core Infrastructure Silver Badge!
  • Happening soon: OpenLineage to apply formally for Incubation status with the LFAI
  • Blog: a post by Ernie Ostic about MANTA’s OpenLineage integration
  • Website: a new Ecosystem page
  • Workshops repo: An Intro to Dataset Lineage with Jupyter and Spark
  • Airflow docs: guidance on creating custom extractors to support external operators
  • Spark docs: improved documentation of column lineage facets and extensions
• Recent release 0.16.1 [Michael R.] 

  • Added

    • Airflow: add dag_run information to Airflow version run facet #1133 @fm100
      Adds the Airflow DAG run ID to the taskInfo facet, making this additional information available to the integration.
    • Airflow: add LoggingMixin to extractors #1149 @JDarDagran
      Adds a LoggingMixin class to the custom extractor to make the output consistent with general Airflow and OpenLineage logging settings.
    • Airflow: add default extractor #1162 @mobuchowski
      Adds a DefaultExtractor to support the default implementation of OpenLineage for external operators without the need for custom extractors.
    • Airflow: add on_complete argument in DefaultExtractor #1188 @JDarDagran
      Adds support for running another method on extract_on_complete.
    • SQL: reorganize the library into multiple packages #1167 @StarostaGit @mobuchowski
      Splits the SQL library into a Rust implementation and foreign language bindings, easing the process of adding language interfaces. Also contains a CI fix.

    Changed

    • Airflow: move get_connection_uri as extractor's classmethod #1169 @JDarDagran
      The get_connection_uri method allowed for too many params, resulting in unnecessarily long URIs. This changes the logic to whitelisting per extractor.
    • Airflow: change get_openlineage_facets_on_start/complete behavior #1201 @JDarDagran
      Splits up the method for greater legibility and easier maintenance.

  • Removed

    • Airflow: remove support for Airflow 1.10 #1128 @mobuchowski
      Removes the code structures and tests enabling support for Airflow 1.10.

  • Bug fixes and more details 

    • https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md

• Update on LFAI & Data progress [Michael R.]
  • LFAI & Data: a single funding effort to support technical projects hosted under the [Linux] foundation
  • Current status: applying soon for Incubation, will be ready to apply for Graduation soon (dates TBD).
  • Incubation stage requirements:

    • 
      

      2+ organizations actively contributing to the project	23 organizations
      A sponsor who is an existing LFAI & Data member	To do
      300+ stars on GitHub	1.1K GitHub stars
      A Core Infrastructure Initiative Best Practices Silver Badge	Silver Badge earned on November 2
      Affirmative vote of the TAC and Governing Board	Pending
      A defined TSC with a chairperson	TSC with chairperson: Julien Le Dem

      Graduation stage requirements:

    • 
      

      5+ organizations actively contributing to the project	23 organizations
      Substantial flow of commits for 12 months	Commit growth rate (12 mo.): 155.53% Avg commits pushed by active contributors (12 mo.): 2.18K
      1000+ stars on GitHub	1.1K GitHub stars
      Core Infrastructure Initiative Best Practices Gold Badge	Gold Badge in progress (57%)
      Affirmative vote of the TAC and Governing Board	Pending
      1+ collaboration with another LFAI project	Marquez, Egeria, Amundsen
      Technical lead appointed on the TAC	To do

      
      

October 13, 2022 (10am PT)

...