...
- Announcements [Julien]
- OpenLineage earned the OSSF Core Infrastructure Silver Badge!
- Happening soon: OpenLineage to apply formally for Incubation status with the LFAI
- Blog: a post by Ernie Ostic about MANTA’s OpenLineage integration
- Website: a new Ecosystem page
- Workshops repo: An Intro to Dataset Lineage with Jupyter and Spark
- Airflow docs: guidance on creating custom extractors to support external operators
- Spark docs: improved documentation of column lineage facets and extensions
- Recent release 0.16.1 [Michael R.]
Added
- Airflow: add dag_run information to Airflow version run facet #1133 @fm100
Adds the Airflow DAG run ID to the taskInfo facet, making this additional information available to the integration. - Airflow: add LoggingMixin to extractors #1149 @JDarDagran
Adds a LoggingMixin class to the custom extractor to make the output consistent with general Airflow and OpenLineage logging settings. - Airflow: add default extractor #1162 @mobuchowski
Adds a DefaultExtractor to support the default implementation of OpenLineage for external operators without the need for custom extractors. - Airflow: add on_complete argument in DefaultExtractor #1188 @JDarDagran
Adds support for running another method on extract_on_complete. - SQL: reorganize the library into multiple packages #1167 @StarostaGit @mobuchowski
Splits the SQL library into a Rust implementation and foreign language bindings, easing the process of adding language interfaces. Also contains a CI fix.
Changed
- Airflow: move get_connection_uri as extractor's classmethod #1169 @JDarDagran
The get_connection_uri method allowed for too many params, resulting in unnecessarily long URIs. This changes the logic to whitelisting per extractor. - Airflow: change get_openlineage_facets_on_start/complete behavior #1201 @JDarDagran
Splits up the method for greater legibility and easier maintenance.
- Airflow: add dag_run information to Airflow version run facet #1133 @fm100
Removed
- Airflow: remove support for Airflow 1.10 #1128 @mobuchowski
Removes the code structures and tests enabling support for Airflow 1.10.
- Airflow: remove support for Airflow 1.10 #1128 @mobuchowski
Bug fixes and more details
- Update on LFAI & Data progress [Michael R.]
- LFAI & Data: a single funding effort to support technical projects hosted under the [Linux] foundation
- Current status: applying soon for Incubation, will be ready to apply for Graduation soon (dates TBD).
- Incubation stage requirements:
2+ organizations actively contributing to the project
23 organizations
A sponsor who is an existing LFAI & Data member
To do
300+ stars on GitHub
1.1K GitHub stars
A Core Infrastructure Initiative Best Practices Silver Badge
Silver Badge earned on November 2
Affirmative vote of the TAC and Governing Board
Pending
A defined TSC with a chairperson
TSC with chairperson: Julien Le Dem
Graduation stage requirements:
5+ organizations actively contributing to the project
23 organizations
Substantial flow of commits for 12 months
Commit growth rate (12 mo.): 155.53%
Avg commits pushed by active contributors (12 mo.): 2.18K
1000+ stars on GitHub
1.1K GitHub stars
Core Infrastructure Initiative Best Practices Gold Badge
Gold Badge in progress (57%)
Affirmative vote of the TAC and Governing Board
Pending
1+ collaboration with another LFAI project
Marquez, Egeria, Amundsen
Technical lead appointed on the TAC
To do
October 13, 2022 (10am PT)
...