...
- Announcements [Julien]
- Two meetups will be happening soon:
- Data Lineage Meetup cohosted with Collibra, Providence, RI, March 9 at 6 PM ET
- OpenLineage Meetup at Data Council Austin on March 30th at 12:15 PM CST
- Talk happening soon:
- Julien Le Dem, "Ten Years of Building Open Source Standards: From Parquet to Arrow to OpenLineage," Data Council Austin, March 30th, 10 AM CST
- Two meetups will be happening soon:
- Recent releases 0.20.6, 0.21.1
- 0.20.6
Added
- Changed
- Airflow: make extractors for async operators work #1601 @JDarDagran
- 0.21.1
- Added
- Clients: add DEBUG logging of events to transports #1633 by @mobuchowski
- Spark: add CustomEnvironmentFacetBuilder class #1545 by New contributor@Anirudh181001
- Spark: introduce the new output visitors AlterTableAddPartitionCommandVisitor and AlterTableSetLocationCommandVisitor#1629 by New contributor@nataliezeller1
- Spark: add column lineage for JDBC relations #1636 by @tnazarew
- SQL: add Linux-aarch64 native library to Java SQL parser #1664 by @mobuchowski
- Changed
- Fixed
- Added
- Thanks to all our contributors!
- More details:
- 0.20.6
- Custom env var support in the Spark integration [Anirudh]
- adds ability to capture environment variables from a Spark cluster
- required the addition of a new class to extend an existing class
- does not override variables already being captured
- desired variables must be specified by the user
- variables are visible in environment properties of OpenLineage events
- Q & A
- Q: is it possible to accidentally include sensitive data in these variables?
- A: users must "opt in" by selecting variables in advance
- Q: what was the experience like interacting with the community?
- A: really great! I got a lot of help from a lot of people, including Pawel
February 9, 2023 (10am PT)
...