Page History

...

• Announcements [Julien]
  • Two meetups will be happening soon:
    • Data Lineage Meetup cohosted with Collibra, Providence, RI, March 9 at 6 PM ET
    • OpenLineage Meetup at Data Council Austin on March 30th at 12:15 PM CST
  • Talk happening soon:
    • Julien Le Dem, "Ten Years of Building Open Source Standards: From Parquet to Arrow to OpenLineage," Data Council Austin, March 30th, 10 AM CST
• Recent releases 0.20.6, 0.21.1
  • 0.20.6

    • Added

      • Airflow: add new extractor for FTPFileTransmitOperator #1603 @sekikn
    • Changed
      • Airflow: make extractors for async operators work #1601 @JDarDagran
  • 0.21.1
    • Added
      • Clients: add DEBUG logging of events to transports #1633 by @mobuchowski
      • Spark: add CustomEnvironmentFacetBuilder class #1545 by New contributor@Anirudh181001
      • Spark: introduce the new output visitors AlterTableAddPartitionCommandVisitor and AlterTableSetLocationCommandVisitor#1629 by New contributor@nataliezeller1
      • Spark: add column lineage for JDBC relations #1636 by @tnazarew
      • SQL: add Linux-aarch64 native library to Java SQL parser #1664 by @mobuchowski
    • Changed
      • Airflow: get table database in Athena extractor #1631 by New contributor@rinzool
    • Fixed
      • dbt: add dbt seed to the list of dbt-ol events #1649 by New contributor @pohek321
  • Thanks to all our contributors!
  • More details: 
    • https://github.com/OpenLineage/OpenLineage/releases
    • https://github.com/OpenLineage/OpenLineage/blob/0.21.1/CHANGELOG.md
• Custom env var support in the Spark integration [Anirudh]
  
  • adds ability to capture environment variables from a Spark cluster
  • required the addition of a new class to extend an existing class
  • does not override variables already being captured
  • desired variables must be specified by the user
  • variables are visible in environment properties of OpenLineage events
  • Q & A
    • Q: is it possible to accidentally include sensitive data in these variables?
    • A: users must "opt in" by selecting variables in advance
    • Q: what was the experience like interacting with the community?
    • A: really great! I got a lot of help from a lot of people, including Pawel

February 9, 2023 (10am PT)

...