Page History

...

• TSC:
  • Mike Collado: Staff Software Engineer, Datakin
  • Maciej Obuchowski: Software Engineer, GetInData, OpenLineage contributor
  • Julien Le Dem: OpenLineage Project lead
  • Willy Lulciuc: Co-creator of Marquez
• And:
  • Ernie Ostic: SVP of Product, Manta 
  • Sandeep Adwankar: Senior Technical Product Manager, AWS
  • Paweł Leszczyński, Software Engineer, GetinData
  • Howard Yoo: Staff Project Manager, Astronomer
  • Michael Robinson: Developer Relations Engineer, Astronomer
  • Ross Turk: Senior Director of Community, Astronomer
  • Minkyu Park: Senior Software Engineer, Astronomer
  • Will Johnson: Senior Cloud Solution Architect, Azure Cloud, Microsoft

Meeting info:

...

...

Notes:

• Releases
  • 0.8.2

    • Added

      • openlineage-airflow now supports getting credentials from Airflows secrets backend (#723) @mobuchowski
      • openlineage-spark now supports Azure Databricks Credential Passthrough (#595) @wjohnson
      • openlineage-spark detects datasets wrapped by ExternalRDDs (#746) @collado-mike

      Fixed

      • PostgresOperator fails to retrieve host and conn during extraction (#705) @sekikn
      • SQL parser accepts lists of sql statements (#734) @mobuchowski
  • 0.8.1

    • Added

      • Airflow integration uses new TaskInstance listener API for Airflow 2.3+ (#508) @mobuchowski
      • Support for HiveTableRelation as input source in Spark integration (#683) @collado-mike
      • Add HTTP and Kafka Client to openlineage-java lib (#480) @wslulciuc, @mobuchowski
      • New SQL parser, used by Postgres, Snowflake, Great Expectations integrations (#644) @mobuchowski

      Fixed

      GreatExpectations: Fixed bug when invoking GreatExpectations using v3 API (#683) @collado-mike
  • 0.7.1

    • Added

      • Python implements Transport interface - HTTP and Kafka transports are available (#530) @mobuchowski
      • Add UnknownOperatorAttributeRunFacet and support in lineage backend (#547) @collado-mike
      • Support Spark 3.2.1 (#607) @pawel-big-lebowski
      • Add StorageDatasetFacet to spec (#620) @pawel-big-lebowski
      • README.md created at OpenLineage/integrations for compatibility matrix (#663) @howardyoo

      Fixed

      • Airflow: custom extractors lookup uses only get_operator_classnames method (#656) @mobuchowski
      • Dagster: handle updated PipelineRun in OpenLineage sensor unit test (#624) @dominiquetipton
      • Delta improvements (#626) @collado-mike
      • Fix SqlDwDatabricksVisitor for Spark2 (#630) @wjohnson
      • Airflow: remove redundant logging from GE import (#657) @mobuchowski
      • Fix Shebang issue in Spark's wait-for-it.sh (#658) @mobuchowski
      • Update parent_run_id to be a uuid from the dag name and run_id (#664) @collado-mike
      • Spark: fix time zone inconsistency in testSerializeRunEvent (#681) @sekikn
• Communication reminders [Julien]
• Agenda [Julien]
• Column-level lineage [Paweł]
  • Linked to 4 PRs, the first being a proposal
  • The second has been merged, but the core mechanism is turned off
  • 3 requirements:
    • Outputs labeled with expression IDs
    • Inputs with expression IDs
    • Dependencies
  • Once it is turned on, each OL event will receive a new JSON field
  • It would be great to be able to extend this API (currently on the roadmap)
  • Q & A
    • Will: handling user-defined functions: is the solution already generic enough?
      • The answer will depend on testing, but I suspect that the answer is yes
      • The team at Microsoft would be excited to learn that the solution will handle UDFs
    • Julien: the next challenge will be to ensure that all the integrations support column-level lineage
• Open discussion
  • Willy: in Mqz we need to start handling col-level lineage, and has anyone thought about how this might work?
    • Julien: lineage endpoint for col-level lineage to layer on top of what already exists
    • Willy: this makes sense – we could use the method for input and output datasets as a model
    • Michael C.: I don't know that we need to add an endpoint – we could augment the existing one to do something with the data
    • Willy: how do we expect this to be visualized?
      • Julien: not quite sure
      • Michael C.: there are a number of different ways we could do this, including isolating relevant dataset fields 

...