...
- TSC:
Michael Collado: Datakin
Julien Le Dem: OpenLineage Project Lead, Datakin
Maciej Obuchowski: GetInData, OpenLineage
Willy Lulciuc: Marquez, OpenLineage
Mandy Chessel: Egeria Project Lead, working on OpenLineage
- And:
Ross Turk: VP marketing at Datakin talk about the website
Minkyu Park: interested in contributing to Datakin
Peter Hicks: Marquez contributor, OpenLineage user
Widget Connector | ||
---|---|---|
|
- Notes:
- OpenLineage website: https://openlineage.io/
- Gatsby based (markdown) in OpenLineage/website repo
- generates a static site hosted in github pages. OpenLineage/OpenLineage.github.io
- deployment is currently manual. Automation in progress
- Please open PRs on /website to contribute a blog posts.
- Getting started with Egeria?
- Suggestions:
- Add page on open governance and how to join the project.
- Add LFAI & data banner to the website?
- Egeria is using MKdocs: very nice to navigate documentation.
- upcoming 0.3.0:
- Facet versioning:
- each facet schema is versioned individually.
- client/server code generation to facilitate producing/consuming openlineage events
- Spark 3.x support
- new mechanism for airflow 2.x
- working with airflow maintainer to improve that.
- Facet versioning:
- Proxy Backend update (planned for OL 0.4.0):
- mapping to egeria backend
- planning to release for the Egeria webinar on the 8th of November
- Willy provided a base module for ProxyBackend
- Monthly release is a good cadence
Open discussions:
Azure purview team hackathon ongoing to consumer OpenLineage events
Design docs discussion:
proposal to add design doc for proposal.
goal:
Similar to the process of projects like Kafka, Flink: for specs and bigger features
not for bug fixes.
options:
proposal directory for docs as markdown
Open PRs against wiki pages: proposals wiki.
Manage status:
list of designs that are implemented vs pending.
table of open proposals.
vote for prioritization:
Every proposal design doc has an issue opened and link back to it.
good start for the blog talking about that feature
New committee on data ops: Mandy will be speaking about Egeria and OpenLineage
Scope:
How the foundation projects should work together around the topic.
Establish OpenLineage is important.
https://wiki.lfaidata.foundation/display/DL/DataOps+Committee
- OpenLineage website: https://openlineage.io/
...