...
Next meeting: Dec 8th 2021 (9am PT)
Tentative agenda:
- SPDX headers [Mandy Chessel]
- OpenLineage and Azure Purview [Shrikanth]+ OpenLineage [Will Johnson, Mark Taylor]
- Logging backend (OpenTelemetry, ...) [Julien Le Dem]
- Open discussion
Nov 10th 2021 (9am PT)
Attendees:
...
Open Questions:
when facets are sent?
do we send them immediately? Do we wait?
iceberg not creating a facet until Spark asks for the splits
preference to sending events as they go.
does that it fit with the OpenLineage view of the world? => yes
do we send them immediately? Do we wait?
iceberg not creating a facet until Spark asks for the splits
Spark, bound to a context thread:
the "logger backend can grab the sql execution id"
loggers depend on thread
listener is on different thread
Report for a given job run
Ryan: runcontext is threadlocal: sets the executionid.
the "logger backend can grab the sql execution id"The client side should be able to send an event immediately vs sent when you get a chance.
Who needs to do this?
Need to have a guide to defining a facet.
Who needs to do this?
Michael C.: TODO: Design Doc on logging
Willy: Do we need a "RUNNING" event?
...