Parent issue: #2058
Summary
Add minimum Spark Connect event-log awareness so Connect server logs can be accepted and correlated by the tools.
This is the Phase 2 work item from #2058.
Problem
Spark Connect introduces additional listener event types that appear in standard event logs, but the current parser configuration filters them out. Without at least accepting these events, the tools cannot reliably detect Connect mode or correlate operations with existing SQL/job events.
Scope
- Detect Connect mode via presence of
SparkListenerConnectServiceStarted
- Extend parser/event configuration to accept Connect-specific listener event types
- Parse and preserve the
jobTag correlation signal needed to link Connect operations with SQL executions and Spark jobs
- Ensure Connect logs no longer fail or lose key metadata at parse time solely because of unsupported event names
Acceptance Criteria
- Connect-specific event classes are accepted by parser configuration
- The tools can identify an app/event log as a Spark Connect server log
jobTag correlation data is available to downstream processing
- Existing non-Connect event-log parsing remains unchanged
- Tests cover parser acceptance and jobTag correlation extraction
Notes
Relevant analysis in repo:
core/docs/spark-connect-events-analysis.md
core/docs/spark-connect-modifiedConfigs-analysis.md
Parent issue: #2058
Summary
Add minimum Spark Connect event-log awareness so Connect server logs can be accepted and correlated by the tools.
This is the Phase 2 work item from #2058.
Problem
Spark Connect introduces additional listener event types that appear in standard event logs, but the current parser configuration filters them out. Without at least accepting these events, the tools cannot reliably detect Connect mode or correlate operations with existing SQL/job events.
Scope
SparkListenerConnectServiceStartedjobTagcorrelation signal needed to link Connect operations with SQL executions and Spark jobsAcceptance Criteria
jobTagcorrelation data is available to downstream processingNotes
Relevant analysis in repo:
core/docs/spark-connect-events-analysis.mdcore/docs/spark-connect-modifiedConfigs-analysis.md