Skip to content

[FEA] Add minimum Spark Connect event-log awareness to parser #2064

@sayedbilalbari

Description

@sayedbilalbari

Parent issue: #2058

Summary

Add minimum Spark Connect event-log awareness so Connect server logs can be accepted and correlated by the tools.

This is the Phase 2 work item from #2058.

Problem

Spark Connect introduces additional listener event types that appear in standard event logs, but the current parser configuration filters them out. Without at least accepting these events, the tools cannot reliably detect Connect mode or correlate operations with existing SQL/job events.

Scope

  • Detect Connect mode via presence of SparkListenerConnectServiceStarted
  • Extend parser/event configuration to accept Connect-specific listener event types
  • Parse and preserve the jobTag correlation signal needed to link Connect operations with SQL executions and Spark jobs
  • Ensure Connect logs no longer fail or lose key metadata at parse time solely because of unsupported event names

Acceptance Criteria

  • Connect-specific event classes are accepted by parser configuration
  • The tools can identify an app/event log as a Spark Connect server log
  • jobTag correlation data is available to downstream processing
  • Existing non-Connect event-log parsing remains unchanged
  • Tests cover parser acceptance and jobTag correlation extraction

Notes

Relevant analysis in repo:

  • core/docs/spark-connect-events-analysis.md
  • core/docs/spark-connect-modifiedConfigs-analysis.md

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions