Skip to content

Move TailLogs endpoint to dataproxy#7194

Open
katrogan wants to merge 10 commits intov2from
katrina/eng26-371-move-runlogsservice-to-dataproxy
Open

Move TailLogs endpoint to dataproxy#7194
katrogan wants to merge 10 commits intov2from
katrina/eng26-371-move-runlogsservice-to-dataproxy

Conversation

@katrogan
Copy link
Copy Markdown
Contributor

@katrogan katrogan commented Apr 10, 2026

And add a GetActionLogContext to fetch log context from RunsService

Why are the changes needed?

Required to reduce dataflow in a secure environment.

What changes were proposed in this pull request?

See PR title

How was this patch tested?

Ran the demo cluster locally. Tested with flyte-sdk and this script

@env.task
async def spew_logs(duration_sec: int = 300, interval_sec: int = 5) -> str:
    elapsed = 0
    iteration = 0
    while elapsed < duration_sec:
        iteration += 1
        print(f"[{elapsed:>4}s] iteration {iteration} - still running...", flush=True)
        await asyncio.sleep(interval_sec)
        elapsed += interval_sec
    print(f"Done after {elapsed}s ({iteration} iterations)", flush=True)
    return f"completed {iteration} iterations"


if __name__ == "__main__":
    import sys

    config_path = sys.argv[1] if len(sys.argv) > 1 else None
    if config_path:
        from flyte.config import Config

        flyte.init_from_config(Config.auto(config_path))
    else:
        flyte.init_from_config()
    run = flyte.run(spew_logs)
    print(f"Run: {run.name}")
    print(f"URL: {run.url}")
    print("Tailing logs...")
    run.action.show_logs(raw=True)

Verified output worked fine

 python examples/basics/long_running.py .flyte/config-oss-local.yaml
  > Bundling code...
  ✓ Code bundle: 1 files, 0.009765625 MB (compressed 0.0006895065307617188 MB)
  > Uploading code bundle...
Run: r8xqdw87c7l266b9j99x
URL: http://localhost:30080/v2/domain/development/project/flytesnacks/runs/r8xqdw87c7l266b9j99x
Tailing logs...
 [flyte] Flyte runtime started for action a0 with run name r8xqdw87c7l266b9j99x [   0s] iteration 1 - still running... [   5s] iteration 2 - still running... [  10s] iteration 3 - still running...

Labels

Please add one or more of the following labels to categorize your PR:

  • added: For new features.
  • changed: For changes in existing functionality.
  • deprecated: For soon-to-be-removed features.
  • removed: For features being removed.
  • fixed: For any bug fixed.
  • security: In case of vulnerabilities

This is important to improve the readability of release notes.

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Docs link

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
@github-actions github-actions bot mentioned this pull request Apr 10, 2026
3 tasks
Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
Comment on lines +1 to +19
package logs

import (
"context"
"fmt"

"connectrpc.com/connect"
"github.com/samber/lo"

"github.com/flyteorg/flyte/v2/gen/go/flyteidl2/core"
"github.com/flyteorg/flyte/v2/gen/go/flyteidl2/workflow"
)

// LogStreamer abstracts log fetching from different backends.
type LogStreamer interface {
TailLogs(ctx context.Context, logContext *core.LogContext, stream *connect.ServerStream[workflow.TailLogsResponse]) error
}

// GetPrimaryPodAndContainer finds the primary pod and container from a LogContext.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: could we move this file to dataproxy/ instead of adding a new folder at the project root

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done but now run logs service depends on logs service - is that okay? eventually we can rip out the run logs service implementation once the sdk changes are in, so i guess it's temporary

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup

@pingsutw pingsutw added this to the V2 GA milestone Apr 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants