[Flyte 7198] Enabling flyte plugins

[0yukali0]

Tracking issue

Close #7198

Why are the changes needed?

Flyte v2 OSS won't trigger ray, dask, spark, torch, tensorflow and mpi plugins

What changes were proposed in this pull request?

1. Register CRD
2. Initialize informer and client for operate CRD
3. Filtering unset plugins

How was this patch tested?

1. make sandbox-build && make sandbox-run FLYTE_DEV=True
2. Installing crd opreators
3. make -C manager run

Labels

• added: For new features.

This is important to improve the readability of release notes.

Setup process

1. Update complete.yaml

enabled_plugins:
  tasks:
    task-plugins:
        enabled-plugins:
          - container
          - sidecar
          - connector-service
          - echo
          - ray
          - spark
          - dask
          - pytorch
          - tensorflow
          - mpi
        default-for-task-types:
          container: container
          sidecar: sidecar
          container_array: k8s-array

2. make devbox-start
3. Installing crd opreators

spark_role.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: spark
  namespace: flyte
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: spark-testproject-development
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: edit
subjects:
  - kind: ServiceAccount
    name: spark
    namespace: flyte

Screenshots

1. spark
2. ray
3. dask
4. torch

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

• main
  • Flyte 2: Cut over to Flyte 2 new backend implementation #6583
    • [Flyte 7198] Enabling flyte plugins 👈

[pingsutw]

@0yukali0 is it ready to be reviewed?

[0yukali0]

Hi @pingsutw
No, lack a filtering implementation when setting yaml of enabling plugins.
Current implementation registers all plugins by default, some plugins are not listed in enabled plugins should be skipped.
i will complete this function later, this PR will be reviewable tomorrow.

[0yukali0]

Hi @pingsutw , this PR is ready.
If there are new suggestions, pls tag me and let me know.
thx.

[pingsutw]

Could we keep the same (tasks/task-plugins)? will make people easy to migrate to v2

[pingsutw]

flyte/flyteplugins/go/tasks/plugins/k8s/dask/dask.go

Line 488 in 1dd0464

if err := daskAPI.AddToScheme(scheme.Scheme); err != nil {

do we still need to add schema here?

[0yukali0]

Hi @pingsutw.
I encountered into a problem that scheme not found when i removed these.
i think is that controller-runtime manager won't recognize schemes if initialization in setup.go doesn't register them.

[pingsutw]

could we add these to loader.go
https://github.com/flyteorg/flyte/blob/master/flytepropeller/plugins/loader.go

[pingsutw]

We could add a helper function and return a PluginConfigMeta

flyte/flytepropeller/pkg/controller/nodes/task/config/config.go

Lines 66 to 75 in cd7371f

	func (p TaskPluginConfig) GetEnabledPlugins() (PluginsConfigMeta, error) {
	enabledPluginsNames := sets.NewString()
	for _, pluginName := range p.EnabledPlugins {
	cleanedPluginName := cleanString(pluginName)
	enabledPluginsNames.Insert(cleanedPluginName)
	}
	pluginDefaultForTaskType := make(map[PluginID][]TaskType)
	// Reverse the DefaultForTaskTypes map. Having the config use task type as a key guarantees only one default plugin can be specified per
	// task type but now we need to sort for which tasks a plugin needs to be the default.