Support multiple --tasks directories and absolute paths

[vdemeester]

Current Behavior

Chisel currently supports a single --tasks flag that specifies where to look for task definitions. Task lookup is limited to paths relative to either:

• The directory containing the PipelineRun YAML, or
• The single directory specified by --tasks=<dir>

File discovery pattern (from pkg/parser/parser.go:470-496):

patterns := []string{
    filepath.Join(baseDir, name+".yaml"),
    filepath.Join(baseDir, name+".yml"),
    filepath.Join(baseDir, "tasks", name+".yaml"),
    filepath.Join(baseDir, "tasks", name+".yml"),
}

Limitations

1. Single search location: Can only search one base directory
2. No absolute paths: Cannot specify absolute paths for task locations
3. No multiple sources: Cannot combine tasks from different directories
4. Inflexible organization: Forces specific directory structure

Desired Behavior

Multiple --tasks Flags

Allow specifying multiple directories to search for tasks:

chisel run pipeline.yaml \
  --tasks=./common-tasks \
  --tasks=./project-tasks \
  --tasks=/shared/company-tasks

Tasks would be searched in order, first match wins.

Absolute Path Support

Support absolute paths in addition to relative:

chisel run pipeline.yaml --tasks=/opt/tekton/tasks

Use Cases

1. Shared task library across projects:

/shared/tekton-tasks/       # Company-wide tasks
  ├── security-scan.yaml
  └── deploy-k8s.yaml

/project/my-app/            # Project-specific tasks
  ├── pipeline.yaml
  └── tasks/
      └── build-app.yaml

Run with:

chisel run pipeline.yaml \
  --tasks=/shared/tekton-tasks \
  --tasks=./tasks

2. Monorepo with multiple task sets:

monorepo/
  ├── common-tasks/
  ├── frontend-tasks/
  └── backend-tasks/

Run backend pipeline with access to common tasks:

chisel run backend/pipeline.yaml \
  --tasks=./common-tasks \
  --tasks=./backend-tasks

3. Local development with overrides:

# Override prod tasks with local versions
chisel run pipeline.yaml \
  --tasks=./local-overrides \
  --tasks=./prod-tasks

Implementation Notes

CLI Changes (cmd/chisel/run.go)

Change from single string to string slice:

// Current
var tasksDir string
rootCmd.Flags().StringVar(&tasksDir, "tasks", "", "directory containing task definitions")

// Proposed
var tasksDirs []string
rootCmd.Flags().StringArrayVar(&tasksDirs, "tasks", nil, "directories to search for task definitions (can be specified multiple times)")

Parser Changes (pkg/parser/parser.go)

Update Options and loadTask:

type Options struct {
    TasksDirs []string  // Changed from single TasksDir
    Debug     bool
}

func (p *Parser) loadTask(name string, baseDirs []string) (*TektonTask, error) {
    // Check cache first
    if task, ok := p.taskCache[name]; ok {
        return task, nil
    }

    // Search in each baseDir in order
    for _, baseDir := range baseDirs {
        patterns := []string{
            filepath.Join(baseDir, name+".yaml"),
            filepath.Join(baseDir, name+".yml"),
            filepath.Join(baseDir, "tasks", name+".yaml"),
            filepath.Join(baseDir, "tasks", name+".yml"),
        }

        for _, pattern := range patterns {
            if data, err := os.ReadFile(pattern); err == nil {
                var task TektonTask
                if err := yaml.Unmarshal(data, &task); err != nil {
                    return nil, err
                }
                p.taskCache[name] = &task
                return &task, nil
            }
        }
    }

    return nil, fmt.Errorf("task %s not found in any of: %v", name, baseDirs)
}

Absolute Path Handling

Paths should be used as-is if absolute, converted to absolute if relative:

func resolveTaskDirs(dirs []string, pipelineDir string) []string {
    var resolved []string
    for _, dir := range dirs {
        if filepath.IsAbs(dir) {
            resolved = append(resolved, dir)
        } else {
            resolved = append(resolved, filepath.Join(pipelineDir, dir))
        }
    }
    return resolved
}

Backward Compatibility

• If no --tasks flags specified, default to pipeline directory (current behavior)
• Single --tasks flag works exactly as before
• Multiple flags enable new search behavior

Alternative Considered

Could use path separator to specify multiple directories in single flag:

chisel run pipeline.yaml --tasks="./tasks1:./tasks2:/shared/tasks"

But multiple flags is more idiomatic for CLI tools.

Related

This enhancement would work well with future remote resolution support (#TODO), where one could specify both local and remote task sources.