Update docs, schemas and tests to v2

Signed-off-by: Alberto Gutierrez <aljesusg@gmail.com>

Author: aljesusg

README.md

@@ -442,8 +442,8 @@ In case multi-cluster support is enabled (default) and you have access to multip
 - **kiali_manage_istio_config_read** - Read-only Istio config: list or get objects. For action 'list', returns an array of objects with {name, namespace, type, validation}. For create, patch, or delete use manage_istio_config.
   - `action` (`string`) **(required)** - Action to perform (read-only)
   - `clusterName` (`string`) - Optional cluster name. Defaults to the cluster name in the Kiali configuration.
-  - `group` (`string`) - API group of the Istio object (e.g., 'networking.istio.io', 'gateway.networking.k8s.io'). Required for 'get' action.
-  - `kind` (`string`) - Kind of the Istio object (e.g., 'VirtualService', 'DestinationRule'). Required for 'get' action.
+  - `group` (`string`) - API group of the Istio object. Required for 'get' action.
+  - `kind` (`string`) - Kind of the Istio object. Required for 'get' action.
   - `namespace` (`string`) - Namespace containing the Istio object. For 'list', if not provided, returns objects across all namespaces. For 'get', required.
   - `object` (`string`) - Name of the Istio object. Required for 'get' action.
   - `serviceName` (`string`) - Filter Istio configurations (VirtualServices, DestinationRules, and their referenced Gateways) that affect a specific service. Only applicable for 'list' action
@@ -452,9 +452,8 @@ In case multi-cluster support is enabled (default) and you have access to multip
 - **kiali_manage_istio_config** - Create, patch, or delete Istio config. For list and get (read-only) use manage_istio_config_read.
   - `action` (`string`) **(required)** - Action to perform (write)
   - `clusterName` (`string`) - Optional cluster name. Defaults to the cluster name in the Kiali configuration.
-  - `confirmed` (`boolean`) - CRITICAL: If 'true', the destructive action (create/patch/delete) is executed. If 'false' (or omitted) for create/patch, the tool returns a YAML PREVIEW. Display it to the user and ask for confirmation before calling again with confirmed=true.
   - `data` (`string`) - Complete JSON or YAML data to apply or create the object. Required for create and patch actions. You MUST provide a COMPLETE and VALID manifest with ALL required fields for the resource type. Arrays (like servers, http, etc.) are REPLACED entirely, so you must include ALL required fields within each array element.
-  - `group` (`string`) **(required)** - API group of the Istio object (e.g., 'networking.istio.io', 'gateway.networking.k8s.io').
+  - `group` (`string`) **(required)** - API group of the Istio object
   - `kind` (`string`) **(required)** - Kind of the Istio object (e.g., 'VirtualService', 'DestinationRule').
   - `namespace` (`string`) **(required)** - Namespace containing the Istio object
   - `object` (`string`) **(required)** - Name of the Istio object
@@ -466,6 +465,17 @@ In case multi-cluster support is enabled (default) and you have access to multip
   - `resourceName` (`string`) - Optional. The specific name of the resource. If left empty, the tool returns a list of all resources of the specified type. If provided, the tool returns deep details for this specific resource.
   - `resourceType` (`string`) **(required)** - The type of resource to query.
 
+- **kiali_list_traces** - Lists distributed traces for a service in a namespace. Returns a summary (namespace, service, total_found, avg_duration_ms) and a list of traces with id, duration_ms, spans_count, root_op, slowest_service, has_errors. Use get_trace_details with a trace id to get full hierarchy.
+  - `clusterName` (`string`) - Optional cluster name. Defaults to the cluster name in the Kiali configuration.
+  - `errorOnly` (`boolean`) - If true, only consider traces that contain errors. Default false.
+  - `limit` (`integer`) - Maximum number of traces to return. Default 10.
+  - `lookbackSeconds` (`integer`) - How far back to search. Default 600 (10m).
+  - `namespace` (`string`) **(required)** - Kubernetes namespace of the service.
+  - `serviceName` (`string`) **(required)** - Service name to search traces for (required). Returns multiple traces up to limit.
+
+- **kiali_get_trace_details** - Fetches a single distributed trace by trace_id and returns its call hierarchy (service tree with duration, status, and nested calls). Use this after list_traces to drill into a specific trace.
+  - `traceId` (`string`) **(required)** - Trace ID to fetch and summarize. If provided, namespace/service_name are ignored.
+
 - **kiali_get_pod_performance** - Returns a human-readable text summary with current Pod CPU/memory usage (from Prometheus) compared to Kubernetes requests/limits (from the Pod spec). Useful to answer questions like 'Is this workload using too much memory?'
   - `clusterName` (`string`) - Optional. Name of the cluster to get resources from. If not provided, will use the default cluster name in the Kiali KubeConfig
   - `namespace` (`string`) **(required)** - Kubernetes namespace of the Pod.
@@ -474,16 +484,6 @@ In case multi-cluster support is enabled (default) and you have access to multip
   - `timeRange` (`string`) - Time window used to compute CPU rate (Prometheus duration like '5m', '10m', '1h', '1d'). Defaults to '10m'.
   - `workloadName` (`string`) - Kubernetes Workload name (e.g. Deployment/StatefulSet/etc). Tool will look up the workload and pick one of its Pods. If not found, it will fall back to treating this value as a podName.
 
-- **kiali_get_traces** - Fetches a distributed trace (Jaeger/Tempo) by trace_id or searches by service_name (optionally only error traces) and summarizes bottlenecks and error spans.
-  - `clusterName` (`string`) - Optional cluster name. Defaults to the cluster name in the Kiali configuration.
-  - `errorOnly` (`boolean`) - If true, only consider traces that contain errors (e.g. error=true / non-200 status). Default false.
-  - `limit` (`integer`) - Max number of traces to consider when searching by service_name. Default 10.
-  - `lookbackSeconds` (`integer`) - How far back to search when using service_name. Default 600 (10m).
-  - `maxSpans` (`integer`) - Max number of spans to return in each summary section (bottlenecks, errors, roots). Default 7.
-  - `namespace` (`string`) - Kubernetes namespace of the service (required when trace_id is not provided).
-  - `serviceName` (`string`) - Service name to search traces for (required when trace_id is not provided).
-  - `traceId` (`string`) - Trace ID to fetch and summarize. If provided, namespace/service_name are ignored.
-
 - **kiali_get_logs** - Get the logs of a Kubernetes Pod (or workload name that will be resolved to a pod) in a namespace. Output is plain text, matching kubernetes-mcp-server pods_log.
   - `clusterName` (`string`) - Optional. Name of the cluster to get the logs from. If not provided, will use the default cluster name in the Kiali KubeConfig
   - `container` (`string`) - Optional. Name of the Pod container to get the logs fro 
@@ -495,7 +495,7 @@ In case multi-cluster support is enabled (default) and you have access to multip
   - `tail` (`integer`) - Number of lines to retrieve from the end of the logs (Optional, defaults to 50). Cannot exceed 200 lines.
   - `workload` (`string`) - Optional. Workload name override (used when name lookup fails).
 
-- **kiali_get_metrics** - Returns metrics for the given resource type, namespaces and resource name.
+- **kiali_get_metrics** - Returns a compact JSON summary of Istio metrics (latency quantiles, traffic trends, throughput, payload sizes) for the given resource.
   - `byLabels` (`string`) - Comma-separated list of labels to group metrics by (e.g., 'source_workload,destination_service'). Optional
   - `clusterName` (`string`) - Cluster name to get metrics from. Optional, defaults to the cluster name in the Kiali configuration (KubeConfig)
   - `direction` (`string`) - Traffic direction. Optional, defaults to 'outbound'

evals/tasks/kiali/Makefile

@@ -0,0 +1,6 @@
+SHELL := /usr/bin/bash
+
+.PHONY: update-tasks
+
+update-tasks:
+	@./scripts/update_tasks.sh

evals/tasks/kiali/README.md

@@ -0,0 +1,106 @@
+# Kiali Task Stack
+
+Kiali-focused MCP tasks live here. Each folder under this directory represents a self-contained scenario that exercises the Kiali toolset (Istio config, topology, observability, troubleshooting).
+
+## Adding a New Task
+
+1. Create a new subdirectory (e.g., `status-foo/`) and place the scenario YAML plus any helper scripts or artifacts inside it.
+2. Make sure the YAML’s `metadata` block includes `name`, `category`, and `difficulty` so it shows up correctly in the catalog below.
+3. Keep prompts concise and action-oriented; verification commands should rely on Kiali MCP tools whenever possible.
+
+## Updating the Catalog
+
+After adding or editing tasks, regenerate this README’s catalog with:
+
+```bash
+make update_tasks
+```
+
+The `update_tasks` target runs `scripts/update_tasks.sh`, which parses every scenario and rewrites the section below automatically. Always run it before committing so the list stays in sync.
+
+## Tasks defined
+<!-- TASKS-START -->
+- Configuration Management
+  - [easy] istio-list (List and Validate VirtualServices)
+        **Prompt:** *|*
+  - [easy] istio-list-destination-rules (List and Validate DestinationRules)
+        **Prompt:** *"List all DestinationRules in the 'bookinfo' namespace and report if any of them have validation errors."*
+  - [medium] istio-create (Create Istio Gateway)
+        **Prompt:** *"Create a new Istio Gateway named 'my-gateway' in the 'istio-system' namespace."*
+  - [medium] istio-delete (Remove Fault Injection)
+        **Prompt:** *"The 'bookinfo' namespace seems to have some fault injection configured that's causing issues. Please identify and remove any fault injection from the 'ratings' VirtualService."*
+  - [medium] istio-patch (Update Traffic Shifting)
+        **Prompt:** *"I need to implement a 50/50 traffic split between versions for the 'reviews' service in the 'bookinfo' namespace. Apply the necessary patch to the 'reviews' VirtualService without asking for confirmation."*
+- Mesh Health & Status
+  - [easy] obs-unhealthy-namespaces (Identify Degraded Namespaces)
+        **Prompt:** *"Check the health of my mesh and let me know if there are any unhealthy namespaces right now."*
+  - [easy] resource-mesh-status (Comprehensive Mesh Health Audit)
+        **Prompt:** *|*
+  - [easy] status-kiali-istio (Audit Control Plane Connectivity)
+        **Prompt:** *"Report on the current status and connectivity between Kiali and the Istio control plane components."*
+- Performance Analysis
+  - [easy] metrics-service-request-rate (Analyze Service Traffic Volume)
+        **Prompt:** *|*
+  - [easy] metrics-workload-latency (Analyze Workload Response Times)
+        **Prompt:** *|*
+- Resource Inspection
+  - [easy] resource-get-namespaces (List Mesh-Enabled Namespaces)
+        **Prompt:** *"Provide a list of all namespaces currently included in my Istio service mesh."*
+  - [easy] resource-get-service-detail (Inspect Service Details)
+        **Prompt:** *"Get the full configuration details and current health status for the 'reviews' service in the 'bookinfo' namespace."*
+  - [easy] resource-get-workload-detail (Inspect Workload Details)
+        **Prompt:** *"Inspect the 'reviews-v1' workload in the 'bookinfo' namespace and provide its detailed status and health information."*
+  - [easy] resource-list-services (Inventory Namespace Services)
+        **Prompt:** *"List all services available in the 'bookinfo' namespace."*
+  - [easy] resource-list-workloads (Inventory Workloads with Sidecar Status)
+        **Prompt:** *"Identify any workloads in the 'bookinfo' namespace that are missing the Istio sidecar proxy."*
+- Traffic Observability
+  - [easy] show-topology (Visualize Namespace Traffic)
+        **Prompt:** *"Show me the traffic topology graph for the 'bookinfo' namespace."*
+  - [easy] topology-mesh-namespaces (Visualize Cross-Namespace Traffic)
+        **Prompt:** *|*
+  - [easy] topology-workload-graph (Visualize Workload-Level Topology)
+        **Prompt:** *|*
+- Troubleshooting & Diagnostics
+  - [easy] troubleshooting-log (Debug Service Errors via Logs)
+        **Prompt:** *|*
+  - [easy] troubleshooting-trace-lagging (Analyze Latency with Distributed Tracing)
+        **Prompt:** *|*
+  - [easy] troubleshooting-workload-logs (Retrieve Recent Workload Logs)
+        **Prompt:** *"Retrieve the last 20 log lines for the 'productpage-v1' workload in the 'bookinfo' namespace."*
+<!-- TASKS-END -->
+
+
+<!-- SUMMARY-OUTPUT-START -->
+=== Evaluation Summary ===
+
+  ✓ Create Istio Gateway (assertions: 3/3)
+  ✓ Remove Fault Injection (assertions: 3/3)
+  ✓ List and Validate VirtualServices (assertions: 3/3)
+  ✓ List and Validate DestinationRules (assertions: 3/3)
+  ✓ Update Traffic Shifting (assertions: 3/3)
+  ✓ Analyze Service Traffic Volume (assertions: 3/3)
+  ✓ Analyze Workload Response Times (assertions: 3/3)
+  ✓ Identify Degraded Namespaces (assertions: 3/3)
+  ✓ List Mesh-Enabled Namespaces (assertions: 3/3)
+  ✓ Inspect Service Details (assertions: 3/3)
+  ✓ Inspect Workload Details (assertions: 3/3)
+  ✓ Inventory Namespace Services (assertions: 3/3)
+  ✓ Inventory Workloads with Sidecar Status (assertions: 3/3)
+  ✓ Comprehensive Mesh Health Audit (assertions: 3/3)
+  ✓ Visualize Namespace Traffic (assertions: 3/3)
+  ✓ Audit Control Plane Connectivity (assertions: 3/3)
+  ✓ Visualize Cross-Namespace Traffic (assertions: 3/3)
+  ✓ Visualize Workload-Level Topology (assertions: 3/3)
+  ✓ Debug Service Errors via Logs (assertions: 3/3)
+  ✓ Analyze Latency with Distributed Tracing (assertions: 3/3)
+  ✓ Retrieve Recent Workload Logs (assertions: 3/3)
+
+Tasks: 21/21 passed (100.00%)
+Assertions: 63/63 passed (100.00%)
+Tokens: ~82147 (incomplete - some counts failed)
+MCP schemas: ~59787 (included in token total)
+Judge used tokens:
+Input: 98980 tokens
+Output: 3199 tokens
+<!-- SUMMARY-OUTPUT-END -->

evals/tasks/kiali/istio-create/istio-create.yaml

@@ -0,0 +1,24 @@
+kind: Task
+apiVersion: mcpchecker/v1alpha2
+metadata:
+  labels:
+    suite: kiali
+  name: "Create Istio Gateway"
+  category: "Configuration Management"
+  description: "Creates a new Istio Gateway in the istio-system namespace to manage ingress traffic."
+  difficulty: medium
+spec:
+  requires:
+  - extension: kubernetes
+    as: k8s
+  verify:
+  - script:
+      file: ./verify.sh
+      timeout: 10s
+  cleanup:
+  - script:
+      inline: |
+        #!/usr/bin/env bash
+        kubectl delete gw my-gateway -n istio-system --ignore-not-found
+  prompt:
+    inline: "Create a new Istio Gateway named 'my-gateway' in the 'istio-system' namespace."

evals/tasks/kiali/istio-create/verify.sh

@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+set -euo pipefail
+NS="istio-system"
+NAME="my-gateway"
+if kubectl get gw "$NAME" -n "$NS" >/dev/null 2>&1; then
+  echo "Verified: Gateway '$NAME' exists in namespace '$NS'."
+else
+  echo "Gateway '$NAME' not found in namespace '$NS'."
+  exit 1
+fi

evals/tasks/kiali/istio-delete/cleanup.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+set -euo pipefail
+NS="bookinfo"
+LABEL="gevals.kiali.io/test=gevals-testing"
+kubectl delete virtualservice -n "$NS" -l "$LABEL" --ignore-not-found
+kubectl delete destinationrule -n "$NS" -l "$LABEL" --ignore-not-found

evals/tasks/kiali/istio-delete/istio-delete.yaml

@@ -0,0 +1,71 @@
+kind: Task
+apiVersion: mcpchecker/v1alpha2
+metadata:
+  labels:
+    suite: kiali
+  name: "Remove Fault Injection"
+  category: "Configuration Management"
+  description: "Identifies and removes fault injection configurations (aborts/delays) from a VirtualService."
+  difficulty: medium
+spec:
+  requires:
+  - extension: kubernetes
+    as: k8s
+  setup:
+  - k8s.create:
+      apiVersion: networking.istio.io/v1
+      kind: DestinationRule
+      metadata:
+        namespace: bookinfo
+        name: ratings
+        labels:
+          gevals.kiali.io/test: gevals-testing
+      spec:
+        host: ratings.bookinfo.svc.cluster.local
+        subsets:
+        - name: v1
+          labels:
+            version: v1
+  - k8s.create:
+      apiVersion: networking.istio.io/v1
+      kind: VirtualService
+      metadata:
+        namespace: bookinfo
+        name: ratings
+        labels:
+          gevals.kiali.io/test: gevals-testing
+      spec:
+        hosts:
+        - ratings.bookinfo.svc.cluster.local
+        http:
+        - route:
+          - destination:
+              host: ratings.bookinfo.svc.cluster.local
+              subset: v1
+            weight: 100
+        fault:
+          abort:
+            percentage:
+              value: 100
+            httpStatus: 503
+  verify:
+  - script:
+      file: ./verify.sh
+      timeout: 10s
+  cleanup:
+  - k8s.delete:
+      apiVersion: networking.istio.io/v1
+      kind: VirtualService
+      metadata:
+        name: ratings
+        namespace: bookinfo
+      ignoreNotFound: true
+  - k8s.delete:
+      apiVersion: networking.istio.io/v1
+      kind: DestinationRule
+      metadata:
+        name: ratings
+        namespace: bookinfo
+      ignoreNotFound: true
+  prompt:
+    inline: "The 'bookinfo' namespace seems to have some fault injection configured that's causing issues. Please identify and remove any fault injection from the 'ratings' VirtualService."

evals/tasks/kiali/istio-delete/setup.sh

@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+set -euo pipefail
+cat <<'EOF' | kubectl apply -f -
+apiVersion: networking.istio.io/v1
+kind: DestinationRule
+metadata:
+  namespace: bookinfo
+  name: ratings
+  labels:
+    gevals.kiali.io/test: gevals-testing
+spec:
+  host: ratings.bookinfo.svc.cluster.local
+  subsets:
+  - name: v1
+    labels:
+      version: v1
+---
+apiVersion: networking.istio.io/v1
+kind: VirtualService
+metadata:
+  namespace: bookinfo
+  name: ratings
+  labels:
+    gevals.kiali.io/test: gevals-testing
+spec:
+  hosts:
+  - ratings.bookinfo.svc.cluster.local
+  http:
+  - route:
+    - destination:
+        host: ratings.bookinfo.svc.cluster.local
+        subset: v1
+      weight: 100
+    fault:
+      abort:
+        percentage:
+          value: 100
+        httpStatus: 503
+EOF

evals/tasks/kiali/istio-delete/verify.sh

@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+NAMESPACE="${EVAL_NAMESPACE:-bookinfo}"
+# Fail if any VirtualService still has fault injection (fault.abort or fault.delay).
+# The agent may patch the VS to remove the fault block, so the VS can still exist.
+vs_with_fault="$(kubectl get virtualservice -n "${NAMESPACE}" -o json \
+  | jq -r '[.items[] | select(any(.spec.http[]?; (.fault != null) and ((.fault | has("abort")) or (.fault | has("delay"))))) | .metadata.name] | .[]?' 2>/dev/null || true)"
+if [[ -n "${vs_with_fault}" ]]; then
+  echo "VirtualService(s) still have fault injection (fault.abort or fault.delay): ${vs_with_fault}"
+  exit 1
+fi

evals/tasks/kiali/istio-list-destination-rules/istio-list-destination-rules.yaml

@@ -0,0 +1,33 @@
+kind: Task
+apiVersion: mcpchecker/v1alpha2
+metadata:
+  labels:
+    suite: kiali
+  name: "List and Validate DestinationRules"
+  category: "Configuration Management"
+  description: "Lists all DestinationRules in a namespace and reports any Istio validation errors."
+  difficulty: easy
+spec:
+  requires:
+  - extension: kubernetes
+    as: k8s
+  setup:
+  - k8s.create:
+      apiVersion: networking.istio.io/v1
+      kind: DestinationRule
+      metadata:
+        namespace: bookinfo
+        name: ratings
+        labels:
+          gevals.kiali.io/test: gevals-testing
+      spec:
+        host: ratings.bookinfo.svc.cluster.local
+        subsets:
+        - name: v1
+          labels:
+            version: v1
+  verify:
+  - llmJudge:
+      contains: "DestinationRule"
+  prompt:
+    inline: "List all DestinationRules in the 'bookinfo' namespace and report if any of them have validation errors."