[OPIK-5692] [BE][FE] feat: add agent config & runner analytics events (#6338)

* [OPIK-5692] [BE][FE] feat: add agent config & runner analytics events

Add PostHog instrumentation for agent configs and sandbox runner to
track Opik 2.0 launch metrics defined in OPIK-5245.

Backend (3 events via AnalyticsService):
- opik_agent_config_saved: fired in createAgentConfig() and
  updateAgentConfig() to capture both UI and SDK config creation.
- opik_agent_config_deployed: fired in createOrUpdateEnvs() and
  setEnvByBlueprintName() to capture all deployment paths.
- opik_sandbox_job_created: fired in createJob() to capture runner
  adoption across UI and API.

Frontend (1 event via trackEvent):
- opik_agent_config_ui_deployed: fired from DeployToPopover with
  UI-specific context (is_new_prod) not available at the BE endpoint.

All BE calls wrapped in try/catch to avoid disrupting primary
operations. User identity resolved automatically via the updated
AnalyticsService.resolveIdentity().

* fix(analytics): address Baz review — dedup deploy helpers and fix blueprint_id

- Consolidated trackAgentConfigDeployed and trackAgentConfigDeployedByName
  into a shared trackAgentConfigDeployedEvent helper (dedup)
- Changed blueprint_id to collect all distinct IDs instead of findFirst(),
  avoiding mis-attribution when envs reference different blueprints

* fix(analytics): add opik_ prefix to event names and fix blueprint_id type inconsistency

Address PR review comments: prefix all analytics events with opik_ per
instrumentation guidelines, and separate blueprint_id (UUID) from
blueprint_name (display name) to prevent type mixing in deployed events.
Also update the analytics skill to document property naming conventions.

* fix(analytics): address Boris's PR review comments

- Remove try/catch wrappers around analytics calls (fire-and-forget)
- Add workspace_id to all analytics events
- Emit one event per env instead of comma-separated aggregation
- Track implicit prod deployment on first config creation
- Soften SKILL.md _id naming rule (thread_id is human-readable)

* chore: revert unrelated package-lock.json changes

* fix(analytics): move tracking to service layer for consistent event properties

Move analytics tracking from AgentConfigsResource to AgentConfigServiceImpl
so both deploy paths (by-ID and by-name) emit consistent event properties
(always blueprint_id + blueprint_name).

* fix(analytics): coerce deployed_to_prod to string in frontend

Aligns frontend event type with backend String.valueOf() serialization
so deployed_to_prod is consistently a string across all emitters.

* fix(analytics): move sandbox_job_created tracking to service layer

Move opik_sandbox_job_created event from LocalRunnersResource to
EndpointJobServiceImpl for consistency with agent config tracking.

* fix: add missing AnalyticsService param to test constructors

* fix: use resolved projectId in analytics callbacks to avoid NPE

Move doOnNext callbacks inside flatMap scope so they use the resolved
projectId variable instead of blueprint.projectId() which is null.
Verified with 66 passing AgentConfigsResourceTest tests.

Author: nLamprok

.agents/skills/analytics-instrumentation/SKILL.md

@@ -59,7 +59,7 @@ private final @NonNull AnalyticsService analyticsService;
 
 2. Call `trackEvent`:
 ```java
-analyticsService.trackEvent("onboarding_first_trace",
+analyticsService.trackEvent("opik_onboarding_first_trace",
     Map.of("trace_id", traceId, "project_id", projectId));
 ```
 
@@ -89,6 +89,12 @@ Backend events:          Java → comet-stats → Segment → PostHog
 PostHog native:          Browser → posthog-js → PostHog (pageviews, feature flags, identification)
 ```
 
+## Event Property Conventions
+
+- **Consistent typing per property**: A given property key should always carry the same kind of value. Don't pass a UUID in one code path and a human-readable name in another for the same key.
+- **Separate ID and name properties**: When both a UUID and a display name exist, use distinct keys (e.g. `blueprint_id` for the UUID, `blueprint_name` for the display name). If one is unavailable in a code path, omit the key or send an empty string — don't repurpose the other key.
+- **Include `workspace_id`**: All backend analytics events should include the workspace ID for segmentation.
+
 ## Deciding Frontend vs Backend
 
 - **Frontend**: UI interactions (button clicks, wizard steps, form submissions, page visits)

apps/opik-backend/src/main/java/com/comet/opik/api/resources/v1/priv/AgentConfigsResource.java

@@ -9,7 +9,6 @@
 import com.comet.opik.domain.AgentBlueprint;
 import com.comet.opik.domain.AgentConfig;
 import com.comet.opik.domain.AgentConfigService;
-import com.comet.opik.infrastructure.auth.RequestContext;
 import com.fasterxml.jackson.annotation.JsonView;
 import io.swagger.v3.oas.annotations.Operation;
 import io.swagger.v3.oas.annotations.headers.Header;
@@ -19,7 +18,6 @@
 import io.swagger.v3.oas.annotations.responses.ApiResponse;
 import io.swagger.v3.oas.annotations.tags.Tag;
 import jakarta.inject.Inject;
-import jakarta.inject.Provider;
 import jakarta.validation.Valid;
 import jakarta.validation.constraints.Min;
 import jakarta.validation.constraints.NotNull;
@@ -55,7 +53,6 @@
 public class AgentConfigsResource {
 
     private final @NonNull AgentConfigService agentConfigService;
-    private final @NonNull Provider<RequestContext> requestContext;
 
     @POST
     @Path("/blueprints")

apps/opik-backend/src/main/java/com/comet/opik/api/resources/v1/priv/LocalRunnersResource.java

@@ -181,6 +181,7 @@ public Response createJob(
         String workspaceId = requestContext.get().getWorkspaceId();
         String userName = requestContext.get().getUserName();
         UUID jobId = endpointJobService.createJob(workspaceId, userName, request);
+
         var uri = uriInfo.getAbsolutePathBuilder().path("/{jobId}").build(jobId);
         return Response.created(uri).build();
     }

apps/opik-backend/src/main/java/com/comet/opik/domain/AgentConfigService.java

@@ -7,6 +7,7 @@
 import com.comet.opik.api.error.ErrorMessage;
 import com.comet.opik.api.validation.HasProjectIdentifier;
 import com.comet.opik.infrastructure.AgentConfigConfiguration;
+import com.comet.opik.infrastructure.bi.AnalyticsService;
 import com.comet.opik.infrastructure.lock.LockService;
 import com.comet.opik.utils.JsonUtils;
 import com.comet.opik.utils.WorkspaceUtils;
@@ -82,6 +83,7 @@ class AgentConfigServiceImpl implements AgentConfigService {
     private final @NonNull ProjectService projectService;
     private final @NonNull LockService lockService;
     private final @NonNull AgentConfigConfiguration agentConfigConfiguration;
+    private final @NonNull AnalyticsService analyticsService;
 
     @Override
     public Mono<AgentBlueprint> createConfig(@NonNull AgentConfigCreate request) {
@@ -127,7 +129,12 @@ public Mono<AgentBlueprint> createConfig(@NonNull AgentConfigCreate request) {
 
                             return blueprint;
                         })).subscribeOn(Schedulers.boundedElastic()),
-                        agentConfigConfiguration.getBlueprintLockDuration().toJavaDuration()));
+                        agentConfigConfiguration.getBlueprintLockDuration().toJavaDuration())
+                        .doOnNext(blueprint -> {
+                            trackAgentConfigSaved(workspaceId, projectId, blueprint);
+                            trackAgentConfigDeployed(workspaceId, projectId,
+                                    blueprint.id(), String.valueOf(blueprint.name()), "prod");
+                        }));
     }
 
     @Override
@@ -155,7 +162,8 @@ public Mono<AgentBlueprint> updateConfig(@NonNull AgentConfigCreate request) {
                             return createBlueprint(dao, request, existingConfig.id(), projectId, workspaceId,
                                     userName);
                         })).subscribeOn(Schedulers.boundedElastic()),
-                        agentConfigConfiguration.getBlueprintLockDuration().toJavaDuration()));
+                        agentConfigConfiguration.getBlueprintLockDuration().toJavaDuration())
+                        .doOnNext(blueprint -> trackAgentConfigSaved(workspaceId, projectId, blueprint)));
     }
 
     @Override
@@ -494,7 +502,13 @@ public Mono<Void> createOrUpdateEnvs(@NonNull AgentConfigEnvUpdate request) {
                     upsertEnvs(dao, workspaceId, projectId, userName, request.envs());
 
                     return null;
-                })).subscribeOn(Schedulers.boundedElastic()));
+                })).subscribeOn(Schedulers.boundedElastic()))
+                .doOnSuccess(v -> {
+                    for (var env : request.envs()) {
+                        trackAgentConfigDeployed(workspaceId, projectId,
+                                env.blueprintId(), "", env.envName());
+                    }
+                });
     }
 
     @Override
@@ -506,9 +520,9 @@ public Mono<Void> setEnvByBlueprintName(@NonNull UUID projectId, @NonNull String
         log.info("Setting environment '{}' to blueprint '{}' for project '{}' in workspace '{}'",
                 envName, blueprintName, projectId, workspaceId);
 
-        return lockService.<Void>executeWithLock(
+        return lockService.<UUID>executeWithLock(
                 new LockService.Lock(ENV_LOCK_FORMAT.formatted(workspaceId, projectId)),
-                Mono.<Void>fromRunnable(() -> transactionTemplate.inTransaction(WRITE, handle -> {
+                Mono.fromCallable(() -> transactionTemplate.inTransaction(WRITE, handle -> {
                     AgentConfigDAO dao = handle.attach(AgentConfigDAO.class);
 
                     AgentBlueprint blueprint = requireBlueprintByName(dao, workspaceId, projectId, blueprintName);
@@ -519,8 +533,11 @@ public Mono<Void> setEnvByBlueprintName(@NonNull UUID projectId, @NonNull String
                                     .blueprintId(blueprint.id())
                                     .build()));
 
-                    return null;
-                })).subscribeOn(Schedulers.boundedElastic()));
+                    return blueprint.id();
+                })).subscribeOn(Schedulers.boundedElastic()))
+                .doOnNext(blueprintId -> trackAgentConfigDeployed(workspaceId, projectId,
+                        blueprintId, blueprintName, envName))
+                .then();
     }
 
     private void upsertEnvs(AgentConfigDAO dao, String workspaceId, UUID projectId, String userName,
@@ -676,4 +693,23 @@ public Mono<AgentBlueprint> createBlueprintFromMask(@NonNull UUID projectId, @No
 
         return updateConfig(request);
     }
+
+    private void trackAgentConfigSaved(String workspaceId, UUID projectId, AgentBlueprint blueprint) {
+        analyticsService.trackEvent("opik_agent_config_saved", Map.of(
+                "workspace_id", workspaceId,
+                "project_id", projectId.toString(),
+                "blueprint_id", blueprint.id().toString(),
+                "blueprint_name", String.valueOf(blueprint.name())));
+    }
+
+    private void trackAgentConfigDeployed(String workspaceId, UUID projectId,
+            UUID blueprintId, String blueprintName, String envName) {
+        analyticsService.trackEvent("opik_agent_config_deployed", Map.of(
+                "workspace_id", workspaceId,
+                "project_id", projectId.toString(),
+                "blueprint_id", blueprintId.toString(),
+                "blueprint_name", blueprintName,
+                "environment", envName,
+                "deployed_to_prod", String.valueOf("prod".equalsIgnoreCase(envName))));
+    }
 }

apps/opik-backend/src/main/java/com/comet/opik/domain/EndpointJobService.java

@@ -10,6 +10,7 @@
 import com.comet.opik.api.runner.LocalRunnerLogEntry;
 import com.comet.opik.api.runner.RunnerType;
 import com.comet.opik.infrastructure.LocalRunnerConfig;
+import com.comet.opik.infrastructure.bi.AnalyticsService;
 import com.comet.opik.infrastructure.redis.StringRedisClient;
 import com.comet.opik.utils.JsonUtils;
 import com.fasterxml.jackson.databind.JsonNode;
@@ -130,16 +131,18 @@ static String runnerCancellationsKey(UUID runnerId) {
     private final @NonNull IdGenerator idGenerator;
     private final @NonNull RunnerService runnerService;
     private final @NonNull LocalRunnerConfig runnerConfig;
+    private final @NonNull AnalyticsService analyticsService;
 
     @Inject
     EndpointJobServiceImpl(@NonNull StringRedisClient redisClient, @NonNull RedissonReactiveClient reactiveRedisClient,
             @NonNull IdGenerator idGenerator, @NonNull RunnerService runnerService,
-            @NonNull LocalRunnerConfig runnerConfig) {
+            @NonNull LocalRunnerConfig runnerConfig, @NonNull AnalyticsService analyticsService) {
         this.redisClient = redisClient;
         this.reactiveRedisClient = reactiveRedisClient;
         this.idGenerator = idGenerator;
         this.runnerService = runnerService;
         this.runnerConfig = runnerConfig;
+        this.analyticsService = analyticsService;
     }
 
     @Override
@@ -239,6 +242,12 @@ public UUID createJob(@NonNull String workspaceId, @NonNull String userName,
         RScoredSortedSet<String> runnerJobs = redisClient.getScoredSortedSet(runnerJobsKey(runnerId));
         runnerJobs.add(Instant.now().toEpochMilli(), jobId.toString());
 
+        analyticsService.trackEvent("opik_sandbox_job_created", Map.of(
+                "workspace_id", workspaceId,
+                "project_id", request.projectId().toString(),
+                "job_id", jobId.toString(),
+                "agent_name", request.agentName()));
+
         return jobId;
     }
 

apps/opik-backend/src/test/java/com/comet/opik/domain/LocalRunnerReaperIntegrationTest.java

@@ -9,6 +9,7 @@
 import com.comet.opik.api.runner.RunnerType;
 import com.comet.opik.infrastructure.LocalRunnerConfig;
 import com.comet.opik.infrastructure.auth.RequestContext;
+import com.comet.opik.infrastructure.bi.AnalyticsService;
 import com.comet.opik.infrastructure.redis.StringRedisClient;
 import com.redis.testcontainers.RedisContainer;
 import io.dropwizard.util.Duration;
@@ -95,7 +96,7 @@ void setUp() {
         runnerService = new RunnerServiceImpl(stringRedis, idGenerator, projectService, runnerConfig,
                 () -> endpointJobService, () -> connectBridgeService, () -> requestContext);
         endpointJobService = new EndpointJobServiceImpl(stringRedis, redisClient.reactive(), idGenerator,
-                runnerService, runnerConfig);
+                runnerService, runnerConfig, Mockito.mock(AnalyticsService.class));
     }
 
     @BeforeEach

apps/opik-backend/src/test/java/com/comet/opik/domain/LocalRunnerServiceImplTest.java

@@ -18,6 +18,7 @@
 import com.comet.opik.api.runner.RunnerType;
 import com.comet.opik.infrastructure.LocalRunnerConfig;
 import com.comet.opik.infrastructure.auth.RequestContext;
+import com.comet.opik.infrastructure.bi.AnalyticsService;
 import com.comet.opik.infrastructure.redis.StringRedisClient;
 import com.fasterxml.jackson.databind.ObjectMapper;
 import com.fasterxml.jackson.databind.node.ObjectNode;
@@ -112,7 +113,7 @@ void setUp() {
         runnerService = new RunnerServiceImpl(stringRedis, idGenerator, projectService, runnerConfig,
                 () -> endpointJobService, () -> connectBridgeService, () -> requestContext);
         endpointJobService = new EndpointJobServiceImpl(stringRedis, redisClient.reactive(), idGenerator,
-                runnerService, runnerConfig);
+                runnerService, runnerConfig, Mockito.mock(AnalyticsService.class));
         connectBridgeService = new ConnectBridgeServiceImpl(stringRedis, idGenerator, runnerService, runnerConfig);
     }
 

apps/opik-frontend/src/lib/analytics/tracking.ts

@@ -5,6 +5,7 @@ export const OpikEvent = {
   ONBOARDING_FIRST_TRACE_RECEIVED: "opik_onboarding_first_trace_received",
   ONBOARDING_SKIPPED: "opik_onboarding_skipped",
   EVAL_SUITE_UI_CONFIGURED: "opik_eval_suite_ui_configured",
+  AGENT_CONFIG_UI_DEPLOYED: "opik_agent_config_ui_deployed",
 } as const;
 
 type OpikEventValues = (typeof OpikEvent)[keyof typeof OpikEvent];

apps/opik-frontend/src/v2/pages/AgentConfigurationPage/AgentConfigurationTab/DeployToPopover.tsx

@@ -17,6 +17,7 @@ import {
   isProdTag,
   isStageTag,
 } from "@/utils/agent-configurations";
+import { OpikEvent, trackEvent } from "@/lib/analytics/tracking";
 
 type DeployToPopoverProps = {
   item: ConfigHistoryItem;
@@ -93,6 +94,14 @@ const DeployToPopover: React.FC<DeployToPopoverProps> = ({
       });
     }
 
+    trackEvent(OpikEvent.AGENT_CONFIG_UI_DEPLOYED, {
+      project_id: projectId,
+      blueprint_id: item.id,
+      environments: selectedStages,
+      deployed_to_prod: String(prodSelected),
+      is_new_prod: isNewProd,
+    });
+
     setOpen(false);
   };