Skip to content

Home

Jump to bottom
Benjamin Paige edited this page Sep 16, 2025 · 6 revisions

Appian Connector 🏢

MACPRO Appian → BigMAC Workflow Data Streaming Platform

Quick Navigation: Architecture | Getting Started | Operations | External Links

Overview

The Appian Connector enables real-time streaming of MACPRO workflow and package data from Appian's Oracle database to CMS BigMAC using Kafka Connect technology. This microservice bridges legacy business processes with modern data streaming infrastructure.

System Summary

Property Value Details
Data Source Appian Oracle Database MCP_SPA_PCKG table monitoring
Connector Type Confluent JDBC Source Connector Version 10.5.1 with Oracle drivers
Target Platform CMS BigMAC Enterprise Kafka streaming platform
Deployment AWS ECS Fargate + Lambda Containerized with serverless management
Data Processing Real-time streaming 2-second polling intervals, 1000-row batches
Monitoring CloudWatch + SNS Automated alerting and dashboards
Repository macpro-appian-connector Complete source code and documentation

Architecture

graph TB
    subgraph "Appian Oracle Database"
        ORACLE[(Oracle Production<br/>MCP_SPA_PCKG Table<br/>Package Management Data)]
    end

    subgraph "Appian Connector (ECS Fargate)"
        CONNECT[Kafka Connect Worker<br/>JDBC Source Connector]
        CONFIG[Configure Connector Lambda]
        ORACLE_CLIENT[Oracle Instant Client<br/>JDBC connectivity]
    end

    subgraph "BigMAC Kafka Platform"
        MSK[MSK Cluster<br/>Topic: aws.appian.cmcs.MCP_SPA_PCKG]
    end

    subgraph "Monitoring & Alerting"
        CW[CloudWatch Metrics]
        SNS[SNS Alerts]
        DASH[Custom Dashboard]
    end

    ORACLE -->|JDBC Polling<br/>2s intervals| CONNECT
    CONNECT -->|Oracle Client| ORACLE_CLIENT
    CONFIG -->|REST API| CONNECT
    CONNECT -->|SSL/TLS| MSK
    CONNECT --> CW
    CW --> SNS
    CW --> DASH

    classDef database fill:#e1f5fe,stroke:#01579b,stroke-width:2px
    classDef connector fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
    classDef kafka fill:#fff3e0,stroke:#e65100,stroke-width:2px
    classDef monitoring fill:#fce4ec,stroke:#880e4f,stroke-width:2px

    class ORACLE database
    class CONNECT,CONFIG,ORACLE_CLIENT connector
    class MSK kafka
    class CW,SNS,DASH monitoring
Loading

Services

The Appian Connector project consists of 2 core services:

Service Purpose Technology
alerts SNS notification system CloudWatch + SNS + KMS
connector Appian data streaming ECS Fargate + Lambda

Service Architecture

graph TB
    subgraph "Service Layer"
        ALERTS[alerts<br/>🚨 Monitoring Foundation]
        CONNECTOR[connector<br/>🔄 Data Streaming]
    end

    ALERTS -->|ecsFailureTopicArn| CONNECTOR

    classDef foundation fill:#ffebee,stroke:#d32f2f,stroke-width:2px
    classDef processing fill:#e8f5e9,stroke:#388e3c,stroke-width:2px

    class ALERTS foundation
    class CONNECTOR processing
Loading

Data Integration

Appian Package Data

  • Source Table: MCP_SPA_PCKG (Oracle database)
  • Data Type: MACPRO workflow and package management data
  • Key Fields: PCKG_ID (incrementing), REPLICA_TIMESTAMP (timestamp)
  • Update Pattern: Package status changes, workflow transitions
  • Target Topic: aws.appian.cmcs.MCP_SPA_PCKG

Integration Pattern

  • Polling Mode: timestamp+incrementing for reliable change detection
  • Frequency: 2-second intervals for near real-time processing
  • Batch Size: 1000 rows maximum per poll
  • Data Format: JSON with PCKG_ID as message key

Getting Started

Quick Start Guide

Step Action Documentation
1 Understand the architecture Architecture - Technical overview and data flow
2 Configure environment Configuration - Environment and connector settings
3 Deploy the service Deployment - Step-by-step deployment guide
4 Monitor operations Monitoring - Observability and alerting setup

Technical Documentation

Core Components

  • Services - Service architecture and component details
  • Integration - BigMAC platform integration and data mapping
  • Operations - Daily operational procedures and troubleshooting

Setup & Configuration

Quick Reference

Common Issues & Solutions

Issue Quick Fix Documentation
Connector not running Check ECS service status Operations - Service Management
Oracle connection failed Verify credentials in Secrets Manager Operations - Database Management
No data in BigMAC Check connector configuration Operations - Package Data Flow
Deployment failed Validate environment configuration Deployment - Troubleshooting

Support & Resources

Resource Purpose Link
Documentation Site Complete technical docs enterprise-cmcs.github.io
Team Communication Real-time support Slack #cms-bigmac
Issue Tracking Bug reports & features Jira OY2 Board
Code Quality Maintainability metrics CodeClimate
Source Code Complete repository GitHub Repository

💡 Getting Help: For technical issues, check Operations first. For architecture questions, start with Architecture. For urgent support, use Slack #cms-bigmac.

Clone this wiki locally