ADME-Fabric Control Plane

A Streamlit-based dashboard for orchestrating data synchronization between Azure Data Manager for Energy (ADME) instances through Microsoft Fabric Lakehouses.

Architecture

┌─────────────────────────────────────────────────────────────────────────────────┐
│                            ADME-Fabric Control Plane                            │
├─────────────────────────────────────────────────────────────────────────────────┤
│                                                                                 │
│  ┌─────────────────────────┐         ┌─────────────────────────┐               │
│  │    SOURCE ENVIRONMENT   │         │   TARGET ENVIRONMENT    │               │
│  │                         │         │                         │               │
│  │  ┌─────────────────┐    │         │    ┌─────────────────┐  │               │
│  │  │   Source ADME   │    │         │    │   Target ADME   │  │               │
│  │  │  (Azure Data    │    │         │    │  (Azure Data    │  │               │
│  │  │   Manager for   │    │         │    │   Manager for   │  │               │
│  │  │    Energy)      │    │         │    │    Energy)      │  │               │
│  │  └────────┬────────┘    │         │    └────────▲────────┘  │               │
│  │           │             │         │             │           │               │
│  │           │ Search &    │         │    Storage  │           │               │
│  │           │ Storage API │         │    API PUT  │           │               │
│  │           ▼             │         │             │           │               │
│  │  ┌─────────────────┐    │         │    ┌─────────────────┐  │               │
│  │  │  Fabric         │    │  Sync   │    │  Fabric         │  │               │
│  │  │  Lakehouse      │────┼────────────▶│  Lakehouse      │  │               │
│  │  │  (Delta Tables) │    │  (EDS)  │    │  (Delta Tables) │  │               │
│  │  └─────────────────┘    │         │    └─────────────────┘  │               │
│  │                         │         │                         │               │
│  └─────────────────────────┘         └─────────────────────────┘               │
│                                                                                 │
│  ┌──────────────────────────────────────────────────────────────────────────┐  │
│  │                         CONTROL PLANE (This App)                         │  │
│  │                                                                          │  │
│  │   ┌──────────────┐   ┌──────────────┐   ┌──────────────────────────┐    │  │
│  │   │  Streamlit   │   │   SQLite /   │   │   Fabric REST API        │    │  │
│  │   │  Dashboard   │◀─▶│   Azure SQL  │   │   (Notebook Triggers)    │    │  │
│  │   │              │   │   Database   │   │                          │    │  │
│  │   └──────────────┘   └──────────────┘   └──────────────────────────┘    │  │
│  │                                                                          │  │
│  └──────────────────────────────────────────────────────────────────────────┘  │
│                                                                                 │
└─────────────────────────────────────────────────────────────────────────────────┘

Data Flow:
  1. Export: Source ADME ──▶ Source Lakehouse (via Export Notebook)
  2. Sync:   Source Lakehouse ──▶ Target Lakehouse (via External Data Share)
  3. Ingest: Target Lakehouse ──▶ Target ADME (via Ingest Notebook)

📊 View Interactive Mermaid Diagrams

flowchart TB
    subgraph SourceEnv["Source Environment"]
        ADME_S[("Source ADME<br/>Azure Data Manager<br/>for Energy")]
        subgraph FabricS["Source Fabric Workspace"]
            LH_S[("Source Lakehouse<br/>Delta Tables")]
            NB_Export["Export Notebook<br/>(ADME → Fabric)"]
        end
    end

    subgraph TargetEnv["Target Environment"]
        subgraph FabricT["Target Fabric Workspace"]
            LH_T[("Target Lakehouse<br/>Delta Tables")]
            NB_Ingest["Ingest Notebook<br/>(Fabric → ADME)"]
        end
        ADME_T[("Target ADME<br/>Azure Data Manager<br/>for Energy")]
    end

    subgraph ControlPlane["Control Plane Dashboard"]
        UI["Streamlit UI"]
        DB[("SQLite<br/>Jobs & Connections")]
        API["Fabric REST API<br/>Notebook Trigger"]
    end

    ADME_S -->|"1. Search & Storage API"| NB_Export
    NB_Export -->|"2. Write Delta"| LH_S
    LH_S -->|"3. EDS Sync"| LH_T
    LH_T -->|"4. Read Delta"| NB_Ingest
    NB_Ingest -->|"5. Storage API PUT"| ADME_T

    UI -->|"Configure & Monitor"| DB
    UI -->|"Trigger Jobs"| API
    API -->|"Execute"| NB_Export
    API -->|"Execute"| NB_Ingest

Data Flow Sequence

sequenceDiagram
    participant User
    participant Dashboard as Control Plane
    participant SrcADME as Source ADME
    participant SrcFabric as Source Fabric
    participant TgtFabric as Target Fabric
    participant TgtADME as Target ADME

    User->>Dashboard: Configure Sync Job
    User->>Dashboard: Click "Sync Now"

    rect rgb(240, 248, 255)
        Note over Dashboard,SrcFabric: Phase 1: Export
        Dashboard->>SrcFabric: Trigger Export Notebook
        SrcFabric->>SrcADME: Search API (get record IDs)
        SrcADME-->>SrcFabric: Record IDs + Cursor
        SrcFabric->>SrcADME: Storage API (batch records)
        SrcADME-->>SrcFabric: Full Record Data
        SrcFabric->>SrcFabric: Write to Delta Table
        SrcFabric-->>Dashboard: Export Complete (N records)
    end

    rect rgb(255, 248, 240)
        Note over SrcFabric,TgtFabric: Phase 2: Fabric Sync (EDS)
        Dashboard->>SrcFabric: Trigger Sync Notebook
        SrcFabric->>TgtFabric: External Data Share
        TgtFabric-->>Dashboard: Sync Complete
    end

    rect rgb(240, 255, 240)
        Note over TgtFabric,TgtADME: Phase 3: Ingest
        Dashboard->>TgtFabric: Trigger Ingest Notebook
        TgtFabric->>TgtFabric: Read from Delta Table
        TgtFabric->>TgtFabric: Transform Records
        TgtFabric->>TgtADME: Storage API PUT (batch)
        TgtADME-->>TgtFabric: Ingest Response
        TgtFabric-->>Dashboard: Ingest Complete (N records)
    end

    Dashboard-->>User: Job Complete ✓

Features

Connection Management: Configure ADME-Fabric environment pairs with encrypted secrets
Sync Wizard: Multi-step wizard with cascading dropdowns for kind/partition selection
Real-time Dashboard: Monitor active jobs with live logs and progress tracking
Job History: View past jobs, clone configurations, re-run with identical settings
Error Browser: Analyze errors with categorization and full stack traces
Comprehensive Logging: Full API request/response tracking for debugging
Mock Mode: Develop and test without real ADME/Fabric connections

Tech Stack

Component	Technology	Version
Frontend/Backend	Streamlit	1.40.0
Database	SQLite + SQLAlchemy	2.0.25
Authentication	MSAL	1.26.0
HTTP Client	Requests + urllib3	2.31.0
Encryption	Fernet (cryptography)	41.0.7
Validation	Pydantic	2.5.3

🚀 Getting Started

Prerequisites

Python 3.11+ (Download)
Git (Download)
Azure Service Principal with permissions to:
- Azure Data Manager for Energy (ADME)
- Microsoft Fabric workspace
Microsoft Fabric workspace with a Lakehouse

Step 1: Clone the Repository

git clone https://github.com/mhadaily/ADME-FABRIC-ControlPlane.git
cd ADME-FABRIC-ControlPlane

Step 2: Create Virtual Environment

Windows (PowerShell):

python -m venv .venv
.\.venv\Scripts\Activate.ps1

Linux/macOS:

python3 -m venv .venv
source .venv/bin/activate

Step 3: Install Dependencies

pip install -r requirements.txt

Step 4: Configure Environment

# Copy the example environment file
copy .env.example .env   # Windows
# cp .env.example .env   # Linux/macOS

Generate an encryption key:

python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

Edit .env and add your encryption key:

ENCRYPTION_KEY=your-generated-key-here
LOG_LEVEL=INFO
MOCK_MODE=false

Step 5: Run the Application

streamlit run app.py

The application will open at http://localhost:8501

🧪 Development

Mock Mode (Recommended for Development)

Enable mock mode to develop without real ADME/Fabric connections:

# .env
MOCK_MODE=true

Mock mode provides:

Simulated ADME kinds and partitions
Fake record counts
Delayed responses to simulate network latency
No actual API calls

Running with Debug Logging

# .env
LOG_LEVEL=DEBUG
DEBUG=true

Project Structure

adme-fabric-control-plane/
├── app.py                          # Streamlit entry point
├── requirements.txt                # Python dependencies
├── Dockerfile                      # Container configuration
├── .env.example                    # Environment template
├── LICENSE                         # MIT License
├── CONTRIBUTING.md                 # Contribution guidelines
│
├── config/
│   ├── settings.py                 # Pydantic settings with defaults
│   └── logging_config.py           # Logging configuration
│
├── database/
│   ├── models.py                   # SQLAlchemy models
│   └── db.py                       # Database session management
│
├── clients/
│   ├── base_client.py              # HTTP client with retry logic
│   ├── adme_client.py              # ADME API client
│   └── fabric_client.py            # Fabric API client
│
├── services/
│   ├── auth_service.py             # Token management & caching
│   ├── connection_service.py       # Environment CRUD operations
│   ├── job_service.py              # Job tracking & management
│   ├── sync_service.py             # 3-phase sync orchestration
│   ├── scheduler_service.py        # Job scheduling
│   └── error_service.py            # Centralized error handling
│
├── components/
│   ├── connection_form.py          # Environment configuration form
│   ├── sync_wizard.py              # Multi-step sync wizard
│   ├── query_builder.py            # LUCENE query builder
│   ├── job_card.py                 # Job status card
│   ├── log_viewer.py               # Real-time log viewer
│   └── error_display.py            # Error banners & details
│
├── pages/
│   ├── 1_🔗_Connections.py         # Connection management
│   ├── 2_📝_Data_Product_Definitions.py  # Data product definitions
│   ├── 3_🚀_Publish.py             # Publish data products
│   ├── 4_📥_Consume.py             # Consume data products
│   ├── 5_📊_Dashboard.py           # Real-time monitoring
│   ├── 6_📜_History.py             # Job history
│   └── 8_⏰_Scheduled_Jobs.py      # Scheduled jobs
│
├── utils/
│   ├── exceptions.py               # Custom exception classes
│   ├── validators.py               # Input validation
│   ├── encryption.py               # Fernet encryption helpers
│   ├── keyvault.py                 # Azure Key Vault integration
│   └── secrets_manager.py          # Secrets management
│
├── notebooks/                      # Fabric notebook templates
│   ├── README.md                   # Deployment instructions
│   ├── adme_export_notebook_v2.py  # ADME → Fabric export
│   └── adme_ingest_notebook_v2.py  # Fabric → ADME ingest
│
├── docs/                           # Additional documentation
│   ├── COMPLETE_WORKFLOW_GUIDE.md
│   ├── SECRETS_MANAGEMENT.md
│   └── FABRIC_API_VERIFICATION.md
│
└── tests/                          # Test suite
    ├── conftest.py
    ├── test_encryption.py
    ├── test_fabric_api.py
    └── test_validators.py

Code Style

The project uses:

Black for code formatting
MyPy for type checking
Pytest for testing

# Format code
black .

# Type check
mypy .

# Run tests
pytest

Adding a New Page

Create a file in pages/ with emoji prefix (e.g., 6_⚙️_Settings.py)

Add page config at the top:

import streamlit as st
st.set_page_config(page_title="Settings", page_icon="⚙️", layout="wide")

Streamlit auto-discovers pages from the pages/ folder

🧪 Testing

Run All Tests

pytest

Run with Coverage

pytest --cov=. --cov-report=html
# Open htmlcov/index.html to view report

Test Categories

# Unit tests only
pytest tests/unit/

# Integration tests
pytest tests/integration/

# Specific test file
pytest tests/unit/test_validators.py

Manual Testing Checklist

Connections Page
- Add new environment
- Test ADME connection
- Test Fabric connection
- Edit existing environment
- Delete environment
Sync Page
- Select source environment
- Select data partition
- Select kind (record type)
- Configure filters
- Select target environment
- Execute sync job
Dashboard
- View active jobs
- View job progress
- Cancel running job
- View logs
History
- View completed jobs
- Clone job configuration
- Re-run job
- Export job logs

🐳 Docker

Build Image

docker build -t adme-fabric-control-plane .

Run Container

docker run -d \
  --name adme-control-plane \
  -p 8501:8501 \
  -e ENCRYPTION_KEY=your-fernet-key \
  -e LOG_LEVEL=INFO \
  -v $(pwd)/data:/app/data \
  adme-fabric-control-plane

Docker Compose

Create docker-compose.yml:

version: '3.8'
services:
  control-plane:
    build: .
    ports:
      - '8501:8501'
    environment:
      - ENCRYPTION_KEY=${ENCRYPTION_KEY}
      - LOG_LEVEL=INFO
      - MOCK_MODE=false
    volumes:
      - ./data:/app/data
    restart: unless-stopped

Run:

docker-compose up -d

☁️ Deployment

Recommended: Deploy with PowerShell Script

The easiest way to deploy to Azure Container Apps is using the included deployment script:

# Deploy to Azure (creates all resources automatically)
./deploy.ps1

# Or with custom parameters
./deploy.ps1 -ResourceGroup "my-rg" -Location "eastus" -AppName "my-control-plane"

The script automatically:

Creates Azure Container Registry
Builds and pushes the Docker image
Creates Azure Container Apps environment
Configures managed identity for Key Vault access
Sets up Azure Key Vault for secrets storage
Deploys the application

Alternative: Azure App Service

Option 1: Deploy from Container Registry

Push to Azure Container Registry:

az acr login --name <your-acr>
docker tag adme-fabric-control-plane <your-acr>.azurecr.io/adme-control-plane:latest
docker push <your-acr>.azurecr.io/adme-control-plane:latest

Create App Service:

az webapp create \
  --resource-group <rg-name> \
  --plan <plan-name> \
  --name <app-name> \
  --deployment-container-image-name <your-acr>.azurecr.io/adme-control-plane:latest

Configure Environment:

az webapp config appsettings set \
  --resource-group <rg-name> \
  --name <app-name> \
  --settings \
    ENCRYPTION_KEY="your-fernet-key" \
    LOG_LEVEL="INFO" \
    WEBSITES_PORT="8501"

Option 2: Deploy from Source

Create App Service (Python 3.11):

az webapp create \
  --resource-group <rg-name> \
  --plan <plan-name> \
  --name <app-name> \
  --runtime "PYTHON:3.11"

Configure Startup Command:

az webapp config set \
  --resource-group <rg-name> \
  --name <app-name> \
  --startup-file "streamlit run app.py --server.port 8000 --server.address 0.0.0.0 --server.headless true"

Deploy Code:

az webapp deployment source config-local-git \
  --resource-group <rg-name> \
  --name <app-name>

git remote add azure <git-url-from-output>
git push azure main

Environment Variables for Production

Variable	Description	Required
`ENCRYPTION_KEY`	Fernet encryption key	✅ Yes
`DATABASE_PATH`	SQLite path (use persistent storage)	✅ Yes
`LOG_LEVEL`	`INFO` or `WARNING` for production	No
`MOCK_MODE`	Must be `false`	No
`WEBSITES_PORT`	Set to `8501` for Azure	✅ Yes (Azure)

Persistent Storage (Azure)

For production, mount Azure Files for SQLite persistence:

az webapp config storage-account add \
  --resource-group <rg-name> \
  --name <app-name> \
  --custom-id data \
  --storage-type AzureFiles \
  --share-name <share-name> \
  --account-name <storage-account> \
  --mount-path /app/data \
  --access-key <access-key>

📒 Fabric Notebooks

Deployment Steps

Navigate to your Fabric workspace
Create a new Notebook for each:
- adme_export_notebook.py → Export Notebook
- fabric_sync_notebook.py → Sync Notebook
- adme_ingest_notebook.py → Ingest Notebook
Copy the content from notebooks/ into each Fabric notebook
Update environment configuration in your Control Plane with the notebook IDs

Notebook Parameters

The notebooks accept parameters via the Fabric REST API when triggered from the Control Plane:

Notebook	Key Parameters
Export	`dataPartitionId`, `admeServerUrl`, `searchApiKind`, credentials
Sync	`source_workspace_id`, `target_workspace_id`, `sync_mode`
Ingest	`osduEndpoint`, `dataPartitionId`, `batch_size`, credentials

See notebooks/README.md for detailed deployment instructions.

🔒 Security

Secrets Management

All secrets (client secrets) are encrypted using Fernet symmetric encryption
The encryption key is stored as an environment variable (never in code)
Secrets are decrypted only when needed for API calls

Generate Encryption Key

from cryptography.fernet import Fernet
print(Fernet.generate_key().decode())

Best Practices

✅ Never commit .env files
✅ Use Azure Key Vault for production secrets
✅ Rotate encryption keys periodically
✅ Enable HTTPS at the load balancer/App Service level
✅ Restrict network access to the App Service

🐛 Troubleshooting

Common Issues

"ENCRYPTION_KEY not set"

Make sure .env file exists and contains ENCRYPTION_KEY

Database locked error

SQLite doesn't support concurrent writes. This usually resolves itself.
For production, consider using PostgreSQL.

"Connection refused" to ADME

1. Check ADME server URL (must be HTTPS)
2. Verify Service Principal has permissions
3. Check firewall rules

Streamlit "AxiosError"

Usually a browser cache issue. Clear cache or try incognito mode.

Logs

View application logs:

# Local development
streamlit run app.py 2>&1 | tee app.log

# Docker
docker logs adme-control-plane

# Azure App Service
az webapp log tail --name <app-name> --resource-group <rg-name>

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🤝 Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit changes (git commit -m 'Add amazing feature')
Push to branch (git push origin feature/amazing-feature)
Open a Pull Request

📚 References

Data Product Flow

Steps & APIs

Create Data Product
- User selects source environment, partition, kinds, and query.
- APIs:
  - GET /api/partition/v1/partitions (ADME): List partitions
  - POST /api/search/v2/query (ADME): Aggregate kinds with counts
  - POST /api/search/v2/query (ADME): Preview/filter records
- Result: Data product definition is saved.
Publish Data Product
- User triggers export/publish for a data product.
- APIs:
  - POST /notebooks/{export_notebook_id}/execute (Fabric): Start export
  - GET /notebooks/{export_notebook_id}/status (Fabric): Poll for export
  - POST /notebooks/{sync_notebook_id}/execute (Fabric): (Optional) Sync
  - POST /notebooks/{ingest_notebook_id}/execute (Fabric): Start ingest
  - GET /notebooks/{ingest_notebook_id}/status (Fabric): Poll for ingest
- Result: Data is exported from ADME, processed in Fabric, and ready for sharing.
Consume Data Product
- User selects a published data product to ingest into their own ADME.
- APIs:
  - GET /api/data-products (Fabric/Custom): List data products
  - POST /api/consume (Fabric/Custom): Trigger ingestion
- Result: Data product is ingested into the consumer’s ADME environment.

Mermaid Diagram

flowchart TD
    A[User: Create Data Product] -->|Selects Source, Partition, Kind| B[ADME API: /partitions, /query]
    B -->|Save Config| C[Control Plane: Save Data Product]
    C -->|User Publishes| D[Fabric API: Execute Export Notebook]
    D -->|Exported Data| E[Fabric Lakehouse]
    E -->|Execute Ingest Notebook| F[Fabric API: Ingest to Target ADME]
    F -->|Data Ready| G[Data Product Published]
    G -->|User Consumes| H[Fabric API: List/Consume Data Product]
    H -->|Ingested| I[Target ADME]

Text Summary

Create Data Product: The user defines a data product by selecting the source ADME, partition, and kinds. The app calls ADME APIs to list partitions and kinds, and saves the configuration.
Publish Data Product: When the user publishes, the app triggers Fabric notebooks via the Fabric API to export data from ADME, process it in Fabric, and prepare it for sharing.
Consume Data Product: Users will be able to browse and subscribe to published data products, triggering ingestion into their own ADME environment.

Name		Name	Last commit message	Last commit date
Latest commit History 8 Commits
clients		clients
components		components
config		config
database		database
docs		docs
notebooks		notebooks
pages		pages
scripts		scripts
services		services
tests		tests
utils		utils
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
app.py		app.py
azure.yaml		azure.yaml
dashboard.png		dashboard.png
deploy.ps1		deploy.ps1
migrate-to-private-link.ps1		migrate-to-private-link.ps1
requirements.txt		requirements.txt

Folders and files

Latest commit

History

Repository files navigation

ADME-Fabric Control Plane

Architecture

Data Flow Sequence

Features

Tech Stack

🚀 Getting Started

Prerequisites

Step 1: Clone the Repository

Step 2: Create Virtual Environment

Step 3: Install Dependencies

Step 4: Configure Environment

Step 5: Run the Application

🧪 Development

Mock Mode (Recommended for Development)

Running with Debug Logging

Project Structure

Code Style

Adding a New Page

🧪 Testing

Run All Tests

Run with Coverage

Test Categories

Manual Testing Checklist

🐳 Docker

Build Image

Run Container

Docker Compose

☁️ Deployment

Recommended: Deploy with PowerShell Script

Alternative: Azure App Service

Option 1: Deploy from Container Registry

Option 2: Deploy from Source

Environment Variables for Production

Persistent Storage (Azure)

📒 Fabric Notebooks

Deployment Steps

Notebook Parameters

🔒 Security

Secrets Management

Generate Encryption Key

Best Practices

🐛 Troubleshooting

Common Issues

Logs

📝 License

🤝 Contributing

📚 References

Data Product Flow

Steps & APIs

Mermaid Diagram

Text Summary

About

Resources

License

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Packages