PsyQueue

The psychic job queue that adapts to your needs. A micro-kernel distributed job queue platform where everything is a plugin.

Why PsyQueue?

• Zero-infra start -- npm install @psyqueue/core @psyqueue/backend-sqlite and go. No Redis, no Docker.
• Everything is a plugin -- Use only what you need. The kernel is ~500 lines.
• Scales with you -- Start with SQLite, graduate to Redis/Postgres without rewriting code.
• Production-grade performance -- Competitive with BullMQ on Redis. Benchmark suite included — run it yourself.
• Built for SaaS -- Multi-tenant fair scheduling, per-tenant rate limits, noisy-neighbor protection.
• Workflow orchestration -- DAG workflows with conditional branching and Saga compensation.
• Enterprise-ready -- Circuit breakers, adaptive backpressure, exactly-once delivery, audit logs.

Quick Start

npm install @psyqueue/core @psyqueue/backend-sqlite

import { PsyQueue } from '@psyqueue/core'
import { sqlite } from '@psyqueue/backend-sqlite'

const q = new PsyQueue()
q.use(sqlite({ path: ':memory:' }))

// Register a handler
q.handle('email.send', async (ctx) => {
  const { to, subject, body } = ctx.job.payload as any
  await sendEmail(to, subject, body)
  return { sent: true }
})

// Start the queue
await q.start()

// Enqueue a job
const jobId = await q.enqueue('email.send', {
  to: 'user@example.com',
  subject: 'Welcome',
  body: 'Hello from PsyQueue!',
})

// Option A: Process one job at a time (simple / testing)
await q.processNext('email.send')

// Option B: Start a continuous worker pool (production)
q.startWorker('email.send', { concurrency: 10 })

// Stop the queue (also stops all workers)
await q.stop()

Features at a Glance

Feature	Plugin	Description
SQLite Backend	@psyqueue/backend-sqlite	Zero-config embedded storage
Redis Backend	@psyqueue/backend-redis	High-throughput production backend
Postgres Backend	@psyqueue/backend-postgres	ACID-compliant relational backend
Scheduling	@psyqueue/plugin-scheduler	Delayed jobs, cron scheduling
Deadline Priority	@psyqueue/plugin-deadline-priority	Dynamic priority boosting near deadlines
Workflows	@psyqueue/plugin-workflows	DAG-based workflow orchestration
Saga Compensation	@psyqueue/plugin-saga	Automatic rollback on workflow failure
Multi-Tenancy	@psyqueue/plugin-tenancy	Per-tenant rate limits and fair scheduling
Circuit Breaker	@psyqueue/plugin-circuit-breaker	Protect downstream services
Backpressure	@psyqueue/plugin-backpressure	Adaptive load shedding
Exactly-Once	@psyqueue/plugin-exactly-once	Idempotency key deduplication
Crash Recovery	@psyqueue/plugin-crash-recovery	WAL-based recovery of orphaned jobs
OTel Tracing	@psyqueue/plugin-otel-tracing	OpenTelemetry distributed tracing
Metrics	@psyqueue/plugin-metrics	Prometheus metrics (prom-client)
Audit Log	@psyqueue/plugin-audit-log	Tamper-evident hash-chained audit trail
Dashboard	@psyqueue/dashboard	Web UI for monitoring and management
Schema Versioning	@psyqueue/plugin-schema-versioning	Payload migration and Zod validation
gRPC Workers	@psyqueue/plugin-grpc-workers	Distribute work to remote gRPC workers
HTTP Workers	@psyqueue/plugin-http-workers	Distribute work to remote HTTP workers
Chaos Testing	@psyqueue/plugin-chaos	Inject failures for resilience testing
Offline Sync	@psyqueue/plugin-offline-sync	Local-first with background sync
Job Fusion	@psyqueue/plugin-job-fusion	Batch similar jobs into one
CLI	@psyqueue/cli	Migration, replay, and audit commands

Benchmark Results

PsyQueue ships with a comprehensive benchmark suite so you can measure performance in YOUR environment:

git clone https://github.com/ayush-jadaun/psyqueue.git
cd psyqueue && pnpm install && pnpm build

# Throughput benchmark (needs Redis)
npx tsx benchmarks/_perf.ts

# Feature-by-feature comparison vs BullMQ
npx tsx benchmarks/feature-comparison.ts

# 10-battle reliability test
npx tsx benchmarks/battle-test.ts

# k6 load test (needs k6 installed)
npx tsx benchmarks/k6/psyqueue-server.ts  # terminal 1
k6 run benchmarks/k6/load-test.js         # terminal 2

Key architectural advantages over BullMQ:

• Fused ackAndFetch — ack + dequeue in one Redis call (2 round-trips vs 3)
• Hybrid list + sorted set — O(1) dequeue for priority=0 jobs
• Hash field packing — 13 fields vs 30 per HGETALL
• 10 features BullMQ doesn't have — workflows, saga, tenancy, fusion, chaos testing, etc.

Architecture

                        +-----------------------+
                        |      PsyQueue         |
                        |      (Kernel)         |
                        +-----------+-----------+
                                    |
                  +-----------------+-----------------+
                  |                 |                 |
           +------+------+  +------+------+  +------+------+
           | Event Bus   |  | Middleware   |  | Plugin      |
           | (wildcards) |  | Pipeline     |  | Registry    |
           +------+------+  +------+------+  +------+------+
                  |                 |                 |
                  |     +-----------+-----------+     |
                  |     |  Lifecycle Phases     |     |
                  |     |  guard -> validate -> |     |
                  |     |  transform -> observe |     |
                  |     |  -> execute -> final  |     |
                  |     +-----------+-----------+     |
                  |                 |                 |
     +------------+-----------------+-----------------+-----------+
     |            |            |            |            |        |
  +--+---+   +---+---+   +---+---+   +---+---+   +---+---+  +--+---+
  |SQLite|   |Redis  |   |Sched- |   |Work-  |   |Tenant |  | ...  |
  |Back- |   |Back-  |   |uler   |   |flows  |   |cy     |  |Plug- |
  |end   |   |end    |   |       |   |       |   |       |  |ins   |
  +------+   +-------+   +-------+   +-------+   +-------+  +------+

The kernel provides four core services:

1. Plugin Registry -- Dependency resolution (topological sort), lifecycle management (init/start/stop/destroy).
2. Event Bus -- Pub/sub with wildcard support (job:* matches job:completed, job:failed, etc.).
3. Middleware Pipeline -- Ordered execution across 6 phases: guard, validate, transform, observe, execute, finalize.
4. Backend Adapter -- Pluggable storage with a unified interface (enqueue, dequeue, ack/nack, scheduling, locking).

Presets

PsyQueue ships with three presets that list recommended plugin combinations.

Lite -- Get started fast

const { queue, config } = PsyQueue.from('lite')
// config.plugins: ['backend-sqlite', 'scheduler', 'crash-recovery']

queue
  .use(sqlite({ path: './jobs.db' }))
  .use(scheduler())
  .use(crashRecovery())

SaaS -- Multi-tenant applications

const { queue, config } = PsyQueue.from('saas')
// config.plugins: ['backend-redis', 'tenancy', 'workflows', 'saga',
//   'circuit-breaker', 'backpressure', 'dashboard', 'metrics']

queue
  .use(redis({ host: 'localhost' }))
  .use(tenancy({ tiers: { free: { ... }, pro: { ... } }, ... }))
  .use(workflows())
  .use(saga())
  .use(circuitBreaker({ breakers: { ... } }))
  .use(backpressure({ signals: { ... } }))
  .use(dashboard({ port: 3001 }))
  .use(metrics())

Enterprise -- Full-featured deployment

const { queue, config } = PsyQueue.from('enterprise')
// config.plugins: ['backend-postgres', 'tenancy', 'workflows', 'saga',
//   'exactly-once', 'audit-log', 'otel-tracing', 'schema-versioning',
//   'circuit-breaker', 'backpressure', 'dashboard', 'metrics']

Documentation

• Getting Started -- Installation, first job, and step-by-step guide
• Architecture -- Kernel design, plugin system, middleware pipeline
• API Reference -- Full API documentation for the core package
• Comparison -- PsyQueue vs BullMQ vs Celery vs Temporal vs pg-boss

Plugin Guides

• Backends -- SQLite, Redis, Postgres
• Scheduling -- Delayed jobs, cron, deadline priority
• Workflows -- DAG workflows and Saga compensation
• Multi-Tenancy -- Tiers, fair scheduling, rate limiting
• Reliability -- Circuit breaker, backpressure, exactly-once, crash recovery
• Observability -- OTel tracing, Prometheus metrics, audit log, dashboard
• Transport -- gRPC and HTTP remote workers
• Advanced -- Chaos testing, offline sync, job fusion, schema versioning

Contributing

Contributions are welcome. Please open an issue or pull request on GitHub.

git clone https://github.com/ayush-jadaun/psyqueue.git
cd psyqueue
pnpm install
pnpm build
pnpm test

License

MIT