Worker Pool Architecture

Cloudillo’s Worker Pool provides a three-tier priority thread pool for executing CPU-intensive and blocking operations. This system complements the async runtime by handling work that shouldn’t block async tasks, ensuring responsive performance even under heavy computational load.

Architecture Overview

Three-Tier Priority System

┌───────────────────────────────────────────┐
│ High Priority Queue (1 dedicated thread)  │
│  - User-facing operations                 │
│  - Time-sensitive tasks                   │
│  - Cryptographic signing                  │
└───────────────────────────────────────────┘
                ↓
┌───────────────────────────────────────────┐
│ Medium Priority Queue (2 threads)         │
│  - Processes: High + Medium               │
│  - Image resizing for posts               │
│  - File compression                       │
│  - Normal background work                 │
└───────────────────────────────────────────┘
                ↓
┌───────────────────────────────────────────┐
│ Low Priority Queue (1 thread)             │
│  - Processes: High + Medium + Low         │
│  - Batch operations                       │
│  - Cleanup tasks                          │
│  - Non-urgent processing                  │
└───────────────────────────────────────────┘

Default Configuration:

  • High priority: 1 dedicated thread
  • Medium priority: 2 threads (also process High)
  • Low priority: 1 thread (also processes High + Medium)
  • Total threads: 4

Priority Cascading

Higher priority workers process lower priority work when idle:

Thread 1 (High):     High tasks only
Threads 2-3 (Med):   High → Medium (if no High)
Thread 4 (Low):      High → Medium → Low (if none above)

Benefits:

  • High-priority work always has dedicated capacity
  • Lower-priority work still gets done when system is idle
  • Automatic load balancing

Core Components

WorkerPool Structure

The pool holds one flume sender per priority queue. Worker threads are spawned in new() and run for the process lifetime.

pub struct WorkerPool {
    high: flume::Sender<Box<dyn FnOnce() + Send>>,
    med:  flume::Sender<Box<dyn FnOnce() + Send>>,
    low:  flume::Sender<Box<dyn FnOnce() + Send>>,
}

A job is just a boxed closure (Box<dyn FnOnce() + Send>); results are returned to the caller through a per-call oneshot channel, not stored in the job type.

Initialization

WorkerPool::new(n1, n2, n3) creates three unbounded flume channels and spawns OS threads:

  • n1 threads listen on High only.
  • n2 threads listen on High + Medium.
  • n3 threads listen on High + Medium + Low.

Each thread shares the same receivers (via Arc), so any idle thread on a queue can pick up the next job.

Worker Loop

Every worker — regardless of how many queues it serves — runs the same loop:

1. Priority pass: for each queue in priority order (high → ...),
   try_recv() (non-blocking). If a job is found, run it and restart.
2. If all queues are empty, block on a flume Selector across all of the
   thread's queues until any queue delivers a job.
3. Run the job inside catch_unwind so a panicking job cannot kill the thread.

The non-blocking priority pass ensures higher-priority work is always preferred; the Selector then parks the thread efficiently instead of busy-waiting.

API Usage

Each submission method takes a CPU-bound closure, sends it to a priority queue, and returns a Future that resolves with the closure’s result via a oneshot channel — so async callers offload CPU work without blocking the Tokio runtime.

Method Queue Notes
spawn(priority, f) chosen by Priority Generic entry point
run_immed(f) High “Run now”
run(f) Medium Default choice
run_slow(f) Low Background work
try_run*(f) same as above For closures returning ClResult<T>; flattens ClResult<ClResult<T>> 
pub enum Priority { High, Medium, Low }

If a worker panics, its oneshot sender is dropped and the awaiting future resolves to an Internal error rather than hanging.

Priority Guidelines

Priority Use For Characteristics
High Crypto during login, profile picture processing, real-time compression User actively waiting, <100ms typical, <100 jobs/sec
Medium Image variants for posts, file compression, data transforms Background but needed soon, 100ms-10s, default choice
Low Batch processing, cleanup, pre-generation, optimization No user waiting, can be delayed indefinitely

Integration Patterns

With Task Scheduler

Tasks often use the worker pool for CPU-intensive work following this pattern:

Algorithm: Task with Worker Pool Integration

1. Async I/O: Load data from blob storage
2. CPU work: Execute on worker pool:
   - Load image from memory
   - Resize using high-quality filter (Lanczos3)
   - Encode to desired format
   - Return resized buffer
3. Async I/O: Store result to blob storage

This separates I/O (async) from CPU (worker thread), allowing:
- The async runtime to continue processing other tasks during image resizing
- Background tasks (Low priority) not to starve user-facing operations
- Efficient resource utilization

Configuration

The pool is constructed in server/src/main.rs with hardcoded thread counts:

WorkerPool::new(1, 2, 1)   // high = 1, medium = 2, low = 1

This 1/2/1 split (4 threads total) suits mixed I/O and CPU workloads while leaving cores for the Tokio runtime. Adjusting it currently requires editing the source; there is no environment variable or runtime setting for worker counts.

See Also