System Architecture Overview

This document provides a technical overview of the Cloudillo system architecture, explaining the core patterns and components that enable its federated, privacy-focused design.

Workspace Structure

Cloudillo is organized as a Rust workspace with feature-specific crates:

cloudillo-rs/
├── server/                      # Binary: cloudillo-server
├── crates/
│   ├── cloudillo/               # Main integrator (routes, websocket, bootstrap)
│   ├── cloudillo-types/         # Shared types, adapter traits, error types
│   ├── cloudillo-core/          # Infrastructure (scheduler, worker, middleware, ACME)
│   ├── cloudillo-auth/          # Authentication (login, WebAuthn, QR, API keys)
│   ├── cloudillo-action/        # Federation actions (posts, delivery, verification)
│   ├── cloudillo-file/          # File processing (images, video, audio, PDF, SVG)
│   ├── cloudillo-crdt/          # CRDT collaborative editing (Yjs WebSocket)
│   ├── cloudillo-rtdb/          # Real-time database
│   ├── cloudillo-push/          # Web Push notifications
│   ├── cloudillo-email/         # Email notifications (SMTP, templates)
│   ├── cloudillo-idp/           # Identity provider, tenant management
│   ├── cloudillo-profile/       # Profile management
│   ├── cloudillo-admin/         # System administration
│   ├── cloudillo-ref/           # Reference/lookup API
│   ├── cloudillo-contact/       # Contacts, vCard, CardDAV
│   ├── cloudillo-calendar/      # Calendar, CalDAV
│   ├── cloudillo-dav/           # Shared WebDAV/DAV plumbing
│   └── cloudillo-proxy/         # Reverse proxy with TLS
└── adapters/
    ├── auth-adapter-sqlite/     # Authentication & cryptography (SQLite)
    ├── meta-adapter-sqlite/     # Metadata storage (SQLite)
    ├── blob-adapter-fs/         # Binary blob storage (Filesystem)
    ├── rtdb-adapter-redb/       # Real-time database (Redb)
    └── crdt-adapter-redb/       # Collaborative editing CRDT (Redb)

Crate Architecture

The workspace follows a three-layer architecture:

• cloudillo-types: Foundation layer — adapter trait definitions (AuthAdapter, MetaAdapter, BlobAdapter, RtdbAdapter, CrdtAdapter), shared domain types, and error types
• cloudillo-core: Infrastructure layer — task scheduler, worker pool, authentication middleware, ACME certificate management, custom extractors, WebSocket infrastructure, and HTTP client
• cloudillo-*: Feature crates — each domain (auth, action, file, etc.) is an independent crate with its own handlers, tasks, and settings
• cloudillo: Integrator — assembles all feature crates into HTTP routes, WebSocket handling, and application bootstrap
• server: Binary crate (cloudillo-server) — integrates all feature crates and adapters into a runnable server
• adapters: Five pluggable storage backends implementing the traits defined in cloudillo-types

Adapter Types

The five adapters separate concerns and enable flexible deployments:

1. AuthAdapter - Authentication, JWT tokens, certificate management, cryptographic operations
2. MetaAdapter - Tenant/profile data, action tokens, file metadata, tasks
3. BlobAdapter - Content-addressed immutable binary data (files, images, snapshots)
4. RtdbAdapter - Real-time hierarchical JSON database with queries and subscriptions
5. CrdtAdapter - Collaborative document editing with conflict-free merges

Core Architecture Patterns

The Five-Adapter Architecture

Cloudillo’s architecture is built on five fundamental adapters that separate concerns and enable flexible deployments:

1. AuthAdapter

Purpose: Authentication, authorization, and cryptographic operations

Responsibilities:

• JWT/token validation and creation
• TLS certificate management (ACME integration)
• Profile signing key storage and rotation
• VAPID keys for push notifications
• Password hashing and WebAuthn credentials
• Tenant management

Key Operations: Validate tokens, create access tokens, manage TLS certificates, handle profile keys, store passwords/credentials

Why separate? Authentication and cryptography require special security considerations and may need different storage backends (HSM, vault services, etc.).

2. MetaAdapter

Purpose: Structured metadata storage

Responsibilities:

• Tenant and profile information
• Action tokens (posts, comments, reactions, etc.)
• File metadata with variants
• Task persistence for the scheduler
• Database metadata (for RTDB)

Key Operations: Create/query tenants, store/retrieve actions, manage file metadata, persist tasks, handle profiles

Why separate? Metadata can be stored in different databases (SQLite, PostgreSQL, MongoDB) based on scale and requirements.

3. BlobAdapter

Purpose: Immutable binary data storage

Responsibilities:

• Content-addressed blob storage
• File data (images, videos, documents)
• Database snapshots (for RTDB)
• Both buffered and streaming I/O

Key Operations: Write/read blobs (buffered or streaming), check blob existence and size

Why separate? Blob storage can use filesystem, S3, CDN, or specialized object storage based on scale and cost considerations.

4. RtdbAdapter

Purpose: Real-time structured database

Responsibilities:

• Path-based hierarchical storage (Firebase-like)
• Query with filtering, sorting, pagination
• Real-time subscriptions via WebSocket
• Transaction support for atomic writes
• Secondary indexes for performance
• Multi-tenant database isolation

Key Operations: Query/create/update/delete documents, transactions, real-time subscriptions

Why separate? Real-time database functionality can use different backends (redb, PostgreSQL, MongoDB) based on query requirements and scale.

Learn more: RTDB with redb

5. CrdtAdapter

Purpose: Collaborative document storage with CRDTs

Responsibilities:

• Binary CRDT update storage (Yjs protocol)
• Document metadata management
• Real-time change subscriptions
• Conflict-free merge guarantees
• Awareness state (presence, cursors)
• Multi-tenant document isolation

Key Operations: Create/read documents, append updates, retrieve update history, subscribe to changes

Why separate? CRDT storage can use different backends (redb, dedicated CRDT stores) and has different performance characteristics than traditional databases.

Learn more: CRDT Collaborative Editing

Benefits of the Adapter Pattern

• ✅ Flexible Deployment: Switch storage backends without changing core logic
• ✅ Separation of Concerns: Security, metadata, and binary data have different requirements
• ✅ Testing: Easy to create in-memory adapters for testing
• ✅ Scalability: Can distribute adapters across different services
• ✅ Cost Optimization: Use appropriate storage for each data type

Content-Addressed Architecture

Cloudillo uses content-addressing throughout its architecture, where resource identifiers are cryptographic hashes of their content. This creates a merkle tree structure that provides cryptographic proof of authenticity and immutability.

Hash-Based Identifiers

All resource IDs are SHA-256 hashes with versioned prefixes:

Version Scheme

Format: {prefix}{version}~{base64_encoded_hash}

• Version 1: SHA-256 with base64url encoding (no padding)
• Future versions: Can upgrade to SHA-3, BLAKE3, etc. without breaking old content
• Backward compatibility: Old content remains valid forever
• Algorithm agility: Migrate to new algorithms without breaking existing references

Example upgrade path:

a1~...  (SHA-256)
a2~...  (SHA-3)
a3~...  (BLAKE3)

Six-Level Merkle Tree

Content-addressing creates a hierarchical merkle tree:

Level 1: Blob Data (raw bytes)
   ↓ SHA-256 hash
Level 2: Blob ID (b1~hash)
   ↓ collected in descriptor
Level 3: File Descriptor (d2,class.variant:b1,hash:f=format:s=size:r=resolution;...)
   ↓ SHA-256 hash of descriptor
Level 4: File ID (f1~hash)
   ↓ referenced in action
Level 5: Action Token (JWT with content, parent, attachments)
   ↓ SHA-256 hash of entire JWT
Level 6: Action ID (a1~hash)

Properties

• ✅ Immutable: Content cannot change without changing the ID
• ✅ Tamper-Evident: Any modification is immediately detectable
• ✅ Deduplicatable: Identical content produces identical IDs
• ✅ Verifiable: Anyone can recompute and verify hashes
• ✅ Cacheable: Content-addressed data can be cached forever
• ✅ Trustless: No need to trust storage providers—verify the hash

Integration with Adapters

Content-addressing is implemented across multiple adapters:

BlobAdapter:

• Stores blobs indexed by blob_id (b1~...)
• Blob IDs are SHA-256 hashes of the blob bytes
• Enables deduplication and integrity verification

MetaAdapter:

• Stores action tokens indexed by action_id (a1~...)
• Action IDs are SHA-256 hashes of the JWT token
• Stores file metadata with file_id (f1~...)
• File IDs are SHA-256 hashes of the descriptor

AuthAdapter:

• Signs action tokens with profile keys (ES384)
• Signature + content hash = complete authenticity proof

Learn more: Content-Addressing & Merkle Trees

Task-Based Asynchronous Processing

Complex operations in Cloudillo are modeled as persistent tasks that can execute asynchronously, survive restarts, and depend on other tasks.

Task System Components

Tasks

Tasks implement the Task<S> trait, whose core methods are kind() (a static type name), build()/serialize() (for persistence), run(&self, state) (the work itself, returning ClResult<()>), and on_failed() (cleanup after retries are exhausted). Dependencies and retry policy are supplied when a task is scheduled, not declared on the trait.

Built-in Task Types (non-exhaustive):

• ActionCreatorTask: Creates and signs action tokens for federation
• ActionVerifierTask: Validates incoming federated actions
• ActionDeliveryTask: Delivers actions to remote instances with retry
• FileIdGeneratorTask: Generates content-addressed file IDs
• ImageResizerTask: Creates image variants (thumbnails, etc.)
• VideoTranscoderTask: Transcodes video to web-optimized formats
• AudioExtractorTask: Extracts audio metadata
• PdfProcessorTask: Extracts text and metadata from PDFs
• EmailSenderTask: Sends emails asynchronously
• CertRenewalTask: Handles ACME certificate renewal
• ProfileRefreshBatchTask: Batch-refreshes remote profiles
• TenantImageUpdaterTask: Updates tenant avatar images

Scheduler

The scheduler manages task lifecycle with dependency resolution:

Features:

• Task registry with dynamic builders
• Dependency resolution (DAG-based)
• Scheduled execution (cron-like)
• Persistence via MetaAdapter (survives restarts)
• Notification system for task completion

Example Flow:

ActionCreatorTask (depends on FileIdGeneratorTask)
    ↓
    waits for file processing to complete
    ↓
FileIdGeneratorTask completes
    ↓
ActionCreatorTask auto-starts
    ↓
Creates signed JWT, stores in MetaAdapter

Worker Pool

A priority-based thread pool for CPU-intensive and blocking operations:

Architecture:

• Three priority tiers: High > Medium > Low
• Per-tier worker thread counts set at construction (hardcoded in the server)
• Uses flume MPMC channels for work distribution
• Returns futures for async integration

Default Configuration (cloudillo-server):

• 1 high-priority worker
• 2 medium-priority workers
• 1 low-priority worker

Use Cases:

• Image processing (CPU-intensive)
• Cryptographic operations
• File compression
• Blocking I/O

Application State Management

AppState Structure

The core application state contains: scheduler, worker pool, HTTP client, TLS certificates, and all five adapters (auth, meta, blob, rtdb, crdt).

AppBuilder Pattern

Configuration uses a fluent builder API with mode, identity, domain, data directory, and adapter selections.

Configuration Options:

• Server mode (Standalone, Proxy, StreamProxy)
• Network binding (HTTPS/HTTP ports)
• Domain configuration
• Directory paths (dist, tmp, data)
• Adapter injection
• Worker pool sizing

Crate Organization

The workspace is organized into feature-specific crates under crates/:

cloudillo-types - Foundation Layer

Shared types and trait definitions used across all crates:

• Adapter trait definitions (AuthAdapter, MetaAdapter, BlobAdapter, RtdbAdapter, CrdtAdapter)
• Core domain types and type aliases
• Error types with unified error handling
• HTTP extractors and utility types

cloudillo-core - Infrastructure Layer

Core system components providing foundational services:

• scheduler.rs: Task scheduling with dependencies
• app.rs: Application state and builder
• acme.rs: Let’s Encrypt/ACME certificate management
• middleware.rs: Authentication middleware
• extract.rs: Custom Axum extractors (TnId, IdTag, Auth)
• ws_bus.rs: WebSocket message bus
• ws_broadcast.rs: WebSocket broadcast manager
• rate_limit/: Rate limiting (governor-based)
• request.rs: HTTP client for federation
• roles.rs: Role definitions
• settings/: Global settings system

cloudillo-auth - Authentication

• Login/logout endpoints, token generation, password management
• WebAuthn passwordless authentication
• QR code-based login flow
• API key generation and validation

cloudillo-action - Federation Actions

• Action creation, verification, and delivery tasks
• JWT verification and token processing
• Action CRUD endpoints and federation inbox
• Federated delivery with retry and audience computation

cloudillo-file - File Storage & Processing

• File upload/download endpoints
• Image processing (resize, format conversion, SVG rasterization)
• Video transcoding and audio extraction (via FFmpeg)
• PDF processing and file descriptor management
• Variant generation and preset system

cloudillo-profile - Profile Management

• Tenant profile endpoints
• Profile federation and sync
• Avatar and banner image processing

cloudillo-rtdb - Real-Time Database

• Database CRUD endpoints
• WebSocket subscription-based data sync

cloudillo-crdt - Collaborative Editing

• Yjs WebSocket protocol handler for CRDT sync

cloudillo-push - Push Notifications

• Web Push notification delivery (RFC 8291 + VAPID)
• Subscription management and per-user preferences

cloudillo-email - Email Notifications

• SMTP delivery via lettre
• Handlebars template rendering
• Async email sender task

cloudillo-idp - Identity Provider

• Tenant registration and lifecycle management

cloudillo-admin - Administration

• System admin endpoints for instance management

cloudillo-ref - Reference API

• Namespace lookups and reference resolution

cloudillo-proxy - Reverse Proxy

• HTTP/WebSocket proxying with TLS termination

cloudillo - Integrator

Assembles all feature crates into a runnable application:

• routes.rs: HTTP endpoint definitions (public and protected route groups)
• websocket.rs: WebSocket protocol handler
• bootstrap.rs: First-run tenant setup
• webserver.rs: Server configuration (TLS, CORS, compression)

Concurrency Model

Multi-threaded Tokio Runtime

Cloudillo uses Tokio’s multi-threaded async runtime for handling concurrent requests.

Concurrency Layers

1. Async Layer (Tokio): HTTP request handling, WebSocket connections, I/O
2. Worker Pool: CPU-intensive tasks, blocking operations
3. Scheduler: Background task execution with dependencies

Interaction Example:

HTTP Request → Tokio async handler
    ↓
Spawn ImageResizerTask on scheduler
    ↓
Scheduler dispatches to worker pool (CPU-intensive)
    ↓
Worker completes, updates MetaAdapter
    ↓
Scheduler notifies waiting tasks
    ↓
Response returned to client

Request Handling Flow

Middleware Pipeline

HTTP requests flow through these layers:

1. HTTPS/SNI Resolution: CertResolver selects TLS certificate by domain
2. Tracing/Logging: Request tracing with structured logging
3. Authentication: require_auth or optional_auth middleware
4. Custom Extractors: TnId, IdTag, Auth extract from request context
5. Handler: Business logic execution
6. Response: JSON serialization, error handling

Custom Extractors

Axum extractors provide typed access to request context:

• TnId: Tenant ID (database primary key, u32)
• IdTag: Tenant identifier string (e.g., “alice.example.com”)
• Auth: Full authentication context (tn_id, id_tag, scope, etc.)

Error Handling

Custom error enum with automatic HTTP response conversion: NotFound (404), PermissionDenied (403), DbError/Unknown/Parse/Io (500).

Server Modes

Cloudillo supports different deployment modes:

Standalone (Default)

Self-contained single instance:

• HTTPS on configured port
• Optional HTTP for ACME challenges
• All adapters run locally

Use case: Personal servers, small communities

Proxy

Used if Cloudillo is behind a reverse proxy:

• Listens on HTTP port
• Certificate handled is the responsibility of the proxy

Use case: Managed hosting providers, self-hosting with multiple services on one IP address

Security Architecture

Implemented in Rust

Maximal memory and concurrency safety. Minimal attack surface.

No Unsafe Code

Cloudillo enforces memory safety:

#![forbid(unsafe_code)]

ABAC Permission System

Cloudillo uses Attribute-Based Access Control (ABAC) for fine-grained permissions across all resources. ABAC provides flexible permission rules based on:

• User attributes (identity, roles, relationships)
• Resource attributes (owner, visibility, type)
• Contextual factors (time, environment)

Key Features:

• Six visibility levels: Public (P), Verified (V), SecondDegree (2), Follower (F), Connected (C), Direct (NULL)
• Policy-based access control (TOP/BOTTOM policies)
• Relationship-aware (following, connections)
• Time-based permissions
• Custom policy support

Learn more: ABAC Permission System

Cryptographic Algorithms

• P384: Elliptic curve for action signing
• ES384: JWT signature algorithm
• SHA256: Content addressing and hashing
• bcrypt: Password hashing

Security Layers

1. TLS/HTTPS: All connections encrypted (Rustls)
2. ACME: Automatic certificate management (Let’s Encrypt)
3. JWT: Cryptographically signed tokens
4. Content Addressing: Tamper detection via SHA256
5. Permission Checks: Authorization at every access point

Bootstrap Process

Initial setup when starting a new instance:

1. Create tenant with base_id_tag
2. Set password (if provided)
3. Generate profile signing key (P384)
4. Initiate ACME for TLS certificates (if email provided)
5. Start scheduler and worker pool
6. Start HTTP/HTTPS servers

Example configuration:

BASE_ID_TAG=alice.example.com        # Required
BASE_PASSWORD=secret                 # Initial password
ACME_EMAIL=alice@example.com         # Let's Encrypt email
MODE=standalone                      # Server mode
LISTEN=0.0.0.0:8443                  # HTTPS binding
DATA_DIR=./data                      # storage path

Key Dependencies

Web Framework

• axum (0.8): Async web framework
• tower: Service abstractions
• tower-http: CORS, static files

TLS & Crypto

• rustls (0.23): Pure Rust TLS
• instant-acme (0.8): ACME client
• jsonwebtoken (10.4): JWT handling
• p384: Elliptic curve operations

Async Runtime

• tokio (1): Multi-threaded async

Serialization

• serde (1.0): Serialization framework
• serde_json: JSON support

Database

• sqlx (0.8): Async SQL (SQLite support)

Utilities

• image (0.25): Image processing
• sha2: SHA256 hashing
• croner (3.0): Cron expressions
• flume: MPMC channels

Architectural Strengths

• ✅ Pluggable Adapters: Easy to swap storage backends
• ✅ Self-Contained: No external dependencies required
• ✅ Federated: Communicates with other instances
• ✅ Task-Based: Persistent, resumable async execution
• ✅ Type-Safe: Leverages Rust’s type system
• ✅ Memory-Safe: Complete #![forbid(unsafe_code)]
• ✅ Observable: Built-in tracing integration

Next Steps

• Identity System - DNS-based identity and key management
• Access Control - Token validation and permissions
• ABAC Permissions - Attribute-based access control system
• Actions & Federation - Action tokens and cross-instance communication
• File Storage - Content-addressed storage and processing
• RTDB with redb - Query-based real-time database
• CRDT Collaborative Editing - Conflict-free collaborative documents
• Real-Time Systems Overview - Introduction to RTDB and CRDT

Identity System & User Profiles

Cloudillo Profiles are located using a DNS-based identity system. Each Cloudillo Identity is associated with a specific API endpoint, which can be accessed via the “cl-o” subdomain of the identity. For example, the API domain of the cloudillo.net identity is available at https://cl-o.cloudillo.net/.

Retrieving a Cloudillo Profile

Fetch a profile by requesting the /api/me endpoint of the identity’s API domain. It is unauthenticated, so any peer (or client) can read it:

curl https://cl-o.cloudillo.net/api/me

The response is wrapped in the standard API envelope (data, time):

{
  "data": {
    "idTag": "cloudillo.net",
    "name": "Cloudillo",
    "type": "community",
    "profilePic": "QoEYeG8TJZ2HTGhVlrtTDBpvBGOp6gfGhq4QmD6Z46w",
    "keys": [
      {
        "keyId": "250205",
        "publicKey": "MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE..."
      }
    ]
  },
  "time": "2025-02-05T12:00:00Z"
}

Profile Fields

• idTag: The identity tag (DNS-based identifier) of the profile.
• name: The display name.
• type: "person" for personal profiles or "community" for communities.
• profilePic: File ID of the profile picture, retrievable via the storage API.
• keys: The profile’s public signing keys (see below).

For the owner-facing view with cover image and extended metadata, clients use /api/me/full.

Profile Picture Retrieval

The profilePic field is a file ID that resolves through the Cloudillo Storage API:

curl https://cl-o.nikita.cloudillo.net/api/store/NMzE4gNI29aEkiV6Q7I1UNWh4x2gFZ7753Pl74veYtU

Key Management

A profile can hold multiple signing keys, enabling periodic key rotation while older keys remain valid for verifying past actions.

Key Structure

Each entry in the keys array is published with three fields:

pub struct AuthKey {
    key_id: Box<str>,              // "keyId": date-based ID, e.g. "250205"
    public_key: Box<str>,          // "publicKey": P-384 public key (PKCS#8 PEM)
    expires_at: Option<Timestamp>, // "expiresAt": optional expiration
}

Private keys are held only in the AuthAdapter and never appear in the response.

Cryptographic Algorithm

Signing keys use the P-384 elliptic curve (NIST P-384 / secp384r1). Action tokens are signed with ES384 (ECDSA over P-384 with SHA-384).

Key Rotation

The key ID is the creation date in YYMMDD format (e.g. 250205). To rotate, the server generates a new key, adds it alongside the existing ones, and marks the old key as expired. Because each action token records the keyId it was signed with, old tokens stay verifiable against the matching (expired) key.

Push Notification Keys

Web push uses a separate per-tenant VAPID key pair (ES256 / P-256), stored in the AuthAdapter. These are independent from the signing keys above and are not part of the /api/me response. See Network & Security for the full cryptography table.

Identity Resolution

DNS-Based Discovery

Cloudillo uses DNS to discover the API endpoint for an identity:

1. Identity: alice.example.com
2. API Domain: cl-o.alice.example.com
3. API Endpoint: https://cl-o.alice.example.com/api/me

Cloudillo Identity Providers (CIP)

Users without their own domain can use a Cloudillo Identity Provider:

CIP Responsibilities:

• Domain management (subdomains or custom domains)
• DNS configuration
• Dynamic DNS support
• CIPs never store or access any user data, nor have any responsibility for it.

Example CIP: cloudillo.net

• Provides identities like alice.cloudillo.net
• API available at https://cl-o.alice.cloudillo.net
• Users can migrate to their own domain later

Custom Domain Setup

To use your own domain with Cloudillo:

1. DNS Configuration:

cl-o.alice.example.com.  A      <your-server-ip>
cl-o.alice.example.com.  AAAA   <your-server-ipv6>
app.example.com.  A             <your-server-ip>
app.example.com.  AAAA          <your-server-ipv6>

2. TLS Certificate: Automatic via ACME (Let’s Encrypt)

ACME_EMAIL=admin@example.com

3. Bootstrap: Configure tenant

BASE_ID_TAG=alice.example.com
BASE_APP_DOMAIN=app.example.com
BASE_PASSWORD=<secure-password>

Profile Metadata & Synchronization

Display metadata (name, type, profile picture, extended sections) lives in the MetaAdapter; the ProfileType enum is either Person or Community. Signing keys live in the AuthAdapter. The /api/me handler combines both.

When an instance interacts with a remote identity, it fetches https://cl-o.{remote_id_tag}/api/me and caches the result locally. The response carries an ETag, so frequent re-syncs answer 304 Not Modified when nothing has changed.

Security Considerations

Public Key Infrastructure

• Public keys are publicly accessible via /api/me
• Private keys are stored securely in AuthAdapter
• No private key export - keys never leave the server
• Key verification happens on every action token

Identity Trust Model

Trust is established through:

1. DNS ownership: Control of domain proves identity ownership
2. Key signatures: Private key proves control of identity
3. HTTPS: TLS proves server authenticity
4. Action signatures: ES384 signatures prove action authenticity

Attack Mitigation

DNS Hijacking:

• Old signatures remain valid (past actions can’t be forged)
• Users can verify key changes

Key Compromise:

• Rotate immediately to a new key and mark the compromised key as expired
• Legitimate past actions remain verifiable (each token pins its keyId)

Server Compromise:

• Private keys in AuthAdapter may need HSM/vault for high-security deployments
• Separate AuthAdapter storage from other adapters
• Regular security audits

API Reference

GET /api/me

Retrieve public profile information, including the signing keys used to verify this identity’s action tokens. Unauthenticated.

Request:

curl https://cl-o.example.com/api/me

Response (200 OK, wrapped in the standard data envelope):

{
  "data": {
    "idTag": "example.com",
    "name": "Alice",
    "type": "person",
    "profilePic": "f1~Qo...46w",
    "keys": [
      { "keyId": "250205", "publicKey": "MHYwEAYHKoZIzj0CAQ..." }
    ]
  }
}

GET /api/me/full returns the same fields plus the cover image and the tier-filtered extended metadata (x) map for the owner-facing UI.

See Also

• System Architecture - Overall system design
• Access Control - Token validation using profile keys
• Actions - Action token signing with profile keys

Content-Addressing & Merkle Trees

Cloudillo implements a merkle tree structure using content-addressed identifiers throughout its architecture. Every action, file, and data blob is identified by the cryptographic hash of its content, creating an immutable, verifiable chain of trust.

What is Content-Addressing?

Content-addressing means identifying data by what it is (its content) rather than where it is (its location). Instead of using arbitrary IDs or URLs, Cloudillo computes a cryptographic hash of the content itself and uses that hash as the identifier.

Benefits

• ✅ Immutable: Content cannot change without changing its identifier
• ✅ Tamper-Evident: Any modification is immediately detectable
• ✅ Deduplicatable: Identical content produces identical identifiers
• ✅ Verifiable: Anyone can recompute and verify hashes independently
• ✅ Cacheable: Content-addressed data can be cached forever
• ✅ Trustless: No need to trust storage providers—verify the hash

Hash Function

Cloudillo uses SHA-256 for all content-addressing:

• Algorithm: SHA-256 (256-bit Secure Hash Algorithm)
• Encoding: Base64url without padding (URL-safe)
• Output: 43-character base64-encoded string
• Collision Resistance: Cryptographically secure

compute_hash(prefix, data):
    hash = SHA256(data)
    encoded = base64url_encode(hash)  // URL-safe, no padding
    return "{prefix}1~{encoded}"

// Example:
compute_hash("b", blob_bytes) → "b1~abc123def456..." (43 chars)
compute_hash("f", descriptor)  → "f1~Qo2E3G8TJZ..." (43 chars)
compute_hash("a", jwt_token)   → "a1~8kR3mN9pQ2vL..." (43 chars)

Merkle Tree Structure

Cloudillo’s content-addressing creates a variable-depth merkle tree where actions can reference other actions recursively. The example below shows a six-level hierarchy for a POST action with image attachments:

┌─────────────────────────────────────────────────────────┐
│ Level 6: Action ID (a1~8kR3mN9pQ2vL6xW...)              │
│   ↑ SHA-256 hash of ↓                                   │
├─────────────────────────────────────────────────────────┤
│ Level 5: Action Token (JWT)                             │
│   Header: {"alg":"ES384","typ":"JWT"}                   │
│   Payload: {                                            │
│     "iss": "alice.example.com",                         │
│     "t": "POST:IMG",                                    │
│     "c": "Amazing photo!",                              │
│     "a": ["f1~Qo2E3G8TJZ..."],                          │
│     "iat": 1738483100                                   │
│   }                                                     │
│   Signature: <ES384 signature>                          │
│   ↓ references ↓                                        │
├─────────────────────────────────────────────────────────┤
│ Level 4: File ID (f1~Qo2E3G8TJZ2HTGhVlrtTDBp...)        │
│   ↑ SHA-256 hash of ↓                                   │
├─────────────────────────────────────────────────────────┤
│ Level 3: File Descriptor (d2,...)                       │
│   "d2,vis.tn:b1,abc:f=avif:s=4096:r=150x150;            │
│        vis.sd:b1,def:f=avif:s=32768:r=640x480;          │
│        vis.md:b1,ghi:f=avif:s=262144:r=1920x1080"       │
│   ↓ references ↓                                        │
├─────────────────────────────────────────────────────────┤
│ Level 2: Variant IDs                                    │
│   b1~abc123... (vis.tn)                                 │
│   b1~def456... (vis.sd)                                 │
│   b1~ghi789... (vis.md)                                 │
│   ↑ SHA-256 hash of ↓                                   │
├─────────────────────────────────────────────────────────┤
│ Level 1: Blob Data (raw bytes)                          │
│   <AVIF encoded image data - tn variant>                │
│   <AVIF encoded image data - sd variant>                │
│   <AVIF encoded image data - md variant>                │
└─────────────────────────────────────────────────────────┘

Level 1: Blob Data

Raw bytes of actual content (images, videos, documents).

Level 2: Variant IDs (b1~…)

Content-addressed blob identifiers computed as SHA256(blob_bytes).

Level 3: File Descriptor (d2,…)

Encoded string listing all available variants of a file.

Format (variants separated by ;, fields by :):

d2,{class}.{variant}:{variant_id}:f={format}:s={size}:r={width}x{height};...

Example:

d2,vis.tn:b1,abc123:f=avif:s=4096:r=150x150;vis.sd:b1,def456:f=avif:s=32768:r=640x480

The d2 format uses , as the ID separator inside variant IDs (b1,abc123). An optional leading R={root} field carries the document-tree root.

Level 4: File ID (f1~…)

Content-addressed file identifier computed as SHA256(descriptor_string).

Level 5: Action Token (JWT)

Cryptographically signed JSON Web Token representing a user action.

Structure:

{
  "header": {
    "alg": "ES384",
    "typ": "JWT"
  },
  "payload": {
    "iss": "alice.example.com",
    "k": "250101",
    "t": "POST:IMG",
    "c": "Check out this amazing photo!",
    "a": ["f1~Qo2E3G8TJZ..."],
    "iat": 1738483100
  },
  "signature": "<ES384 signature with alice's private key>"
}

Level 6: Action ID (a1~…)

Content-addressed action identifier computed as SHA256(complete_jwt_token).

Hash Versioning Scheme

All identifiers use a versioned prefix format for future-proofing:

{prefix}{version}~{base64_encoded_hash}

Current Prefixes

Version Scheme

• Version 1: SHA-256 with base64url encoding (no padding)
• Future versions: Can upgrade to SHA-3, BLAKE3, etc.
• Backward compatibility: Old content remains valid forever
• Algorithm agility: Migrate to new algorithms without breaking existing references

Example upgrade path:

a1~...  (SHA-256)
a2~...  (SHA-3)
a3~...  (BLAKE3)

Merkle Tree Properties

Content-addressing gives the tree several properties for free:

• Immutability: Changing any content changes its ID, so the original is never overwritten — edits produce a new action with a new a1~... ID.
• Tamper-evidence: A modification anywhere propagates upward. Altering a thumbnail changes its b1~... ID, which changes the descriptor, the f1~... file ID, and breaks the post’s attachment reference — verification fails.
• Deduplication: Identical content yields identical IDs, so a file shared by many users is stored once.

Chain of Trust

Each reply references its parent by content hash (the p claim), binding a thread together cryptographically:

Post (a1~abc...)
  ↑ p (parent)
Comment (a1~def...)
  ↑ p (parent)
Reply (a1~ghi...)

Modifying the Post would change its ID and break the Comment’s reference, which would break the Reply’s — the whole thread is bound together.

Proof of Authenticity

Cloudillo provides two complementary layers of proof:

Layer 1: Author Identity (Cryptographic Signatures)

Action tokens are signed with ES384 (ECDSA with P-384 curve and SHA-384 hash):

Action Token = Header.Payload.Signature

Signature = ECDSA_sign(SHA384(Header.Payload), alice_private_key)

Verification:

1. Fetch alice’s public keys from https://cl-o.alice.example.com/api/me (the keys array)
2. Verify signature using public key
3. ✓ Proves Alice created this action

Layer 2: Content Integrity (Content Hashes)

All identifiers are SHA-256 hashes:

action_id = SHA256(entire JWT token)
file_id = SHA256(descriptor string)
variant_id = SHA256(blob bytes)

Verification:

1. Download content
2. Recompute hash
3. Compare with claimed identifier
4. ✓ Proves content hasn’t been tampered with

Verification Example

Scenario: Verify a LIKE action on a POST with image attachment.

verify_like_action(like_id):
    // 1. Verify LIKE action
    like_token = fetch_action_token(like_id)
    verify_signature(like_token, bob_public_key)
    verify like_id == SHA256(like_token)
    ✓ Bob authored this LIKE, content is intact

    // 2. Verify parent POST action (p = parent claim)
    post_id = like_token.p
    post_token = fetch_action_token(post_id)
    verify_signature(post_token, alice_public_key)
    verify post_id == SHA256(post_token)
    ✓ Alice authored this POST, content is intact

    // 3. Verify file attachment (a = attachments claim)
    file_id = post_token.a[0]
    descriptor = fetch_descriptor(file_id)
    verify file_id == SHA256(descriptor)
    ✓ File descriptor is intact

    // 4. Verify each variant blob
    variants = parse_descriptor(descriptor)
    for each variant:
        blob_data = fetch_blob(variant.blob_id)
        verify variant.blob_id == SHA256(blob_data)
        verify blob_data.size == variant.size
    ✓ All image variants are intact

    RESULT: Complete chain verified!

DAG Structure

Cloudillo’s action system forms a Directed Acyclic Graph (DAG) with these properties:

Multiple Roots

Unlike a traditional tree with one root, Cloudillo has multiple independent threads:

Post 1 (a1~abc...)                Post 2 (a1~xyz...)
│                                  │
├─ Comment 1.1 (a1~def...)        ├─ Comment 2.1 (a1~uvw...)
│  │                               │
│  ├─ Reply 1.1.1 (a1~ghi...)     └─ Like 2.1 (a1~rst...)
│  │                                  (parent: a1~uvw...)
│  └─ Reply 1.1.2 (a1~jkl...)
│
├─ Comment 1.2 (a1~mno...)
│
└─ Like 1 (a1~pqr...)
   (parent: a1~abc...)

Each top-level post is a root node. Comments and reactions form child nodes.

Performance Implications

Caching Strategy

Content-addressed data is immutable, enabling aggressive caching:

GET /api/files/f1~Qo2E3G8TJZ...

Response Headers:
  Cache-Control: public, max-age=31536000, immutable
  Content-Type: image/avif

Benefits:

• Browsers cache forever (max-age = 1 year)
• CDN cache forever
• No cache invalidation needed
• Reduces bandwidth for federated instances

Security Considerations

Trust Model

Cloudillo’s merkle tree creates a trustless verification model:

Storage providers don’t need to be trusted: Even if a storage provider is malicious, they cannot:

• Modify content without breaking hashes ✗
• Forge signatures without private keys ✗
• Create false parent references ✗
• Tamper with attachments without detection ✗

Users verify everything cryptographically—no trust required.

Attack Resistance

Known attacks and mitigations:

See Also

• Actions & Action Tokens - How action tokens are created and verified
• File Storage & Processing - How file content-addressing works
• Identity System - Cryptographic signing keys for actions
• Access Control - How access tokens work alongside content-addressing
• System Architecture - Overall system design

Network & Security

Cloudillo’s network layer uses Rustls with the AWS-LC-RS cryptographic provider for TLS, automatic certificate management via ACME, and runs a dual-server architecture.

TLS architecture

Cloudillo uses Rustls for TLS termination, configured with HTTP/2 (preferred) and HTTP/1.1 via ALPN negotiation.

SNI-based certificate resolution

A custom CertResolver serves the correct certificate for each domain using SNI (Server Name Indication). This is essential for multi-tenant hosting where many users share a single server.

The resolver maintains an in-memory RwLock<HashMap<domain, CertifiedKey>> cache, prepopulated on startup from the database. On a TLS handshake:

1. Check in-memory cache (fast path, read lock)
2. On cache miss, load from database on a blocking worker thread
3. Parse PEM certificate and private key, insert into cache
4. Return certificate for TLS handshake

Each tenant gets entries for both its canonical domain (cl-o.{id_tag}) and any custom domain.

Certificate management

Cloudillo uses instant-acme to automatically provision and renew TLS certificates from Let’s Encrypt using the HTTP-01 challenge method.

Dual-server setup

• HTTPS server (primary): Handles all application traffic with TLS via Rustls and the SNI-based certificate resolver
• HTTP server (optional): Serves ACME HTTP-01 challenge responses at /.well-known/acme-challenge/{token} and redirects everything else to HTTPS

Certificate renewal

A scheduled CertRenewalTask checks all certificates periodically. Certificates expiring within 30 days are automatically renewed. The renewal uses exponential backoff (1s initial, 1.5x factor, 90s timeout) for resilience.

Certificates are stored via the AuthAdapter trait, which provides create_cert, read_cert_by_domain (and by tenant or id_tag), list_all_certs, and list_tenants_needing_cert_renewal.

Cryptography

Algorithms

Key management

Profile signing keys use the P-384 elliptic curve (ES384). Keys are identified by date-based key IDs (format: YYMMDD) and stored in PKCS#8 PEM format. Each tenant has its own signing key pair:

• Public key: Published for other instances to verify action tokens
• Private key: Used to sign outgoing action tokens

A key failure cache (default size: 100 entries) prevents repeated fetch attempts for unreachable remote keys during federation.

Security policies

Memory safety

Cloudillo enforces unsafe_code = "forbid" as a workspace-wide lint, along with strict clippy rules (unwrap_used = "deny", expect_used = "deny", panic = "deny").

CORS

API endpoints use a permissive CORS policy (CorsLayer::very_permissive()). This is intentional: Cloudillo apps run in sandboxed iframes served from the /apps/ directory and need cross-origin access to the API.

Request size limits

File upload size is configurable per tenant via the file.max_file_size_mb setting (default: 50 MiB).

See also

• Rate Limiting - Hierarchical rate limiting and proof-of-work
• Federation - Inter-instance communication
• Identity System - Cryptographic identity and keys
• Access Control - JWT validation and authorization
• System Architecture - Overall architecture

Blob Storage

Cloudillo’s blob storage holds immutable binary data (files, images, videos) under content-addressed IDs — every blob is named by the SHA-256 hash of its bytes. This gives automatic deduplication, integrity verification, and permanent cacheability. Uploaded media is additionally split into multiple size/quality variants grouped by a file descriptor.

Content-Addressed Storage

File Identifier Format

Cloudillo uses multiple identifier types in its content-addressing system:

{prefix}{version}~{base64url_hash}

Components:

• {prefix}: Resource type indicator (a, f, b, d)
• {version}: Hash algorithm version (currently 1 = SHA-256)
• ~: Separator
• {base64url_hash}: Base64url-encoded hash (43 characters, no padding)

Identifier Types

Important: d2, is not a content-addressed identifier—it’s the actual encoded descriptor string. The file ID (f1~) is the hash of this descriptor.

Examples

Blob ID:       b1~QoEYeG8TJZ2HTGhVlrtTDBpvBGOp6gfGhq4QmD6Z46w
File ID:       f1~m8Z35EIa3prvb3bhjsVjdg9SG98xd0bkoWomOHQAwCM
Descriptor:    d2,vis.tn:b1~xRAVuQtgBx_kLqZnoOSd5XqCK_aQolhq1XeXk73Zn8U:f=avif:s=1960:r=90x128
Action ID:     a1~8kR3mN9pQ2vL6xWpYzT4BjN5FqGxCmK9RsH2VwLnD8P

All file and blob IDs use SHA-256 content-addressing. See Content-Addressing & Merkle Trees for hash computation details.

File Types

Cloudillo supports four file types, each handled by different adapters based on mutability and use case:

File Type Selection

Files are created via different API endpoints based on their type:

Available presets for BLOB uploads: default, profile-picture, cover, high_quality, mobile, archive, podcast, video, orig-only, thumbnail-only, apkg

Per-File Access Control

Each file has independent access control:

See Access Control for detailed permission handling.

File Variants

Concept

A single uploaded image automatically generates multiple variants optimized for different use cases:

• pf (profile): Profile picture icon (~80px)
• tn (thumbnail): Small preview (~256px)
• sd (standard definition): Mobile/low bandwidth (~720px)
• md (medium definition): Desktop viewing (~1280px)
• hd (high definition): High quality display (~1920px)
• xd (extra definition): 4K/maximum quality (~3840px)

File Descriptor Encoding

A file descriptor encodes all available variants in a compact format.

File Descriptor Format Specification

Format

d2,{class}.{variant}:{blob_id}:f={format}:s={size}:r={width}x{height}[:{optional}];...

Components

• d2, - Descriptor prefix (version 2)
• {class} - Media class:
  • vis - Visual (images: jpeg, png, webp, avif)
  • vid - Video (mp4/h264)
  • aud - Audio (opus, mp3)
  • doc - Documents (pdf)
  • raw - Original unprocessed file
• {variant} - Quality tier: pf, tn, sd, md, hd, or xd
• {blob_id} - Content-addressed ID of the blob (b1~...)
• f={format} - Format: avif, webp, jpeg, png, mp4, opus, pdf
• s={size} - File size in bytes (integer, no separators)
• r={width}x{height} - Resolution in pixels (width × height)
• ; - Semicolon separator between variants (no spaces)

The original is encoded as the bare token orig (no class prefix), regardless of its media class — e.g. orig:b1~...:f=jpeg:.... A descriptor may also begin with an optional R={root_id}; field that links the file to its document-tree access-control root.

Optional Fields

For video, audio, and document files:

• dur={seconds} - Duration in seconds (floating point, video/audio only)
• br={kbps} - Bitrate in kbps (integer, video/audio only)
• pg={count} - Page count (integer, documents only)

Example

d2,vis.tn:b1~abc123:f=avif:s=4096:r=150x150;vis.sd:b1~def456:f=avif:s=32768:r=640x480

This descriptor encodes two variants:

• Thumbnail: AVIF format, 4096 bytes, 150×150 pixels, blob ID b1~abc123
• Standard: AVIF format, 32768 bytes, 640×480 pixels, blob ID b1~def456

Video Example

d2,vis.tn:b1~abc:f=avif:s=4096:r=150x84;vid.sd:b1~def:f=mp4:s=5242880:r=720x404:dur=120.5:br=350;vid.hd:b1~ghi:f=mp4:s=20971520:r=1920x1080:dur=120.5:br=1400

This descriptor includes:

• Thumbnail: AVIF image preview
• SD Video: 720p MP4, 120.5 seconds, 350 kbps
• HD Video: 1080p MP4, 120.5 seconds, 1400 kbps

Parsing Rules

1. Check prefix: Verify descriptor starts with d2,
2. Split by semicolon (;): Get individual variant entries
3. For each variant, split by colon (:) to get components:
   • Component [0] = class.variant (vis.tn, vis.sd, vid.hd)
   • Component [1] = blob_id (b1~...)
   • Components [2..] = key=value pairs
4. Parse key=value pairs:
   • f={format} → Format string
   • s={size} → Parse as u64 (bytes)
   • r={width}x{height} → Split by x, parse as u32 × u32
   • dur={seconds} → Parse as f64 (optional)
   • br={kbps} → Parse as u32 (optional)
   • pg={count} → Parse as u32 (optional)

Parsing logic: split by semicolons for variants, then by colons for fields, then parse key=value pairs.

Variant Size Classes - Exact Specifications

Cloudillo generates image variants at specific size targets to optimize bandwidth and storage:

Generation Rules

Which variants are generated depends on the preset configuration. The default preset generates: tn, sd, md, hd. The high_quality preset adds xd. Variants larger than the original image are automatically skipped (smaller originals are never upscaled).

Properties:

• Each variant maintains the original aspect ratio
• Uses Lanczos3 filter for high-quality downscaling
• Maximum dimension constraint prevents oversizing
• Smaller originals don’t get upscaled

Variant Selection

Clients request a specific variant:

GET /api/files/f1~Qo2E3G8TJZ...?variant=hd

Response: Returns HD variant if available, otherwise falls back to smaller variants.

Automatic Fallback

If the requested variant doesn’t exist, the server returns the best available:

1. Try requested variant (e.g., hd)
2. Fall back to next smaller (e.g., md)
3. Continue until variant found
4. Return smallest if none larger

Fallback order: xd → hd → md → sd → tn

Content-Addressing Flow

File storage uses a three-level content-addressing hierarchy:

Level 1: Blob Storage

Upload image → Save as blob → Compute SHA256 of blob bytes → Store blob with ID: b1~{hash}

blob_data = read_file("thumbnail.avif")
blob_id = compute_hash("b", blob_data)
// Result: "b1~abc123..." (thumbnail blob ID)

Example: b1~abc123... identifies the thumbnail AVIF blob

See Content-Addressing & Merkle Trees for hash computation details.

Level 2: Variant Collection

Generate all variants (tn, sd, md, hd) → Each variant gets its own blob ID (b1~...) → Collect all variant metadata → Create descriptor string encoding all variants

variants = [
    { class: "vis.tn", blob_id: "b1~abc123", format: "avif", size: 4096, width: 150, height: 150 },
    { class: "vis.sd", blob_id: "b1~def456", format: "avif", size: 32768, width: 640, height: 480 },
    { class: "vis.md", blob_id: "b1~ghi789", format: "avif", size: 262144, width: 1920, height: 1080 },
]

descriptor = build_descriptor(variants)
// Result: "d2,vis.tn:b1~abc123:f=avif:s=4096:r=150x150;vis.sd:b1~def456:f=avif:s=32768:r=640x480;vis.md:b1~ghi789:f=avif:s=262144:r=1920x1080"

Level 3: File Descriptor

Build descriptor → Compute SHA256 of descriptor string → Final file ID: f1~{hash} → This file ID goes into action attachments

descriptor = "d2,vis.tn:b1~abc:f=avif:s=4096:r=150x150;vis.sd:b1~def:f=avif:s=32768:r=640x480"
file_id = compute_hash("f", descriptor.as_bytes())
// Result: "f1~Qo2E3G8TJZ..." (file ID)

Example Complete Flow

1. User uploads photo.jpg (3MB, 3024x4032px)

2. System generates variants:
   vis.tn:  150x200px → 4KB   → b1~abc123
   vis.sd:  600x800px → 32KB  → b1~def456
   vis.md:  1440x1920px → 256KB → b1~ghi789
   vis.hd:  2880x3840px → 1MB → b1~jkl012

3. System builds descriptor:
   "d2,vis.tn:b1~abc123:f=avif:s=4096:r=150x200;
       vis.sd:b1~def456:f=avif:s=32768:r=600x800;
       vis.md:b1~ghi789:f=avif:s=262144:r=1440x1920;
       vis.hd:b1~jkl012:f=avif:s=1048576:r=2880x3840"

4. System hashes descriptor:
   file_id = f1~Qo2E3G8TJZ2... = SHA256(descriptor)

5. Action references file:
   POST action attachments = ["f1~Qo2E3G8TJZ2..."]

6. Anyone can verify:
   - Download all variants
   - Verify each blob_id = SHA256(blob)
   - Rebuild descriptor
   - Verify file_id = SHA256(descriptor)
   - Cryptographic proof established ✓

File attachments integrate into Cloudillo’s merkle tree structure. See Content-Addressing & Merkle Trees for how files fit into the verification chain.

Image Processing Pipeline

Upload Flow

When a client uploads an image:

1. Client Request

POST /api/files/default/profile-picture.jpg
Authorization: Bearer <access_token>
Content-Type: image/jpeg
Content-Length: 2458624

<binary image data>

2. Dimension Extraction & Variant Selection

The image dimensions are extracted and the preset’s image variant list (e.g. ["vis.tn", "vis.sd", "vis.md", "vis.hd"] for default) is walked from smallest to largest. Each variant’s bounding box is capped at the original’s longest side — the original is never upscaled. A variant is then skipped if its capped size is less than 10% larger than the last variant actually created, so a small original collapses to just the thumbnail plus one or two distinct sizes instead of several near-identical blobs.

The intermediate steps (task scheduling, hash computation, blob storage, variant generation, and metadata storage) are shown in the Complete Upload Flow Diagram below.

3. Response

The upload responds immediately with a temporary local ID (@{f_id}) plus the synchronously-generated thumbnail blob ID and original dimensions. The remaining variants are still being generated asynchronously:

{
  "fileId": "@1234",
  "thumbnailVariantId": "b1~QoE...46w",
  "dim": [3024, 4032]
}

The final content-addressed file ID (f1~...) is only known once all variant tasks finish. The server then pushes a FILE_ID_GENERATED WebSocket event ( { tempId, fileId, rootId } ) so clients can swap the temporary @{f_id} for the permanent f1~ ID.

Complete Upload Flow Diagram

Client uploads image
  ↓
POST /api/files/{preset}/filename.jpg
  ↓
Read image, extract dimensions, allocate local f_id
  ↓
Generate thumbnail synchronously
  ├─ Resize with Lanczos3 → encode → SHA256 → b1~ blob
  ├─ Store blob in BlobAdapter
  └─ Record file_variants row
  ↓
Respond immediately: { fileId: "@<f_id>", thumbnailVariantId, dim }
  ↓
Schedule image.resize task per remaining variant
  ├─ Resize / encode / hash / store blob
  └─ Record file_variants row
  ↓
file.id-generate task (depends on all variant tasks)
  ├─ Collect variant rows → build d2 descriptor
  ├─ file_id = SHA256(descriptor) → f1~...
  ├─ Finalize file (status P → A)
  └─ Broadcast FILE_ID_GENERATED over WebSocket

Download Flow

Client Request

GET /api/files/f1~...?variant=hd
Authorization: Bearer <access_token>

Server Processing

1. Parse Descriptor

variants = parse_file_descriptor(file_id)
# Returns list of VariantInfo

2. Select Best Variant

selected = get_best_file_variant(
    variants,
    requested_variant,   # "hd"
)

# Falls back by quality within the same class if the
# requested variant isn't locally available:
# hd → md → sd → tn

3. Stream from BlobAdapter

stream = blob_adapter.read_blob_stream(tn_id, selected.variant_id)

# Set response headers
response.headers["Content-Type"] = content_type_from_format(selected.format)
response.headers["X-Cloudillo-Variant"] = selected.variant_id
response.headers["X-Cloudillo-Variants"] = descriptor
response.headers["Content-Length"] = selected.size

# Stream response
return stream_response(stream)

Response

HTTP/1.1 200 OK
Content-Type: image/avif
Content-Length: 16384
X-Cloudillo-Variant: b1~m8Z35EIa3prvb3bhjsVjdg9SG98xd0bkoWomOHQAwCM
X-Cloudillo-Variants: d2,vis.tn:b1~xRAVuQtgBx_kLqZnoOSd5XqCK_aQolhq1XeXk73Zn8U:f=avif:s=1960:r=90x128;vis.sd:b1~m8Z35EIa3prvb3bhjsVjdg9SG98xd0bkoWomOHQAwCM:f=avif:s=8137:r=256x364;orig:b1~5gU72rRGiaogZuYhJy853pBd6PsqjPOjS__Kim9-qE0:f=avif:s=15012:r=256x364
Cache-Control: public, max-age=31536000, immutable

<binary image data>

Note: Content-addressed files are immutable, so can be cached forever.

Metadata Structure

File metadata lives in two tables in the MetaAdapter. The files row holds the logical file (type, preset, owner, visibility, folder/document-tree links); file_variants rows hold one entry per generated variant. A local internal f_id integer keys both tables while the file is still being processed; the content-addressed file_id (f1~...) is written only once all variant tasks finish (status transitions P → A).

CREATE TABLE files (
    f_id INTEGER NOT NULL,          -- local internal id (primary key)
    tn_id INTEGER NOT NULL,
    file_id TEXT,                   -- f1~... descriptor hash (set when finalized)
    file_tp CHAR(4),                -- BLOB / CRDT / RTDB
    status CHAR(1),                 -- A active, P pending, D deleted
    preset TEXT,
    content_type TEXT,
    file_name TEXT,
    visibility CHAR(1),             -- NULL / P / V / 2 / F / C
    parent_id TEXT,                 -- folder hierarchy
    root_id TEXT,                   -- document-tree access-control root
    created_at INTEGER,
    PRIMARY KEY(f_id)
);

CREATE TABLE file_variants (
    tn_id INTEGER NOT NULL,
    f_id INTEGER NOT NULL,
    variant_id TEXT,                -- b1~... blob id
    variant TEXT,                   -- 'vis.sd', 'vid.hd', 'orig', ...
    res_x INTEGER,
    res_y INTEGER,
    format TEXT,
    size INTEGER,
    available BOOLEAN,              -- blob present locally
    global BOOLEAN,                 -- stored in shared global cache
    duration REAL,                  -- video/audio
    bitrate INTEGER,                -- video/audio (kbps)
    page_count INTEGER,             -- documents
    PRIMARY KEY(f_id, variant_id, tn_id)
);

The available flag matters for federation: a synced file lists all variants in its descriptor, but only the variants whose blobs have actually been fetched are marked available locally. See Access Control for how visibility and root_id drive permission checks.

File Presets

Presets control which variants are generated and whether the original is stored. Files are uploaded with a preset in the path:

POST /api/files/{preset}/{filename}

POST /api/files/default/avatar.jpg       // standard image variants
POST /api/files/archive/document.pdf     // keep original, minimal processing

Available presets: default, profile-picture, cover, high_quality, mobile, archive, podcast, video, orig-only, thumbnail-only, apkg. See File Processing for the full per-preset variant matrix.

Storage Organization

BlobAdapter Layout

Blobs are stored on disk under a per-tenant directory, sharded into two levels by the first four characters of the hash (after the ~). The filename is the full blob ID:

{data_dir}/
├── {tn_id}/
│   ├── {h0h1}/                      // first 2 hash chars
│   │   └── {h2h3}/                  // next 2 hash chars
│   │       └── b1~QoEYeG8TJ...46w   // blob, filename = full ID
│   └── ...
└── {other_tn_id}/
    └── ...

Each variant (and the original, when stored) is an independent blob with its own ID. The file descriptor — not the filesystem — is what groups variants into a logical file. File metadata is stored separately in the MetaAdapter (see Metadata Structure above).

See Also

• System Architecture - Task system and worker pool
• Actions - File attachments in action tokens
• Access Control - File permission checking

RTDB (Real-Time Database)

Query-oriented database system providing Firebase-like functionality for structured JSON data with real-time subscriptions.

Overview

The RTDB system enables:

• JSON document storage and retrieval
• Query filters (equals, greater than, less than)
• Sorting and pagination
• Computed values (increment, aggregate, functions)
• Atomic transactions
• Real-time subscriptions via WebSocket

Documents

• Overview - Introduction to RTDB architecture and features
• redb Implementation - How RTDB is implemented using the lightweight redb embedded database

Use Cases

• User profiles and settings
• Task lists and project management
• E-commerce catalogs
• Analytics and reporting
• Structured forms and surveys

CRDT (Collaborative Editing)

Conflict-free replicated data types enabling true collaborative editing with automatic conflict resolution.

Overview

The CRDT system provides:

• Conflict-free replicated data types (CRDTs)
• Rich data structures (Text, Map, Array, XML)
• Automatic conflict resolution
• Real-time synchronization
• Offline editing support
• Yrs/Yjs ecosystem compatibility

Documents

• Overview - Introduction to CRDTs and Yrs implementation
• redb Implementation - How CRDT updates are stored persistently using redb