Architecture Overview
canaryRoutingMiddleware runs as a Fastify preHandler hook on every request. If the resolved platformId is in the canary list, the request is proxied in-process to the canary app and the response is returned directly to the caller — the primary app never processes it further.
Canary membership: DB-backed with Redis cache
Canary platform membership is stored in theplatform_plan.canary boolean column. On each lookup, the list of canary platform IDs is fetched from Redis (canary-platform-ids key). On cache miss the list is read from the database and cached. When the canary flag is changed via the API, the cache is invalidated immediately.
This removes the need to keep AP_CANARY_PLATFORM_IDS in sync across services and allows runtime changes without a redeploy.
Components
Canary App
A second instance of the server API running a different image tag. No special env vars are needed for canary membership — it is driven entirely by the DB. Configure it with:
The canary app runs as a full app instance. Global queue consumers (
runsMetadata, system-job-queue) run on both the primary and canary app — this is safe because BullMQ ensures each job is processed by exactly one consumer.
Canary Workers
Dedicated workers that connect to the canary app instead of the primary app:
Workers use
AP_FRONTEND_URL for their Socket.IO RPC channel and for posting engine results back to the app. Canary workers are registered with the canary app, so the RPC path is fully isolated.
Primary App Config
The primary app only needs to know where to proxy canary requests:WebSocket / Real-time Updates
Socket.IO is configured with a Redis adapter (@socket.io/redis-adapter). Events emitted on any app instance (primary or canary) are broadcast through Redis pub/sub to all connected instances. This means:
- Users connected to the primary app receive real-time flow run updates even when the execution happened on the canary app.
- No WebSocket proxying is required.
Queue Isolation
Jobs for canary platforms are enqueued to
canaryWorkerJobs, which only canary workers poll — fully isolated from the primary worker fleet.
Deploying a New Canary Build
The canary has no workflow of its own. Thedeploy job of the Cloud gate (continuous-delivery-cloud.yml) deploys it, after the approval and a few minutes before the rest of production. There is no daily schedule and no manual dispatch: the canary shares the production database and runs migrations at boot, so a daily build of main ran newer migrations under the older fleet.
The deploy job:
- Checks if any migration that the canary does not have yet is breaking. It fails if one is, unless the Cloud run sets
skip_migration_check. - Deploys the candidate image to the canary app (
config/app-canary.yml) and canary workers. - Deploys the same image to the fleet.
promote-production lock from step 1 to step 3, so no other gate run or rollback can run between the canary and the fleet.
Required GitHub secrets: AP_API_KEY (the primary app’s AP_API_KEY value).
Rolling Back a Canary Deployment
The next Cloud deploy sets the canary image. The Cloud rollback workflow does not touch the canary hosts, so after a Cloud rollback the canary keeps the newer image until the next Cloud deploy. There is no separate canary rollback, because running migrations down on the shared database can drop data the newer code wrote.Promoting canary build to production
Canary is not a promotion path. The Cloud gate (continuous-delivery-cloud.yml) deploys each approved candidate to the canary and then to the fleet in the same run.
Managing Platform Routing
Enable canary routing for a platform
Disable canary routing for a platform
platform_plan.canary in the database and invalidate the Redis cache (canary-platform-ids). The change takes effect on the next request — no restart required.