The parts
Collabase plugs run in the application. Plugs from the catalog and your own plugs run in the sandbox. Without the sandbox, flows that use only Collabase plugs keep working.
How a run is carried out
1
A trigger puts the run in the queue
The run exists in the database before it is queued, with its trigger data kept in storage.
2
A worker takes it
The worker takes a lease on the run and renews it every 10 seconds while a step runs. A workspace may have 10 runs going at once by default. A run that finds its workspace full goes back to the queue after two seconds.
3
Steps run one after another
After every step the worker writes the result to the step journal. A step may run for up to 10 minutes.
4
The run finishes or pauses
A run ends as succeeded, failed, cancelled or timed out. A run that waits for a delay, a call or an approval is paused and holds no worker.
After a crash
The step journal makes a run survive a restart of the application, of Redis or of the host.- A worker that stops without releasing its run loses the lease. Within about 45 seconds another worker takes the run over.
- The new worker reads the journal and skips every step that already finished. It repeats only the step that was in progress.
- A sweep every minute puts runs that have had no write for two minutes back in the queue, so a run is not lost when Redis loses its queue.
- If the database or the sandbox is unavailable, the run waits and tries again up to ten times, five seconds apart, and then the sweep picks it up.
Concurrency
The limit for one workspace is 10 runs at once by default, and can be set between 1 and 200. It keeps one busy workspace from using all workers. The limit is a counter in Redis, so it holds across all replicas. A slot that a lost worker never gave back frees itself after 30 minutes.Retention
A daily sweep deletes finished runs that are older than the retention of their workspace, together with their step logs and files. The retention is 30 days by default and can be set between 1 and 365 days in the workspace settings. Runs that are going are never removed. Back up the database and the file storage together, because the step logs and large results are in storage.Settings the services need
The installers create the secrets for a new installation, and the updater adds them to an existing one. Set them by hand only for an installation that you assemble yourself. If one is missing, plugs from the catalog and your own plugs are not available, and the application says so. Collabase plugs keep working.
What to watch
A flow that fails ten times in a row is switched off, and the editors and admins of its workspace are told.