Background processing – what runs without a user¶
Some work in nexxcite does not run in the browser of a user but in the background on the server: actions started by business rules after a save, scheduled actions, mailbox listeners and actions started through the REST API. This chapter explains from the administrator's point of view what runs in the background, how the job queue behaves, what happens on errors and where you can follow every run.
It is intended for administrators and operators of a nexxcite installation. The pages of the Admin Studio for this are described in detail in Monitoring.
1. Background processing at a glance¶
| What runs in the background | Started by | Runs as | Runs on | Configured in |
|---|---|---|---|---|
| Background action of an event-based business rule | Saving a record that matches the rule | The user of the rule, or – with "Execute in user context" – the user who saved | The server where the record was saved | Business rules |
| Process of an event based workflow rule | Saving a record that matches the rule; continued when its wait time has passed or its event occurs | The "Identity" of the rule | The server that saved the record; elapsed wait times on the servers with ProcessScheduledActions = true |
Event based workflow rule |
| Background action through the REST API | A call with waitForCompletion = false |
The API user | The server that received the call | Run an action |
| Scheduled action | Its schedule | Its "Identity" | The server with ProcessScheduledActions = true |
Scheduled actions |
| Mailbox listener | Polling about every 20 seconds | Its "Identity" | The server with ProcessMailboxListeners = true |
Mailbox listeners |
Summary¶
- Background work runs on the server, without a browser and without a user interface. Commands that need one – dialogs, forms, navigation, messages, input prompts – cannot work there.
- It runs with the permissions of a user (the "runs as" column): that user's roles decide which records the action may read and change.
- Every server has its own job queue in memory. Jobs waiting or running on a server are lost when the server stops.
- Every run is recorded in the execution history, on every server, in the database: Admin Studio → Monitoring → Background Processes.
2. Background actions of business rules¶
An "Event Based Background Action Rule" starts an action in the background when a record changes in a certain way. In short:
- While a user changes a record, nexxcite notes the rules whose triggers apply.
- After the record was saved, each of these rules is checked once more: it is skipped if it is switched off, has no action, or its condition is no longer true.
- For each remaining rule, one job is put into the job queue of the server – once per rule and save, with the details of the change.
- The job runs as soon as a worker of the queue is free – or after "Execution Delay (Seconds)", if the rule sets one.
- The action runs against the saved record, as the user of the rule.
All background actions started by one save share a common cause; the history shows them together under "Started by the same cause".
| Setting of the rule | Effect in the background |
|---|---|
| "User" | The user the action runs as |
| "Execute in user context" | Runs as the user who saved the record instead |
| "Execution Delay (Seconds)" | Waits this many seconds after the save. Empty, zero or a negative value: no delay. |
| "Execution Limit" | OnlyOnCreation / OnlyOnUpdate restrict the rule to new or to existing records. LimitedNumberOfTimes is not evaluated and behaves like None. |
| "Max Executions" | How often the rule may run for one record in total. 0: unlimited. A start beyond the limit is recorded as "Skipped" with the reason. |
| "Store Execution Log" | Whether the log of the run is kept (section 5) |
The user who saved the record does not wait for the background action and does not see its result. An error of the action does not undo the save. The rules themselves are described in Business rules.
A delayed job lives in the memory of the server, too. A job with a delay of an hour is lost if the server restarts within that hour. For work that has to happen at a later point in time reliably, use an event based workflow rule.
Processes of event based workflow rules¶
A "Event Based Workflow Rule" does not queue a job; it starts a process for the record, which waits for a point in time the record stores or for a further change, runs actions and can repeat. The state of the process is kept in the database, not in the job queue:
- a wait of days or weeks survives restarts of every server,
- the servers that process scheduled actions look about every 10 seconds for processes whose wait time has passed and continue them,
- a matching change of the record continues a waiting process on the server that saved it,
- the actions a process runs appear in the execution history with the origin "Event based workflow rule".
The processes themselves, with every decision they took, are shown under Monitoring → Background Processes → "Workflow Rules". The rule is described in Event based workflow rule.
3. The job queue¶
Every server has a job queue with a number of workers that take jobs one after the other. The queue holds:
- background actions (rules, REST API),
- the scheduled actions and mailbox listeners registered on this server, as recurring jobs,
- a small test job that every server puts into its own queue about every 5 minutes, to see that the
queue still executes jobs (
ExecutionHistory:JobQueueProbeIntervalMinutes).
The queue is kept in memory¶
The queue is not stored in the database. That has consequences:
| Situation | Effect |
|---|---|
| A server stops or fails | Jobs that were waiting or running on it are lost. The history shows them as "Abandoned". |
| A server starts | It begins with an empty queue. Scheduled actions and mailbox listeners are registered again. |
| Several servers | Each server has its own queue. A background action runs on the server where it was started and does not move to another one. |
Abandoned jobs are not repeated automatically. Check Monitoring → Background Processes with the state filter "Problems (failed, abandoned)" after a restart, and start what is needed again: "Requeue" for a background action, "Run now" for a scheduled action.
Retries¶
| Kind | Repeated automatically? |
|---|---|
| Background action that ends with an unexpected error (the action stops with an exception) | Yes, by the job queue, several times with growing intervals. In our reading of the current configuration, up to 10 attempts. |
| Background action whose protocol reports an error, but which ran to its end | No |
| Scheduled action | No – the next run is the next scheduled time. Use "Failure Action" to react to errors. |
| Mailbox listener | Not as a retry – a mail that stays unread is fetched again in the next round, about 20 seconds later. |
Because an action may run more than once, actions that run in the background should be safe to repeat – for example, check whether the record they would create already exists.
4. Execution history¶
nexxcite writes every run of background processing into the database – on every server, including servers no user can reach. You see it under Admin Studio → Monitoring → Background Processes:
| Kind | Recorded |
|---|---|
| "Background action" | Every background action – of a business rule or the REST API – when queued, when started and with its result |
| "Scheduled action" | Every run, with the protocol of the failure action |
| "Mailbox listener" | Every round that handled at least one mail or failed, with one line per mail |
| "Traced action" | Actions that users start, if the action has "Trace Executions" switched on |
A run passes through the states "Queued" or "Scheduled", "Running" and ends as "Succeeded", "Succeeded with warnings", "Failed", "Skipped" or "Abandoned".
The details of a run show the protocol of the action step by step, the arguments, the error, the log and the runs "Started by the same cause". From there, "Requeue" queues a background action again with its original arguments, and "Run now" / "Poll now" start a scheduled action or a mailbox listener again. How the pages work: Monitoring.
Admin Studio → Monitoring → Background Jobs shows the job queue itself – but only that of the server serving your browser tab, and only since its last start. The history under Background Processes is the complete, permanent record.
Tracing user actions¶
The switch "Trace Executions" in the action editor records every run of the action that a user starts – with the user, the record, the protocol and, depending on "Store Execution Log", the log. Use it for a limited time to find the cause of an error that users report; the history grows with every click.
5. Logs of a run¶
Besides the protocol of the action, nexxcite can keep the application log that was written during a run. Whether it is kept is decided in this order – the first level that decides wins:
- The definition that started the run: "Store Execution Log" of the scheduled action, the business rule or the mailbox listener
- "Store Execution Log" of the action
ExecutionHistory:LogCaptureModein the configuration (default:OnFailure)
| Choice in the editor | Configuration value | The log is kept |
|---|---|---|
| "Inherit (use the next level)" | – | as the next level decides |
| "Never" | Never |
never |
| "On failure or warnings" | OnFailure |
for failed runs, runs with warnings and abandoned runs |
| "Always" | Always |
for every run |
Example: a scheduled action has "Inherit", its action has "Always", the configuration says
OnFailure. The log of every run of this scheduled action is kept, because the action decides.
Use "Always" for a limited time while you look for an error – logs take up space in the database.
6. Settings for operation¶
The following values are set in the configuration of the application (appsettings.json or
environment variables). The complete list is in
Installation and configuration.
| Setting | Default | Meaning |
|---|---|---|
ApplicationServer:ProcessScheduledActions |
true |
This server runs scheduled actions |
ApplicationServer:ProcessMailboxListeners |
true |
This server polls the mailboxes of mailbox listeners |
ExecutionHistory:Enabled |
true |
Records the history. Off, nothing new is recorded. |
ExecutionHistory:LogCaptureMode |
OnFailure |
Default for keeping logs: Never, OnFailure, Always |
ExecutionHistory:MaxLogEntries |
2000 | Lines of the log kept per run |
ExecutionHistory:LogCaptureMinLevel |
Debug |
Lowest level of the kept log |
ExecutionHistory:MaxProtocolNodes |
2000 | Size limit of the stored protocol of a run |
ExecutionHistory:JobQueueProbeIntervalMinutes |
5 | Interval of the test job in the job queue. 0 switches it off. |
ExecutionHistory:HousekeepingIntervalMinutes |
5 | Interval of the clean-up: abandoned jobs, retention |
RuleProcesses:PollIntervalSeconds |
10 | How often the servers that process scheduled actions continue processes of event based workflow rules whose wait time has passed |
How long runs and logs (30 and 14 days by default) and finished workflow processes (kept for good by default) are kept is set in the global settings, once for the whole installation.
What runs in the background, and when, is configured in the definitions – not here. With several servers, exactly one server may process scheduled actions and mailbox listeners; see Running several servers.
7. Limitations¶
- Jobs are kept in memory. Waiting, delayed and running jobs of a server are lost when it stops, and they are not repeated automatically.
- No load sharing for background actions: a background action runs on the server where it was started.
- "Max Executions" counts the starts of a rule per record from the moment this version runs; what ran before the update is not counted.
- An action command that runs a part of an action in the background at a later time is not available in the current version. Use an event based workflow rule, a background rule with "Execution Delay (Seconds)" or a scheduled action.
- Monitoring → Background Jobs shows only the queue of one server; its counters start again with every restart.
8. Frequently asked questions¶
A user saved a record, but the background action did not run. Where do I look? In Monitoring → Background Processes → "Executions", filtered by kind "Background action". If there is no entry, the rule did not fire: its trigger did not apply, it is switched off, its condition was false after the save, or it has no user. If the entry is "Abandoned", the server stopped before the job ran.
Can I see which user a background action ran as? Yes. The details of a run show "Triggered by" – the user who started it – and "Runs as", when the action ran as a different user.
Does background processing continue while nobody is signed in? Yes. Scheduled actions and mailbox listeners run independently of users and also when the license has expired. Background actions of rules only arise when someone saves a record.
We restart the servers every night. Do we lose anything? Jobs that wait or run at the moment of the restart are lost – including background actions with a delay that has not passed yet. Scheduled actions whose time falls into the restart are not made up. Plan restarts outside the times of your scheduled actions, and check for "Abandoned" runs afterwards.