Running several servers – load balancing and Kubernetes¶

This chapter describes how to run several nexxcite servers side by side with one database – behind a load balancer or on Kubernetes. It explains what the servers share, what stays with each server, which server runs the background processing and what happens when a server stops or fails.

It is intended for operators. The basic installation is described in Installation and configuration.

1. Several servers at a glance¶

Summary¶

All servers work with the same database. Definitions, records, licenses and the history are the same for everybody.
Every open browser tab is connected to exactly one server and stays with it. The state of the tab lives in the memory of that server. The load balancer therefore needs sticky sessions and WebSockets.
Exactly one server runs the scheduled actions and polls the mailboxes of the mailbox listeners. All others serve users only.
A shared Redis keeps the definition cache of all servers in step and carries the messages between them.

What is shared and what is not¶

Shared by all servers	Kept per server
Database: definitions, records, licenses, execution history, server states	The connections of the browser tabs and everything they show, including unsaved input
Keys for sign-in cookies (shared directory)	The queue of background jobs (background actions, scheduled actions, mailbox polls)
Definition cache and message bus (Redis)	Global user values
The license – every license is handed out only once	Collaborative editing of a sketch in several tabs

A typical setup¶

                  Browser
                     │  HTTPS
          ┌──────────▼───────────┐
          │ Load balancer        │  sticky sessions, WebSockets
          └───┬────────┬─────────┘
              │        │
        ┌─────▼──┐ ┌───▼────┐        ┌───────────────────┐
        │ web-1  │ │ web-2  │  ...   │ worker            │  scheduled actions,
        └───┬────┘ └───┬────┘        │ (no user traffic) │  mailbox listeners
            │          │             └─────────┬─────────┘
            └──────────┴──────────┬────────────┘
                     SQL Server · Redis · shared key directory

The servers are started from the same image. They differ only in their configuration.

2. Setting up¶

Settings that are the same on every server¶

Setting	Value	Why
`ApplicationServer:MultiServerMode`	`true`	Makes every server treat only its own sessions as its own (see section 4)
`DataAccess:ConnectionString`	the same database	–
`DataProtection:KeyRingPath`	a directory all servers share	Otherwise a sign-in on one server is not accepted by another. With `MultiServerMode`, nexxcite does not start without it.
`DataProtection:ApplicationName`	the same value (default `nexxcite`)	–
`DefinitionSecrets:Enabled`, `DefinitionSecrets:Key` and `DefinitionSecrets:PreviousKeys`	the same values	A server with another key cannot read the passwords and API keys the others stored (see Secrets)
`DefinitionCache:Provider` and `DefinitionCache:RedisConnectionString`	`Redis` and its address	A change reaches every server immediately
`Messaging:Provider` and `Messaging:ConnectionString`	`Redis` and its address	Servers inform each other about changes and see each other's state
`Messaging:SubjectPrefix`	the same value	Servers with different prefixes do not see each other
`ApiAuthentication:SigningKey`	the same key	Otherwise an access token of one server is rejected by another
`ForwardedHeaders:*`	according to your proxies	See Installation and configuration

The shared directory for the keys must be writable by all servers at the same time. On Kubernetes, that is a volume with the access mode ReadWriteMany (for example NFS or Azure Files).

Settings that differ per server¶

Setting	Web servers	The one worker
`ApplicationServer:ProcessScheduledActions`	`false`	`true`
`ApplicationServer:ProcessMailboxListeners`	`false`	`true`
`ApplicationServer:InstanceId`	unique per server	unique, and stable across restarts

Both switches are true by default. Set them to false explicitly on every web server. A web server whose setting is missing would run every scheduled action a second time and fetch the same mails as the worker.

The InstanceId names the server in monitoring and in the history. If it is not set, every start creates a new one. On Kubernetes, the name of the pod is a good choice. A stable id for the worker has an advantage: after a restart, the worker immediately marks the jobs it left unfinished as abandoned (see section 5).

Kubernetes¶

The worker must never run twice, not even for a moment during an update. Run it as a StatefulSet with one replica or as a Deployment with the strategy Recreate. The default rolling update starts the new pod before it stops the old one – during that time, scheduled actions would run twice.
Probes: use /health/startup, /health/live and /health/ready as described in Installation and configuration. Allow the startup probe enough time for the schema migration (up to 10 minutes per server).
Graceful shutdown: give the pods time to end their requests and clear their sessions, for example a termination grace period of a few minutes and a short preStop delay so that the load balancer stops sending new connections before the server stops.
If you replace the entry point of the image with your own script, start nexxcite with exec as the last command. Otherwise the stop signal reaches the script and not nexxcite, and the server cannot shut down cleanly.

The names and effects of these Kubernetes settings depend on your cluster and its ingress controller; their documentation is authoritative.

3. Load balancer: sticky sessions and WebSockets¶

Every browser tab keeps a permanent connection to one server. The state of the tab – the open form, the records shown, unsaved input – exists only in the memory of that server.

Requirement	Why
WebSockets passed through	The permanent connection uses WebSockets.
Sticky sessions (session affinity, usually by cookie)	After a short interruption, the browser reconnects. It has to reach the same server – only that server knows the tab.
Long idle timeouts	The connection stays open while the user works. Browser and server exchange a sign of life about every 15 seconds.

Example: A user works on web-1. The Wi-Fi drops for a minute. With sticky sessions, the browser reconnects to web-1 and the user continues without loss. Without them, the browser may reach web-2, which does not know the tab: the page reloads and unsaved input is lost.

Monitoring → Servers shows the server your own tab is connected to ("your connection").

4. Behaviour in detail¶

Changes to definitions¶

Configuration	A change saved on one server reaches the others
`DefinitionCache:Provider = Redis` and `Messaging:Provider = Redis`	immediately
without Redis	when the cached entry expires, after up to 30 minutes (`DefinitionCache:TimeToLiveSeconds`)

The same applies to users and roles: a deactivated user or a changed role takes effect on all servers at once with Redis.

If Redis cannot be reached, nexxcite keeps working: definitions are read from the database (slower), and messages between the servers are lost until Redis is back. Changes may then reach the other servers late.

Licenses¶

The license applies to the whole installation, not per server. A user who works on two servers takes one license. See Licensing.

Sessions at startup¶

When a server starts, it removes the leftover session records of its earlier run. With MultiServerMode = true, it removes only its own. Without it, a starting server removes the sessions of all servers – the license usage of users on other servers is then counted wrongly. This is why MultiServerMode is required as soon as there is more than one server.

Scheduled actions and mailbox listeners¶

Only servers with ProcessScheduledActions = true run scheduled actions; only servers with ProcessMailboxListeners = true poll mailboxes. Mailbox listeners poll about every 20 seconds.
The same servers continue the processes of event based workflow rules whose wait time has passed, about every 10 seconds. Several such servers are safe: each process is continued by exactly one of them. A change of the record continues a waiting process on the server that saved it.
Important: the schedule of a scheduled action and the polling of a mailbox listener are set up when the worker starts. A scheduled action or mailbox listener that is created or whose schedule is changed on a web server reaches the worker only when the worker is restarted. Changes to the content of an existing one (for example the action it runs) take effect without a restart. A scheduled action that is switched off stops running at once.
A scheduled action or mailbox listener that is deleted stops on the worker as soon as the deletion is published – the worker learns about it through the message bus. Without a message bus (Messaging:Provider = None), a deleted scheduled action is reported as a failed run at every scheduled time, and a deleted mailbox listener keeps being polled, until the worker restarts.
A mailbox listener that is switched off keeps being polled by the worker until the worker restarts. In the current version, this also applies to a single server.
Monitoring → Background Processes → Calendar lists every scheduled action with the server it is registered on and warns when an enabled schedule is registered on no server or when no server processes scheduled actions at all.

Background actions¶

Background actions (started by business rules or through the REST API) run on the server where they were started, in its job queue. They do not move to another server.

"Run now", "Poll now" and the developer console¶

These functions reach other servers through the message bus:

"Run now" of a scheduled action is carried out by the worker, "Poll now" of a mailbox listener as well – from whichever server you click it on.
The developer console lists the sessions of all servers.
A deleted scheduled action or mailbox listener is removed from the worker (see above).
A new, changed or deleted authentication provider (single sign-on) is offered on all servers as soon as it is published. See Authentication providers.

Without a message bus (Messaging:Provider = None), they only work for the server that serves your tab.

5. When a server stops or fails¶

Situation	Users on that server	Licenses	Background jobs of that server	Shown as
Orderly stop (scale down, update, restart)	The connection ends. The browser connects to another server and reloads the page. Unsaved input is lost.	released immediately	Jobs that had not finished are lost and marked "Abandoned" with the next clean-up (every 5 minutes)	"Stopped"
Server killed or crashed	same	released after about 15–20 minutes	marked "Abandoned" about 5–10 minutes after the last sign of life	"Unreachable" after about 45 seconds, "Lost" after 10 minutes
Restart with the same `InstanceId`	same	–	The server marks its own unfinished jobs as "Abandoned" when it starts	"Healthy" again
The worker is down	not affected	–	Scheduled actions do not run, mailboxes are not polled	–
Database cannot be reached	Users cannot work	–	Jobs fail	"Unhealthy"; `/health/ready` fails

Abandoned jobs are not repeated automatically. In Monitoring → Background Processes, a background action can be started again with "Requeue" from the details of its run; a scheduled action with "Run now".

A stopped or lost server is still shown for 6 hours (ExecutionHistory:ServerDisplayHours) and removed after 2 days (ExecutionHistory:ServerRetentionDays). "Remove" on its card removes it earlier.

6. Scaling¶

What to scale by: the number of open connections per server is the most meaningful measure. Each open browser tab occupies memory on its server, while the processor is mostly idle between two clicks. The metrics endpoint offers the number of active connections; see Installation and configuration. Processor and memory load are a simpler alternative.
Scaling down affects users: every removed server takes its connections with it, and its users lose unsaved input (section 5). Scale down slowly and preferably outside working hours.
Scale the worker? No. There is always exactly one worker.

The right number of connections per server depends on your application and hardware. Determine it with a load test.

7. Limitations¶

Unsaved work does not move. When a server stops, its users reload the page on another server.
Background jobs are kept in memory. Jobs that are waiting or running on a server are lost when it stops. They are visible as "Abandoned" in the history.
Exactly one worker. If it fails, scheduled actions and mailbox listeners pause until it is back. There is no automatic takeover by another server.
New or rescheduled scheduled actions and mailbox listeners reach the worker only with its next restart (section 4).
Global user values are kept per server. A value set in a tab on one server is not visible in a tab on another server.
Collaborative editing of a sketch in several tabs works only between tabs on the same server.
Monitoring → Background Jobs shows only the job queue of the server that serves your tab. Use Monitoring → Background Processes for the picture of all servers.
Different versions: the server cards show the version of every server. nexxcite does not prevent servers of different versions from working with the same database at the same time, for example during a rolling update.

8. Frequently asked questions¶

Do I need Redis for two servers? Not strictly, but we recommend it. Without Redis, a change saved on one server reaches the other only after up to 30 minutes, and functions such as "Run now" work only on the server that serves your tab.

Can several servers process scheduled actions to share the load? No. Every server that processes them runs every scheduled action. Exactly one server may have ProcessScheduledActions and ProcessMailboxListeners switched on.

Does the worker need to be reachable for users? No. It needs no route from the load balancer. Its state is shown on every server under Monitoring → Servers and Monitoring → Background Processes.

A server was killed. When are its licenses free? After about 15–20 minutes. When a server is stopped in an orderly way, they are free immediately.

Is the license counted per server? No. All servers share the licenses of the installation.

I created a scheduled action, but it does not run. If it was saved on a web server, restart the worker. Monitoring → Background Processes → Calendar shows whether it is registered.