Monitoring – servers, background processing and diagnosis¶
This chapter describes the area Monitoring of the Admin Studio: the state of the servers, the history of the background processing, the job queue, the data inspector and the developer console. It also lists the endpoints that monitoring systems outside nexxcite can query.
It is intended for administrators and operators of a nexxcite installation.
1. Monitoring at a glance¶
| Page | What it is for | Permission |
|---|---|---|
| Servers | Health, load and activity of every server, license usage, history charts | "View Servers" |
| Background Processes | What ran in the background, when and on which server – with errors, protocols and logs | "View Background Processes" |
| Authentication Log | Every attempt to sign in, with result, reason, IP address and browser – see Authentication log | "View Authentication Log" |
| Background Jobs | The job queue of the server that serves your browser tab | "Background Job Administration" |
| Data Inspector | Browsing and editing the records of any entity with any of its grids and forms | "Data Inspector" |
| Developer Console | Watching another browser session live: log, database statements, actions, data | "Developer Console" |
All pages are found in the Admin Studio under Monitoring and through the spotlight search (Ctrl+K). All permissions except "Developer Console" are part of "System Administrator"; "Developer Console" is only part of "System Developer". See Roles and permissions.
Interventions – "Run now", "Poll now", "Requeue", "Remove" – additionally require "Background Job Administration". Watching does not require it.
The terms¶
| Term | Meaning |
|---|---|
| Server (instance) | One running nexxcite process. With several servers, each one is listed. See Running several servers. |
| Heartbeat | The sign of life every server sends about every 15 seconds, with its state and figures |
| Execution | One run of background processing: a background action, a scheduled action, a mailbox listener round that handled mails, or an action with "Trace Executions" |
| Job queue | Where a server keeps the background jobs that wait or run. It exists in the memory of each server. |
2. Servers¶
Admin Studio → Monitoring → Servers shows the state of every server. The page refreshes about every 15 seconds; "Auto refresh on" / "Auto refresh off" switches that.
At the top, the page names the server your browser tab is connected to ("You are connected to … – this instance serves this page").
Tiles¶
The tiles sum up all servers that are currently reporting:
| Tile | Shows |
|---|---|
| "Servers" | Healthy servers / reporting servers |
| "Licenses" | Licenses in use / licensed, and the number of reserved licenses. See Licensing. |
| "Connections" | Open browser tabs on all servers, signed in or not |
| "Unique users" | Signed-in users with at least one open tab, each counted once |
| "Actions", "Expressions", "Queries" | Executed per minute |
| "Form renders", "Dashboard renders" | Per minute |
Activity figures are measured over the last minute. The small curves show the values of about the last ten minutes – since the server that serves the page was started.
Instance cards¶
Below "Instances", every server has a card:
| Part | Content |
|---|---|
| Head | Instance id, host name, version, health state; "your connection" on the card of your own server |
| Roles | "Scheduler" (runs scheduled actions), "Mailboxes" (polls mailbox listeners), "Web" (neither) |
| Reasons | Why the server is not healthy, in plain text |
| Activity | The figures of the tiles for this server |
| Figures | Heartbeat, uptime, CPU, memory, threads, database response time, message bus, workers, queued / running jobs, scheduled and recurring jobs, jobs OK / failed since start |
| Mailbox listeners | Per listener: state, last poll, mails handled since start, last error |
| Buttons | "Executions" (the runs of this server under Background Processes), "Remove" (only for a server that no longer reports) |
Health states¶
| State | Meaning |
|---|---|
| Healthy | Everything in order |
| Degraded | A run failed within the last hour; a mailbox listener failed on its last poll; jobs pile up (more than 50 per worker); a test job has waited more than 10 minutes; part of the heartbeat does not answer |
| Unhealthy | The database cannot be reached; the job server is not running or has no workers; a mailbox listener failed 10 times in a row |
| Unreachable | No heartbeat for three intervals (about 45 seconds). The server is down, hanging or cut off. |
| Lost | No heartbeat for 10 minutes (ExecutionHistory:ServerLostAfterMinutes). Treated as gone. |
| Stopped | The server was shut down in an orderly way. |
Every few minutes, each server puts a small test job into its own job queue. If it waits longer than 10 minutes, the queue does not execute new jobs, and the server is shown as "Degraded".
Stopped and lost servers are shown for 6 hours and then hidden; the page says how many are hidden. "Remove" removes such a server from the list of all servers at once. Unfinished jobs of that server are marked as abandoned. If the server is in fact still running, it reappears with its next heartbeat.
History¶
The section "History" shows charts over 24 h, 7 days, 30 days or 90 days, for all servers or for one:
| Chart | Shows |
|---|---|
| "License usage" | Concurrent licenses in use in the whole installation – the highest value per interval |
| "Licensed users per server" | Users with a license and a session on each server |
| "CPU" | Share of all cores used by the process |
| "Memory" | Memory used by the process |
| "Open sessions" | Browser tabs connected to each server |
| "Database latency" | Time a server needs to write its heartbeat to the database |
| "Actions" | Executed actions per minute |
Every server stores its figures every 5 minutes (ExecutionHistory:MetricsSnapshotIntervalMinutes);
they are kept for 90 days (ExecutionHistory:MetricsRetentionDays). Each point is the average of an
interval, the license usage its peak.
3. Background Processes¶
Admin Studio → Monitoring → Background Processes shows what ran in the background – on every server, including servers that no user can reach (such as the worker of an installation with several servers). The page refreshes about every 20 seconds.
What is recorded¶
| Kind | Recorded |
|---|---|
| "Background action" | Every action that runs in the background – started by a business rule or through the REST API |
| "Scheduled action" | Every run of a scheduled action |
| "Mailbox listener" | Rounds that handled at least one mail or failed, with one line per mail |
| "Traced action" | Runs of actions that users start, if the action has "Trace Executions" switched on |
Not recorded: mailbox rounds without mails (every 20 seconds per listener), scheduled actions that are switched off or outside their planned period, and repetitions of the same mailbox error (the first error, a changed error and then once per hour).
An execution passes through the states "Queued" or "Scheduled", "Running" and one of the results "Succeeded", "Succeeded with warnings", "Failed" or "Skipped". "Abandoned" means the job waited or ran on a server that stopped or failed before the job finished – the job queue lives in the memory of the server, so the job was lost.
Tabs¶
| Tab | Content |
|---|---|
| "Overview" | Tiles for the last 24 hours (executions, succeeded, failed, abandoned, in flight, servers), the server cards, "Problems - last 24 hours" and "Running and waiting" |
| "Executions" | All executions, filtered by "Kind", "State" (including "Problems (failed, abandoned)"), "Server", "Period" and a search over action, source, error text and job id |
| "Calendar" | A month with runs and errors per day. For a chosen day, one line per source on a 24-hour axis – which job ran on which server when. Above it, the table "Scheduled actions" with schedule, next and last run and warnings. |
| "Workflow Rules" | The processes of event based workflow rules: what waits right now – rule, record, current item, the time it waits for, state – and what finished, with its outcome. Filtered by rule, state and outcome. |
Details of an execution¶
A click on an execution opens its details on the right: all facts, the protocol of the action step by step (errors expanded), the mails of a mailbox round with the rule that handled them, the captured log and the executions "Started by the same cause" – for example all background actions that one save triggered.
| Button | Effect |
|---|---|
| "Open …" | Opens the definition that triggered the run |
| "All runs" | Shows all runs of that definition |
| "Run now" / "Poll now" | Runs the scheduled action again or polls the mailbox, on the server responsible for it |
| "Requeue" | Queues a background action again with its original arguments |
| "Open action" | Opens the action in its editor |
A manual run ignores the planned period of a scheduled action, but not "Disabled". Traced actions that a user started cannot be repeated. When the target is another server, the page waits up to 15 seconds for its confirmation; without one, check the history instead of trying again.
Details of a process¶
A click on a process in "Workflow Rules" opens its details on the right: the facts of the process and its trace – every decision it took, with the item, the time and the server.
| Button | Effect |
|---|---|
| "Open rule" | Opens the event based workflow rule in its editor |
| "Actions of this process" | Shows the executions of the actions the process ran |
| "Advance now" | Treats the wait time as elapsed and continues the process now on the server serving the page. Not offered while the process waits for an event only. |
| "Cancel process" | Ends the process after a confirmation; it moves to the finished processes with the outcome "Cancelled" |
"Advance now" and "Cancel process" need the permission "Background Job Administration". A process another server is continuing at that very moment cannot be cancelled; try again a moment later.
Logs of executions¶
Whether the log of a run is kept is decided by ExecutionHistory:LogCaptureMode (default: only for
failed runs and runs with warnings). Scheduled actions, business rules, mailbox listeners and actions
can override it. See Installation and configuration.
"Local job queue" at the top opens Background Jobs (section 4).
4. Background Jobs¶
Admin Studio → Monitoring → Background Jobs shows the job queue of the server that serves your browser tab – with several servers, not the queues of the others. It is the tool for direct work on that queue:
| Section | Content and actions |
|---|---|
| Tiles | Counters of the queue |
| "Failed" | The newest failed jobs; "Retry" queues one again |
| "Recurring jobs" | Scheduled actions and mailbox listeners registered on this server, with cron, last and next run. "Run now" starts one at once, "History" shows its runs under Background Processes. |
| "Servers" | The job server of this process, its workers and queues |
| "Jobs" | The jobs of the queue by state. "Requeue" queues a job again, "Delete" removes it. A click on a job shows its details, including its arguments. |
The page shows the notice "Volatile storage.": the queue lives in memory, and its counters and contents are lost when the server restarts. The permanent record is the history under Background Processes; "Execution history" leads there.
5. Data Inspector¶
Admin Studio → Monitoring → Data Inspector shows the records of any entity, regardless of whether the application offers a page for them. It is meant for checking and correcting data while you build an application.
- Choose the "Entity".
- Choose the "Grid" that shows the records – by default the default grid of the entity.
- Choose the "Form" that opens a record – by default the default form of the entity.
A click on a row opens the record with the chosen form. "New" creates a record with that form. The address of the page contains the selection, so you can bookmark or share it.
Records are read and saved with your own permissions; read restrictions of your roles apply as everywhere else. "Delete" is shown but not available – delete records through a form or an action of the application.
6. Developer Console¶
The developer console watches another browser tab live – your own in another window, or the tab of another user – and shows what happens in it. It is the tool for finding out why a form, an action or a rule behaves as it does.
Opening: Admin Studio → Monitoring → Developer Console opens it in the lower part of the Admin Studio. The spotlight entry "Developer Console" opens it full screen in a new browser tab, so the tab you want to watch stays as it is.
Choosing a session¶
The tab "Session" lists the sessions that can be watched, grouped into "This tab", "Your sessions" and "Other users". Choose one and press "Connect" to start recording. "Look for sessions again" refreshes the list – for example after the watched tab was reloaded, which ends its session. "Buffer limit" sets how many lines the console keeps.
What it shows¶
| Tab | Content |
|---|---|
| "Telemetry" | Browser, address, screen size, language and the current page of the watched session |
| "Log" | The log of the session, filterable by level |
| "Database" | The database statements of the session with duration and number of rows |
| "Actions" | Every action that ran, with its protocol as a tree |
| "Data Set" | The record shown in the form of the session |
| "App State" | The navigation state of the session |
Lists can be searched, copied and exported as JSON.
Privacy and transparency¶
Watching a session shows the data of its user: the records they open and – in the database tab – the values they type. Two global settings control this (see Global settings):
- "Send SQL statements to the developer console" – off, the database tab stays empty.
- "Show users that they are being observed" – on (default), the watched tab shows a frame and the
notice "Observed by
".
The permission is checked on the server of the watched session and again about every 10 seconds; withdrawing it ends the recording.
With several servers, the console sees the sessions of all servers only when a message bus is
configured (Messaging:Provider = Redis). Session data then travels through Redis – secure Redis like
any system that holds business data.
7. Endpoints for external monitoring¶
| Endpoint | Purpose |
|---|---|
/health/startup |
Answers once the server has started (schema migration included) |
/health/live |
Answers as long as the process works |
/health/ready |
Fails while the database cannot be reached |
/metrics |
Metrics in Prometheus format |
The endpoints answer without sign-in; the health endpoints answer only with the status word. With
ApplicationServer:ManagementPort, they answer only on a separate port. Details and settings in
Installation and configuration.
The figures of the pages in this chapter (health, activity, executions) are not part of /metrics.
8. Limitations¶
- Background Jobs shows only the queue of the server serving your tab. The overall picture is under Background Processes.
- Activity curves on the Servers page start with the server that serves the page. After it restarts, they begin again; the History charts are not affected.
- "Run now", "Poll now" and the developer console reach other servers only with a message bus.
- The Data Inspector cannot delete records.
- The history of executions is kept for 30 days and logs for 14 days by default. Both are set in the global settings for the whole installation.
9. Frequently asked questions¶
A server shows "Degraded" because of an error an hour ago. Is it broken? No. A failed run keeps the server at "Degraded" for one hour so that it gets noticed. The reasons on the card say what happened; the details are under Background Processes.
A server shows "Unreachable", but users can work. Why? The heartbeat of that server does not reach the others – for example because it cannot write to the database or reach the message bus. Check the reasons and the log of that server.
Can I see which server ran a scheduled action? Yes. The column for the server in the executions, and the calendar of a day, show it.
Does the developer console slow down the watched session? Only while it is recording. Without a connected console, a session prepares and sends nothing.
Are users told that they are being watched? Yes, by default. The global setting "Show users that they are being observed" can switch the notice off.