Sentinel

Monitor devices, core nodes, network assets and custom metrics, raise alerts and notify people by email, SMS, push, webhooks, Slack or Teams.

Sentinel is the monitoring built into tangle-core. It watches devices, core nodes, probes and metrics that any script pushes, raises alerts from rules you define, and notifies people. There is nothing extra to install. Sentinel is part of the Pro and Ultimate editions.

#What Sentinel watches

Sentinel runs on the primary node of a cluster, or on a standalone node. Backups keep their view of the system up to date but do not evaluate rules or send anything until they become primary. Probes and notifications run in separate tasks with timeouts, so Sentinel never slows the show.

#Where you configure it

Sentinel uses four kinds of project resources:

Edit them in the designer's Sentinel section (/designer/sentinel), or through REST (/api/v1/alertRules, /api/v1/notificationChannels, /api/v1/maintenanceWindows, /api/v1/probes) and MCP (create_alertRule, update_alertRule and so on), like any other project resource.

#Alert rules

{
  "id": "server-room-hot",
  "name": "Server room hot",
  "severity": "critical",
  "condition": { "kind": "metric", "name": "room.temperature", "labels": { "room": "server" }, "above": 30, "clear": 28 },
  "forSeconds": 60,
  "channels": ["ops-email"]
}

A rule produces one alert per subject. A status rule over all devices raises one alert for each device that goes offline, not one alert for the whole rule. Alert ids look like <ruleId>/<subject kind>:<subject id>, for example projectors-offline/device:projector-1.

#Alert conditions

status: devices whose status is in statuses. Default: offline, timeout and error.

{ "kind": "status", "rooms": ["lobby"], "statuses": ["offline", "timeout"] }

state: a device state value compared to a threshold, with above, below, equals or notEquals. clear is the value that resolves the alert (hysteresis).

{ "kind": "state", "rooms": ["room-a"], "capability": "lamp", "key": "hours", "above": 1800 }

metric: a pushed or probe metric compared to a threshold, per label set. Same comparisons and clear as state.

{ "kind": "metric", "name": "probe.latencyMs", "labels": { "probe": "web" }, "above": 500 }

stale: no update for seconds. Watch a device state key (capability and key) or a metric (metric and labels).

{ "kind": "stale", "seconds": 300, "metric": "room.temperature" }

A device that never reported the key is only watched when you name it in devices.

coreNode: problems in a redundant cluster, with on:

{ "kind": "coreNode", "on": "down", "nodes": ["core-b", "core-c"] }

Without nodes, all nodes are watched. On a standalone node, coreNode rules never fire.

status, state and stale conditions select devices with devices, rooms or modules (module ids). With none of them, every device is watched.

#Severities

Channels have a minSeverity, so you can send everything to email and only critical alerts to SMS.

#Alert lifecycle

An alert moves through these states:

1. pending: the condition holds, and forSeconds has not passed yet. If the condition clears now, the alert is dropped and nothing is sent.
2. firing: the condition held for forSeconds. A notification goes out.
3. resolved: the condition was false for clearForSeconds (and, with clear, the value passed the clear threshold). A resolve notification goes out if notifyOnResolve is on.

With "above": 30, "clear": 28, an alert fires at 31, stays firing at 29.5 and resolves at 27.9. This stops an alert from flapping around one value.

An alert also ends when its rule is disabled or removed, or its subject is removed from the project. These endings are not notified.

#Notification channels

Every channel has these fields:

Note: Passwords, tokens and webhook URLs marked secret are stored in the OS credential store of the machine, not in the project file. The project keeps ******** in their place.

#Email

{
  "id": "ops-email", "name": "Ops email", "kind": "email", "minSeverity": "warning",
  "settings": { "host": "smtp.example.com", "username": "alerts@example.com", "password": "...", "from": "alerts@example.com", "to": ["ops@example.com"] }
}

The subject reads like [Tangle Lobby] FIRING critical: Server room hot on room.temperature{room=server}, where Tangle Lobby is the project name.

#SMS (Twilio)

SMS and push messages are shortened to 160 characters.

#Push (ntfy, Pushover)

Critical alerts are sent with a higher priority than warnings and info.

#Webhook

Core POSTs JSON:

{
  "event": "firing",
  "site": "Tangle Lobby",
  "title": "[Tangle Lobby] FIRING critical: Server room hot on room.temperature{room=server}",
  "text": "room.temperature{room=server} is 31.5\n\nRule: Server room hot\nSeverity: critical\n...",
  "severity": "critical",
  "alert": { "id": "server-room-hot/metric:room.temperature{room=server}", "state": "firing", "...": "..." },
  "at": 1760362800000
}

event is firing, repeat, resolved, summary or test. With a signingKey, the request carries X-Tangle-Signature: sha256=<hex>, where <hex> is the HMAC-SHA256 of the raw body with the signing key. To verify, compute the same HMAC over the body you received and compare:

import crypto from "node:crypto";

function verify(rawBody, header, key) {
  const expected = Buffer.from("sha256=" + crypto.createHmac("sha256", key).update(rawBody).digest("hex"));
  const received = Buffer.from(header ?? "");
  return expected.length === received.length && crypto.timingSafeEqual(expected, received);
}

#Slack

#Microsoft Teams

Teams receives an adaptive card coloured by severity.

#Delivery, retries and rate limits

• Notifications go into a queue and are sent in the background. Each attempt has a 10-second timeout.
• A failed delivery is retried after 5 seconds, 30 seconds and 2 minutes, then recorded as failed in the history.
• Each channel sends at most 20 messages per minute. The rest are counted and summarised in one message once the minute is over ("20 more alerts were not sent to this channel in the last minute").

#Test a channel

In the designer, open the channel and choose Test. Through the API:

curl -X POST http://127.0.0.1:7340/api/v1/sentinel/channels/ops-email/test \
  -H "Authorization: Bearer $TOKEN"

The test is sent at once, without queue or retry, and the answer reports success or the provider's error, for example an SMTP authentication failure.

#Maintenance windows

A maintenance window mutes notifications. Alerts still fire and show in the designer, marked muted.

One-off window (times in milliseconds since the epoch):

{ "id": "lamp-swap", "name": "Lamp swap", "devices": ["projector-1"], "start": 1760346000000, "end": 1760353200000 }

Weekly window:

{ "id": "nightly", "name": "Nightly shutdown", "rooms": ["lobby"], "days": ["mon", "tue", "wed", "thu", "fri"], "from": "23:00", "to": "07:00" }

Scope a window with rules, devices and rooms. With none of them, the window mutes everything.

If a problem is still there when the window ends, Sentinel sends one firing notification for it.

#Probes

Probes check network assets that have no device module.

[
  { "id": "web", "name": "Booking site", "kind": "http", "url": "https://example.com/health", "expectStatus": [200], "intervalSeconds": 30 },
  { "id": "nas", "name": "Media NAS", "kind": "tcp", "host": "10.0.0.20", "port": 445, "room": "control" }
]

Each probe produces metrics labelled probe=<id>: probe.up (1 or 0), probe.latencyMs, and for HTTP probe.httpStatus. Alert on them with a metric condition:

{ "id": "web-down", "name": "Booking site down", "severity": "critical", "forSeconds": 60,
  "condition": { "kind": "metric", "name": "probe.up", "labels": { "probe": "web" }, "below": 1 } }

#Pushing metrics

Any script can push metric samples (scope control):

curl -X POST http://localhost:7340/api/v1/sentinel/metrics \
  -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" \
  -d '[{"name":"room.temperature","value":31.5,"labels":{"room":"server"}}]'

Each sample has name (letters, digits, ., _ and -, up to 128 characters), a numeric value, optional labels (string values) and an optional at timestamp in milliseconds. A label room that matches a room id ties the metric to that room.

Push to the primary node. Core keeps one sample per minute for 24 hours, and forgets a metric that has not been updated for a day. GET /api/v1/sentinel/metrics returns the current values; add ?samples=true for the last 24 hours of minute samples.

Pushing also works over WebSocket (sentinel.metrics with samples) and MCP (push_metrics).

#Acknowledging and silencing

Both need the control scope and are refused on a backup node.

Acknowledge an alert to stop its repeat notifications. The alert keeps firing until the condition clears and then resolves normally.

curl -X POST http://127.0.0.1:7340/api/v1/sentinel/ack \
  -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" \
  -d '{"alertId":"lamp-hours/device:projector-1","note":"lamp ordered"}'

Silence one alert, or every alert of a rule (current and future), until a time in milliseconds since the epoch. Silenced alerts send no notifications.

curl -X POST http://127.0.0.1:7340/api/v1/sentinel/silence \
  -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" \
  -d '{"ruleId":"projectors-offline","until":1760400000000}'

In the designer's Alerts tab, acknowledge with a note, or silence for 15 minutes, 1 hour, 4 hours or 1 day.

#History

Every alert transition is recorded: pending, firing, resolved, dropped, repeat, acknowledged, silenced, notified and notifyFailed (with the channel and the error).

• GET /api/v1/sentinel/alerts?history=true returns current alerts and the latest transitions, newest first. ?limit= sets how many (up to 500).
• Events: subscribe to sentinel.alert:<ruleId> (or sentinel.alert:*).
• On disk, history is a JSON Lines file at <data>/sentinel/<project id>/history.jsonl in the core data folder (see Operations). It rotates at 5 MB, keeping one old file, and survives restarts.

#The designer's Sentinel section

Open Sentinel in the designer. It has these tabs:

On a backup node the section says the core is a backup and that the primary sends notifications. Without a monitoring license it says so; you can still set up rules and channels.

#API and MCP reference

See REST API, WebSocket and MCP.

#Edition

Sentinel needs the Pro or Ultimate edition (the MONITORING entitlement). It also runs in demo mode. On Lite and in the free designer:

• No rules are evaluated, no probes run and nothing is sent.
• GET /api/v1/sentinel reports licensed: false.
• Acknowledge, silence, metric pushes and channel tests answer 403 not_licensed.
• Alert rules, channels, maintenance windows and probes stay editable, so you can set them up before upgrading.

Turning monitoring on or off never affects devices, timelines, automations or cues. See Licensing.

#Coming later: Tangle Cloud

Tangle Cloud, a hosted remote monitoring service, is not available yet. It is planned to receive heartbeats, alerts and metrics from each site, alert when a whole site goes silent, and give a view across many sites. Local Sentinel works on its own and does not need it.