Redundancy

Run the same show on two or more tangle-core machines in lockstep, with automatic failover in under a second.

A redundant cluster runs the same project on two or more machines. Every node computes the show, so timelines run on all of them at the same moment, but only the primary drives devices. If the primary stops, the next node takes over within a second, already at the same point of the show.

Redundancy needs Tangle Ultimate, or Tangle Pro with the redundancy add-on. See Edition requirement.

#Concepts

Primary and backups. One node is the primary: it sends commands to devices, runs automations and Sentinel, and accepts changes. The other nodes are backups. A node without redundancy is standalone.

Lockstep. Each backup keeps a connection to the primary and receives the project, device state, and every transport operation with the cluster time at which it happened. Each backup then plays every timeline itself, in shadow: cues go to its cue log, not to devices. When the primary fails, a backup is already where the show is.

Cluster clock. All nodes share one time scale, measured against the primary's clock. Every node computes the same execution time for every cue. A new primary keeps its clock offset, so cluster time never jumps.

Epochs. The epoch is a counter that goes up on each failover. If two nodes ever both claim to be primary, the higher epoch wins (on a tie, the smaller node id), and the other steps down.

Quorum. With 3 or more nodes, a node can only be or become primary while it reaches a majority of the cluster (itself included). A network split therefore never gives two primaries. With 2 nodes no majority can survive a failure, so the quorum is 1.

Cue fingerprints. Each node keeps a cue log: the count of cues fired (or shadow-fired) and a digest of that sequence. Equal digests at equal counts prove two nodes computed the same show. Heartbeats carry this position, and backups compare theirs with the primary's.

Diverged. A backup whose cue log differs from the primary's reports diverged, resyncs from the primary, and cannot take over until it is in sync again.

Failover timing. Nodes send heartbeats every 100 ms. When the primary is silent for 500 ms, the first candidate takes over. Each next candidate in election order waits 200 ms longer, so only one takes over.

Who is primary. The node started with --role primary goes first, then the others by node id. Only healthy, synced backups are candidates. Every node starts as a backup and listens before becoming primary, so a node that comes back after a failover never fights the one that took over.

#Requirements

• The same project on every node. By default a backup adopts the primary's project and saves it to its own project file, so you edit the project on the primary only.
• A cluster key shared by every node. When you add a core from the designer, core creates it for you. For a cluster set up by hand, choose one of at least 16 characters.
• A unique node id per node.
• UDP and TCP port 7341 open between all nodes (heartbeats over UDP, state mirroring over TCP). --cluster-listen changes the address and port.
• For automatic discovery and pairing: UDP and TCP port 7342 open, and the nodes on the same local network (discovery uses multicast, which does not cross routers).
• The same device modules installed on every node.
• A license activated on every node. See Edition requirement.

Heartbeats and mirror messages are encrypted and authenticated with the cluster key (XChaCha20-Poly1305). Nothing else on the network can pose as a node or read the project, device state or device passwords that cross between nodes. A message that fails authentication is dropped.

Device passwords and keys, and notification channel secrets, follow the primary: it copies them to every backup over the encrypted channel when the backup joins and whenever they change, and each backup stores them in its own credential store. Cores kept separate get nothing. A backup that takes over logs in to password-protected devices with the same passwords.

#Add a core from the designer

Every core announces itself on the local network. The main core (the primary, or a standalone core) lists the cores it hears and asks you what to do with each new one. This is the easiest way to build a cluster: no node ids, peers or keys to type.

1. Install tangle-core, a license and the same device modules on the new machine, and start it. Its project does not matter: it will take the main core's.
2. Find the new core's 6-digit pairing code. It is in the new core's log (pairing code of this core code=482913) and on the Nodes page of its own designer. A new code is made at each start, and after 5 wrong tries.
3. In the designer of the main core, a banner tells you a new core was found. Open Nodes: the new core is listed with its name, host and address.
4. Choose:
   • Keep separate: the new core runs its own, independent show. The main core remembers your choice and does not ask again. Nothing changes on the new core.
   • Add as redundant: type the pairing code. The two cores pair, and the new core joins this core's cluster as a backup.

Accepting a core needs a token with the admin scope, and a license that allows the new cluster size. If it does not, core refuses with the reason.

What happens when you pair:

• The two cores prove they know the code without sending it (a password-authenticated key exchange, SPAKE2). A wrong code fails with wrong_pairing_code and changes nothing on either core.
• The main core creates a random cluster key and hands it to the new core inside that exchange. From then on all traffic between the cores is encrypted.
• The main core becomes the primary without stopping its devices or timelines. The new core stops its own devices, takes the primary's project (its previous project file is kept as <name>.before-cluster.<ext>), syncs, and receives the device passwords.
• Every member learns about the new node and saves the membership. Quorum follows the new size: a third node makes the quorum 2.
• Each core saves its membership and key in cluster.json in its data folder, readable by the Tangle account only. At the next start it rejoins its cluster without any option.

tangle-core cluster show prints this machine's saved cluster. To take a machine out of its cluster, stop core on it and run tangle-core cluster leave. This forgets the saved membership on that machine only; the other members keep it in their peers, which is harmless while it is off.

Note: A designer sign-in made on the primary works on a newly paired backup only after both cores restart.

Cores in another cluster are listed as clustered and cannot be paired. A backup does not list discovered cores; open the designer on the primary. To stop a core from announcing itself, start it with --no-discovery. Across routed networks, where multicast does not reach, set up the cluster by hand as below.

#Set up a 3-node cluster

You can also build a cluster by hand, with options on each node. Three nodes are the recommended setup: a network split can never produce two primaries.

1. Install tangle-core, the same modules and the same project file on the three machines. In this example they are 10.0.0.1, 10.0.0.2 and 10.0.0.3.

2. Choose a shared key and set it on every machine:

   export TANGLE_CLUSTER_KEY=<same secret on every node, 16+ characters>

3. Start each node with its own --node-id and the cluster addresses of the other nodes in --peer:

   # machine A (10.0.0.1): goes first
   tangle-core run -p show.yaml --node-id core-a --peer 10.0.0.2:7341,10.0.0.3:7341
   # machine B (10.0.0.2)
   tangle-core run -p show.yaml --node-id core-b --peer 10.0.0.1:7341,10.0.0.3:7341 --role backup
   # machine C (10.0.0.3)
   tangle-core run -p show.yaml --node-id core-c --peer 10.0.0.1:7341,10.0.0.2:7341 --role backup

4. Check the cluster (see Check the cluster): one primary, two backups with synced: true.

To accept designer and dashboard connections from other machines, add the remote access options to each node: --listen 0.0.0.0:7340 --allow-remote --insecure-http. See Operations.

#Set up a 2-node cluster

export TANGLE_CLUSTER_KEY=<same secret on both nodes>
# machine A (10.0.0.1)
tangle-core run -p show.yaml --node-id core-a --peer 10.0.0.2:7341
# machine B (10.0.0.2)
tangle-core run -p show.yaml --node-id core-b --peer 10.0.0.1:7341 --role backup

Warning: With 2 nodes, if the link between them drops, both become primary and both drive devices until the link is back. Then the higher epoch wins and the other steps down. Use 3 nodes, or connect the two machines with a dedicated link.

#Cluster options

#Running as a service

Set TANGLE_CLUSTER_KEY when you install the service. The key goes to a file only the service can read (cluster.key in the data folder), not into the service definition:

sudo TANGLE_CLUSTER_KEY=<shared key> tangle-core service install -- \
  --project /shows/main.yaml --node-id core-a --peer 10.0.0.2:7341,10.0.0.3:7341

On Windows, set the key in an administrator PowerShell first ($env:TANGLE_CLUSTER_KEY = "<shared key>"), then run tangle-core service install -- ....

See Operations.

#Sharing the work

By default a cluster runs in standby mode: the primary drives every networked device and the other nodes wait to take over. A large site can instead spread the devices across the nodes, in shared mode, to lighten the load on one machine. The primary stays the one "brain": it runs timelines, automations, intents, variables and the lighting programmer; the other nodes run device modules and keep shadowing the show, so any of them can still take over.

Set it on the Nodes page, under Sharing the work (stored in the project as cluster, so every node agrees):

How devices are placed:

• Hardware attached to a node (node) always runs there and never moves.
• Runs on (runsOn) or a folder rule gives a networked device a preferred node. It moves to another node while that one is down.
• Everything else is placed automatically, balanced by device weight across the healthy nodes. A device does not move again unless its node fails or you rebalance.
• When a node stops, the primary moves its devices to the least loaded survivors in about 2 seconds. If the primary itself fails, the new primary keeps the placement and only re-places the old primary's devices.
• A node only runs the devices of the latest placement from the current primary, and a node that loses the majority stops its devices first, so two nodes never drive the same device in a cluster of three or more.

Intents and client connections still go to the primary, which forwards actions for a device placed elsewhere over the cluster link (one network hop, typically under 2 ms). Device state comes back the same way and is mirrored to every node.

API: GET /api/v1/cluster/placement, POST /api/v1/cluster/rebalance, PUT /api/v1/project/cluster; MCP get_cluster_placement, rebalance_cluster, set_cluster_settings; event cluster.placement. Shared mode needs no extra license: every node is a host, as in standby mode.

#Hardware attached to one machine

Networked devices are driven by whichever node is primary. A device wired to one machine (serial port, USB DMX interface, LTC output) can only be driven from that machine. Give it a node in the project:

{ "id": "dmx-usb", "name": "USB DMX", "moduleId": "misc.usbdmx.lighting", "node": "core-b", "config": {} }

Only core-b runs that device's module, whatever its role. The primary sends actions for it to core-b over the cluster, and its state comes back the same way, so every node sees it.

If core-b is down, that device is offline and its actions fail with an error naming the node. Everything else continues. For anything that must survive a machine failure, prefer network gateways: Art-Net or sACN nodes, serial-over-IP converters.

On a standalone node the node field is ignored.

#Check the cluster

GET /api/v1/cluster (scope read) or the MCP tool get_cluster_status returns this node's view:

{
  "nodeId": "core-b",
  "role": "backup",
  "epoch": 3,
  "health": "ok",
  "synced": true,
  "diverged": false,
  "primary": "core-a",
  "size": 3,
  "quorum": 2,
  "clockOffsetMs": -0.42,
  "nodes": [
    { "nodeId": "core-a", "role": "primary", "epoch": 3, "health": "ok", "synced": true,
      "cueLog": { "count": 128, "digest": "..." }, "address": "10.0.0.1:7341", "ageMs": 40, "alive": true },
    { "nodeId": "core-c", "role": "backup", "epoch": 3, "health": "ok", "synced": true,
      "cueLog": { "count": 128, "digest": "..." }, "address": "10.0.0.3:7341", "ageMs": 65, "alive": true }
  ]
}

Before a show, check that every backup is synced and that cue log counts and digests match the primary's.

To be alerted when a node goes down, falls out of step or the cluster has no primary, add a coreNode alert rule in Sentinel.

#Clients and the primary

Changes go to the primary. A backup refuses configuration changes, transport commands, intents, manual automation runs and Sentinel actions with 409 not_primary. The answer includes the primary's address:

{
  "type": "https://tangle.dev/problems/not_primary",
  "title": "Conflict",
  "status": 409,
  "detail": "this node is a backup; send changes to the primary",
  "code": "not_primary",
  "primary": "http://10.0.0.1:7340"
}

Reading (project, state, cluster status) works on every node.

Every node serves the dashboards (/d), end-user views (/view) and their live connection. A dashboard follows the primary by itself: see One address for operators.

The primary copies API tokens, including enrolled panel tokens, to every backup, so a client keeps working after a failover. Dashboard and view PIN sessions are signed with the cluster key and are accepted by every node.

#One address for operators

Operators should not need to know which machine is the primary. Tangle gives you three layers. The first is always on. Add the others when you need them.

#1. Any node's address works

Open a dashboard on any node, for example http://10.0.0.2:7340/d/lobby. The page asks that node for the other members of the cluster and remembers them on the device. Then:

• it connects to the primary, even when you opened a backup's address;
• when the primary fails, it reconnects to the new primary within a few seconds, without a reload and without asking for the PIN again;
• when the node you opened goes down, the page keeps working through the others, and after a reload it starts from the nodes it remembers.

You can still list the nodes yourself with the cores parameter, for example /d/lobby?cores=10.0.0.1:7340,10.0.0.2:7340. This helps a panel that has never reached the cluster before. See Dashboards.

The one limit: the browser must load the page from a node that is up. A bookmark to a machine that is switched off does not open. The cluster address fixes this.

#2. The cluster address

Give the cluster a name in the designer, under Nodes > Cluster address, for example riverside. The current primary then answers for riverside.local on the local network, and operators bookmark http://riverside.local:7340/d/lobby.

• Only the primary answers. When a backup takes over, it announces the name within a second, and the old primary sends a goodbye if it is still running. A standalone core answers for the name too, so you can set it up before adding a backup.
• If another machine already uses the name, the Nodes page shows a conflict and Tangle does not announce it. Pick another name.
• .local names use mDNS (Bonjour). macOS, iOS, iPadOS and Linux with Avahi resolve them. Windows 10 and 11 resolve them in most browsers. Android support depends on the device, so give Android panels a node's IP address and let layer 1 do the failover.
• mDNS stays on one network segment. It does not cross routers or VLANs.

#3. A floating IP or a reverse proxy

For sites with a network administrator, or panels that cannot resolve .local:

Floating IP (Linux, keepalived). One virtual IP moves to whichever machine holds it. Make keepalived follow the Tangle primary with a check script, so the IP and the primary stay on the same machine:

#!/bin/sh
# /usr/local/bin/tangle-primary: succeeds when this node is the Tangle primary.
# TOKEN is an API token with the read scope.
curl -sf -H "Authorization: Bearer $TOKEN" http://127.0.0.1:7340/api/v1/cluster | grep -q '"role":"primary"'

# /etc/keepalived/keepalived.conf on every node
vrrp_script tangle_primary {
  script "/usr/local/bin/tangle-primary"
  interval 1
  fall 1
  rise 1
}
vrrp_instance TANGLE {
  state BACKUP
  interface eth0
  virtual_router_id 73
  priority 100
  advert_int 1
  virtual_ipaddress { 10.0.0.100/24 }
  track_script { tangle_primary }
}

Panels then use http://10.0.0.100:7340. keepalived needs root and runs on Linux only. On macOS and Windows, use a reverse proxy.

Reverse proxy on another machine (Caddy). The proxy sends requests to whichever node answers, and the dashboard then finds the primary by itself:

:80 {
  reverse_proxy 10.0.0.1:7340 10.0.0.2:7340 10.0.0.3:7340 {
    lb_policy first
    health_uri /d
    health_interval 1s
    fail_duration 2s
  }
}

The proxy machine is then a single point of failure for the first page load, so keep it simple and on a UPS. Open dashboards are not affected if it fails, because they talk to the nodes directly.

#Which one to use

#Failure scenarios

What a new primary does on promotion:

• Timelines continue from their position. Chasing timelines start their own chase, and timecode outputs are sent the transport again.
• Automations start running and startup triggers fire. Runs that were in progress on the old primary are not resumed, and time triggers missed during the outage are not replayed.
• Sentinel starts evaluating. Firing notifications in the first two minutes say "after failover".

#Edition requirement

One license covers one installation: a main host plus its redundant hosts. Activate the same license key on each machine. The first machine becomes the main host, the next ones join as redundant hosts, up to the number the license allows. Each host keeps its own signed license file, so a backup takes over licensed with no network call.

If the license does not allow redundancy, or the cluster lists more other nodes than the license allows, core logs the reason and runs standalone. A machine activated as a redundant host runs licensed only as part of its cluster (with --peer).

See Licensing.

#Recommendations

• Use 3 nodes. With 2 nodes, put them on a dedicated link.
• Give the machine you prefer as primary --role primary and every other node --role backup.
• Keep all nodes on the same version of tangle-core and the same module versions.
• Keep hardware that must survive a failure off single machines: use network gateways instead of node-attached devices.
• Before the show, fail over on purpose and check that every device comes online on each node.
• Install each node as a service so it restarts after a crash.
• Check GET /api/v1/cluster before each show, and add Sentinel coreNode rules so the crew hears about a lost backup.
• Give the cluster an address, and test a failover from an operator's tablet before the show.