NATS Logo by Example

Online-Peer-Remove in JetStream

CLI Go Python Deno Node Rust C# Java Ruby Elixir C
Jump to the output or the recording
$ nbe run jetstream/online-peer-remove/cli
View the source code or learn how to run this example yourself

Code

#!/bin/bash


set -euo pipefail


nats-server --version
nats --version


NATS_URL="nats://localhost:4222"

Shared config.

cat <<- EOF > shared.conf
accounts: {
  SYS: {
    users: [{user: sys, pass: sys}]
  }
  APP: {
    jetstream: true
    users: [{user: app, pass: app}]
  }
}


system_account: SYS
EOF

Define the server configs for cluster.

cat <<- EOF > "n1.conf"
server_name: n1
port: 4222
http_port: 8222
include shared.conf
jetstream {
  store_dir: "./n1"
}
cluster: {
  name: hub 
  port: 6222
  routes: [
    nats-route://127.0.0.1:6222,
    nats-route://127.0.0.1:6223,
    nats-route://127.0.0.1:6224,
  ]
}
EOF


cat <<- EOF > "n2.conf"
server_name: n2
port: 4223
include shared.conf
jetstream {
  store_dir: "./n2"
}
cluster: {
  name: hub
  port: 6223
  routes: [
    nats-route://127.0.0.1:6222,
    nats-route://127.0.0.1:6223,
    nats-route://127.0.0.1:6224,
  ]
}
EOF


cat <<- EOF > "n3.conf"
server_name: n3
port: 4224
include shared.conf
jetstream {
  store_dir: "./n3"
}
cluster: {
  name: hub
  port: 6224
  routes: [
    nats-route://127.0.0.1:6222,
    nats-route://127.0.0.1:6223,
    nats-route://127.0.0.1:6224,
  ]
}
EOF

Bring up the cluster

Start a server for each configuration and sleep a second in between so the seeds can startup and get healthy.

echo "Starting n1..."
nats-server -c n1.conf -P n1.pid &
sleep 1


echo "Starting n2..."
nats-server -c n2.conf -P n2.pid &
sleep 1


echo "Starting n3..."
nats-server -c n3.conf -P n3.pid &
sleep 1

Wait until the servers up and healthy.

echo 'Cluster healthy?'
curl --fail --silent \
  --retry 5 \
  --retry-delay 1 \
  http://localhost:8222/healthz; echo

Complaing about too many credential types.. nats context save sys –user sys –password sys nats context save app –user app –password app

Observe the current JS report.

nats --user sys --password sys \
  server report jetstream

Create a stream to see how an asset behaves.

cat <<- EOF > stream.json
{
  "name": "events",
  "subjects": [
    "events.*"
  ],
  "retention": "limits",
  "max_consumers": -1,
  "max_msgs_per_subject": -1,
  "max_msgs": -1,
  "max_bytes": -1,
  "max_age": 0,
  "max_msg_size": -1,
  "storage": "file",
  "discard": "old",
  "num_replicas": 3,
  "duplicate_window": 120000000000,
  "sealed": false,
  "deny_delete": false,
  "deny_purge": false,
  "allow_rollup_hdrs": false,
  "allow_direct": false,
  "mirror_direct": false
}
EOF


nats --user app --password app \
  stream add --config stream.json > /dev/null


nats --user app --password app \
  req --count 100 'events.test'

Remove the peer while online.

nats --user sys --password sys \
  server raft peer-remove -f n3

Add more data after a peer-remove.

nats --user app --password app \
  req --count 100 'events.test'

Stop the server.

nats-server --signal ldm=n3.pid

Add more data.

nats --user app --password app \
  req --count 100 'events.test'

Start the server.

nats-server -c n3.conf -P n3.pid &

Wait to observe the behavior.

sleep 3

Report again.

nats --user sys --password sys \
  server report jetstream

Output

Network f35b01bb_default  Creating
Network f35b01bb_default  Created
Container f35b01bb-nats-1  Creating
Container f35b01bb-nats-1  Created
Container f35b01bb-nats-1  Starting
Container f35b01bb-nats-1  Started
nats-server: v2.9.10-beta.1
v0.0.36-0.20221219182759-9c401f4863da
Starting n1...
[24] 2022/12/19 19:03:17.427752 [INF] Starting nats-server
[24] 2022/12/19 19:03:17.427799 [INF]   Version:  2.9.10-beta.1
[24] 2022/12/19 19:03:17.427802 [INF]   Git:      [not set]
[24] 2022/12/19 19:03:17.427805 [INF]   Cluster:  hub
[24] 2022/12/19 19:03:17.427807 [INF]   Name:     n1
[24] 2022/12/19 19:03:17.427809 [INF]   Node:     fjFyEjc1
[24] 2022/12/19 19:03:17.427811 [INF]   ID:       NBB25L77STHHDNXYND32BYQTCUS2S2UVYOIVQTCMCNZ5JUFU5MLE54CI
[24] 2022/12/19 19:03:17.428905 [WRN] Plaintext passwords detected, use nkeys or bcrypt
[24] 2022/12/19 19:03:17.429019 [INF] Using configuration file: n1.conf
[24] 2022/12/19 19:03:17.429887 [INF] Starting http monitor on 0.0.0.0:8222
[24] 2022/12/19 19:03:17.429984 [INF] Starting JetStream
[24] 2022/12/19 19:03:17.430774 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[24] 2022/12/19 19:03:17.430849 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[24] 2022/12/19 19:03:17.431297 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[24] 2022/12/19 19:03:17.431338 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[24] 2022/12/19 19:03:17.431378 [INF] 
[24] 2022/12/19 19:03:17.431419 [INF]          https://docs.nats.io/jetstream
[24] 2022/12/19 19:03:17.431460 [INF] 
[24] 2022/12/19 19:03:17.432118 [INF] ---------------- JETSTREAM ----------------
[24] 2022/12/19 19:03:17.432659 [INF]   Max Memory:      5.09 GB
[24] 2022/12/19 19:03:17.433163 [INF]   Max Storage:     25.92 GB
[24] 2022/12/19 19:03:17.433633 [INF]   Store Directory: "n1/jetstream"
[24] 2022/12/19 19:03:17.434180 [INF] -------------------------------------------
[24] 2022/12/19 19:03:17.434424 [INF] Starting JetStream cluster
[24] 2022/12/19 19:03:17.434428 [INF] Creating JetStream metadata controller
[24] 2022/12/19 19:03:17.434893 [INF] JetStream cluster bootstrapping
[24] 2022/12/19 19:03:17.435112 [INF] Listening for client connections on 0.0.0.0:4222
[24] 2022/12/19 19:03:17.435217 [INF] Server is ready
[24] 2022/12/19 19:03:17.435415 [INF] Cluster name is hub
[24] 2022/12/19 19:03:17.435435 [INF] Listening for route connections on 0.0.0.0:6222
[24] 2022/12/19 19:03:17.435837 [ERR] Error trying to connect to route (attempt 1): dial tcp 127.0.0.1:6223: connect: connection refused
[24] 2022/12/19 19:03:17.435925 [INF] 127.0.0.1:6222 - rid:7 - Route connection created
[24] 2022/12/19 19:03:17.436010 [INF] 127.0.0.1:33582 - rid:8 - Route connection created
[24] 2022/12/19 19:03:17.436042 [ERR] Error trying to connect to route (attempt 1): dial tcp 127.0.0.1:6224: connect: connection refused
[24] 2022/12/19 19:03:17.436102 [INF] 127.0.0.1:6222 - rid:7 - Router connection closed: Duplicate Route
[24] 2022/12/19 19:03:17.436182 [INF] 127.0.0.1:33582 - rid:8 - Router connection closed: Duplicate Route
[24] 2022/12/19 19:03:17.535424 [WRN] Waiting for routing to be established...
Starting n2...
[33] 2022/12/19 19:03:18.428371 [INF] Starting nats-server
[33] 2022/12/19 19:03:18.428393 [INF]   Version:  2.9.10-beta.1
[33] 2022/12/19 19:03:18.428396 [INF]   Git:      [not set]
[33] 2022/12/19 19:03:18.428398 [INF]   Cluster:  hub
[33] 2022/12/19 19:03:18.428415 [INF]   Name:     n2
[33] 2022/12/19 19:03:18.428419 [INF]   Node:     44jzkV9D
[33] 2022/12/19 19:03:18.428421 [INF]   ID:       NCYOQ3RGLUERDDZQ2KUMAW6HNDCZK4EVJ6JEHHGLSIL5T64P7ULG7REO
[33] 2022/12/19 19:03:18.428425 [WRN] Plaintext passwords detected, use nkeys or bcrypt
[33] 2022/12/19 19:03:18.428429 [INF] Using configuration file: n2.conf
[33] 2022/12/19 19:03:18.429372 [INF] Starting JetStream
[33] 2022/12/19 19:03:18.429656 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[33] 2022/12/19 19:03:18.429660 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[33] 2022/12/19 19:03:18.429662 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[33] 2022/12/19 19:03:18.429663 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[33] 2022/12/19 19:03:18.429665 [INF] 
[33] 2022/12/19 19:03:18.429667 [INF]          https://docs.nats.io/jetstream
[33] 2022/12/19 19:03:18.429668 [INF] 
[33] 2022/12/19 19:03:18.429670 [INF] ---------------- JETSTREAM ----------------
[33] 2022/12/19 19:03:18.429674 [INF]   Max Memory:      5.09 GB
[33] 2022/12/19 19:03:18.429676 [INF]   Max Storage:     25.92 GB
[33] 2022/12/19 19:03:18.429678 [INF]   Store Directory: "n2/jetstream"
[33] 2022/12/19 19:03:18.429680 [INF] -------------------------------------------
[33] 2022/12/19 19:03:18.429940 [INF] Starting JetStream cluster
[33] 2022/12/19 19:03:18.429943 [INF] Creating JetStream metadata controller
[33] 2022/12/19 19:03:18.430435 [INF] JetStream cluster bootstrapping
[33] 2022/12/19 19:03:18.430873 [INF] Listening for client connections on 0.0.0.0:4223
[33] 2022/12/19 19:03:18.430991 [INF] Server is ready
[33] 2022/12/19 19:03:18.431078 [INF] Cluster name is hub
[33] 2022/12/19 19:03:18.431097 [INF] Listening for route connections on 0.0.0.0:6223
[33] 2022/12/19 19:03:18.431542 [INF] 127.0.0.1:6223 - rid:7 - Route connection created
[33] 2022/12/19 19:03:18.431618 [INF] 127.0.0.1:32870 - rid:8 - Route connection created
[33] 2022/12/19 19:03:18.431657 [INF] 127.0.0.1:6222 - rid:9 - Route connection created
[33] 2022/12/19 19:03:18.431689 [ERR] Error trying to connect to route (attempt 1): dial tcp 127.0.0.1:6224: connect: connection refused
[33] 2022/12/19 19:03:18.431737 [INF] 127.0.0.1:6223 - rid:7 - Router connection closed: Duplicate Route
[33] 2022/12/19 19:03:18.431816 [INF] 127.0.0.1:32870 - rid:8 - Router connection closed: Duplicate Route
[24] 2022/12/19 19:03:18.431968 [INF] 127.0.0.1:55638 - rid:9 - Route connection created
[24] 2022/12/19 19:03:18.436697 [INF] 127.0.0.1:6223 - rid:10 - Route connection created
[33] 2022/12/19 19:03:18.436869 [INF] 127.0.0.1:32882 - rid:10 - Route connection created
[24] 2022/12/19 19:03:18.436958 [INF] 127.0.0.1:6223 - rid:10 - Router connection closed: Duplicate Route
[33] 2022/12/19 19:03:18.437060 [INF] 127.0.0.1:32882 - rid:10 - Router connection closed: Duplicate Route
[33] 2022/12/19 19:03:18.531331 [WRN] Waiting for routing to be established...
[24] 2022/12/19 19:03:18.531740 [INF] Self is new JetStream cluster metadata leader
[33] 2022/12/19 19:03:18.532046 [INF] JetStream cluster new metadata leader: n1/hub
Starting n3...
[42] 2022/12/19 19:03:19.429220 [INF] Starting nats-server
[42] 2022/12/19 19:03:19.429278 [INF]   Version:  2.9.10-beta.1
[42] 2022/12/19 19:03:19.429295 [INF]   Git:      [not set]
[42] 2022/12/19 19:03:19.429400 [INF]   Cluster:  hub
[42] 2022/12/19 19:03:19.429458 [INF]   Name:     n3
[42] 2022/12/19 19:03:19.429614 [INF]   Node:     BXScrY9i
[42] 2022/12/19 19:03:19.429695 [INF]   ID:       NAVZQ6MTUILVPNR2DPZOCKJKLMWKNIDG27EIYAUUWAT6N26LMTNB4H2R
[42] 2022/12/19 19:03:19.429723 [WRN] Plaintext passwords detected, use nkeys or bcrypt
[42] 2022/12/19 19:03:19.429832 [INF] Using configuration file: n3.conf
[42] 2022/12/19 19:03:19.430304 [INF] Starting JetStream
[42] 2022/12/19 19:03:19.430693 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[42] 2022/12/19 19:03:19.430879 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[42] 2022/12/19 19:03:19.431004 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[42] 2022/12/19 19:03:19.431118 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[42] 2022/12/19 19:03:19.431239 [INF] 
[42] 2022/12/19 19:03:19.431353 [INF]          https://docs.nats.io/jetstream
[42] 2022/12/19 19:03:19.431483 [INF] 
[42] 2022/12/19 19:03:19.431592 [INF] ---------------- JETSTREAM ----------------
[42] 2022/12/19 19:03:19.431714 [INF]   Max Memory:      5.09 GB
[42] 2022/12/19 19:03:19.431827 [INF]   Max Storage:     25.92 GB
[42] 2022/12/19 19:03:19.431944 [INF]   Store Directory: "n3/jetstream"
[42] 2022/12/19 19:03:19.432056 [INF] -------------------------------------------
[42] 2022/12/19 19:03:19.432446 [INF] Starting JetStream cluster
[42] 2022/12/19 19:03:19.432707 [INF] Creating JetStream metadata controller
[42] 2022/12/19 19:03:19.433364 [INF] JetStream cluster bootstrapping
[42] 2022/12/19 19:03:19.434011 [INF] Listening for client connections on 0.0.0.0:4224
[42] 2022/12/19 19:03:19.434344 [INF] Server is ready
[42] 2022/12/19 19:03:19.434399 [INF] Cluster name is hub
[42] 2022/12/19 19:03:19.434559 [INF] Listening for route connections on 0.0.0.0:6224
[24] 2022/12/19 19:03:19.435027 [INF] 127.0.0.1:55642 - rid:11 - Route connection created
[42] 2022/12/19 19:03:19.435034 [INF] 127.0.0.1:6223 - rid:7 - Route connection created
[42] 2022/12/19 19:03:19.435105 [INF] 127.0.0.1:6224 - rid:8 - Route connection created
[42] 2022/12/19 19:03:19.435174 [INF] 127.0.0.1:56228 - rid:9 - Route connection created
[42] 2022/12/19 19:03:19.435281 [INF] 127.0.0.1:56228 - rid:9 - Router connection closed: Duplicate Route
[42] 2022/12/19 19:03:19.435365 [INF] 127.0.0.1:6224 - rid:8 - Router connection closed: Duplicate Route
[33] 2022/12/19 19:03:19.435460 [INF] 127.0.0.1:32894 - rid:11 - Route connection created
[42] 2022/12/19 19:03:19.435723 [INF] 127.0.0.1:6222 - rid:10 - Route connection created
[24] 2022/12/19 19:03:19.438273 [INF] 127.0.0.1:6224 - rid:12 - Route connection created
[42] 2022/12/19 19:03:19.438990 [INF] 127.0.0.1:56236 - rid:11 - Route connection created
[24] 2022/12/19 19:03:19.439532 [INF] 127.0.0.1:6224 - rid:12 - Router connection closed: Duplicate Route
[42] 2022/12/19 19:03:19.440411 [INF] 127.0.0.1:56236 - rid:11 - Router connection closed: Duplicate Route
[42] 2022/12/19 19:03:19.535024 [WRN] Waiting for routing to be established...
[42] 2022/12/19 19:03:19.535713 [INF] JetStream cluster new metadata leader: n1/hub
Cluster healthy?
{"status":"ok"}
[33] 2022/12/19 19:03:20.433114 [INF] 127.0.0.1:6224 - rid:12 - Route connection created
[42] 2022/12/19 19:03:20.433128 [INF] 127.0.0.1:56252 - rid:12 - Route connection created
[33] 2022/12/19 19:03:20.433469 [INF] 127.0.0.1:6224 - rid:12 - Router connection closed: Duplicate Route
[42] 2022/12/19 19:03:20.433884 [INF] 127.0.0.1:56252 - rid:12 - Router connection closed: Duplicate Route
╭───────────────────────────────────────────────────────────────────────────────────────────────╮
│                                       JetStream Summary                                       │
├────────┬─────────┬─────────┬───────────┬──────────┬───────┬────────┬──────┬─────────┬─────────┤
│ Server │ Cluster │ Streams │ Consumers │ Messages │ Bytes │ Memory │ File │ API Req │ API Err │
├────────┼─────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
│ n1*    │ hub     │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ n2     │ hub     │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ n3     │ hub     │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
├────────┼─────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
│        │         │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
╰────────┴─────────┴─────────┴───────────┴──────────┴───────┴────────┴──────┴─────────┴─────────╯

╭────────────────────────────────────────────────────────────╮
│                RAFT Meta Group Information                 │
├──────┬──────────┬────────┬─────────┬────────┬────────┬─────┤
│ Name │ ID       │ Leader │ Current │ Online │ Active │ Lag │
├──────┼──────────┼────────┼─────────┼────────┼────────┼─────┤
│ n1   │ fjFyEjc1 │ yes    │ true    │ true   │ 0.00s  │ 0   │
│ n2   │ 44jzkV9D │        │ true    │ true   │ 0.94s  │ 0   │
│ n3   │ BXScrY9i │        │ true    │ true   │ 0.94s  │ 0   │
╰──────┴──────────┴────────┴─────────┴────────┴────────┴─────╯
[42] 2022/12/19 19:03:21.080177 [INF] JetStream cluster new stream leader for 'APP > events'

 12 / 100 [=======>----------------------------------------------------------------------]    0s
 22 / 100 [===============>--------------------------------------------------------------]    0s
 32 / 100 [=======================>------------------------------------------------------]    0s
 44 / 100 [=================================>--------------------------------------------]    0s
 55 / 100 [==========================================>-----------------------------------]    0s
 67 / 100 [===================================================>--------------------------]    0s
 77 / 100 [===========================================================>------------------]    0s
 89 / 100 [=====================================================================>--------]    0s
 97 / 100 [===========================================================================>--]    0s
100 / 100 [==============================================================================]    0s

[24] 2022/12/19 19:03:21.271601 [WRN] JetStream cluster could not replace peer for stream 'APP > events'
[42] 2022/12/19 19:03:21.272844 [ERR] JetStream being DISABLED, our server was removed from the cluster
[42] 2022/12/19 19:03:21.274569 [INF] Initiating JetStream Shutdown...
[24] 2022/12/19 19:03:21.275310 [WRN] JetStream cluster detected duplicate assignment for stream "events" for account "APP"
[33] 2022/12/19 19:03:21.275814 [WRN] JetStream cluster detected duplicate assignment for stream "events" for account "APP"
[42] 2022/12/19 19:03:21.276930 [INF] JetStream Shutdown

19:03:21 No responders are available
  1 / 100 [------------------------------------------------------------------------------]    0s

[42] 2022/12/19 19:03:21.323273 [INF] Entering lame duck mode, stop accepting new clients
[42] 2022/12/19 19:03:21.323505 [INF] Initiating Shutdown...
[42] 2022/12/19 19:03:21.324358 [INF] 127.0.0.1:6223 - rid:7 - Router connection closed: Server Shutdown
[42] 2022/12/19 19:03:21.324534 [INF] 127.0.0.1:6222 - rid:10 - Router connection closed: Server Shutdown
[33] 2022/12/19 19:03:21.325334 [INF] 127.0.0.1:32894 - rid:11 - Router connection closed: Client Closed
[42] 2022/12/19 19:03:21.326043 [INF] Server Exiting..
[24] 2022/12/19 19:03:21.326255 [INF] 127.0.0.1:55642 - rid:11 - Router connection closed: Client Closed

19:03:21 No responders are available
  1 / 100 [------------------------------------------------------------------------------]    0s

[97] 2022/12/19 19:03:21.377007 [INF] Starting nats-server
[97] 2022/12/19 19:03:21.377027 [INF]   Version:  2.9.10-beta.1
[97] 2022/12/19 19:03:21.377030 [INF]   Git:      [not set]
[97] 2022/12/19 19:03:21.377035 [INF]   Cluster:  hub
[97] 2022/12/19 19:03:21.377037 [INF]   Name:     n3
[97] 2022/12/19 19:03:21.377040 [INF]   Node:     BXScrY9i
[97] 2022/12/19 19:03:21.377042 [INF]   ID:       NBGRCT6CHR3C7GUMDPO6ZWU33JPVCMK2AOOK2X3SUL3FLWQBHIWRIUXU
[97] 2022/12/19 19:03:21.377045 [WRN] Plaintext passwords detected, use nkeys or bcrypt
[97] 2022/12/19 19:03:21.377049 [INF] Using configuration file: n3.conf
[97] 2022/12/19 19:03:21.377598 [INF] Starting JetStream
[97] 2022/12/19 19:03:21.377903 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[97] 2022/12/19 19:03:21.377913 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[97] 2022/12/19 19:03:21.377915 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[97] 2022/12/19 19:03:21.377917 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[97] 2022/12/19 19:03:21.377919 [INF] 
[97] 2022/12/19 19:03:21.377921 [INF]          https://docs.nats.io/jetstream
[97] 2022/12/19 19:03:21.377922 [INF] 
[97] 2022/12/19 19:03:21.377924 [INF] ---------------- JETSTREAM ----------------
[97] 2022/12/19 19:03:21.377929 [INF]   Max Memory:      5.09 GB
[97] 2022/12/19 19:03:21.377932 [INF]   Max Storage:     25.92 GB
[97] 2022/12/19 19:03:21.377934 [INF]   Store Directory: "n3/jetstream"
[97] 2022/12/19 19:03:21.377936 [INF] -------------------------------------------
[97] 2022/12/19 19:03:21.378839 [INF]   Starting restore for stream 'APP > events'
[97] 2022/12/19 19:03:21.379334 [INF]   Restored 100 messages for stream 'APP > events'
[97] 2022/12/19 19:03:21.379388 [INF] Starting JetStream cluster
[97] 2022/12/19 19:03:21.379393 [INF] Creating JetStream metadata controller
[97] 2022/12/19 19:03:21.379936 [INF] JetStream cluster bootstrapping
[97] 2022/12/19 19:03:21.380439 [INF] Listening for client connections on 0.0.0.0:4224
[97] 2022/12/19 19:03:21.380728 [INF] Server is ready
[97] 2022/12/19 19:03:21.381302 [INF] Cluster name is hub
[97] 2022/12/19 19:03:21.381332 [INF] Listening for route connections on 0.0.0.0:6224
[97] 2022/12/19 19:03:21.381829 [INF] 127.0.0.1:56258 - rid:10 - Route connection created
[24] 2022/12/19 19:03:21.382165 [INF] 127.0.0.1:55656 - rid:24 - Route connection created
[33] 2022/12/19 19:03:21.382505 [INF] 127.0.0.1:32900 - rid:17 - Route connection created
[97] 2022/12/19 19:03:21.382805 [INF] 127.0.0.1:6223 - rid:11 - Route connection created
[24] 2022/12/19 19:03:21.383751 [INF] 127.0.0.1:6224 - rid:25 - Route connection created
[97] 2022/12/19 19:03:21.383813 [INF] 127.0.0.1:6222 - rid:12 - Route connection created
[97] 2022/12/19 19:03:21.384079 [INF] 127.0.0.1:6224 - rid:13 - Route connection created
[97] 2022/12/19 19:03:21.384936 [INF] 127.0.0.1:6224 - rid:13 - Router connection closed: Duplicate Route
[97] 2022/12/19 19:03:21.385689 [INF] 127.0.0.1:56258 - rid:10 - Router connection closed: Duplicate Route
[97] 2022/12/19 19:03:21.386473 [INF] 127.0.0.1:56260 - rid:14 - Route connection created
[97] 2022/12/19 19:03:21.386550 [INF] 127.0.0.1:56260 - rid:14 - Router connection closed: Duplicate Route
[24] 2022/12/19 19:03:21.387240 [INF] 127.0.0.1:6224 - rid:25 - Router connection closed: Duplicate Route
[97] 2022/12/19 19:03:21.481327 [WRN] Waiting for routing to be established...
[33] 2022/12/19 19:03:22.413081 [INF] 127.0.0.1:6224 - rid:18 - Route connection created
[97] 2022/12/19 19:03:22.413241 [INF] 127.0.0.1:56276 - rid:15 - Route connection created
[97] 2022/12/19 19:03:22.413452 [INF] 127.0.0.1:56276 - rid:15 - Router connection closed: Duplicate Route
[24] 2022/12/19 19:03:22.413649 [INF] 127.0.0.1:6224 - rid:26 - Route connection created
[33] 2022/12/19 19:03:22.413351 [INF] 127.0.0.1:6224 - rid:18 - Router connection closed: Duplicate Route
[97] 2022/12/19 19:03:22.413838 [INF] 127.0.0.1:56284 - rid:16 - Route connection created
[24] 2022/12/19 19:03:22.413934 [INF] 127.0.0.1:6224 - rid:26 - Router connection closed: Duplicate Route
[97] 2022/12/19 19:03:22.414262 [INF] 127.0.0.1:56284 - rid:16 - Router connection closed: Duplicate Route
[24] 2022/12/19 19:03:22.533536 [WRN] JetStream cluster detected duplicate assignment for stream "events" for account "APP"
[33] 2022/12/19 19:03:22.534301 [WRN] JetStream cluster detected duplicate assignment for stream "events" for account "APP"
╭────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                         JetStream Summary                                          │
├────────┬─────────┬─────────┬───────────┬──────────┬─────────┬────────┬─────────┬─────────┬─────────┤
│ Server │ Cluster │ Streams │ Consumers │ Messages │ Bytes   │ Memory │ File    │ API Req │ API Err │
├────────┼─────────┼─────────┼───────────┼──────────┼─────────┼────────┼─────────┼─────────┼─────────┤
│ n1*    │ hub     │ 1       │ 0         │ 100      │ 4.5 KiB │ 0 B    │ 4.5 KiB │ 1       │ 0       │
│ n2     │ hub     │ 1       │ 0         │ 100      │ 4.5 KiB │ 0 B    │ 4.5 KiB │ 0       │ 0       │
│ n3     │ hub     │ 1       │ 0         │ 0        │ 0 B     │ 0 B    │ 0 B     │ 0       │ 0       │
├────────┼─────────┼─────────┼───────────┼──────────┼─────────┼────────┼─────────┼─────────┼─────────┤
│        │         │ 3       │ 0         │ 200      │ 9.0 KiB │ 0 B    │ 9.0 KiB │ 1       │ 0       │
╰────────┴─────────┴─────────┴───────────┴──────────┴─────────┴────────┴─────────┴─────────┴─────────╯

╭────────────────────────────────────────────────────────────╮
│                RAFT Meta Group Information                 │
├──────┬──────────┬────────┬─────────┬────────┬────────┬─────┤
│ Name │ ID       │ Leader │ Current │ Online │ Active │ Lag │
├──────┼──────────┼────────┼─────────┼────────┼────────┼─────┤
│ n1   │ fjFyEjc1 │ yes    │ true    │ true   │ 0.00s  │ 0   │
│ n2   │ 44jzkV9D │        │ true    │ true   │ 1.89s  │ 0   │
│ n3   │ BXScrY9i │        │ true    │ true   │ 1.89s  │ 0   │
╰──────┴──────────┴────────┴─────────┴────────┴────────┴─────╯

Recording

Note, playback is half speed to make it a bit easier to follow.