Operations

This page is about running a server day to day. How to install and start it is in Installation, and the exit codes a supervisor sees are there too.

Remote access#

There are three ways in from outside the process, all configured under [network] in config/server.toml.

Management API#

REST under /v1, and a WebSocket console. Every route needs a bearer token.

[network.api]
enabled = true
bind = "127.0.0.1"
port = 25580
tls = false
tls-cert = ""
tls-key = ""
max-failures = 5
lockout-seconds = 30
requests-per-minute = 120
max-connections = 32
max-connections-per-address = 8
max-body-bytes = 65536
idle-seconds = 60

The token is generated on the first start and kept in .basalt/remote-token, readable only by its owner. The log says Management API token generated and stored in .basalt/remote-token. once, and never prints the token. Delete the file and restart to replace it.

TOKEN=$(cat .basalt/remote-token)
curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:25580/v1/status

Errors are { "error": "..." }: 401 for a missing or wrong token, 404 for an unknown route, 405 for the wrong method, 413 for a body over the limit, 429 with Retry-After for a lockout or when requests-per-minute is exceeded. Chunked request bodies are not supported.

On the WebSocket the server sends every log line as it is written, as {"type": "log", "line": "..."}. Send a command either as plain text or as {"command": "...", "id": ...}; the answer is {"type": "result", "id": ..., "cancelled": ..., "output": "..."} with your id echoed. A browser cannot set a header on a WebSocket, so the token may be passed as the subprotocol bearer.<token> instead.

A command from the API runs as the source Api, with all permissions, and is logged: Remote command from <address> (api): <command>.

RCON#

[network.rcon]
enabled = false
bind = "127.0.0.1"
port = 25575
password = ""
max-failures = 5
lockout-seconds = 30
max-connections-per-address = 8

RCON is plaintext: the password and every command cross the network unencrypted, so keep it on loopback or behind a VPN. It refuses to start with an empty password:

RCON stays off: [network.rcon] has enabled = true but an empty password, and an open console with no password is never safe. Set a password in config/server.toml.

When it is on, the log warns that it is plaintext at every start. A failed login is logged as Failed RCON login from <address>., and repeated failures lock the address out, the lockout doubling each time up to a day. max-connections-per-address is 1 to 16.

The API and RCON cannot share a port: [network.rcon] and [network.api] cannot share one port. A key written in these two tables that the server does not know stops the start: unknown key "x" in [network.rcon].

Query#

[network.query]
enabled = false
port = 25565

Answers GameSpy 4 status queries over UDP, IPv4 only. It reports the first line of the MOTD of the default server list profile as the server name, the default world's name, the version, the port, the player counts and the names. It may use the same number as the Java port, since one is TCP and the other UDP.

Logging#

[logging]
level = "info"
log-ips = true
file = true
keep-days = 14
color = "auto"

On every start the previous latest.log is compressed into .basalt/logs/archive/<date>-<n>.log.gz and archives older than keep-days are deleted. Rotation happens at start: a server that stays up for months writes one latest.log for all of it. The logging settings are read once at start, so changing them needs a restart.

To follow a problem, set level = "debug" while you reproduce it and put it back after.

Performance#

The targets are 20 ticks per second, steadily, with players moving all over the world, and almost no CPU when nobody is online. Basalt measures a tick against a 50 ms budget.

What to watch#

The number to react to is the 95th percentile and the maximum of the tick time, not the mean: players feel the tick that was slow, not the average one. A mean close to 50 ms means the server is out of headroom.

With nobody online for pause-when-empty-seconds (60 by default) the server stops ticking and only serves connections; the log says Server empty for 60 seconds, pausing and Server resumed. Set it to 0 to never pause.

Knobs#

In order of how much each usually matters:

generation-metrics in [performance] counts noise evaluations and per-phase timings while generating chunks and logs a report once the startup chunks are ready. It also forces startup generation onto one thread, so leave it off unless you are measuring.

The generation threads are sized from the machine and have no setting; they yield to the tick, so the tick stays at its rate while the world generates.

The watchdog#

watchdog-seconds in [performance] (60 by default; 0 turns it off) is the longest one tick may take before the server decides it has crashed. It logs the following and exits:

A single server tick took 63.2 seconds (should be max 0.05); considering it to be crashed, the server will forcibly shut down.

The process exits with code 1 without saving, so run the server under a supervisor that restarts it. A plugin or a data pack function that never returns is the usual cause.

Profiling#

Reports are JSON files in .basalt/timings/, named after the time. A report holds the server's version, the tick rates and durations, counts of what is loaded (chunks, worlds, caches), CPU use and memory, and a list of sections with the calls, total, average and maximum time of each. A section nobody called is absent: if the report has no sections, start measuring, let the server do the slow thing, then report. /timings report says so when that is the case.

Backups#

Everything the server writes is in its directory:

To copy a running server consistently:

/save-all flush
/save-off
(copy the files)
/save-on

/save-off stops automatic saving so the files hold still while you copy; do not forget /save-on. players.db is a database in write-ahead mode: while the server runs, copy it together with players.db-wal and players.db-shm, whichever exist. The simplest consistent backup is the one taken after /stop.

Player data deliberately lives outside the worlds, so a world can be restored without touching anyone's inventory, and the reverse. See Worlds for restoring one.

Hardening#

• Only open what you use. The ports are listed in Installation. The API, RCON and query ship loopback or off.
• Keep the management API on loopback and put a reverse proxy with TLS in front of it, or set tls = true with a certificate. The server warns at start when the API is reachable beyond loopback without TLS.
• Prefer the API to RCON. RCON is plaintext. If you need it, bind it to loopback or a VPN and give it a long password.
• Keep online-mode = true unless a proxy forwards your players, and then see Running behind a proxy to make the backend unreachable except through it.
• Give out permissions narrowly. Nodes like basalt.command.stop, basalt.command.op and basalt.command.group are the whole server. Build moderator groups from named nodes rather than *, and leave out basalt.admin.broadcast unless they should count as operators. See Groups and permissions.
• Limit abuse at the edge. connection-throttle-ms and max-connections-per-address in [network.java] slow a flood of logins from one address (loopback is never throttled), and limits.toml bounds packets per second and chat or command spam.
• Restrict names with username-pattern (a regular expression every player name must match) when you want only a known set of characters.
• Whitelist with whitelist = true and enforce-whitelist = true so that removing a name also removes the player.
• Plugins get what their [plugins.allow] entry grants and nothing more: packets, network and commands are the three powers, and a plugin that asks for one that is not listed loads without it. Only list what you have read.
• Back up the proxy key and the API token like any secret, and never commit config/ with a secret, a password or a token in it.
• Run as an unprivileged user, as in the systemd example in Installation.

Troubleshooting#

When something is wrong the log says what, and the first line to read is the one with ERROR or WARN. These are the messages you will actually meet.

The server will not start#

A data pack that fails validation does not stop the start: the server logs the errors and falls back to vanilla's data alone. Start with --safeMode to do that on purpose.

Players cannot join#

Commands and text#

• A player without the node for a command is told it does not exist, as if it were mistyped. Check Groups and permissions, and /reload if you just edited groups.toml.
• An edit to commands.toml [commands.*] or [messages] does nothing until the server restarts. /reload re-reads groups.toml, the server list, the tab list, the sidebar, the boss bar, animations, messages and limits.
• A placeholder that shows as %basalt_...% could not be filled in. The log explains why once (... cannot be filled in here: ... or ... is not a placeholder this server knows.). Test it with /placeholder parse me <text>.
• Condition '...' of server list profile 'x' was skipped: ... means the profile's when used a placeholder that has no value in a ping. Use the protocol, host and ip placeholders there.
• Server list icon ... not found or ... is not a 64x64 PNG means the icon file is not where icon says, relative to config/.
• Boss bar profile 'x' in config/boss_bar.toml has an unknown colour 'y'; using white. is only a warning. The same goes for an unknown overlay.

The server is slow#

1. /tick query and /timings: is the tick over budget all the time, or does it spike?
2. If it spikes, start /timings start, wait for the spike, /timings report, and read the sections that cost the most.
3. If the cost is world ticking, lower simulation-distance. If it is chunk sending or generation, lower view-distance or set chunk-send-rate.
4. If it is a plugin, /plugins shows off after N in a row for one that has failed repeatedly and was switched off by the server, and N error(s) for one that is failing but running. A plugin that overruns its budget fails alone, it does not stall the tick.
5. A tick of many seconds is the watchdog's business, see above.