Troubleshooting
The failure modes a cell actually has.
solitary doctor answers the host-level half of this page — Lima, the hypervisor, memory, disk, the config tree and a proxy the cells will not inherit — without starting anything. Start there when a cell will not come up at all, rather than when a cell that runs is behaving oddly. See doctor checks the host.
A restricted cell cannot build or clone
An allow list has to be complete. A restricted cell cannot pull its own image unless the registry is listed, and a build cannot install packages unless its package sources are. What you see is a name that does not resolve:
dial tcp: lookup production.cloudfront.docker.com on 127.0.0.1:53: no such hostThe name in the error is the entry to add. The dashboard's traffic view (t) shows the same thing live, and a refused line names it directly.
Changing the list takes effect when the machine next starts:
solitary down my-cell
solitary up my-cellA machine has to fit the host
On Linux, a guest's entire memory is a file on /dev/shm, which is usually half of RAM. A machine asking for more than that filesystem holds boots, reports itself running, and then dies the moment it touches enough pages — the process stays alive and Lima still calls it running, so every command hangs instead of failing.
up refuses to create such a machine and says what to lower vm.memory to:
solitary: this machine asks for 16.0GiB of memory, but /dev/shm on this host holds 7.8GiB, and that is where the guest's memory lives
A machine larger than that starts, reports itself running and then dies with no error
Lower vm.memory to 7.8GiB or less, or raise /dev/shmIt also warns — without refusing — when a machine fits the filesystem but not the space free on it right now. Stop another cell if it then fails to start.
This does not apply on macOS, where the hypervisor does not back guest memory with a file.
A cell that stops answering
Solitary reports more than running and stopped:
| State | Means |
|---|---|
uninitialized | The cell is defined, but no machine was ever created for it. |
stopped | The machine exists but is not running. |
running | The machine is up and the container inside it is running. |
degraded | The machine is up, but its container is not. |
busy | The machine answers, but its container's state took too long to read. |
unreachable | Lima considers the machine running, but nothing inside it answers. |
broken | Lima reports the machine as broken. |
orphaned | The machine is here, but the definition it came from is gone. |
A command inside a machine gets 30 seconds. One that runs out of time is followed by a trivial one, and the answer to that decides which of the two states you see. busy is a machine that still answers but is slow — fresh from boot, its disk can stay busy for minutes, and podman waits as long as a build holds its lock. It clears on its own; leave it be.
unreachable is a machine that does not answer even that, and exists so that a hung machine fails your commands instead of blocking them. solitary down then up recovers it without touching the disk.
A degraded cell has lost its container, usually because command exited — it must not exit; it is the container's life. up starts a replacement.
An orphaned machine is what a deleted or renamed cell directory leaves behind, holding its disk with nothing left to name it. solitary rm --orphans discards them all; solitary init <name> gives one a definition again. doctor reports them too.
A tunnel that will not come up
Three things a WireGuard configuration must not have, all refused when the cell is read rather than at boot:
- A missing
Endpoint. There is nothing to connect to, and nothing for the firewall to allow. - A
DNS =line. A cell resolves through its own resolver; name the address undernetwork.resolversinstead. - An IPv6
Endpoint. The cell's tunnel firewall enforces IPv4 egress and refuses an IPv6 peer. Use the provider's IPv4 endpoint.
If the tunnel is configured and down, the cell reaches nothing at all — that is the kill switch working, not a separate fault. The dashboard's VPN row says so directly. See routing a cell through a VPN.
A tunnel that worked and then stopped
up, no recent handshake in the dashboard, and a cell that reaches nothing — the kill switch doing its job on a tunnel that is no longer carrying anything. The usual cause is the peer's address moving out from under a tunnel that resolved it once at boot.
The machine watches for this and restarts the tunnel after three minutes of silence, and solitary up brings up a tunnel that has stopped answering. A cell whose machine has not booted since that watchdog was added does not have it yet, so if it keeps happening, restart the cell once:
solitary down <cell> && solitary up <cell>What it does is described under staying connected.
A change to vm, ports or network that did not take
Those three are read when the machine boots, and Lima cannot apply them to a running one, so up warns rather than silently ignoring them:
Warning: the machine settings for "claude" changed since it started.
vm, ports and network are read when the machine boots, so the running
cell still uses the old ones. To apply the change:
solitary down claude && solitary up claudeStopping and starting applies them. The disk, the cell's home, and the secrets are untouched.
Removing a setting applies the same way, and applies fully: a machine that was restricted or tunnelled has its resolver, firewall and tunnel taken back out at the next boot, so what the guest enforces is what cell.yaml says.
A smaller vm.disk than the machine has
A disk grows at the next boot and never shrinks — Lima has no way to give the space back. So a vm.disk below what the machine already carries is refused before anything is written:
solitary: cell "vscode" asks for a 20.0GiB disk, but its machine already has 50.0GiB, and a disk cannot be shrunk.
Set vm.disk to 50GiB or more in ~/.config/solitary/cells/vscode/cell.yaml,
or discard the machine and everything on it with 'solitary rm vscode' and run up againBoth ways out are real ones. Raising the setting keeps the machine and everything on it, and costs nothing: a disk is allocated sparsely, so a larger one does not take more space than it holds. rm then up gets a machine built to the smaller size and discards the disk, the cell's home with it.
This bites hardest on a cell that is not the one you think: a cell is named by its directory, so a definition installed under a name a previous cell used is backed by that cell's machine. solitary doctor reports both — the mismatch, and any machine no cell names any more.
A change to vm.provision
The one setting a restart cannot apply, because the old script has already run:
Warning: vm.provision for "claude" changed since its machine was provisioned.
The new script runs at the next start, but what the old one did is
already on the machine's disk and nothing undoes it. For a machine
built by this script alone — discarding the disk, and the cell's home
with it:
solitary rm claude && solitary up claudeThe dashboard reports the same thing in the cell's detail view. Editing the script is usually fine — the new one runs on top, and adding a package is not undone by anything. Removing or replacing what an earlier script did is what needs the rebuild. See changing a cell.
Permission denied inside a cell
A cell runs everything as root by default, and there a global npm install -g, an apt-get install and a write to /usr/local all work. Permission errors mean something in the cell is running as a user that is not root — the login user of an sshd or an editor server the image serves.
That user needs two things, one from each side. The definition names them, so up gives them the cell's home rather than leaving it root's:
user: cellThe image gives them the toolset: sudo for the system, and ownership of the directory node lives in, since npm install -g writes into it. Neither weakens the cell — the machine is the boundary, not the container. See the user inside a cell.
Changing user replaces the container at the next up; nothing in the home is touched.
Work that disappeared
Work belongs in /home/cell, which lives on the machine's disk rather than in the container. It survives a new image, a stop and start, and anything a tool installs into the home directory.
rm is what discards it, and rm means destroy the machine. No host filesystem is mounted or synced back automatically: anything you would be upset to lose belongs either in a git remote, in the outbox, or in secrets, which lives on the host and is passed in on every start.