Taking one node travelling: what running a sovereign AI stack on DGX Spark actually taught us

We work on an energy infrastructure project where energy security and data privacy are not preferences. They are constraints with teeth. Grid data, commercial terms and client identities cannot ride through third-party AI APIs. So the models come to us: an on-prem cluster serving frontier-class work to the people who need it, with no packets leaving the building.

The cluster has tiers: a two-node NVIDIA DGX Spark pair (GB10 Grace Blackwell, 128 GB unified memory each) serving a large MoE model with tensor parallelism across both boxes; a Blackwell workstation as the latency tier; a 4x RTX 3090 box for image generation and burst work. A gateway routes every consumer to the right model. The Sparks have a second job that matters just as much: they are the development tier for client delivery, because they run the same NVIDIA stack we deploy into production. More on that later.

Which raised a question nobody had a good answer for: what happens when you need to take the work on the road?

Consultants travel. Sites get visited. Sometimes there is a data room with no internet and a hard question that needs a good model, offline, that afternoon. Cloud APIs are off the table by policy. So the plan: take one node of the production pair out of the rack and make it a self-contained inference box: a 27B dense model as the orchestrator, a small MoE as the fast sub-agent tier. Run it from a hotel desk or a site office, then slot it back into the pair when home.

Simple idea. What followed was two of the most instructive days of infrastructure work we have had in years, and a set of lessons that apply to anyone running GB10-class hardware as production infrastructure rather than as a demo bench.

Lesson one: unified memory has no seatbelts

The first attempt at traveller testing was also our worst outage of the year, and it is worth telling on ourselves.

A GB10 presents 128 GB of unified memory: one pool, shared between CPU and GPU, with no VRAM boundary. That is the machine's superpower and its trap. One of our AI agent sessions (we let agents operate the cluster; more on that philosophy another time) tried to bring up a test inference engine on a node that was actively serving as one half of the tensor-parallel pair. The node had roughly 3 GB of headroom. The test engine streamed 28 GB of model weights through CPU memory on its way to the GPU.

The box spiralled into out-of-memory thrash, the kernel's OOM killer started choosing victims, and it chose the production rank. On a discrete-GPU box you get VRAM errors and a failed launch. On unified memory, a memory-pressure event anywhere can kill the workload you actually care about. The pair fell over, and because tensor-parallel pairs die ugly, the head node sat wedged retrying a dead rank.

The rule we now operate by: modes are exclusive. A node is either a pair worker or a standalone box, never both, and the transition is a scripted, one-command mode switch rather than an ad-hoc experiment. Leaving the desk looks like this:

# stop pair orchestration (non-blocking; the unit is Type=oneshot)
ssh spark-head 'systemctl --user stop vllm-pair.service'

# tear containers down on BOTH nodes (down, not stop: restart policies
# resurrect anything you merely stopped)
ssh spark-head   'cd ~/pair && docker compose down'
ssh spark-worker 'cd ~/pair && docker compose down'

# enable + start the traveller units, then verify with a real completion
ssh spark-worker 'systemctl --user enable --now traveller-thinker.service'

The return procedure is the mirror image, with the traveller units disabled first so a desk reboot can never race both workloads into the same memory pool. That discipline, boring, scripted and exclusive, is what finally made the whole concept reliable. Every failure before it traced back to violating it.

Lesson two: 200 GbE fabrics are woodwork, not software

When the pair would not come back up, we spent hours in the software before touching the hardware. The symptoms were pure config vocabulary: NCCL ibv_modify_qp failures, “No data available”, GID index errors. Everything looked misconfigured. The truth was physical.

Three separate fabric faults, stacked:

1. RoCE GID tables reshuffle. The RDMA layer indexes its address table by position, and the positions are not stable across reboots, power cuts or link events. Our startup config pinned a GID index that had been correct for forty hours of flawless serving, until an unclean power event shuffled the table and the pin pointed at an empty slot. Every hard power event on these boxes deserves a GID table check before anything else gets debugged:

cat /sys/class/infiniband/rocep1s0f1/ports/1/gids/*
cat /sys/class/infiniband/rocep1s0f1/ports/1/gid_attrs/types/*

1. Hot-plugging a QSFP cable can silently disable the port. Reseating a DAC “hot” left the interface administratively down with its IP-based RDMA addresses withdrawn, while pings to the peer still worked, because ICMP and RDMA disagree about what “up” means. Re-applying the network connection brought the IP back; the RDMA addresses regenerated at a different index (see fault one).
2. A half-seated QSFP reports “No cable” while looking perfect. The final fault was a connector that had not clicked home. ethtool said “No cable”; eyes said “obviously fine”. These connectors need positive force and an audible click. After a power event, trust ethtool, not your eyes.

The debugging heuristic we would offer anyone: when IP works but RDMA dies on a Spark pair, go physical early. Reseat both ends properly, cycle power on the NIC (a hard power-off rather than a reboot: some firmware state survives warm restarts), and then read the config. We burned half a day proving the software was innocent.

One more for the collection: the same day, an unrelated workstation GPU logged Xid 79, “GPU has fallen off the bus”, twice in six hours. When a driver error says “node reboot required”, believe it, and when hardware errors recur across reboots, stop debugging software and reseat the card.

Lesson three: the aarch64 long tail is real

DGX Spark is ARM. Most AI infrastructure documentation assumes x86, and the gaps bite in small, stupid, time-consuming ways. A checklist from our week:

• Official vLLM wheels install cleanly on aarch64, but the ecosystem around them has sharp edges. We hit {env:VAR} template interpolation that silently fails in one tool's ARM build while working on x86, and process spawns that require absolute binary paths (/usr/bin/npx, not npx) only on ARM.
• Node.js version gates arrive via the back door: an Azure tooling package quietly required Node 22 or newer while the distro ships 18.
• First-generation JIT compilation on ARM is measured in minutes, not seconds. A model server that answers health checks but times out on its first completion is not broken. It is compiling. Budget ten minutes of patience before declaring failure.

None of these are hard. All of them cost thirty minutes each if you do not know they are coming.

What the traveller actually looks like

Here is the part where we correct our own first draft, because it matters. The naive build, a 27B dense model in FP8, ran at 7.1 tok/s single stream. That is not a configuration problem, it is arithmetic: the GB10 has roughly 273 GB/s of unified memory bandwidth and the FP8 weights are 29 GiB, which puts the dense ceiling near 9.4 tok/s. We also tried MTP speculative decoding on the dense build and it regressed to 5.3 tok/s. On low-bandwidth unified memory, that class of drafter hurts more than it helps.

The shipped traveller is a different animal. The trick is not a faster model, it is less data per token plus a drafter that actually fits the memory profile:

• Thinker tier: a W4A4 compressed-tensors build of a 27B abliterated model (20.6 GiB on disk, 22.5 GiB loaded) with a DFlash2 speculative decoder at k=7. Measured on our box: 33 to 38 tok/s on code and 22.9 tok/s on prose. Mean draft acceptance length 5.71 of 7.
• Doer tier: Nemotron 3.5 Lightning, a 30B-total MoE with 3B active parameters, in NVFP4. Measured: 61 to 69 tok/s. That is workstation-class speed on portable hardware, and it is the tier an agent loop actually spends most of its time in.

The drafter config, verbatim from the unit file:

vllm serve /models/SuperQwen3.8-27b-abliterated-NVFP4 \
  --served-model-name superqwen-27b --port 8003 \
  --max-model-len 262144 --kv-cache-dtype fp8 \
  --speculative-config '{"method":"dflash",
    "num_speculative_tokens":7,
    "model":"/models/Qwen3.8-27B-DFlash2"}'

Both tiers co-reside in the same 121 GiB pool at 111 GiB, about 92 per cent utilisation, and we stress-tested them with simultaneous benchmark runs: 67 tok/s on the doer and 36 tok/s on the thinker, at the same time, stable. That is not room to spare, it is a full house that works. The whole mode switch, from pair worker to travel box and back, is two commands in each direction.

Two honest caveats for the record. The 262K context window has never been exercised end to end on the doer tier. And the dense FP8 baseline remains the fallback build precisely because it is simple: if something in the speculative stack misbehaves, 7 tok/s that cannot break is the safety net.

The dev box that de-risks production

Here is the part we underappreciated until the traveller forced the issue: a GB10 is not trying to be the fastest box in the rack. It is not. Our workstation serves the same model at two to three times the traveller's speed, and the pair serves a model four times the size. Raw speed was never the Spark's job.

The Spark's job is being the portable tier of an NVIDIA pipeline. It runs the same CUDA, the same vLLM, the same containers as the iron our clients deploy into. So the client workflow is: prototype on a local Spark, with no cloud costs, no queueing behind other tenants and no data leaving the building; prove the workload; then promote the exact same build into the company's production NVIDIA infrastructure with real assurance, because it already ran on the target stack. That is not a hypothetical pipeline. We are doing it with clients now, and the traveller is the same idea with the power cable unplugged and the box in a laptop bag.

A footnote from the bench session: the multimodal path works too. The thinker build handled a document task (the “Pink” test) without ceremony, which matters when the data room contains scans rather than text.

The failover we did not plan for

Here is the part that surprised us. Within hours of the traveller stack coming up, the workstation that normally serves the cluster's default model threw its second hardware fault of the day and went dark. The failover was almost embarrassingly simple: point the gateway's default-model route at the traveller box, restart the gateway, done. The cluster's default model was back in under a minute, served from a machine that had caused an outage that same morning.

A portable node is secretly two pieces of infrastructure: your travel kit and a standing failover node. If you are building a pair, the economic case for the second box includes both roles.

Where this goes next: the Qwen3.8-Flash-Next question

The obvious objection to all of the above: the new Qwen3.8-Flash-Next class of models (huge total parameter counts, tiny active parameters, big retrieval tables) changes the arithmetic. A 180B-total, 6B-active model with a 51-billion-parameter n-gram table is exactly the shape a 128 GB unified-memory box wants: weights that fit with headroom, active parameters small enough to beat the bandwidth wall, and a retrieval table that offloads to storage rather than living in memory. We have the weights staged on the NAS already, waiting for vLLM support to land, and the pair rather than the solo box is its natural home.

Our instinct, having spent two days at this coalface: the models will come, and the box is ready for them. What most teams will not be ready for is everything else: the mode discipline, the fabric behaviour, the ARM long tail. That is the actual moat of competence. The model is the easy part.

The short version

If you are running GB10-class hardware as production infrastructure:

1. Modes are exclusive. Unified memory has no isolation; never co-host workloads on a node that has a job. Script the transitions.
2. Fabric faults masquerade as config faults. Reseat, power-cycle, then debug. Check GID tables after every power event.
3. Trust ethtool over your eyes on QSFP seating.
4. Budget ten minutes for first-generation JIT on aarch64 before calling anything broken.
5. Expect the ARM long tail. Small incompatibilities, each cheap, collectively a day.
6. Beat bandwidth with quantisation and drafting, not hope. Dense FP8 gave us 7.1 tok/s. W4A4 plus a DFlash2 k=7 drafter gave 33 to 38. MTP regressed. Measure, then ship the winner.
7. A portable node is also a failover node, and a dev box is also a de-risking pipeline. Design the gateway so moving the default route is a one-line change. One day you will be glad it is.

Sovereign AI on hardware you can carry in a laptop bag is not a demo any more. It is a slightly feral but entirely real piece of critical infrastructure. With a bit of disciplined operational work, it is ready for the site visit, the data room and the power cut.

Throughput figures are our own measurements on our own hardware during the week of 24 August 2026, from logged bench sessions with full provenance in our internal bench notes. Reference figures such as memory bandwidth are NVIDIA published specifications. We are MarStack: we build on-prem AI infrastructure for energy-sector work where the data cannot leave the building.