Bonsai Ternary LLM on the CPU — no GPU at all

An 8B model with zero GPU

The point of this guide isn’t speed — it’s independence from the GPU. Bonsai 8B loads into about 2.5 GB of RAM and runs on the CPU alone: no NVIDIA driver, no CUDA install, no VRAM budget, no -ngl. If a machine can run llama.cpp, it can run this — a headless server, a laptop, a NAS, a cheap VPS, a workstation whose GPU is busy with something else.

The trick is ternary quantization: every weight is stored as just −1, 0, or +1 (1.58 bits). That collapses the model to an eighth of its FP16 size and turns the heavy matrix math into add-only operations — no per-weight multiply. On a CPU that pays off most during prefill (reading your prompt), where a Ryzen 3900X hits ~440 tok/s, competitive with a modern laptop GPU.

The honest catch is generation. Producing new tokens one at a time is memory-bandwidth-bound, not compute-bound, so a desktop CPU lands around 3 tok/s for the 8B. That’s fine for batch jobs, background summarization, scripted pipelines, and drafting where you don’t sit and watch — and too slow for snappy interactive chat. If you need real-time typing speed, a GPU is still the answer. If you need an 8B model to exist at all on a GPU-less box, this is how. The FAQ explains why prefill flies and generation crawls.

Three steps, no GPU anywhere

curl -sL https://ndgold.com/guides/bonsai-llm/setup.sh | bash

1. Build the Prism llama.cpp fork (CPU-only)

The native ternary kernel (GGUF Q2_0 with g128 grouping) is not in mainstream llama.cpp yet — you need the Prism fork to load Q2_0. Build it with -DGGML_CUDA=OFF so the binary is pure CPU with no CUDA runtime to chase down later.

git clone --depth 1 -b prism https://github.com/PrismML-Eng/llama.cpp.git
cd llama.cpp
cmake -B build-cpu -DGGML_CUDA=OFF -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build-cpu -j"$(nproc)" --target llama-cli llama-bench llama-server

-DGGML_NATIVE=ON lets the compiler target your CPU’s exact instruction set (AVX2 on the Ryzen), which the ternary kernel leans on. Confirm the build has no GPU baggage: ldd build-cpu/bin/llama-cli | grep -c "not found" should print 0. -j"$(nproc)" uses every core — drop to -j4 if you want to keep the machine responsive during the build.

2. Download a model

mkdir -p ~/models/bonsai
curl -L --progress-bar -o ~/models/bonsai/Ternary-Bonsai-8B-Q2_0.gguf \
  "https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf/resolve/main/Ternary-Bonsai-8B-Q2_0.gguf?download=1"

mkdir -p ~/models/bonsai
curl -L --progress-bar -o ~/models/bonsai/Bonsai-8B-Q4_0-lossless.gguf \
  https://huggingface.co/Minarut/Ternary-Bonsai-8B-GGUF-llamacpp-compatible/resolve/main/Ternary-Bonsai-8B-Q4_0-lossless.gguf

3. Run it (CPU-only)

# One-shot prompt, 12 threads, no GPU layers
./build-cpu/bin/llama-cli \
  -m ~/models/bonsai/Ternary-Bonsai-8B-Q2_0.gguf \
  -ngl 0 -t 12 -fa 1 \
  -p "Explain ternary (1.58-bit) neural networks in 3 sentences." \
  -n 150 -e

# Interactive chat (multiline, history, /regen, /clear)
./build-cpu/bin/llama-cli \
  -m ~/models/bonsai/Ternary-Bonsai-8B-Q2_0.gguf \
  -ngl 0 -t 12 -fa 1 -c 4096

No nvidia-smi to check here — watch htop instead. During prefill you’ll see all 12 threads light up; during generation the load drops to roughly one busy core, which is exactly why new tokens come out slowly. Set your thread count to your physical core count, not the SMT/thread count.

Use it day-to-day: an OpenAI-compatible API

Because generation is slow, the most practical setup is to run Bonsai as a persistent local server and talk to it from your tools — fire off a request, let it work, collect the result. llama-server exposes a full OpenAI-compatible HTTP API and ships with a built-in web chat UI.

Start the server

./build-cpu/bin/llama-server \
  -m ~/models/bonsai/Ternary-Bonsai-8B-Q2_0.gguf \
  -ngl 0 -t 12 -fa 1 -c 4096 \
  --host 0.0.0.0 --port 8080
# → "server listening on http://0.0.0.0:8080"

Open http://localhost:8080/ for the built-in web UI — chat history, parameter tuning, and Markdown rendering, with the chat template auto-detected from the GGUF metadata. --host 0.0.0.0 makes it reachable from other machines (see remote access below); drop it to keep the server local-only.

Call the API

Any tool that speaks the OpenAI API — LangChain, Continue.dev, your own scripts — can point at it. Standard chat completions:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Ternary-Bonsai-8B",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What makes ternary neural networks special?"}
    ],
    "temperature": 0.7,
    "max_tokens": 150
  }'

Add "stream": true to get token-by-token server-sent events instead of waiting for the whole response — useful when generation is slow and you want output as it arrives.

# Python — point any OpenAI client at the local server
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="Ternary-Bonsai-8B",
    messages=[{"role": "user", "content": "Hello!"}],
    max_tokens=100,
)
print(resp.choices[0].message.content)

Keep it running (systemd)

To keep the server up across reboots and SSH disconnects, run it as a service:

sudo tee /etc/systemd/system/bonsai.service << 'EOF'
[Unit]
Description=Bonsai ternary LLM (llama-server, CPU-only)
After=network.target

[Service]
Type=simple
User=%i
WorkingDirectory=%h/llama.cpp/build-cpu
ExecStart=%h/llama.cpp/build-cpu/bin/llama-server \
  -m %h/models/bonsai/Ternary-Bonsai-8B-Q2_0.gguf \
  -ngl 0 -t 12 -fa 1 -c 4096 --host 0.0.0.0 --port 8080
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now bonsai.service

Replace %i/%h with a concrete user and home path if your systemd doesn’t expand them, or install it as a user service. Because the build is CPU-only, there’s no LD_LIBRARY_PATH to set — the unit is as simple as it looks.

Reach it remotely

Started with --host 0.0.0.0, the server is reachable from any machine that can route to it. Two clean options:

• Tailscale — if both machines are on the same Tailnet, run tailscale ip -4 on the server and open http://100.x.x.x:8080 from anywhere. Zero config, no port forwarding, encrypted mesh.
• SSH tunnel — no Tailscale? Forward the port from your laptop with ssh -L 8080:localhost:8080 user@server, then use http://localhost:8080 locally. In that case you can leave off --host 0.0.0.0 and keep the server bound to localhost.

The honest CPU numbers

Measured with llama-bench on a Ryzen 9 3900X (12 cores / 24 threads, DDR4), -ngl 0 -fa 1, no GPU. PP512 = 512-token prompt processing (prefill); TG128 = 128-token generation.

Two things jump out. Prefill is fast — ~440 tok/s means the model reads a long prompt in well under a second. Generation is slow and memory-bandwidth-bound: because producing tokens means streaming the whole weight matrix from RAM for each one, the smaller 4B (4.85 ± 0.04 tok/s) generates about 50% faster than the 8B (3.22 ± 0.23 tok/s) — it reads half the bytes per token. The 8B is still usually the model to run: you get double the parameters at only a ~35% generation-speed penalty, not for free but cheaply. Reach for the 4B when generation throughput or RAM matters more than quality.

How prefill scales with threads (8B, PP512)

Prefill scales modestly — a single core already does 80% of peak because each AVX2 core is doing serious work. The sweet spot is 8–12 threads, one per physical core; pushing to all 24 SMT threads actually regresses. Generation, meanwhile, is essentially single-core-bound no matter how many threads you give it.

CPU vs Apple Silicon vs GPU

Where does a desktop x86 CPU land? Same 8B Q2_0 model, same PP512/TG128 test:

The Ryzen’s prefill (440 tok/s) is neck-and-neck with the M4’s GPU (455 tok/s) and 3× the M4’s CPU — the add-only ternary matmul is well-optimized for x86 AVX2. But generation is ~10× slower than the M4 CPU (3.2 vs 32 tok/s): Apple Silicon’s huge unified-memory bandwidth wins the autoregressive loop, which the x86 Q2_0 path hasn’t caught up on. The takeaway for CPU users: budget your prompts freely, but treat generation as a background job, not an interactive one.

Ternary, threads, and common issues

Going further

The setup above is everything you need to run an 8B model on any CPU. If you do have a Pascal GPU and want interactive speed, the other guides cover the GPU path — Gemma-4 vision for multimodal, or Krea 2 Turbo for image generation — to round out a local stack.

← Back to all guides