Run Bonsai Language Models Locally

Bonsai-demo provides scripts and documentation for running PrismML's Bonsai language models locally across Mac, Linux, Windows, and CPU-only setups. It supports Metal, CUDA, Vulkan, and ROCm backends, with links to Bonsai 27B, 1-bit Bonsai, and Ternary-Bonsai model releases and whitepapers. The repo targets users who want to test compact or quantized LLMs on local hardware.

AI / ML llm local-ai quantization shell

Stars: 2K
Forks: 198
Issues: 10
PRs: 0

Open on GitHub Archive stats

Run Bonsai models on local hardware

Bonsai-demo packages the setup path for PrismML's Bonsai model family, including Bonsai 27B, 1-bit Bonsai, and Ternary-Bonsai.

What it supports

The repo focuses on local inference across common environments: Mac with Metal, Linux or Windows with CUDA, Vulkan, or ROCm, and CPU-only machines.

Why it matters

It gives developers a direct way to test quantized Bonsai models without relying on a hosted API. The linked model collections and whitepapers also make it easier to compare the 27B, 1-bit, and ternary variants.