Srota Inference Engine
The disk-streaming LLM inference engine inside Laghu — it streams model weights from disk under a memory ceiling you set, so models larger than your free RAM can still run.
About Srota Inference Engine
Srota is the engine that powers Laghu, Aitele's local LLM runtime for Windows. It reads GGUF models and decodes Q4_K, Q6_K, MXFP4, F16, BF16 and F32 weights through its own portable C kernel (no ggml code) and a byte-level BPE tokeniser written from the public description, streaming weights from disk under a user-set RAM ceiling — trading reply speed for capacity. Laghu serves it as a chat interface on 127.0.0.1 only, with a model catalogue from 1B to 405B parameters and optional web search with cited sources; no account, API key or prompt upload, and no model weights are shipped. Version 1.1 hand-writes the Q4_K/Q6_K dot products in AVX2 and reads streamed weights ahead on 2–4 threads while the CPU computes, making it about three times faster than 1.0 under the same ceiling with word-for-word identical output at temperature 0 — on a 6-core laptop with an NVMe drive, Qwen2.5 7B under a 256 MiB ceiling went from 0.15 to 0.44 tokens a second. Shipped with Laghu 1.1, a per-user, administrator-free installer for Windows 10/11 x64 with signed automatic updates.
Highlights
- ▸Disk-streamed inference under a memory ceiling — model size no longer capped by RAM (a lower ceiling means slower replies)
- ▸Fully local chat on 127.0.0.1 — no account, API key or prompt upload
- ▸Own GGUF reader and portable C kernel for six quantised/float encodings, with no ggml linkage
- ▸Per-user Windows installer with no admin rights; SHA-256 checksums published
- ▸1.1: about 3× faster than 1.0 under the same memory ceiling, same output (measured on a 6-core laptop with NVMe)
More from this lab
AI & Machine Learning Lab.
- software · beta · JUL 27, 2026AXON Agent Engine v0.1.0
The cross-device agent engine inside Sandhi — your desktop and Android devices joined into one end-to-end encrypted network, with the reasoning on your own desktop.
- software · beta · MAR 15, 2026RSN Core v0.4
Reverse Synthetic Neural Network framework — synthesize full transformer architectures from data in seconds, without training.
Publications from this lab
- Technical report · in preparation · NOV 06, 2026Srota: Memory-Ceilinged, Disk-Streamed Inference for Running Large Language Models on Commodity PCs
Running a large language model locally is normally gated by memory: the quantised weights must fit in RAM or VRAM. We describe Srota, the inference engine inside Laghu, Aitele's local LLM application for Windows, which relaxes this constraint by streaming a model's weights from disk under a user-set memory ceiling, so that models larger than available RAM can still be executed — trading generation speed for capacity. Srota reads GGUF containers and decodes the Q4_K, Q6_K, MXFP4, F16, BF16 and F32 block encodings through an independently written portable C kernel that links no ggml code, paired with a byte-level BPE tokeniser implemented from its public description. Version 1.1 hand-vectorises the Q4_K/Q6_K dot products with AVX2, tiles the batched prefill path for cache reuse, and reads streamed blocks ahead on two to four threads inside the ceiling, bypassing the OS file cache when free memory cannot hold the streamed part. On a six-core laptop with an NVMe drive this raised decode speed about threefold over 1.0 at the same ceiling (Qwen2.5 1.5B at 64 MiB: 0.76 → 2.50 tokens/s; Qwen2.5 7B at 256 MiB: 0.15 → 0.44 tokens/s), with word-for-word identical output at temperature 0 and bit-identical logits whether weights are streamed or resident; in the published runs a 7.6-billion-parameter model (4.4 GB) executes in about 0.8 GB at 0.55 tokens/s on a laptop with 1.4 GB free. Laghu ships as a per-user, administrator-free installer that serves a chat interface on the loopback interface only, with signed automatic updates; it requires no account or API key, uploads no prompts and ships no model weights. All measurements come from one machine; characterising the speed–memory-ceiling trade-off across storage media, model sizes and dense versus mixture-of-experts architectures remains the principal open evaluation.
- Technical report · in preparation · JUL 27, 2026Sandhi: A Privacy-Preserving Cross-Device Agent Architecture with Executor-Side Safety Gates
Agents that operate graphical interfaces usually run in a vendor cloud, see a single device, and rely on the model to police itself. We describe Sandhi, a system that links a user's desktop and Android devices into one end-to-end encrypted network in which the desktop is the only place reasoning happens. At each step a vision-language model — local via Ollama, or user-keyed through an OpenAI-compatible API — receives the target device's accessibility tree (capped at 120 nodes) and a downscaled screenshot, and emits exactly one JSON action from a 26-kind schema. Safety is enforced by the executor, not the model: keyword-classified send, purchase, delete and install actions require confirmation; input into focused password fields is refused; and payment apps cause the agent to disable its own accessibility service. Devices pair through a QR-carried token mixed into an X25519 / HKDF / ChaCha20-Poly1305 handshake, and off-LAN traffic crosses a blind relay that sees only an HMAC-derived room identifier. Licences and update manifests are Ed25519-signed and verified offline, so no account or telemetry is required. Sandhi v0.1.0 ships for Windows and Android. Task success, latency and the robustness of the gates against adversarial models are not yet evaluated; those benchmarks are future work.
- Preprint · Aitele Research LLP · MAR 15, 2026Reverse Synthetic Neural Networks (RSN): Training-Free Model Construction by Closed-Form Statistical Synthesis
Modern neural networks are trained: a fixed architecture is initialized at random and fit by gradient descent and backpropagation. We study the opposite construction. In a Reverse Synthetic Network the data synthesizes the model directly — every weight, from embeddings and attention projections to feed-forward maps and output heads, is computed in closed form from corpus statistics, with no loss function, no gradient, and no GPU. We report two results, deliberately separated. Synthesis is competitive with trained baselines on discriminative tasks, reaching 87.0% on 20 Newsgroups in tens of seconds on a CPU with zero training. For generation the same paradigm hits a hard ceiling: it collapses to an n-gram model, which we explain mechanistically — PPMI–SVD representations encode distributional similarity, not predictive composition. Measured in bits-per-byte against a locally-run GPT-2, RSN is best understood as a cheap, strong zero-training floor, not a path to the trained frontier.
Choose the one that fits.
Get your app, venture or R&D done
An app, a venture or R&D — brief us on WhatsApp and get everything built, lightning fast.
Work with the commons For builders & researchersApply as a Fellow
Bring your app, your business — and apply for investment — or your own research. Not built yet? We'll build it with you first.
Apply For investorsInvest in the commons
Back Aitele itself, a lab's research programme, a venture, or a technology release on its way to market — one conversation covers all four.
Invest