First, an important architectural clarification, then the concrete recommendation:
🧠 Layer 1 (NKTg AI — the “processing brain”) should NOT need a generative LLM at all
According to your own product philosophy, Layer 1 is a pure physical algorithm (the NKTg Law, semantic energy measurement, AMP/DAMP decomposition) — this runs fine in plain WASM/JS with no language model required. That’s precisely the selling point: it measures, it doesn’t generate speculation.
However, if you want to add a deep-learning component to Layer 1 (e.g., more accurate AMP/DAMP sentence classification, or sentence ranking), the best-fit architecture is not a generative LLM but:
| Layer 1 component | Recommended architecture | Why |
|---|---|---|
| AMP/DAMP classification | Small encoder (BERT-style ~66–110M: MiniLM, BERT-mini, DistilBERT) via Transformers.js / ONNX Runtime Web | Sentence classification is a classification task — encoders are light, fast, run well on CPU, no WebGPU needed |
| Core Content measurement/ranking | Your algorithm + MiniLM (22M) embeddings | Embeddings capture inter-sentence similarity, extremely lightweight |
🔌 The generative LLM belongs in Layer 2 — best choices for LLM Local
Key fact about in-browser LLMs: WebGPU is the unlock, WebAssembly is only the CPU fallback — WebGPU is stable across every major browser engine as of 2026 (Pinggy, LocalAI Master). The most mature engine is WebLLM (MLC AI, Apache-2.0, supports Llama 3, Phi, Gemma, Mistral, Qwen, with JSON mode and an OpenAI-compatible API) (GitHub mlc-ai/web-llm, WebLLM).
The sweet spot for your published requirements (16 GB RAM, 6–8 GB VRAM GPU):
- Primary recommendation: Qwen 2.5/3 Instruct at 1.5B–4B, or Phi-3.5-mini (3.8B) — quantized 0.5B–3B models download in 300MB–2GB and run at 40–180 tok/s on modern hardware (BuildMVPFast). Notably, Phi-3.5 mini hits ~71 tok/s on Apple Silicon — the fastest decode throughput in WebLLM’s catalog (Learn AI wiki).
- Quantization: q4f16_1 (4-bit) — the best balance between download size and quality.
- Engineering notes: run inference inside a Web Worker (don’t block the UI thread), cache model weights in IndexedDB for offline use, and feature-detect
navigator.gpuandshader-f16before loading (TinyWeights). Pure WASM (wllama) should only be a fallback for machines without a GPU — it’s capped at ~4GB linear memory and only feels interactive with models ≤0.5B (Pinggy). - If you want domain specialization (Vietnamese legal/finance), the standard pipeline is LoRA fine-tune a small model, then compile it to WebGPU via MLC-LLM — a worked example already exists (webSLM, built on Qwen2.5-0.5B) (webSLM).
✅ Recommended architecture summary
textCopy
Layer 1 (NKTg AI): Your algorithm (WASM) + MiniLM/DistilBERT encoder (ONNX) → AMP/DAMP, Core Content
Layer 2 (LLM Local): WebLLM (WebGPU) + Qwen 1.5–4B / Phi-3.5-mini, q4f16 quantized, in a Web Worker, cached in IndexedDB
Fallback: wllama (WASM CPU) with a ≤0.5B model, or switch to LLM Cloud mode
