Tag
local-llm
6 articles
SpeakoFlow: The Open-Source Voice Assistant That Stays Local
SpeakoFlow is a free, MIT-licensed local-first voice assistant for Windows, macOS and Linux, built by solo developer Abhishek Barali as a fork of CJ Pais's Handy. It combines hotkey dictation, a 'Hey Flow' generative writing mode, an assistant panel with screen vision, live translation and AI cleanup. Speech-to-text always runs on-device via whisper.cpp and Parakeet; the assistant can run fully offline through a built-in llama.cpp engine, through Ollama or LM Studio, or through any OpenAI-compatible cloud provider with your own key. There is no account and no telemetry. The trade-offs are real: the binaries are not code-signed on Windows or macOS, requiring a manual quarantine-clearing step on Mac, and the project is at v1.0.1 with 126 commits, six GitHub stars and one maintainer.
By Marcus Rivera · 8 min · Aug 7, 2026
Meetily: The Open-Source AI Notetaker That Runs 100% Local
Meetily is a privacy-first, open-source AI meeting assistant with 27.4K GitHub stars and an MIT license. Built on Rust and Tauri, it runs Whisper or Parakeet transcription and Ollama summarization entirely on your own device, so meeting audio never touches the cloud. It supports macOS and Windows, with flexible summary providers and a commercial PRO tier.
By Marcus Rivera · 5 min · Jul 30, 2026
OpenClaw: The 383K-Star AI Agent With a Security Problem
OpenClaw is a free, self-hosted, model-agnostic AI agent that runs as a persistent background daemon and acts across WhatsApp, Telegram, Slack, and Discord. It became the fastest-growing repo in GitHub history (383K+ stars) but carries serious security flaws: authentication off by default, plaintext credential storage, tens of thousands of internet-exposed instances, and fake installers spreading infostealer malware. Run it only from the official repo, behind a VPN, with auth on and scoped credentials.
By Marcus Rivera · 6 min · Jul 22, 2026
Unsloth: Fine-Tune LLMs 2x Faster on a Single GPU
Unsloth is an open-source library that fine-tunes open LLMs (Llama, Qwen, Mistral, Gemma, gpt-oss) roughly 2x faster and with up to 70% less VRAM than a stock Hugging Face setup, without sacrificing accuracy. It achieves this with custom OpenAI Triton kernels and a manual backpropagation engine, and fuses LoRA with 4-bit quantization. It runs on any NVIDIA GPU with CUDA Capability 7.0+, including the free Colab T4. Install with 'pip install unsloth' and use FastLanguageModel.from_pretrained plus get_peft_model to attach LoRA adapters before training with trl's SFTTrainer.
By Marcus Rivera · 6 min · Jul 10, 2026
vLLM: Serve LLMs 24x Faster Than Hugging Face Transformers
vLLM is the default open-source LLM serving engine in 2026. PagedAttention cuts KV-cache memory waste from 60-80% to under 4%, and continuous batching keeps the GPU full, together delivering 14-24x the throughput of Hugging Face Transformers. Install with pip, launch an OpenAI-compatible server via 'vllm serve', then tune --gpu-memory-utilization, --max-num-batched-tokens, --tensor-parallel-size, and chunked prefill against real traffic.
By Marcus Rivera · 7 min · Jul 1, 2026
Ollama: Run Local LLMs Like a Pro in 2026
A hands-on guide to Ollama, the default local-LLM runner in 2026 (v0.30.10). Covers install, pulling and running models, calling them from the OpenAI SDK at localhost:11434, structured JSON outputs, tool calling, and Modelfiles, plus how to size a model to your hardware.
By Marcus Rivera · 6 min · Jun 25, 2026