Run an LLM on ESP32: Local AI on a $5 Microcontroller

You’re paying $20/month for ChatGPT Plus. Your smart home runs on Alexa, which means Amazon hears everything. Your phone’s AI features send data to Google or Apple. What if you could run a real language model on a $5 chip that fits on your fingertip — no internet, no subscriptions, no data leaving your house?

It’s not science fiction. It’s happening right now.

The Problem — Cloud AI Costs Add Up (And You’re Not Private)

API bills sneak up on you. $20 here, $50 there, suddenly you’re spending hundreds a year on AI you could run locally. Rate limits hit at the worst time. And every prompt you send? It’s logged, analyzed, used to train the next model you’ll pay for.

Privacy isn’t paranoia. It’s knowing your journal entries, your code, your late-night questions aren’t someone else’s training data.

The Solution — TinyML Just Got Real (LLMs on Microcontrollers)

What changed: quantization + better chips

Two years ago, “TinyML” meant a 50KB model that detects “hey google” or classifies a gesture. Cute, but not useful for real language tasks.

Then three things happened:

1. Quantization got aggressive — 4-bit (INT4) and even 2-bit weights that keep 95%+ quality
2. Model architectures got efficient — Phi, SmolLM, TinyLlama designed for edge deployment
3. MCUs got serious RAM — ESP32-S3 with 512KB SRAM + 8MB PSRAM for under $10

The result: a 28-million-parameter language model running at 15 tokens/second on a chip that costs less than a coffee.

28M parameters on a $5 chip — actually useful

Let’s be honest about what “useful” means here. You’re not getting GPT-4. You’re getting a model that can:

• Answer factual questions (“What’s the capital of Kazakhstan?”)
• Write simple code snippets (“Python function to parse JSON”)
• Hold a basic conversation
• Control smart home devices via natural language
• Summarize short texts

What it can’t do: complex reasoning, long-context analysis, creative writing, or anything requiring world knowledge beyond its training cutoff.

For a $10 total investment? That’s insane value.

What You Need (Hardware + Software)

ESP32-S3 with PSRAM (why it matters)

Don’t buy the basic ESP32. You need the ESP32-S3 variant with PSRAM (Pseudo-Static RAM). Here’s why:

Look for: “ESP32-S3-DevKitC-1” or “ESP32-S3-N16R8” (16MB flash, 8MB PSRAM). Available on Amazon, AliExpress, DigiKey, Mouser.

Arduino IDE setup (5 minutes)

• Download Arduino IDE 2.x from arduino.cc
• File → Preferences → Additional Boards Manager URLs:

https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json

• Tools → Board → Boards Manager → Search “esp32” → Install “esp32 by Espressif Systems”
• Tools → Board → ESP32 Arduino → Select “ESP32S3 Dev Module”
• Critical settings for PSRAM:

– PSRAM: “Enabled”
– Flash Size: “16MB (128Mb)”
– Partition Scheme: “16M Flash (3MB APP/9.9MB FATFS)” or similar

Model files: where to download

Phi-2-mini (28M, INT4) — Recommended starter

• Hugging Face: microsoft/phi-2-mini-gguf → phi-2-mini-q4_k_m.gguf
• Size: ~16MB
• Speed: ~15 tok/s on ESP32-S3
• Quality: Surprisingly coherent

TinyLlama-1.1B (INT4) — Better quality, slower

• Hugging Face: TinyLlama/TinyLlama-1.1B-Chat-v1.0-GGUF → tinyllama-1.1b-chat-q4_k_m.gguf
• Size: ~650MB (needs SD card)
• Speed: ~2 tok/s
• Quality: Much closer to “real” LLM

SmolLM-135M (INT8) — Fastest

• Hugging Face: HuggingFaceTB/SmolLM-135M-Instruct-GGUF → smollm-135m-q8_0.gguf
• Size: ~80MB
• Speed: ~25 tok/s
• Quality: Basic but functional

Download tip: Get the .gguf format. It’s the standard for llama.cpp and works with the ESP32 port.

Step-by-Step — Flash Your First LLM in 15 Minutes

1. Install the ESP32 LLM library

In Arduino IDE:

• Sketch → Include Library → Manage Libraries
• Search “esp32-llama” → Install “esp32-llama by espressif” (or similar port)
• Or manually: GitHub → espressif/esp-llm → Download ZIP → Sketch → Include Library → Add .ZIP Library

2. Prepare your model file

Option A: Convert to C array (for models <4MB, fits in flash)

# On your computer
xxd -i phi-2-mini-q4_k_m.gguf > model_data.h

This creates a massive C header file. Include it in your sketch.

Option B: Load from SD card (for larger models, recommended)

• Format microSD as FAT32
• Copy .gguf file to root
• Insert SD into ESP32-S3 board (most dev kits have slot)

3. The minimal sketch

#include 
#include "model_data.h"  // If using C array method


Llama llm;


void setup() {
  Serial.begin(115200);
  while (!Serial) delay(10);


  Serial.println("Initializing LLM...");


  // For C array method:
  llm.begin(model_data, model_data_len);


  // For SD card method:
  // llm.begin("/phi-2-mini-q4_k_m.gguf");


  Serial.println("Ready! Type a prompt:");
}


void loop() {
  if (Serial.available()) {
    String prompt = Serial.readStringUntil('\n');
    prompt.trim();


    if (prompt.length() > 0) {
      Serial.print("You: ");
      Serial.println(prompt);
      Serial.print("AI: ");


      // Stream tokens as they generate
      llm.generate(prompt.c_str(), [](const char* token) {
        Serial.print(token);
      });
      Serial.println();
    }
  }
}

4. Flash and test

• Connect ESP32-S3 via USB
• Tools → Port → Select your COM port
• Upload (Ctrl+U)
• Open Serial Monitor (115200 baud)
• Type: “What is 2+2?” → Press Enter
• Watch it generate!

Real Performance — What to Expect

Speed comparison table

What works / what doesn’t with 28M params

Works well:

• Factual Q&A (capitals, definitions, simple math)
• Code snippets (single functions, regex, JSON parsing)
• Smart home commands (“turn on living room lights”)
• Short summaries (2-3 paragraphs max)
• Translation (simple sentences, common languages)

Struggles with:

• Multi-step reasoning (“If A then B, but C means…”)
• Long context (forgets after ~512 tokens)
• Creative writing (repetitive, generic)
• Specialized knowledge (recent events, niche topics)
• Non-English languages (mostly English training)

Use Cases — What Can You Actually Build?

Offline chatbot

Serial monitor + keyboard = private AI companion. No logs, no cloud, works on a plane.

Voice assistant (add mic + speaker)

• INMP441 I2S microphone (~$2)
• MAX98357A I2S amplifier + speaker (~$3)
• Use esp-skainet or esp-adf for wake word + STT
• Pipeline: Wake word → STT → LLM → TTS → Speaker
• Total hardware: ~$20

Smart home NLU

Replace Home Assistant’s cloud NLU with local intent parsing:

• “Make it cozy” → parses to: dim lights 50%, warm color, play jazz
• Runs entirely on ESP32, controls devices via MQTT/HTTP
• Zero latency, zero cloud dependency

Learning tool

Best way to understand how LLMs work: watch one run on bare metal. See token generation in real-time. Modify temperature, top-k, top-p and watch behavior change. It’s education you can touch.

Cost Comparison — ESP32 vs Pi vs Cloud vs Laptop

Board	SRAM	PSRAM	Flash	LLM Support	Price
——-	——	——-	——-	————-	——-
ESP32 (basic)	520KB	None	4MB	❌ Too little RAM	$3-5
ESP32-S3 (no PSRAM)	512KB	None	8-16MB	❌ Barely runs 10M params	$5-8
ESP32-S3 (with PSRAM)	512KB	2-8MB	8-16MB	✅ Runs 28M+ params	$8-12
Model	Params	Quant	Size	Speed (tok/s)	Quality
——-	——–	——-	——	—————	———
Phi-2-mini	28M	INT4	16MB	~15	Good for basics
SmolLM	135M	INT8	80MB	~25	Fast, basic
TinyLlama	1.1B	INT4	650MB	~2	Best quality
Gemma-2B	2B	INT4	1.2GB	~1	Slow but smart
Approach	Hardware Cost	Ongoing	Privacy	Speed	Best For
———-	—————	———	———	——-	———-
ESP32-S3 + PSRAM	$10-15	$0	Complete	15 tok/s	Embedded, battery, privacy
Raspberry Pi 4 + Ollama	$60-100	Electricity	Complete	20-50 tok/s	Home server, bigger models
Laptop (llama.cpp)	$0 (existing)	$0	Complete	50-200 tok/s	Development, big models
Cloud API (OpenAI)	$0	$20-200/mo	None	Fast	Production, best quality

Troubleshooting — Common Issues & Fixes

“Out of memory” / “Allocation failed”

→ Enable PSRAM in board settings. This is #1 cause. Double-check: Tools → PSRAM → “Enabled”

“Gibberish output” / “Repeating tokens”

→ Check quantization. INT4 models need proper calibration. Try INT8 version if available. Also verify model file isn’t corrupted (check file size matches Hugging Face).

“Won’t flash” / “Failed to connect”

→ Boot mode. Hold BOOT button, press RESET, release BOOT. Try different USB cable (data + power, not charge-only). Check COM port in Device Manager.

“Slow as molasses”

→ You’re using a big model on flash. Move to PSRAM. Or use smaller model (Phi-2-mini, SmolLM). INT4 > INT8 for speed on ESP32.

“Serial monitor shows nothing”

→ Baud rate mismatch. Must be 115200. Also check “Newline” setting in Serial Monitor (bottom right).

Takeaway

You can run a real language model on a $10 microcontroller right now. No cloud. No subscriptions. No data leaving your device. The hardware fits in your pocket. The setup takes 15 minutes.

Is it GPT-4? No. Is it useful for real tasks? Absolutely. Smart home control, offline coding help, private journaling, learning how LLMs work — all for the price of lunch.

Grab an ESP32-S3 with PSRAM. Flash Phi-2-mini. Type your first prompt. That moment when it answers — offline, in your hand — that’s when it clicks: local AI isn’t the future. It’s here.

Ready to start? Order the board today. You’ll have it running by the weekend.