I had a theory that this was both possible and easy and so I just had to test it out and see for myself…
I have Qwen3.8 27b running on an ESP32 :)
How it works
Use an SD card as RAM. Video showing some more detail below:
Performance

Yep, that’s ~50 tokens per week.
There were a couple easy things we could do to increase the tokens per week so let’s do them.
Optimization: MoE

We can use Qwen 3.6 35B A3B, a MoE model, which is slightly more sensible to run on an ESP32 (which isn’t saying much). Since this model has 3B experts it doesn’t have to read the whole model from memory each time which makes it much faster.
Let’s see if we can improve that a bit more.
Optimization: SD Clock Frequency

Change SD card interface clock frequency from 20MHz to 40MHz
Optimization: Other Options
From here on there are a couple other easy ways to improve speed if we really wanted, highlighted below:
Change from 1-bit SDMMC to 4-bit SDMMC

Change from SD card to eMMC

However the ESP32 is definitely the wrong tool for the job and the fun part was just getting it working - so I’m going to leave this here for now.
Moral of the story
AI Summary of Implementation Details
Here’s an AI summary of some implementation details in case you’re interested in recreating this:
The hardware is a Freenove ESP32-S3 N16R8 running at 240 MHz, with 8 MiB of PSRAM, 16 MiB of flash and a 32 GB microSD card. The card uses FAT32 over a one-bit SDMMC connection at 20 MHz which was later changed to 40MHz.
The GGUF is first exported on the PC into separate tensor files and matching tokenizer tables, preserving the quantized weights. The 35B model bundle occupies about 20.54 GB on the card. Splitting it into tensor files also avoids FAT32’s single-file size limit.
MicroPython coordinates tokenization, model layers and file operations on the ESP32. Matrix-vector operations load an input vector and stream a few quantized weight rows into reusable RAM buffers. The normal batch is four rows, falling back to two or one if allocation fails. Custom native C kernels decode the quantized blocks, multiply by the input values and accumulate FP32 results. The expensive arithmetic therefore runs in native code.
Activations, recurrent state and the attention KV cache are also stored in files on SD. Operations read the pieces they need into RAM, compute with them, write results back and reuse the buffers. This trades RAM consumption for substantial storage traffic and latency.
Every text decoder layer runs, and output generation uses the unrestricted vocabulary head. The PC provides the interface and records serial logs; it does not perform inference for the physical ESP32 run.


