Guide
Running a Local Agent on Raspberry Pi 5 with LiteRT and Gemma
Google recently demonstrated a practical Edge AI stack for Raspberry Pi 5 using LiteRT and lightweight Gemma models. For Gemma 4 E2B, Google reports roughly 99 tokens/s prefill, 9 tokens/s decode, and about 1,432 MB peak memory on Raspberry Pi 5. In the Reachy Mini voice demo, effective generation reached about 27.3 characters per second, approximately 300 English words per minute. More importantly, LiteRT can orchestrate more than an LLM: CPU resources can handle Gemma and speech recognition while the Pi GPU continuously runs vision models such as YOLO. This makes it possible to build private, offline multimodal agents without a cloud dependency.
# Running a Local Agent on Raspberry Pi 5 with LiteRT and Gemma
## Article Summary
Google recently demonstrated a practical Edge AI stack for Raspberry Pi 5 using LiteRT and lightweight Gemma models. For Gemma 4 E2B, Google reports roughly 99 tokens/s prefill, 9 tokens/s decode, and about 1,432 MB peak memory on Raspberry Pi 5. In the Reachy Mini voice demo, effective generation reached about 27.3 characters per second, approximately 300 English words per minute. More importantly, LiteRT can orchestrate more than an LLM: CPU resources can handle Gemma and speech recognition while the Pi GPU continuously runs vision models such as YOLO. This makes it possible to build private, offline multimodal agents without a cloud dependency.
---
Local AI does not always require a workstation GPU.
Many edge applications care more about:
- offline operation;
- deterministic latency;
- privacy;
- power consumption;
- hardware cost;
- deployment at scale.
Examples include robots, cameras, factory terminals, retail devices, translators, and industrial gateways.
For those workloads, Raspberry Pi 5 plus LiteRT and Gemma is an increasingly realistic architecture.
## What performance can you expect?
Google reports approximately:
```text
Gemma 4 E2B on Raspberry Pi 5
Prefill: 99 tokens/s
Decode: 9 tokens/s
Peak memory: ~1,432 MB
```
Nine decode tokens per second is not cloud-frontier speed.
But for a low-power board without a discrete GPU, it is fast enough for many interactive tasks.
In Googleβs Reachy Mini voice pipeline, effective text generation reached around:
```text
27.3 characters/s
β 300 words/minute
```
That is faster than normal conversational speaking speed, meaning text generation may no longer be the primary bottleneck for some local voice experiences.
## Choose the smallest model that solves the task
Useful Gemma options include:
- Gemma 3 270M for task-specific classification;
- EmbeddingGemma 300M for local retrieval and semantic search;
- Gemma 3 1B for lightweight generation;
- Gemma 4 E2B for memory-sensitive multimodal or agentic edge workloads;
- Gemma 4 E4B for stronger reasoning with higher resource requirements.
The correct edge strategy is not to force the largest possible model onto the board.
It is to use the smallest model that meets the product requirement.
## What LiteRT provides
LiteRT is an on-device inference runtime and toolchain.
It covers:
```text
conversion
quantization
benchmarking
inference
CPU and GPU backends
```
This matters operationally.
A prototype can tolerate ad-hoc model formats and dependencies.
A fleet of hundreds or thousands of devices cannot.
## Install LiteRT CLI
A basic setup is:
```bash
python -m venv .venv
source .venv/bin/activate
pip install litert-cli
```
Configure a Hugging Face token:
```bash
export HUGGING_FACE_HUB_TOKEN=
```
Then run a compatible LiteRT model from the community repository.
Conceptually:
```bash
litert lm run \
--from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
gemma-4-E2B-it.litertlm \
--prompt="Summarize the latest sensor alert in 3 bullets."
```
## Build narrow agents, not general assistants
Edge hardware is best used for bounded tasks.
A factory diagnostic assistant might receive:
```text
temperature
vibration
error code
```
then use local retrieval and Gemma to produce:
- probable cause;
- inspection steps;
- shutdown recommendation.
That is often more commercially useful than trying to recreate a universal cloud chatbot.
## Split workloads across CPU and GPU
Googleβs Reachy Mini architecture illustrates a good pattern:
```text
camera
β YOLO detection
β GPU
microphone
β local ASR
β CPU
transcript + visual metadata
β Gemma 4 E2B
β CPU
response
β local TTS
```
The goal is to avoid forcing all workloads onto the same compute resource.
Continuous vision workloads can occupy the GPU while language reasoning remains on the CPU.
## Do not send every video frame to the LLM
A better design converts perception into structured state:
```json
{
"objects": [
{"label": "person", "position": "left"},
{"label": "box", "position": "center"}
]
}
```
The LLM reasons over the latest relevant metadata rather than raw frames.
This reduces latency and compute pressure dramatically.
## Privacy is a major advantage
A cloud architecture sends sensor data outward:
```text
sensor
β internet
β cloud model
β response
```
A local architecture can remain:
```text
sensor
β Raspberry Pi
β local models
β response
```
For audio, video, industrial telemetry, and household data, keeping raw data on the device can simplify privacy and resilience.
## Offline does not mean secure
Local agents still need protection against:
- replaced model files;
- stolen storage;
- unauthenticated local APIs;
- prompt injection;
- unsafe hardware actions.
Use signed updates, model hashes, tool allowlists, local authentication, and hard physical safety constraints.
## Local RAG with EmbeddingGemma
A maintenance manual can be indexed locally:
```text
manual
β chunks
β EmbeddingGemma 300M
β local vector index
```
A query such as βWhat does error E37 mean?β can be embedded, retrieved locally, and answered by Gemma without sending the document to a cloud service.
## Keep tool access narrow
A robot might expose:
```text
look_left
look_right
read_sensor
set_indicator
speak
```
Avoid arbitrary shell execution.
A safe pattern is:
```text
LLM
β set_indicator(color="red")
β validation layer
β GPIO
```
not:
```text
LLM
β arbitrary Python
β direct hardware access
```
## Benchmark more than tokens per second
Useful edge metrics include:
- cold start;
- prefill;
- decode;
- peak RAM;
- CPU load;
- temperature;
- power consumption;
- first-audio latency;
- end-to-end task time.
Thermal throttling matters on sustained workloads.
A demo that is fast for one minute may behave differently after thirty minutes at full load.
## A practical starter project
Build an offline device assistant:
```text
microphone
β local ASR
β intent
β EmbeddingGemma retrieval
β Gemma 4 E2B
β constrained tool call
β local TTS
```
Expose only a few safe tools such as sensor readout or status indicators.
This is already a complete local edge agent.
## When Raspberry Pi 5 is not enough
Move to stronger edge hardware when you need:
- large-model reasoning;
- multiple high-resolution video streams;
- large context windows;
- demanding vision-language models;
- high-quality full-duplex voice.
Options include accelerator HATs, Jetson-class devices, NPUs, or hybrid edge/cloud architectures.
## Conclusion
Raspberry Pi 5 is not competing with frontier-cloud inference.
Its value is the combination of:
```text
low latency
+ offline operation
+ privacy
+ low cost
+ fleet deployment
```
When Gemma 4 E2B can operate in roughly 1.4 GB peak memory and LiteRT can coordinate vision, audio, embeddings, and language inference on a small board, many previously cloud-dependent agent workloads become locally feasible.
For more Edge AI, local-model, Gemma, and practical agent engineering guides, visit **Zyentor Picks**: https://www.zyentorpicks.com/.