Small Language Models on the Edge for Real-World Agentic Systems in Industry

Mon, 01 Dec 2025 00:00:00 +0000

01 · The Problem

Large language models are hard to put to work inside a company: they cost a lot to run, they send private data to the cloud, and they stop working the moment the network does.

02 · The Approach

Run small models under 7B parameters on ordinary hardware, and make up for their smaller capacity with agentic architectures that plan, route, and check their own work.

03 · The Finding

On commodity hardware, Phi‑4 reaches C1‑level translation and a 0.883 summarization score, while local and in‑browser hosting answer 3–4× faster than the cloud.

Contribution 1 · A Bias-Resistant Benchmark

Grading a small model is subjective, so two rival graders must agree. Role-play makes them careful; consensus curbs single-judge bias.

Each open model is examined as a language-learning student, then graded by two frontier models playing rival instructors who must reach a single CEFR verdict.

The exam loop, step by step

1Set the exam

The Proctor agent draws a translation exercise, English to Afrikaans, from a fixed exam set.

2Student answers

Each SLM, cast as a student, returns a translation with no idea it is being scored.

3Collect the answer

The Proctor takes the answer back and keeps the grade hidden from the student.

4Hand to graders

The exercise and answer go to two grader agents, each a different frontier model.

5Discuss to consensus

Goddard (GPT-4) and Vorster (Claude) debate the work and agree one CEFR grade.

6Return the verdict

The agreed grade and written review flow back to the Proctor.

7Record the result

The Proctor logs a structured record: student, language pair, agreed CEFR grade, and a review summary, ready to compare across all seven models.

Contribution 2 · Planning a Small Model Can Follow

Asked to plan in one shot, a small model rambles. Split the job into three narrow steps and it stays on track.

a messy user request → three-phase pipeline → a valid, executable plan

1 Subtask identification

Break the request into small, single-purpose steps, so each one is simple enough for a small model to get right.

2 Dependency reasoning

Work out the order: which steps can run now and which must wait for another step’s output first.

3 Schema-constrained output

[
  { "step":1, "needs":[] },
  { "step":2, "needs":[1] }
]

Emit the plan as JSON locked to a fixed schema, so it always parses and can be run without hand-fixing.

Splitting reasoning from formatting is the key move: the model thinks in phases 1 and 2, then only has to fill in a rigid structure in phase 3, which is exactly where small models tend to slip.

What The Small Models Actually Did

Phi-4 led the field, matching advanced-level language work while running on ordinary hardware.

Phi-4 translation, on the CEFR scale (advanced)

0.883

Phi-4 summarization, G-Eval score

3–4×

faster to answer than a cloud service

open models under 7B params, benchmarked

Translation C1 · CEFR

English to Afrikaans, graded by the two rival examiners. Phi-4 reached C1, the advanced tier a proficient human speaker sits at.

Summarization 0.883 · G-Eval

Scored with G-Eval, an LLM-graded quality measure on a 0 to 1 scale. Phi-4’s 0.883 is close-to-frontier work from a sub-7B model.

Latency · running on the device vs. calling the cloud

Cloud API call

baseline

On-device
local & in-browser

3–4× faster

Hosted with WebLLM in the browser or run locally, the models answer 3 to 4 times faster than a round trip to a cloud API, with no per-call fee and no queue.

Where this unlocks deployment

Privacy-sensitive

Prompts and documents never leave the machine, so regulated data stays in-house.

Bandwidth-constrained

No large payloads to the cloud on every call, so it works over weak or metered links.

Air-gapped

Runs with no internet at all, for plants and sites that are offline by design.

Why It Matters

Data stays home

Because the model runs on the device, sensitive data never crosses the network. That opens up privacy-bound and fully offline settings the cloud can’t serve.

Scaffolding beats size

Agentic structure, decomposed planning and multi-agent grading, lets a sub-7B model do work that usually assumes a much larger cloud model.

Judging you can trust

Two rival graders that must agree is a reusable recipe for scoring open-ended output without leaning on a single, biased judge.

About this page

The diagrams above redraw the paper’s benchmarking and planning framework in this site’s style; the exam-loop figure follows the appendix’s evaluation walkthrough. Seven open models under 7B parameters were graded on translation and summarization by two frontier examiners (GPT-4 and Claude) on the CEFR scale, with Phi-4 the standout. Full method, prompts, and results are in the paper and the linked appendix repository.

Agentic AI | David Fernandez

Small Language Models on the Edge for Real-World Agentic Systems in Industry