Small Language Models on the Edge for Real-World Agentic Systems in Industry
David Fernandez
Key Contributions & Takeaways
Large language models are expensive to run, they send private data to the cloud, and stop working the moment the network does. This paper asks whether small models, under 7 billion parameters and running on ordinary hardware, can perform enterprise work if you give them the right support to plan and check their own work. We show that careful small models such as Phi-4 reach advanced language ability while answering several times faster than a cloud call and keeping every prompt on the device.
- A grading method that resists a single judge’s bias. Each small model is examined like a language student, then scored by two frontier models cast as rival instructors (GPT-4 and Claude) that must discuss and agree on one CEFR grade. Role-play makes them careful; forcing consensus curbs the bias of any single judge.
- Planning split into three narrow steps a small model can actually follow. Rather than plan in one shot, the pipeline separates subtask identification, dependency reasoning, and schema-constrained output, so the model reasons first and only fills a rigid JSON structure last, exactly where small models tend to slip.
- Advanced results on commodity hardware, several times faster than the cloud. Across seven open models under 7B parameters, Phi-4 reached C1-level translation and a 0.883 G-Eval summarization score, while local and in-browser (WebLLM) hosting answered 3 to 4 times faster than a cloud API, with no data leaving the device.
Large language models are hard to put to work inside a company: they cost a lot to run, they send private data to the cloud, and they stop working the moment the network does.
Run small models under 7B parameters on ordinary hardware, and make up for their smaller capacity with agentic architectures that plan, route, and check their own work.
On commodity hardware, Phi‑4 reaches C1‑level translation and a 0.883 summarization score, while local and in‑browser hosting answer 3–4× faster than the cloud.
Each open model is examined as a language-learning student, then graded by two frontier models playing rival instructors who must reach a single CEFR verdict.
The Proctor agent draws a translation exercise, English to Afrikaans, from a fixed exam set.
Each SLM, cast as a student, returns a translation with no idea it is being scored.
The Proctor takes the answer back and keeps the grade hidden from the student.
The exercise and answer go to two grader agents, each a different frontier model.
Goddard (GPT-4) and Vorster (Claude) debate the work and agree one CEFR grade.
The agreed grade and written review flow back to the Proctor.
The Proctor logs a structured record: student, language pair, agreed CEFR grade, and a review summary, ready to compare across all seven models.
English to Afrikaans, graded by the two rival examiners. Phi-4 reached C1, the advanced tier a proficient human speaker sits at.
Scored with G-Eval, an LLM-graded quality measure on a 0 to 1 scale. Phi-4’s 0.883 is close-to-frontier work from a sub-7B model.
local & in-browser3–4× faster
Hosted with WebLLM in the browser or run locally, the models answer 3 to 4 times faster than a round trip to a cloud API, with no per-call fee and no queue.
Prompts and documents never leave the machine, so regulated data stays in-house.
No large payloads to the cloud on every call, so it works over weak or metered links.
Runs with no internet at all, for plants and sites that are offline by design.
Because the model runs on the device, sensitive data never crosses the network. That opens up privacy-bound and fully offline settings the cloud can’t serve.
Agentic structure, decomposed planning and multi-agent grading, lets a sub-7B model do work that usually assumes a much larger cloud model.
Two rival graders that must agree is a reusable recipe for scoring open-ended output without leaning on a single, biased judge.
The diagrams above redraw the paper’s benchmarking and planning framework in this site’s style; the exam-loop figure follows the appendix’s evaluation walkthrough. Seven open models under 7B parameters were graded on translation and summarization by two frontier examiners (GPT-4 and Claude) on the CEFR scale, with Phi-4 the standout. Full method, prompts, and results are in the paper and the linked appendix repository.
“Small models on ordinary hardware, graded by two rival AI examiners that have to agree, reach advanced C1-level translation and answer 3 to 4 times faster than the cloud, without a byte of data leaving the device.”
David Fernandez is a PhD candidate in Computer Science at Clemson University and an AI Researcher at BMW Group. His research focuses on safe, efficient, and explainable AI, spanning large language models, vision-language models, agentic systems, and adversarial robustness. His 10 papers include six first-authored studies on component-level explainability, zero-shot reasoning, and adversarial scenario analysis.
At BMW Group, he develops enterprise AI agents, automated prompt optimization, and defenses against multi-turn adversarial attacks. His work also includes edge AI for industrial agentic systems.
Through Clemson’s Virtual Prototyping of Ground Systems (VIPR-GS) program, he develops explainable reasoning and multimodal evaluation systems for the U.S. Army’s Next Generation Combat Vehicle program.