Small Language Models on the Edge for Real-World Agentic Systems in Industry
David Fernandez
Key Contributions & Takeaways
Large language models are expensive to run, they send private data to the cloud, and stop working the moment the network does. This paper asks whether small models, under 7 billion parameters and running on ordinary hardware, can perform enterprise work if you give them the right support to plan and check their own work. We show that careful small models such as Phi-4 reach advanced language ability while answering several times faster than a cloud call and keeping every prompt on the device.
- A grading method that resists a single judge’s bias. Each small model is examined like a language student, then scored by two frontier models cast as rival instructors (GPT-4 and Claude) that must discuss and agree on one CEFR grade. Role-play makes them careful; forcing consensus curbs the bias of any single judge.
- Planning split into three narrow steps a small model can actually follow. Rather than plan in one shot, the pipeline separates subtask identification, dependency reasoning, and schema-constrained output, so the model reasons first and only fills a rigid JSON structure last, exactly where small models tend to slip.
- Advanced results on commodity hardware, several times faster than the cloud. Across seven open models under 7B parameters, Phi-4 reached C1-level translation and a 0.883 G-Eval summarization score, while local and in-browser (WebLLM) hosting answered 3 to 4 times faster than a cloud API, with no data leaving the device.
Large language models are hard to put to work inside a company: they cost a lot to run, they send private data to the cloud, and they stop working the moment the network does.
Run small models under 7B parameters on ordinary hardware, and make up for their smaller capacity with agentic architectures that plan, route, and check their own work.
On commodity hardware, Phi‑4 reaches C1‑level translation and a 0.883 summarization score, while local and in‑browser hosting answer 3–4× faster than the cloud.
The diagrams above redraw the paper’s benchmarking and planning framework in this site’s style; the exam-loop figure follows the appendix’s evaluation walkthrough. Seven open models under 7B parameters were graded on translation and summarization by two frontier examiners (GPT-4 and Claude) on the CEFR scale, with Phi-4 the standout. Full method, prompts, and results are in the paper and the linked appendix repository.
“Small models on ordinary hardware, graded by two rival AI examiners that have to agree, reach advanced C1-level translation and answer 3 to 4 times faster than the cloud, without a byte of data leaving the device.”
David Fernandez is a PhD candidate in Computer Science at Clemson University, working on safe, efficient, and explainable AI for safety-critical systems. His research spans perception, adversarial robustness, and on-device deployment of large foundation models, including LLMs and VLMs, with five first-authored publications on component-level explainability, zero-shot reasoning, and adversarial scenario analysis, alongside collaborative work on edge AI for industrial agentic systems. Much of this research is grounded in autonomous driving, where trustworthiness, latency, and robustness constraints are unforgiving, but the underlying methods transfer broadly to other high-stakes domains.
As a member of Clemson’s VIPR-GS Research Program, he develops hierarchical LLM reasoning frameworks and VLM evaluation systems for the U.S. Army’s Next Generation Combat Vehicle (NGCV) program, focusing on zero-shot reasoning and component-level explainability under real-world deployment constraints.
At BMW Group, he designs agentic AI systems for enterprise environments, building autonomous prompt optimization pipelines that enable continual agent improvement without model retraining and context-aware moderation frameworks that detect coordinated multi-turn adversarial attacks in production deployments.