- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OpenJarvis defines on-device agent harnesses as a combination of five different specs and uses a cloud teacher model-based optimization loop to execute per-benchmark search of the best configurations across the five different specs to achieve near cloud model-level accuracy on the given benchmarks.
Critical strengths
OpenJarvis provides a method to improve the local on-device model's performance to that of the cloud model on the targeted benchmarks through the use of an evolution loop on the on-device agent system driven by a cloud model. OpenJarvis also has a larger search space, targeting the entire agent system stack, including parts such as the inference engine or tools, unlike existing research such as GEPA.
Critical weaknesses
-
The technique itself seems to be a general LLM-based agent system optimization method, and it is unclear why the authors are targeting on-device private AI agent scenarios. Distillation of frontier-model capabilities to smaller models through an optimization loop is a well-studied technique (GEPA), and therefore the five-spec optimization method should work for any agentic harness or other parts of the agentic system stack.
-
Optimizations are task-specific. Users would need to generate and gather a non-private training dataset and run optimization for different sets of their private tasks.
-
The model used for the headline accuracy figures (Qwen3.5-122B) is not an on-device model.
Questions for authors
Why is this method framed as an on-device method? End users rarely have the time and resources to generate and gather a non-private training dataset and run cloud-model-based optimization loops to improve their private workloads. I cannot agree that OpenJarvis would be something that individual users will actively use to improve their personal AI stack.
Wouldn't this method be more beneficial for a batch execution case where such optimization overhead can be amortized across a large number of similar tasks, as many existing LLM-based agentic system optimization methods such as GEPA do?
Thank you for your response.
I worked on both mobile on-device LLMs (smartphones) and inference engines (vLLM on DGX/AMD boxes). Based on the responses, my honest evaluation of OpenJarvis is a combination of existing agent optimizations from the cloud ported to on-device . As a mobile systems researcher, this is quite heartbreaking. I have a few clarifying questions.
(1) Can you please define (1) Personal AI, (2) On-device execution, and (3) Personal Devices you used in your paper?
To be specific:
For personal AI, what is the largest number of parameters you consider as "Personal AI"?
For on-device execution, how high of VRAM or system RAM use end-to-end (including KV cache) do you consider on-device?
For personal devices, how far of a price in USD do you consider as personal devices?
(2) None of the techniques are unique to on-device, personal AI. Many inference providers have a dedicated engineering team optimizing their serving stack over different backends (vLLM, SGLang, ...), quantizations (bf16, FP8, NVFP4, MXFP4), and their clients apply task-specific prompt tuning (GEPA, ...) to optimize their executions. Agent-based hill-climbing/optimization is a well-known technique in all parts of this stack.
I do find the combined optimization space of OpenJarvis to be valuable and interesting. However, it is still very unclear why these techniques are framed as on-device. Each individual technique has its cloud counterparts, and OpenJarvis simply ports them.
A larger model being able to increase a smaller model's accuracy through evolution is well known. These are properties of the models, not on-device or server-specific findings. GEPA achieves accuracy increases in smaller models entirely on server hardware.
(3) The evaluations are far from real-world on-device, personal-AI environments. I do not understand why you think "Search runs without a curated dataset and without the cloud" benchmark numbers with teacher-synthesized tasks answer the personal dataset problem, when I am clearly asking about a more fundamental question of privacy.
Yes, the 122B model can hill-climb an 8B model to a higher benchmark score, but who can run a 122B model on-device? Most personal consumers cannot afford a $4,699 device for AI. Your counterargument of "accuracy measured on devices a person can buy" honestly sounds ridiculous - yes, "A person" can buy "devices" if they pay enough money. But does that justify anything? You are just stating something very obvious, like "water is wet". Reads very AI-generated.
(4) What does your response W3 even mean?
"Table 5’s accuracy, including 122B’s 80.3%, ran on H100 servers. A 122B model on a $4,699 Spark is desktop-class. The best on-device configuration (122B FP4) scores 76.4%, 7.1 pp below Claude’s 83.5% [−10.4, −3.8]. “Within 3.2 pp” describes H100 execution only."
So are the headline numbers on H100 or Spark? Also, my argument is that a 122B model is not an on-device model because it is too big. Are you trying to say it is an on-device model because it ran on a $4,699 Spark (desktop-class)? Why did you add the price?