acceptodds
Under review as a conference paper at ICLR 2027

OpenJarvis: Personal AI, On Personal Devices

Abstract

Personal AI stacks for writing, research, coding, and scheduling are becoming central to daily work, yet most still route every query to cloud-hosted frontier models. This paradigm exposes private data and creates recurring API spend. Excitingly, open-source models and consumer accelerators are significantly improving, enabling the possibility of a personal AI stack running on-device. However, we find that replacing cloud models with local models in existing personal AI stacks, such as OpenClaw and Hermes Agent, leads to significant deterioration in performance. For example, replacing Claude Opus 4.6 with Qwen3.5-9B drops accuracy by 25–39 percentage points across PinchBench and GAIA. This collapse reflects both a model-capability gap and harness incompatibility: agentic prompts, tool descriptions, memory configuration, and inference runtime settings carry model-specific defaults that materially contribute to the substitution gap. Towards building an on-device personal AI stack, we present OpenJarvis, an architecture that represents a personal AI system as a typed spec over five primitives: Intelligence, Engine, Agents, Tools & Memory, and Learning. By exposing each primitive as an independently editable field, the spec makes the stack portable, measurable, and end-to-end optimizable around any choice of model. On-device specs match or exceed cloud accuracy on 4 out of 8 benchmarks, and the best one (Qwen3.5-122B) lands within 3.2 pp of the best cloud baseline on average with up to a 6,600× reduction in marginal API cost; Qwen3.5-35B achieves 4× lower end-to-end latency. To close the remaining accuracy gap between the best cloud model and the best local model, OpenJarvis uses LLM-guided spec search, in which a frontier teacher proposes edits across the spec and a held-out gate accepts only non-regressing improvements. Search improves local students by 13–32 pp on average at 7–11× lower optimization cost than LoRA fine-tuning, with student inference and training on-device.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
personal AI, on-device inference, local language models, agentic systems, agent frameworks, LLM-guided search, configuration search, prompt optimization, inference efficiency, energy measurement, systems benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_peLiOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OpenJarvis defines on-device agent harnesses as a combination of five different specs and uses a cloud teacher model-based optimization loop to execute per-benchmark search of the best configurations across the five different specs to achieve near cloud model-level accuracy on the given benchmarks.

Critical strengths

OpenJarvis provides a method to improve the local on-device model's performance to that of the cloud model on the targeted benchmarks through the use of an evolution loop on the on-device agent system driven by a cloud model. OpenJarvis also has a larger search space, targeting the entire agent system stack, including parts such as the inference engine or tools, unlike existing research such as GEPA.

Critical weaknesses
  1. The technique itself seems to be a general LLM-based agent system optimization method, and it is unclear why the authors are targeting on-device private AI agent scenarios. Distillation of frontier-model capabilities to smaller models through an optimization loop is a well-studied technique (GEPA), and therefore the five-spec optimization method should work for any agentic harness or other parts of the agentic system stack.

  2. Optimizations are task-specific. Users would need to generate and gather a non-private training dataset and run optimization for different sets of their private tasks.

  3. The model used for the headline accuracy figures (Qwen3.5-122B) is not an on-device model.

Questions for authors

Why is this method framed as an on-device method? End users rarely have the time and resources to generate and gather a non-private training dataset and run cloud-model-based optimization loops to improve their private workloads. I cannot agree that OpenJarvis would be something that individual users will actively use to improve their personal AI stack.

Wouldn't this method be more beneficial for a batch execution case where such optimization overhead can be amortized across a large number of similar tasks, as many existing LLM-based agentic system optimization methods such as GEPA do?

Reviewer_nVTVOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
3
Read review
Paper summary

The paper talks about a high level framework on how to work LLM frameworks such as OpenClaw work well with local models. This is not obvious since, dropping a local model into the stack makes it much more disruptive, partly due to model capability gap, but also due to harness imcompatibility.

In the paper they invent some core primitive abstractions that describe the spec of execution on the local machine, and propose LLM guided optimization algorithm in order to do a smart search in the space of specs. The primitives are,

  • Intelligence
  • Engine
  • Agents
  • Tools and Memory
  • Learning One optimization meta function, covers the optimizer step.

Now, for a given task, a teacher LLM helps the setup go through the steps of, Diagnose, Planing and executing. Whenever there is a failure, the teacher model analyzes the failure and suggests edits on the spec. Each edit is tested out on a held out set. This is called a gate, since it is accepted only if it improves on the held out set. It could even suggest a training step which falls under Learning primitive. Hence it provides a LLM guided set up optimal setup search for the local LLM to work on the real task at hand.

They do experiments on 8 benchmarks, 11 local models from 4 families, 3 cloud baselines and 7 platforms. Results are the mean of 5 runs.

The results suggest that retargeting the spec saves 56-77% of the substituiton drop. No single component recovers more than 27%. Also local models match or beat the best cloud score on 4 out of 8 benchmarks. The API cost is heavily saved since its only used in setup time. The paper also notes 4x lower latency for Qwen-35b model on agentic workloads. LLM guided search even improves the local student models on its own benchmarks, (13-32pp) at much lower cost. And the gain is consistent across multiple benchmarks.

Overall the paper seems to have a good abstraction/structure to make a local model optimize towards a local hardware, authonomously guided by a stronger LLM.

Critical strengths
  • Addresses a pretty useful problem for local hardwares. Which has benefits in privacy, API spend etc.
  • Identifies the core issues such as harness incompatibility.
  • More importantly, identifies a high level workflow using LLM intelligence in order to self fix and self optimize, in an autononous way.
  • Identifies core abstractions/primitives in the high level problem. This enforces structure which then LLMs can build upon in order to solve a real problem.
  • Evaluation is well done, covers a broad range and measure impact across several dimensions.
Critical weaknesses
  • The failure clusters are LLM defined and relies on LLM to detect them. Only can fix what the LLM idenfities
  • Search is task specific. And needs a specific held out set to gate it properly, which needs some set up to do. Also expensive upfront.
  • Novelty is somewhat incremental. The paper defines the abstractions a bit more concretely and extracts a high level workflow and also similar to GEPA.
  • How the benchmarks based testing translate to real world usecases, for local models, is not quite clear.
Questions for authors
  • Any attempts to validate the GPT5Mini judge with another model or human labelled sample, especially on benchmark where it grades?
  • Any chance you have confidence intervals for the main table?
  • How do the break even numbers differ when user's workload includes search cost and weigh training also?
Reviewer_Rts5OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies what breaks when a user replaces the cloud model inside a shipping personal-agent stack with a local one, holding everything else fixed

  • in OpenClaw, swapping Claude Opus 4.6 for Qwen3.5-9B drops PinchBench and GAIA
  • in Hermes Agent the same swap also drop eval scores

The authors trace a substantial share of this to named defaults in the two codebases, not to the model itself

  • Hermes only appends its tool-use enforcement prompt
  • OpenClaw silently assigns uncatalogued local models a 200k-token default context window

OpenJarvis makes those hidden defaults explicit as a spec: one TOML configuration with slots for the model and its quantization/decoding , the serving backend and KV-cache settings, the agent loop, system prompt, and turn limit, the enabled tools, their descriptions, and the memory backend, and the optimizer

Critical strengths
  • this paper quantifies overfitting of prompts and tool descriptions. no single component recovers most of the eval score drop. how much of "local models are worse" is actually "the stack was built for a different model"
  • the finding is grounded in concrete mechanism, not just aggregate scores, Hermes's TOOL_USE_ENFORCEMENT_MODELS allowlist omitted Qwen, and OpenClaw's 200k-token context fallback for uncatalogued local models
  • joint optimization across components is demonstrated, not just asserted. The spec would be a mere taxonomy if single-component optimization sufficed
  • a practically important negative result about weight training. non-weight edits are cheap, inspectable, and reversible, this is a useful corrective to a weights-first default
Critical weaknesses
  • the efficiency headline should use the paper's own conservative numbers: 6,600× lower cost compares local marginal API fees against uncached Claude Opus 4.6 pricing with local inference at $0. the appendix does the honest accounting of ~120× once prompt caching and electricity are included
  • Table 20 indicates a couple of points pair accuracy from one quantization with efficiency from another (e.g., Qwen3.5-122B FP8 accuracy / FP4 efficiency). Measured, single-configuration points would make Fig. 3 fully convincing; at minimum the figure caption should state the estimation method
  • Some benchmarks are small or saturated: ToolCall-15 has 15 tasks, PinchBench 23, and GAIA is subsampled to 50. 80.3% vs. 83.5% is best interpreted as parity within measurement precision
  • the search-gain framing should distinguish recovery from new capability. The largest gains in Table 8 occur where the unoptimized baseline was clearly misconfigured for the student. recovering that loss is the point
  • gate protocol needs one clear paragraph. the gate draws on teacher-annotated traces, public agentic datasets, and benchmark splits. What I could not reconstruct in one place is the per-benchmark split accounting: diagnosis vs gate vs reported test
Questions for authors
  • you state the gate uses a held-out split separate from the reported test accuracy, and that adapter training data contains no test items. Could you clarify the same point for the diagnosis traces and the non-weight edits: which data does the teacher see during diagnosis, and is it disjoint from the final test sets?
  • The speed and energy numbers in Figures 1 and 3 appear to be estimates rather than direct measurements. Were any of them measured on real hardware, and if so, which ones?
  • Table 1 shows retargeting alone recovers much of the substitution drop, while Table 8 measures search from an unoptimized OpenJarvis spec. How do these two baselines relate, and does spec search still add gains when started from an already-retargeted spec?

Comments on the paper

2 comments

Thank you for your response.

I worked on both mobile on-device LLMs (smartphones) and inference engines (vLLM on DGX/AMD boxes). Based on the responses, my honest evaluation of OpenJarvis is a combination of existing agent optimizations from the cloud ported to on-device . As a mobile systems researcher, this is quite heartbreaking. I have a few clarifying questions.


(1) Can you please define (1) Personal AI, (2) On-device execution, and (3) Personal Devices you used in your paper?

To be specific:

  • For personal AI, what is the largest number of parameters you consider as "Personal AI"?

  • For on-device execution, how high of VRAM or system RAM use end-to-end (including KV cache) do you consider on-device?

  • For personal devices, how far of a price in USD do you consider as personal devices?


(2) None of the techniques are unique to on-device, personal AI. Many inference providers have a dedicated engineering team optimizing their serving stack over different backends (vLLM, SGLang, ...), quantizations (bf16, FP8, NVFP4, MXFP4), and their clients apply task-specific prompt tuning (GEPA, ...) to optimize their executions. Agent-based hill-climbing/optimization is a well-known technique in all parts of this stack.

I do find the combined optimization space of OpenJarvis to be valuable and interesting. However, it is still very unclear why these techniques are framed as on-device. Each individual technique has its cloud counterparts, and OpenJarvis simply ports them.

A larger model being able to increase a smaller model's accuracy through evolution is well known. These are properties of the models, not on-device or server-specific findings. GEPA achieves accuracy increases in smaller models entirely on server hardware.


(3) The evaluations are far from real-world on-device, personal-AI environments. I do not understand why you think "Search runs without a curated dataset and without the cloud" benchmark numbers with teacher-synthesized tasks answer the personal dataset problem, when I am clearly asking about a more fundamental question of privacy.

Yes, the 122B model can hill-climb an 8B model to a higher benchmark score, but who can run a 122B model on-device? Most personal consumers cannot afford a $4,699 device for AI. Your counterargument of "accuracy measured on devices a person can buy" honestly sounds ridiculous - yes, "A person" can buy "devices" if they pay enough money. But does that justify anything? You are just stating something very obvious, like "water is wet". Reads very AI-generated.


(4) What does your response W3 even mean?

"Table 5’s accuracy, including 122B’s 80.3%, ran on H100 servers. A 122B model on a $4,699 Spark is desktop-class. The best on-device configuration (122B FP4) scores 76.4%, 7.1 pp below Claude’s 83.5% [−10.4, −3.8]. “Within 3.2 pp” describes H100 execution only."

So are the headline numbers on H100 or Spark? Also, my argument is that a 122B model is not an on-device model because it is too big. Are you trying to say it is an on-device model because it ran on a $4,699 Spark (desktop-class)? Why did you add the price?

Reviewer_peLiOpenReview ↗

Fully AI-generated response?

Upon closer reading, the entire response reads as AI-generated.

A few examples:

  • "The review's concerns are about framing."
  • "The headline reports accuracy measured on devices a person can buy."
  • "The mechanism is general; the payoff is not."
  • "Results are in new Table R10. (newline) Table R10."
  • "A 122B model on a $4,699 Spark is desktop-class."

I adjusted my score accordingly. I know this year's ICLR is suffering from a massive influx of AI-generated submissions, but it is quite frustrating to spend my human time on AI-generated responses.

Edit: This response was based on the 08 Nov 2026, 18:27 Pacific Standard Time version of the author's initial response. The authors fixed their response a bit afterwards, such as the table R10 formatting issue. You can still checked the version I read using the revisions button.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.