Results in detail
Prototype results, still work in progress. The short version is on the overview.
Loading results…
Every run: accuracy against output tokens
One dot per run. Color = model weights, shape = where it ran. Runs that returned no usable answer sit below the axis.
Three findings
Finding 1
Image resolution matters most
Finding 2
Same Nemotron weights: fewer failed runs on NVIDIA's endpoint
Finding 3
Nemotron spends far more output tokens, and "low" effort didn't lower them
| Setting sent | Reasoning tokens |
|---|---|
| none (default) | 563 |
reasoning_effort: "low" | 664 |
reasoning_effort: "high" | 821 |
enable_thinking: false | 0 |
Time and cost per run
Time is wall-clock per run, end to end. NVIDIA API catalog (hosted) runs used trial credits, so no price is shown.
What I'd ask NVIDIA
- Honor
reasoning_efforton the NVIDIA API catalog (hosted), or document the supported way to cap reasoning. "low" produced more reasoning tokens than the default in my text test (n=1 per setting), and on drawings some replies used the whole output budget on reasoning and returned no answer. - Document the input resolution Nemotron VL expects: native size, whether large images are tiled or downscaled, and what to send for wide engineering drawings. The same drawing scored very differently at 1024 px and 1642 px.
How results are measured
- A synthetic 735/315 kV single-line diagram with a hand-checked answer key of 66 items.
- Every model gets the same drawing and the same instructions. Each result must form a valid IEC 61850 SCL file under IEC's public schema.
- Scores count items that exactly match the answer key.