Benchmark · arXiv 2606.08034 · 2026

Sci-ρ: A Multilingual Visually-Grounded Symbolic Benchmark for STEM Problems

Muhammad Falensi Azmi1*, Ikhlasul Akmal Hanif1*, Vallerie Alexandra Putra2, Adi Yeltay1, Abdullah Mubarak3, Fajri Koto1

1MBZUAI · 2Binus University · 3Bandung Institute of Technology · *Equal contribution

Symbolic benchmarks have emerged as a key approach to assess model robustness under minor modifications to STEM-related questions. However, existing symbolic benchmarks mostly remain limited to mathematical reasoning, lack visual grounding, and are predominantly in English.

We introduce Sci-ρ (Science Rhobustness), a dynamic benchmark for visually-grounded STEM problems spanning five subjects and seven languages, comprising 4,242 problem templates (606 per language) crafted by domain experts, including Olympiad medalists. Each template is implemented as executable Python code that generates diverse but equivalent problem instances by varying numerical values, visual patterns, geometric shapes, color schemes, and function types, resulting in 42,420 instances in total, each paired with reasoning steps and ground-truth solutions.

Two variants of the same parabola-and-secant problem. GPT-5.4 answers the first correctly (x = -0.40) but on the second variant assumes A and B are symmetric and answers 0 instead of 0.01.
An illustrative failure case of GPT-5.4 when evaluated across problem variants in our dataset, highlighting vulnerabilities in VLM robustness.

Sci-ρ Dataset

Each template consists of one image and a description in seven languages: English, Arabic, Chinese, Hindi, Indonesian, Kazakh, and Swahili. The machine-translated descriptions are verified by native speakers of the corresponding languages.

Left: one circuit diagram paired with descriptions in seven languages. Right: templates per subject (Mathematics 199, Physics 195, Chemistry 137, Biology 23, Computer Science 52), question types (90% short answer, 6% MCQ, 4% free-form) and level (41% GCSE, 59% A-Level). Bottom: variant types: pattern, shape, numerical, function type, colour.
A glimpse of our dataset. (a) An example from our dataset where each template consists of one image and description in seven languages; (b) Statistics of our dataset; (c) Example of variants included in our dataset.
199Mathematics
195Physics
137Chemistry
52Comp. Sci.
23Biology

We report average-case accuracy (Aavg) and worst-case accuracy (Awst), defined as the proportion of problem templates that a model answers correctly across every generated variation. Δ denotes the reduction Awst − Aavg.

English-only Result

Among all evaluated models, GPT-5.4 achieves the highest performance, securing over 90% average accuracy in Mathematics, Physics, Chemistry, and Computer Science. Despite remarkable average performance, frontier models struggle to solve problem variants consistently, exhibiting performance drops (Δ) ranging from −10% to −28% in closed-weight models. Smaller open-weight models suffer even more severe degradation.

English-only results on Sci-ρ
Aavg and Awst per subject; Δ on the right.
AavgAwstclosed-weight

Multilingual Results

Language variants have minimal effect on proprietary and large open-weight models, with GPT-5.4 and Gemini 2.5 showing near-zero degradation across all tested languages. In contrast, smaller open-weight models exhibit notable performance drops in non-English settings, with degradation becoming more pronounced in lower-resource languages.

Grouped bar chart of average-case accuracy gap relative to English for each model and language. Proprietary models stay near zero; small open-weight models show negative gaps, largest for Swahili.
Average-case accuracy gap (ΔAavg) relative to English across languages and models. Negative values indicate performance degradation in non-English settings.

Attention Heads Analysis

We examine the attention heads of Qwen3-VL-8B-Instruct at the final layer and compute the Image Attention Ratio (IAR), the attention directed towards image tokens relative to text tokens during response generation. The IAR varies across languages, with Chinese exhibiting the lowest ratio, while Kazakh and Hindi yield the highest.

Five line charts, one per subject, showing Image Attention Ratio across normalised generation steps for each of seven languages, with 95% confidence intervals.
IAR trends across subjects and languages for Qwen3-VL-8B-Instruct (95% confidence intervals are shown).

Reasoning Step Performance

To analyze reasoning quality, we conduct step-level evaluation on five models, measuring precision (𝒫), recall (ℛ), and F1 (ℱ) of the reasoning steps. Step-level evaluation reflects the same trend, revealing a significant gap between average F1 and worst-case F1 scores.

Reasoning step performance
Each cell shows ℱ / ℱwst.

Data and code

Citation

@article{azmi2026scirho,
  title   = {Sci-Rho: A Multilingual Visually-Grounded Symbolic Benchmark for STEM Problems},
  author  = {Azmi, Muhammad Falensi and Hanif, Ikhlasul Akmal and Putra, Vallerie Alexandra and Yeltay, Adi and Mubarak, Abdullah and Koto, Fajri},
  journal = {arXiv preprint arXiv:2606.08034},
  year    = {2026}
}