Research

What language models can do in Indonesia's languages, and what the benchmarks can't tell you

We ran thirteen language models through every public Indonesian-language benchmark we could score. The best are strong in Indonesian, Javanese and Sundanese; the benchmarks stop being useful right where real conversation starts.

Baibu Research 19 min read

Baca dalam Bahasa Indonesia

Baibu builds AI for Indonesia, in the languages people speak at home. Indonesia has more than 700 languages. Most of the people we build for speak Indonesian and at least one of them, usually Javanese or Sundanese, and they switch between the two inside a single message. We wanted to know two things: how well today’s language models handle those languages, and how much the public benchmarks can actually tell us about it.

So we ran thirteen models, nine with open weights and four closed, through every public Indonesian-language benchmark we could obtain and score reproducibly. That came to seven benchmarks covering 23 language varieties, counting the formal and informal registers of Javanese, Sundanese and Madurese separately. Every model got the same protocol on each benchmark: the same prompts, greedy decoding, and each model’s reasoning set as low as it would go.

The short version: the best models are strong in Indonesian, Javanese and Sundanese, and the benchmarks stop being useful right about where real conversation starts.

This post recommends no model. It describes the field and what is missing from it.

What the models can do

Model Reading: Indonesian Reading: Javanese Reading: Sundanese School exams: 7 other languages Replies in Minangkabau
Sahabat-AI v2 70B · open 92.4 88.3 86.1 47.0 56%
Aya Expanse 32B · open 90.1 72.7 64.6 42.3 0%
Gemma 4 31B 31B · open 93.9 90.7 90.2 59.7 80%
Muse Glimmer 30B · open 92.7 88.0 86.9 45.6 55%
Qwen 3.8 27B · open 93.6 88.4 88.3 43.5 4%
SEA-LION v4.5 27B · open 93.4 90.4 88.3 45.4 1%
Qwen 3.5 9B · open 90.3 83.6 80.0 42.8 1%
Gemma 4 E4B 8B (4B effective) · open 88.3 79.9 77.0 41.2 0%
Tiny Aya 3B · open 77.6 68.0 55.3 35.9 5%
Gemini 3.1 Flash Lite closed 94.1 91.7 89.7 67.5 64%
Gemini 3.5 Flash Lite closed 90.8 89.0 86.9 62.3 85%
GPT-5.6 Luna closed 93.1 89.9 88.8 57.7 5%
GPT-5.6 Terra closed 95.1 92.3 90.4 57.6 12%

Accuracy in %. Reading is Belebele. School exams averages IndoMMLU’s sections on seven other regional languages and cultures. The last column is the share of Minangkabau prompts answered mostly in Minangkabau. Open models are ordered by size, not score. Instruction following is in Appendix A, the full notes in Appendix C.

Indonesian, Javanese and Sundanese are close to solved for reading comprehension. On Belebele, the strongest models score 93–95% in Indonesian and lose only 2–5 points in Javanese and Sundanese. That includes the best open model in the survey, Gemma 4 31B (93.9 / 90.7 / 90.2), which sits level with the closed references: Gemini 3.1 Flash Lite at 94.1 / 91.7 / 89.7 and GPT-5.6 Terra at 95.1 / 92.3 / 90.4. Instruction following on SEA-IFEval shows the same picture, with the top of the field at 92–98% in all three languages (Appendix A).

Size still matters, and it matters most in Sundanese. On Belebele, every model loses more on Sundanese than on Javanese, but by very different amounts: the larger models drop 4–6 points from Indonesian to Sundanese, and the three models under 10B parameters drop 10–22. Tiny Aya scores 77.6 in Indonesian, 68.0 in Javanese and 55.3 in Sundanese, and 58.1% on Sundanese instruction following against 85.7% in Indonesian. Size is not the whole story: Aya Expanse 32B drops 26 points in Sundanese, more than any of the small models.

The regional models are not ahead. Sahabat-AI 70B, trained for Indonesian languages, is competitive but not better. It scores 88.3 and 86.1 on Javanese and Sundanese Belebele, a few points behind general-purpose models less than half its size. It does lead the open models on open-ended questions with no passage to read (29.2% in Indonesian against 23.5% for Gemma 4 31B), which is what training on regional text should buy. SEA-LION v4.5, the Southeast Asian regional model, uses a tokenizer byte-identical to the Qwen model it is built from, so it carries no representation advantage at the level of text encoding.

After Indonesian, Javanese and Sundanese, the field falls off. The benchmarks reach 17 more languages, and every model does worse in them:

  • On IndoMMLU’s school-exam questions about seven other regional languages and cultures, the best model averages 67.5%. Open models average 36–60%. The best models score 79–85% on the Indonesian, Javanese and Sundanese sections.
  • On LoraxBench’s open questions, Batak Toba, Buginese and Makassarese answers are correct 1–17% of the time, against 12–34% in Indonesian.
  • Translating Batak Toba, Buginese or Makassarese into Indonesian scores chrF 44–68, against 67–86 for Javanese and Sundanese.

On knowledge and open questions, the closed models hold up better in the long tail than the open ones, and the gap is widest exactly where there is least training text. Translation is the exception: there the closed models are no more consistent than the open ones.

Javanese and Sundanese cost more to process. Every open model’s tokenizer needs 16–45% more tokens for the same content in Javanese, and 22–51% more in Sundanese, than it needs in Indonesian. That is a direct cost and latency tax on the languages, invisible in every accuracy table. The measures also disagree: Sahabat-AI’s tokenizer has the smallest Javanese and Sundanese penalty relative to its own Indonesian, and is the most expensive in absolute terms in all three languages.

Does the answer come back in the language you asked in?

Nothing in the public suite checks this, so we measured it. We ran a language identifier over the models’ saved answers and asked, for each question put in Javanese, Sundanese or Minangkabau, what language the answer was mostly in. We first validated the classifier on benchmark text of known language: cut to ten words, it is 95–97% accurate for Indonesian, Javanese and Sundanese. It was not validated on model answers themselves.

For Javanese and Sundanese, most models do answer in the language they were asked in. The strongest open and closed models do so 94–100% of the time. Three models do not:

  • Aya Expanse 32B mostly answers Javanese and Sundanese in plain Indonesian: 66–87% of the time, depending on the task.
  • Gemma 4 E4B, the small Gemma, answers in the asked language only 65–84% of the time.
  • Tiny Aya appears to answer Sundanese questions in Javanese. The identifier labels 48% of the words in its Sundanese safety answers as Javanese, and 26% as Sundanese; on instruction following the split is 28% and 46%. Its Javanese answers are 89–95% Javanese. The identifier misreads real Sundanese as Javanese 5 times in 1,591, so this is unlikely to be a classifier artifact, but we have not had a speaker read the answers.

Minangkabau is failed across the field. On IndoSafety’s Minangkabau prompts, 8 of 13 models answer in Minangkabau less than 15% of the time, and answer in Indonesian instead. That includes both GPT-5.6 models and SEA-LION. IndoSafety is a safety prompt set, so we checked a second, unrelated one: LoraxBench’s open questions in Minangkabau. The same models fail there, all but SEA-LION (20%) under 15%, and no model answers in Minangkabau more than 38% of the time. (Only answers of ten or more words are classified; on the open questions that excludes 25–78% of answers, which are names and numbers.) The answers can be correct, and every Minangkabau speaker can read Indonesian, which is exactly why no accuracy score will ever report it. A model can score well on a Minangkabau benchmark without ever answering in Minangkabau.

What the benchmarks can’t tell you

This is the part we think matters most.

Appendix B lists every benchmark we ran and what it cannot tell you. Five problems stand out.

A conversation benchmark you can pass without reading the conversation

PingPong is the only public collection of group chats natively written in mixed Javanese–Indonesian–English and Sundanese–Indonesian–English, and we expected it to be the closest thing to real use in the suite. Its question-answering task asks multiple-choice questions about a chat. Some have an answer in the chat. For the rest, the correct choice is a fixed option, “there is no correct answer”.

So we computed what a rule that never looks at the chat would get. “Pick the longest option” scores 71% and 74% on the unanswerable halves of the Indonesian–English and Javanese sets, and 69% on the answerable half of the Sundanese set. Random guessing scores 19%.

The cause is option length. In the Indonesian–English and Javanese sets the real answers are short, a name or a job, so the fixed “no correct answer” option is almost always the longest. In the Sundanese set the answers are full phrases, so the correct answer itself is usually the longest.

The consequences for model scores:

  • Of the thirteen models, one beats the length rule on the Indonesian–English and Javanese unanswerable questions.
  • On the Sundanese answerable questions, every model scores 91–98%, so the column no longer separates anything. The same models range from 10% to 57% on Indonesian–English. Tiny Aya scores 10.1% on one and 91.3% on the other, which no account of ability explains and option length does.
  • Averaging the two halves, as a leaderboard naturally would, ranks models by how often they say “no answer” rather than by comprehension.

None of this makes the chats less valuable. It means the question set cannot be read as a comprehension score without a no-model baseline beside it, and nobody had published one.

A reading test that scores length, with an answer key that isn’t in the passage

LoraxBench’s reading-comprehension task gives a passage and a question and asks for a short span from the passage. It is scored, as the paper scores it, by whether the reference answer appears anywhere in the response. That rule has two problems, one in each direction.

In Indonesian it rewards saying more. The reference answers are about three words long. A response that contains them anywhere scores, however much else it says. The smallest Gemma tops the Indonesian column at 89% while matching the reference exactly only 23% of the time; Muse Glimmer is third at 86% with 2% exact. A rule that never runs a model, returning the passage sentence most like the question, scores 66%. Across the thirteen models the Indonesian column tends to rank them in the opposite order to Belebele and to instruction following.

Elsewhere the answer often isn’t in the passage. For an extractive task the reference should be a span of the passage. In Indonesian it always is. In Javanese it is 62% of the time, in Sundanese 53%, and across the other 18 varieties 64% (55–81%, depending on the variety). The missing references share about half their words with the passage: the same answer, it seems, translated separately from the text it answers. A model that copies the right span, as instructed, scores zero on those items, and every model does. So the whole field sits between 38% and 59%, under a ceiling set by the answer key. On the items where the answer is in the passage, the same models score 77–93% in Javanese and 73–94% in Sundanese.

The column still orders the models sensibly outside Indonesian. Its level does not mean what a reader would take it to mean, and nothing in the published scores signals it.

A safety judge that goes easy on its own vendor

Safety scores for Indonesian are usually produced by an LLM judge reading each response. We judged every model’s IndoSafety responses with GPT-5.6 and re-judged a stratified sample of 4,613 of them with Gemini 3.5 Flash, which is not one of the models tested.

The two judges disagree, and the disagreement follows the vendor. GPT-5.6 grades every model about 7 points harsher than Gemini does, except OpenAI’s own, where the two judges agree. On the seven models from neither vendor, GPT-5.6 marks 15.5% of answers unsafe and Gemini 8.8%; on OpenAI’s models the figures are 7.7% and 7.4%. Measured against the other models, that is 6.4 points of leniency toward its own vendor [95% CI 4.8, 8.0], and it is robust. The Gemini judge shows a smaller lean toward Google’s models, Gemma included: 2.2 points [0.1, 4.3], only just distinguishable from zero.

Under the GPT judge, the OpenAI models look about twice as safe as the field (7.7% unsafe against 15.5%). Under the Gemini judge they look barely safer (7.4% against 8.8%).

The judges also report nearly identical unsafe rates across Indonesian, Javanese, Sundanese and Minangkabau. We can’t tell that apart from the judges failing to read the regional languages, because IndoSafety ships no gold labels. For both reasons we are not publishing a per-model safety ranking. We had planned a full second judging pass in case the judges disagreed; when they did, we declined it, because it would have sharpened a ranking neither judge can support. That decision was made after we saw the cross-check, in the direction of publishing less. We know of no published Indonesian safety number that controls for either effect.

Scoring artifacts the size of model differences

Twice during the survey a cell moved by more than our measured noise floor, about 5 points, without any model changing:

  • Lead-ins were scored instead of answers. A translation benchmark’s reference scorer read “Here is the translation:” instead of the translation.
  • A menu was scored instead of an answer. A small model answered each translation request with three labelled options, and the scorer graded the label.

In other cases the model could not be scored at all:

  • A token budget sized for short answers left nothing to score. Thinking-enabled models spent a short-answer budget on reasoning and returned empty strings.
  • Reasoning traces landed in the scored text. Three models returned their thinking inside the answer.

We caught each one because we kept per-task protocol checks and reran what they flagged. A published table in this field is fragile at the 5-point level. Most say nothing about it.

Everything is clean, single-turn, and mostly translated

Every benchmark we ran except PingPong is monolingual text. Every one is a single turn: a passage and a question, an exam item, one instruction. Most are translated or machine-rewritten. Belebele is translated from English. The Indonesian–English code-mixing benchmark we considered was rewritten into mixed language by one of the models it would be used to test. IndoMMLU, PingPong and the SEA-HELM metaphor set are the exceptions, written natively; LoraxBench starts from native Indonesian material but renders it into the other varieties.

People don’t talk to an assistant in any of those registers. Mixing Javanese or Sundanese with Indonesian is the default in everyday conversation, not an edge case. No public benchmark measures Indonesian mixed with a regional language. The code-mixing sets that exist cover Indonesian with English, or with Toba Batak over English trivia. There is no multi-turn Indonesian benchmark at all.

Javanese and Sundanese coverage is also thin where it exists. Most of it is multiple-choice reading comprehension, and regional suites often leave both languages out of their headline aggregate, even where individual tasks include them.

What would need to exist

The survey tells us what a useful instrument for these languages would have to do. It should:

  1. Be natively authored. Conversations should be written by speakers in their own mixing habits, not translated or generated.
  2. Be multi-turn and mixed. It should cover Indonesian with Javanese and Sundanese in both directions, at the levels people actually use.
  3. Check the response language on every item. It is nearly free, and as the Minangkabau result shows, it catches a failure no accuracy score will.
  4. Test register as a choice. Given who the speaker is addressing, which register is right? Nothing public asks this, and it is the error that actually offends.
  5. Be scored without a judge wherever possible. Where a judge is unavoidable, it should be validated against native speakers and cross-checked across vendors, with the cross-check published.
  6. Ship its own baselines. It should publish random, constant-answer and “longest option” scores, and separate halves where a single average would hide a bias, before any model is scored.
  7. Check its own answer key in every language. If the answer should be in the text, verify that it is, in each translation, and score exact answers rather than “contains the answer”.

Baibu’s users write in exactly this register, so this is the instrument we intend to build.


Appendix A: Instruction following

Model Indonesian Javanese Sundanese
Sahabat-AI v2 70B · open 93.3 90.5 86.7
Aya Expanse 32B · open 84.8 55.2 48.6
Gemma 4 31B 31B · open 98.1 92.4 92.4
Muse Glimmer 30B · open 89.5 85.7 69.5
Qwen 3.8 27B · open 89.5 90.5 86.7
SEA-LION v4.5 27B · open 93.3 93.3 84.8
Qwen 3.5 9B · open 91.4 85.7 77.1
Gemma 4 E4B 8B (4B effective) · open 91.4 86.7 81.0
Tiny Aya 3B · open 85.7 81.9 58.1
Gemini 3.1 Flash Lite closed 97.1 93.3 91.4
Gemini 3.5 Flash Lite closed 89.5 92.4 86.7
GPT-5.6 Luna closed 93.3 92.4 86.7
GPT-5.6 Terra closed 96.2 93.3 94.3

Accuracy in %, SEA-IFEval. Repeat runs vary by up to 4.8 points, so smaller gaps are noise.

Appendix B: What each benchmark can’t tell you

Benchmark Languages we ran Built from Task What it can’t tell you
Belebele ID, JV, SU Translated from English (FLORES) Multiple-choice reading Near ceiling: the top models score 93–95% in Indonesian and within 5 points in JV and SU
IndoMMLU ID, JV, SU and 7 other regional languages and cultures Indonesian school exams Multiple-choice knowledge Exam knowledge only; nothing is generated
LoraxBench reading 23 varieties Indonesian datasets, rendered into each variety Copy a short answer from a passage Scored by “contains the answer”, which rewards long replies; outside Indonesian, 20–49% of the answer key is not in the passage
LoraxBench open questions and translation 23 varieties As above Answer without a passage; translate into Indonesian Open questions use the same “contains” rule, so levels carry the length bias
SEA-IFEval ID, JV, SU Native-speaker translations of English IFEval Follow checkable format instructions Checks format, not content or register; repeat runs vary by up to 4.8 points
SEA-HELM sentiment JV, SU NusaX, translated by native speakers from Indonesian Three-way sentiment label A label on one sentence; nothing is generated
SEA-HELM metaphor JV, SU Multilingual Fig-QA, written by native speakers Pick the meaning of a figurative phrase Binary choice, so chance is 50%; the JV and SU sets are not parallel
IndoSafety ID, JV, SU, Minangkabau Adapted from English safety prompt sets LLM judge rates each answer No gold labels; each judge favours its own vendor and reports flat rates across languages
PingPong JV–ID–EN, SU–ID–EN, ID–EN Group chats written natively in mixed language Multiple-choice questions about a chat For 12 of 13 models, no better than “pick the longest option” on the unanswerable ID–EN and JV questions
Nothing public ID mixed with JV or SU — Multi-turn conversation; register choice —

Appendix C: Method notes

  • Main results table. Reading is Belebele, run 5-shot. School exams is the average of IndoMMLU’s seven sections on regional languages and cultures other than Indonesian, Javanese and Sundanese: Balinese, Banjarese, Dayak Ngaju, Lampung, Madurese, Makassarese and Minangkabau. Replies in Minangkabau is the share of IndoSafety’s Minangkabau prompts answered mostly in Minangkabau, by language ID; the rest are mostly Indonesian or mixed. Gemini 3.8 Flash could not be scored on the short-answer tasks and is omitted.
  • Models. Gemma 4 31B and E4B, Qwen 3.5 9B and Qwen 3.8 27B, SEA-LION v4.5 27B, Sahabat-AI v2 70B (served in FP8), Aya Expanse 32B, Tiny Aya, and Muse Glimmer 30B. Closed references: Gemini 3.1 and 3.5 Flash Lite, and GPT-5.6 Luna and Terra. Gemini 3.8 Flash could not be scored on short-answer tasks and is reported as unrunnable there. Open models ran at bf16 unless noted, on revisions pinned before any scores were read. Closed models are identified by their API name, called in September 2026; we did not record the vendor’s served snapshot, and vendors can update a model behind its name.
  • Benchmarks. Belebele, IndoMMLU, LoraxBench, SEA-IFEval, IndoSafety, two SEA-HELM Javanese–Sundanese tasks, and PingPong. All were scored generatively with an identical protocol per task. Belebele was run 5-shot; everything else zero-shot. The PingPong scoring is our own implementation, since the paper’s prompt descriptions and tables do not fully reconcile. PingPong’s releases declare no licence; we publish aggregate scores only and reproduce no items.
  • Noise. Measured repeat variation is up to 1.0 point on short-answer tasks and 4.8 points on SEA-IFEval. LoraxBench, SEA-HELM and PingPong have no measured floor.
  • Data. Only aggregate scores are published. No benchmark item is reproduced here.

About the author

  • Baibu Research

    Research team

    Baibu's research team studies how AI can serve Indonesia in the languages people speak at home.

Share this story