Local LLMs for German Diary Summaries: Gemma vs. Qwen

TL;DR: I would use Gemma 4 31B to write my German diary summaries and Qwen 3.8 27B to critique them. Across three weekly reviews per writer, I found 20 issues in Qwen’s revised texts and two in Gemma’s, with different levels of severity. Qwen was the more useful critic, catching problems that Gemma missed when reviewing Qwen’s writing. The synthetic benchmark had barely separated the two. These findings apply to my tested setups and workflow, not the model families in general.

I wanted a local model to turn my German diary entries into useful weekly reviews. That sounds straightforward enough: read the notes, keep the important events, and write a summary. The difficult part is keeping the meaning intact. Something I intended to do should not quietly become something I already did, and an event mentioned on Monday may actually have happened the week before.

I’ve been experimenting with local models for a while, including connecting LM Studio to Home Assistant. For my notes, though, fluent German and a plausible answer aren’t enough. I want to be able to trust the account of what happened without having to reconstruct the whole week myself afterwards.

So I built a synthetic benchmark, initially comparing Qwen and Gemma. I later added two older models for context. Qwen and Gemma remained close enough that I wanted to try both on my actual notes. After checking three weekly reviews from each, I had a clearer answer, but also a distinction the benchmark hadn’t given me: the model I preferred as a writer wasn’t the one I preferred as a critic.

The Synthetic Benchmark

The benchmark used three synthetic German document collections: a short diary with 23 entries, a longer diary with 120 entries, and 55 household and insurance notes. I had considered using manually sanitised real notes as well, but left them out of this benchmark.

Each collection had five tasks: a roughly 200-word summary, a factual lookup, an exhaustive extraction, a chronology, and a question the documents could not answer. That last category mattered to me. If the notes do not establish something, I want the model to say so rather than fill the gap with something convincing.

Every request received the complete relevant collection, without retrieval or conversation history. Sources, tasks and reference answers were generated with Codex, then validated and reviewed before the evaluation inputs were frozen. An anonymous model judge, using gpt-6.1-sol with medium reasoning, scored the final answers against the sources and reference keys. The judge did not see which local model had written the answer.

The local inference ran through llama-swap and llama.cpp on an RTX 3090 with 24 GB VRAM. Preparation and judging used a frontier service, so the benchmark as a whole was not an offline workflow.

I started with Qwen 3.8 27B in Unsloth IQ4_XS and Gemma 4 31B IT in Q4_K_M. In a later run, I added GPT-OSS 20B in MXFP4 and Ministral 3 14B Reasoning 2512 in Q4_K_M. I had used both older models for German before and wanted to see how they compared, even though I expected them to do worse. They were reference points for me, never serious contenders for this workflow.

The final comparison covered 15 tasks per model, 60 judged responses in total. These were particular deployments with model-specific settings, rather than a comparison at identical quantisation and inference settings.

What the Scores Said

The four quality dimensions were scored from 0 to 4, with higher being better. Faithfulness measures whether the answer preserves the source’s meaning; completeness measures whether it includes the required information. Clear German is a separate score because a readable answer can still be wrong.

Model	Faith­fulness	Complete­ness	Sources	German	Severe errors	Incomplete
Qwen 3.8 27B	3.07	3.73	3.33	3.67	2/15	0/15
Gemma 4 31B	3.33	3.47	3.40	4.00	4/15	0/15
Ministral 3 14B	2.33	2.33	2.73	3.40	5/15	1/15
GPT-OSS 20B	1.40	2.07	1.93	3.07	12/15	0/15

Sources means source-reference quality; German means clarity. The last two columns count affected answers out of fifteen. Qwen and GPT-OSS used medium reasoning, Gemma had thinking enabled, and Ministral’s scores come from its corrected thinking setup.

The colours are a visual guide: scores below 2 are red, 2 to below 3 yellow, and 3–4 green. For error and incomplete-answer counts, lower is better: zero is green, some affected answers yellow, and half or more red. These bands help scan the table; they do not change the scores or replace the caveats below.

The Ministral row needs a small caveat: one request used its entire generation budget for reasoning and returned no final answer. The rubric still awarded that empty answer full marks for faithfulness and references because it contained no false claims or citations. That inflates those two averages; it certainly does not make the response useful. The row above retains the original scores and shows the failure separately.

Qwen was strongest on completeness and extracted all ten expected items across the three extraction tasks without adding extra items. Gemma extracted nine of ten, but had better average faithfulness, clearer German, and stronger handling of questions the sources could not answer. Qwen also had fewer answers with severe errors. There was a reasonable case for either model, depending on the task.

A later weighted calculation, giving faithfulness and completeness 40% each and the other dimensions 10% each, put Gemma at 86.5% and Qwen at 85.5%. Those are normalised rating averages, not percentages of correct answers. A one-point gap on this small pilot was hardly enough to settle my choice. For the summary tasks specifically, they even had identical average faithfulness and completeness scores: 2.67 and 3.00.

GPT-OSS was fast, with a median request time of 21.30 seconds, but the errors ruled it out for this use. Ministral improved after correcting its thinking setup, yet still lagged behind the two larger models. Gemma’s median request took 60.80 seconds, compared with Qwen’s 100.64 seconds. Those are observed request times across the task mix, not a controlled throughput comparison.

Qwen’s reasoning setting also made a substantial difference. At xhigh, only five of its fifteen benchmark requests produced a final answer; the others exhausted the 8,192-token output allowance. Switching to medium let all fifteen finish. More thinking was not helping if I never got the summary.

The Test with My Real Notes

Because the two leading models were so close, I built a workflow to turn my actual diary entries into weekly reviews. I initially checked one review from each writer, then expanded the quality check to three reviews per writer. The counts below are totals across those three reviews, replacing my initial single-review findings.

The workflow deliberately uses two different models, so the writer does not critique its own text:

1. The writing model creates an initial summary of my diary entries.
2. The other model critiques that summary.
3. The original writing model receives the critique and updates its summary based on that feedback.
4. The writing model generates tags.

I compared Qwen writes → Gemma critiques → Qwen revises with Gemma writes → Qwen critiques → Gemma revises. I checked the revised reviews, after that feedback loop, against the original notes myself. This compares two complete workflows, so the resulting quality cannot be attributed to the writer alone. I also looked at the critiques and the changes they prompted to understand what each model contributed.

Quality Across Three Reviews

Issues in the revised reviews	Qwen writer	Gemma writer
Source misinterpreted or information hallucinated	9	1
Expressions that did not really make sense	11	0
Minor spelling mistakes	0	1
Total issues found	20	2

Gemma’s source-related error was an event assigned to the wrong time, although the event itself was described correctly. Qwen made the same mistake. Gemma’s only other issue was a minor spelling error.

That was a much clearer result than the synthetic scores. Qwen wrote very thorough summaries, taking a lot of individual details into account. Gemma’s summaries were considerably shorter and more concise. I preferred that for a weekly review, and Gemma’s texts also needed much less correction: parts of Qwen’s accounts were wrong or awkward enough not to make sense. For something that is supposed to save me time reviewing my week, both length and accuracy matter quite a bit.

The totals mix different kinds of issues: a spelling mistake is obviously less serious than a hallucinated event. They are manual findings across three reviews per writer, not error rates or benchmark scores. The synthetic table counts answers containing severe errors, which is a different measurement again. Even keeping those distinctions in mind, the practical result was much more persuasive for my choice of writer than a small difference in aggregate scores.

Writing and Critiquing Are Different Jobs

Looking at the feedback loop changed how I viewed Qwen. As a critic, it requested more changes than Gemma, and its corrections were useful. Many concerned wording, but the revised wording was closer to what I had actually written in the diary.

For example, Gemma initially described something as “totale Überforderung”. Qwen suggested “eine Art der Überforderung”, which better reflected the original note. That is a small change on the page, but it changes how strongly the summary characterises my experience. In another draft, Qwen caught a more substantial problem: Gemma had reversed the meaning of a statement, describing it exactly the wrong way around. The critique gave Gemma a chance to correct it before the final review.

Gemma was much less consistent as a critic. Sometimes it asked for only one or two changes, sometimes none at all, even though Qwen’s text still contained errors. It also once requested an incorrect day as a correction. A review step can introduce a mistake as well as remove one.

It did catch useful things too. In one case, the week’s notes had no Monday entry, and Qwen shifted all the weekday references by a day. Gemma corrected those references throughout the text, including changing Tuesday to Monday where appropriate. So it could catch a systematic chronology problem, while overlooking other errors elsewhere.

Those are observations from the drafts and critiques, separate from the issue totals for the revised reviews above. They explain why my preference is more specific than simply choosing Gemma for everything: Gemma wrote much better summaries, but was a weak critic in these tests. Qwen was the opposite. Keeping Qwen in the reviewer role was useful, even though I would not choose it to write the review.

Tokens, Speed and the Actual Setup

The timing data below comes from the original pair of runs on my calendar week 41 notes. It is not an average across the three reviews used for the expanded quality check. Both timed runs used Unsloth models on the RTX 3090, served through llama-swap and llama.cpp, with MTP enabled. Nothing was cached. The important differences in the configurations were:

• Qwen: Qwen3.8-27B-UD-IQ4_XS.gguf, medium thinking, a 64,000-token context, f16 K/V cache, and up to four MTP draft tokens.
• Gemma: gemma-4-31B-it-Q4_K_M.gguf, thinking enabled, a 48,000-token context, q8_0 K/V cache, and a separate MTP draft model with up to two draft tokens.

Both used one slot, flash attention, batch size 512, micro-batch size 128, temperature 1.0, top-p 0.95 and seed 42. Qwen used top-k 20, Gemma top-k 64. These practical settings differ from the frozen synthetic benchmark, which used a 34,816-token context and an 8,192-token generation allowance.

Here are the recorded requests, in workflow order. Tokens are the logged prompt and generated counts; the generated count should not be read as a word count or a clean measurement of visible summary length, since these are thinking-enabled runs. Different tokenisers also limit direct comparisons.

Qwen Writes, Gemma Critiques

Step	Model	Prompt tokens	Generated tokens	Decode tokens/s	Time
Initial summary	Qwen	8,262	6,041	60.04	108.81 s
Critique	Gemma	10,992	3,090	47.11	101.65 s
Revision	Qwen	11,868	3,918	92.24	64.58 s
Tags	Qwen	3,344	1,384	66.16	25.01 s

Gemma Writes, Qwen Critiques

Step	Model	Prompt tokens	Generated tokens	Decode tokens/s	Time
Initial summary	Gemma	8,276	3,023	45.43	97.79 s
Critique	Qwen	9,109	12,569	76.37	184.01 s
Revision	Gemma	12,310	2,722	49.18	91.88 s
Tags	Gemma	1,707	816	51.13	19.41 s

Comparing the Runs

For the initial summary, Qwen generated almost twice as many tokens as Gemma: 6,041 versus 3,023. Despite its higher decode rate, it took slightly longer to finish. Tokens per second only tell part of the story when the models generate very different amounts of text.

The full workflow gives a different speed result:

Workflow	Total prompt tokens	Total generated tokens	Sum of request times
Qwen writer, Gemma critique	34,466	14,433	300.05 s, about 5 minutes
Gemma writer, Qwen critique	31,402	19,130	393.09 s, about 6 minutes 33 seconds

These totals add the four logged requests; they are not a separate measurement of elapsed workflow time including orchestration and model switching. They also count repeated input across requests, not unique source tokens.

In this measured pair, the workflow with Gemma writing was slower overall and generated more tokens, mainly because Qwen’s critique alone produced 12,569 tokens and took 184.01 seconds. Gemma’s initial summary used fewer generated tokens and finished sooner; the critique stage then outweighed that saving. After inspecting the critiques across all three reviews, though, I have a better reason to keep that step: Qwen found problems I wanted corrected. I would rather wait for useful feedback than get a quick approval of a text that still contains mistakes.

I also tried Qwen with xhigh outside the fixed benchmark budget. It used around 80,000 tokens for shorter summaries with prompts of roughly 2,000 input tokens. That is not a setting I would use for this workflow. Medium was the practical choice.

What I Will Use

For my German diary reviews, I would choose Gemma 4 31B as the writer and Qwen 3.8 27B as the critic, then let Gemma revise the text using Qwen’s feedback. Across three reviews, that combination left me with one misdated event and a minor spelling mistake. The reverse combination left twenty issues, including nine misinterpretations or hallucinations.

Qwen’s completeness in the synthetic tests had made it look promising as a writer, and I expected more from its summaries. Its critiques turned out to be the more useful contribution. Gemma wrote the text I preferred, but I would not rely on its approval of another model’s writing.

The level of detail is a separate reason to try both on your own notes. If you want a thorough account that takes many individual details into consideration, Qwen’s style may suit you better. If you want a shorter review you can read through quickly, Gemma was closer to that in my tests. My preference for concise summaries influenced my choice; someone looking for more detail might weigh that differently, while still needing to check the factual errors I found in Qwen’s output.

The synthetic benchmark was useful for comparing Qwen and Gemma and putting them in context against the two older models. Working through the real reviews showed me how to assign the roles. I still need to check the output myself; using a separate critic helped, but did not remove every error.

I also used the non-QAT Gemma model here. Trying a QAT variant would be a separate follow-up test; I have not measured whether it improves this result. For now, I would keep the tested Gemma writer and Qwen critic, despite the time Qwen spends on its feedback.