Blog

Small language models, explained

Markdown

What makes a language model small, what parameters and effective parameters mean, why small models fit on phones, and what they give up in capability.

What is a small language model?

A small language model (SLM) is a language model compact enough to run on everyday hardware, such as a phone or laptop, instead of a data centre. There is no agreed cut-off: models with a few billion parameters or fewer are usually called small, and one 2024 survey uses a range of 100 million to 5 billion.

Small and large language models work the same way. Both are neural networks, almost always transformers, trained to predict the next token of text, and both are adapted after pre-training to follow instructions; the survey above covers decoder-only transformers of exactly this kind. The difference is scale. A phone-sized model has a few billion parameters, while frontier cloud models are far larger, so a small model holds less knowledge and has less capacity for reasoning (see “Where do small models fall short?”). Small models are what make on-device AI practical.

Size is not the only lever on quality. A small model’s quality therefore depends on its training as much as its size. It can learn from a larger model through distillation, in which it is trained to match the larger model’s outputs, and it gains from carefully chosen data: phi-1, a 1.3-billion-parameter code model trained largely on “textbook quality” data, reached 50.6% pass@1 on the HumanEval coding benchmark.

What do parameters and ‘effective parameters’ mean?

Parameters are the numbers a model learns during training, and their count is the usual measure of its size: the “1.7B” in Qwen3 1.7B means about 1.7 billion. “Effective parameters”, the “E” in Google’s Gemma 4 E2B and E4B, counts the parameters the model computes with and leaves out large embedding tables that it only looks up.

Most parameters sit in the weight matrices of a transformer’s attention and feed-forward layers, which every token passes through. A separate block, the embeddings, maps each token in the vocabulary to a vector, and in a small model it can be a large share of the total. Qwen3 1.7B’s embedding table has 151,936 rows (its configured vocabulary, padded beyond the tokeniser’s 151,669 entries) of 2,048 dimensions, about 311 million embedding parameters, consistent with its model card, which lists 1.7 billion parameters in all and 1.4 billion outside the embeddings 7. Scaling studies often count only the non-embedding parameters.

Gemma 4’s E-series takes the idea further. Google’s model card explains that ‘The “E” in E2B and E4B stands for “effective” parameters’: the models use Per-Layer Embeddings (PLE), which give “each decoder layer its own small embedding for every token”, and “These embedding tables are large but are only used for quick lookups, which is why the effective parameter count is much smaller than the total” 8. The card lists Gemma 4 E2B at 2.3 billion effective parameters, 5.1 billion with embeddings, and E4B at 4.5 billion effective, 8 billion with embeddings.

The distinction matters on a phone because lookups need memory but little computation, and a runtime can leave the tables in storage. “Effective” is different again from “active”: Gemma 4 26B A4B is a mixture-of-experts model with 25.2 billion parameters, of which 3.8 billion are active during inference 8.

Why do small models fit on phones?

A model’s weights take roughly its parameter count times the bits stored per parameter, divided by eight. At 4 bits, 1.7 billion parameters need about 0.85 GB, which leaves room for Android and other apps on a phone with 4 GB of RAM.

Parameter count and precision are the two levers. Figure 1 works through the arithmetic for Qwen3 1.7B, from 32-bit floating point down to a 4-bit model file.

  1. Stored as 32-bit floats, 1.7 billion parameters take 6.8 GB.
  2. At 16 bits, they take 3.4 GB.
  3. At 8 bits, they take 1.7 GB. Each halving of the bits per weight halves the memory the weights need.
  4. At 4 bits, about 0.85 GB. Half a byte per weight is what makes a model of this size practical on a 4 GB phone.
  5. A 4-bit model file can be larger than the raw weights. Block scales and other stored data add to its size.
Figure 1. Memory needed to hold the weights of Qwen3 1.7B in decimal gigabytes. Blue bars are computed as parameters × bits ÷ 8 for 1.7 billion parameters and count weights only; a running model also needs memory for its key–value cache and runtime. The orange bar is the downloaded model file.
PrecisionBits per parameterApproximate weight memory
32-bit float326.8 GB
16-bit float163.4 GB
8-bit integer81.7 GB
4-bit integer40.85 GB

Phone-sized models combine a few billion parameters with 4-bit or mixed-precision weights for this reason; quantization reduces the storage needed for weights.

Weights are not the whole footprint. A running model also needs memory for its key–value cache, which grows with the length of the context window, and for the runtime itself. Speed follows the same arithmetic: each new token requires reading the model’s weights from memory, so fewer and smaller weights mean faster answers on the same phone.

What do small models do well?

Small models are good at tasks where what they need is in the prompt or in common knowledge: rewriting, summarising, drafting, explaining and answering from supplied passages. Because they run on the device, they can also answer without a network connection and without sending the prompt to a server.

Google’s model card calls the Gemma 4 family “well-suited for reasoning, agentic workflows, coding, and multimodal understanding”, with E2B and E4B accepting text, image and audio input 8. These are the developers’ own descriptions and benchmarks; how well a particular task works on a particular phone still needs testing.

Supplying the right text helps with what a small model doesn’t know. Retrieval-augmented generation adds passages fetched at question time, so the model works from sources rather than memory. Small models are also predictable to run: there is no per-question server cost, no queue and no dependence on a connection after the download.

Where do small models fall short?

Small models hold less knowledge, are weaker at long or multi-step reasoning and handle long contexts less well than larger models, and like all language models they can state false things fluently.

The pattern shows within a single family. On Google’s model card, Gemma 4 E2B scores 60.0% on the MMLU Pro knowledge and reasoning benchmark and E4B 69.4%, against 85.2% for the 31-billion-parameter Gemma 4 model; on GPQA Diamond, a set of graduate-level science questions, the scores are 43.4%, 58.6% and 84.3% 8. These are Google’s results for its models.

Small models can also get stuck repeating themselves. Qwen’s model card warns that greedy decoding “can lead to performance degradation and endless repetitions” 7.

Why do small models copy the shape of their instructions?

Small models tend to imitate the form of a prompt as well as follow its content, so a long, structured system prompt can come back as structure in every answer.

A prompt that told a model to “lead with the direct answer, then useful detail” came back as answers with bold labels reading “Direct Answer:” and “Useful Detail:”. The instruction was meant as advice about content; the model treated it as a template.

How do thinking modes work in small models?

A thinking mode makes the model write out intermediate reasoning before its final answer, spending more time and tokens for more careful results on multi-step problems. Qwen3 and Gemma 4 both offer it as a switch within one model rather than as a separate reasoning model.

The idea comes from chain-of-thought prompting. Newer small models are trained to reason, in Qwen3’s case partly by learning from its larger models.

The switch is part of the prompt. Qwen3 1.7B thinks by default and writes its reasoning inside a <think>…</think> block; developers turn thinking off with an enable_thinking setting, and users can add /think or /no_think to a message to change mode from turn to turn 7. Gemma 4 enables thinking when its system prompt starts with a <|think|> token 8.

Thinking costs time and context, because reasoning tokens are part of the model’s output and share its window. The reasoning appears before the answer when a model is set up to expose it, but behavior depends on the model and the application.

How do Qwen3 1.7B, Gemma 4 E2B and Gemma 4 E4B compare?

The three differ in size, inputs and context length. Qwen3 1.7B has 1.7 billion parameters and takes text only; Google lists Gemma 4 E2B and E4B at 2.3 and 4.5 billion effective parameters, with text, image and audio input. All three are released under Apache 2.0.

Qwen3 1.7BGemma 4 E2BGemma 4 E4B
DeveloperAlibaba (Qwen team)Google DeepMindGoogle DeepMind
Parameters1.7B, of which 1.4B outside the embeddings2.3B effective, 5.1B with embeddings4.5B effective, 8B with embeddings
Layers283542
Context length32,768 tokens128K tokens128K tokens
InputTextText, image, audioText, image, audio
Thinking modeYes, on by defaultYesYes
LicenceApache 2.0Apache 2.0Apache 2.0

Table 1. Specifications as published in Qwen’s Qwen3-1.7B model card and Google’s Gemma 4 model card, checked 5 October 2026 7 8.

References

  1. Qwen. “Qwen3-1.7B.” Model card and configuration, Hugging Face. Accessed 5 October 2026. huggingface.co/Qwen/Qwen3-1.7B
  2. Google DeepMind. “Gemma 4 model card.” Google AI for Developers. Accessed 5 October 2026. ai.google.dev/gemma/docs/core/model_card_4

Put this guide to work with Ailoot

Connect AI models to your applications, agents and automations through one Ailoot API. Choose the model that fits your task and follow the Ailoot documentation to get started.