# Small language models, explained

> What makes a language model small, what parameters and effective parameters mean, why small models fit on phones, and what they give up in capability.

Source: https://ailoot.ai/blog/en/small-language-models-explained/
Canonical: https://ailoot.ai/blog/en/small-language-models-explained/
Language: en
Publisher: Ailoot (https://ailoot.ai/)
Translation: guide-100072
Revision: sha256-de485abb5b314f813edb811f2634ac96eafa2d83045a82bd8c26029857b6b1f1


## What is a small language model?

A small language model (SLM) is a language model compact enough to run on everyday hardware, such as a phone or laptop, instead of a data centre. There is no agreed cut-off: models with a few billion parameters or fewer are usually called small, and one 2024 survey uses a range of 100 million to 5 billion.

Small and large language models work the same way. Both are neural networks, almost always transformers, trained to predict the next token of text, and both are adapted after pre-training to follow instructions; the survey above covers decoder-only transformers of exactly this kind. The difference is scale. A phone-sized model has a few billion parameters, while frontier cloud models are far larger, so a small model holds less knowledge and has less capacity for reasoning (see “Where do small models fall short?”). Small models are what make on-device AI practical.

Size is not the only lever on quality. A small model’s quality therefore depends on its training as much as its size. It can learn from a larger model through distillation, in which it is trained to match the larger model’s outputs, and it gains from carefully chosen data: phi-1, a 1.3-billion-parameter code model trained largely on “textbook quality” data, reached 50.6% pass@1 on the HumanEval coding benchmark.

## What do parameters and ‘effective parameters’ mean?

Parameters are the numbers a model learns during training, and their count is the usual measure of its size: the “1.7B” in Qwen3 1.7B means about 1.7 billion. “Effective parameters”, the “E” in Google’s Gemma 4 E2B and E4B, counts the parameters the model computes with and leaves out large embedding tables that it only looks up.

Most parameters sit in the weight matrices of a transformer’s attention and feed-forward layers, which every token passes through. A separate block, the embeddings, maps each token in the vocabulary to a vector, and in a small model it can be a large share of the total. Qwen3 1.7B’s embedding table has 151,936 rows (its configured vocabulary, padded beyond the tokeniser’s 151,669 entries) of 2,048 dimensions, about 311 million embedding parameters, consistent with its model card, which lists 1.7 billion parameters in all and 1.4 billion outside the embeddings 7. Scaling studies often count only the non-embedding parameters.

Gemma 4’s E-series takes the idea further. Google’s model card explains that ‘The “E” in E2B and E4B stands for “effective” parameters’: the models use Per-Layer Embeddings (PLE), which give “each decoder layer its own small embedding for every token”, and “These embedding tables are large but are only used for quick lookups, which is why the effective parameter count is much smaller than the total” 8. The card lists Gemma 4 E2B at 2.3 billion effective parameters, 5.1 billion with embeddings, and E4B at 4.5 billion effective, 8 billion with embeddings.

The distinction matters on a phone because lookups need memory but little computation, and a runtime can leave the tables in storage. “Effective” is different again from “active”: Gemma 4 26B A4B is a mixture-of-experts model with 25.2 billion parameters, of which 3.8 billion are active during inference 8.

## Why do small models fit on phones?

A model’s weights take roughly its parameter count times the bits stored per parameter, divided by eight. At 4 bits, 1.7 billion parameters need about 0.85 GB, which leaves room for Android and other apps on a phone with 4 GB of RAM.

Parameter count and precision are the two levers. Figure 1 works through the arithmetic for Qwen3 1.7B, from 32-bit floating point down to a 4-bit model file.

<figure id="la-fit">
<ol>
<li><strong>Stored as 32-bit floats, 1.7 billion parameters take 6.8 GB.</strong></li>
<li><strong>At 16 bits, they take 3.4 GB.</strong></li>
<li><strong>At 8 bits, they take 1.7 GB.</strong> Each halving of the bits per weight halves the memory the weights need.</li>
<li><strong>At 4 bits, about 0.85 GB.</strong> Half a byte per weight is what makes a model of this size practical on a 4 GB phone.</li>
<li><strong>A 4-bit model file can be larger than the raw weights.</strong> Block scales and other stored data add to its size.</li>
</ol>
<figcaption>Figure 1. Memory needed to hold the weights of Qwen3 1.7B in decimal gigabytes. Blue bars are computed as parameters × bits ÷ 8 for 1.7 billion parameters and count weights only; a running model also needs memory for its key–value cache and runtime. The orange bar is the downloaded model file.</figcaption>
</figure>

| Precision | Bits per parameter | Approximate weight memory |
|---|---:|---:|
| 32-bit float | 32 | 6.8 GB |
| 16-bit float | 16 | 3.4 GB |
| 8-bit integer | 8 | 1.7 GB |
| 4-bit integer | 4 | 0.85 GB |

Phone-sized models combine a few billion parameters with 4-bit or mixed-precision weights for this reason; quantization reduces the storage needed for weights.

Weights are not the whole footprint. A running model also needs memory for its key–value cache, which grows with the length of the context window, and for the runtime itself. Speed follows the same arithmetic: each new token requires reading the model’s weights from memory, so fewer and smaller weights mean faster answers on the same phone.

## What do small models do well?

Small models are good at tasks where what they need is in the prompt or in common knowledge: rewriting, summarising, drafting, explaining and answering from supplied passages. Because they run on the device, they can also answer without a network connection and without sending the prompt to a server.

Google’s model card calls the Gemma 4 family “well-suited for reasoning, agentic workflows, coding, and multimodal understanding”, with E2B and E4B accepting text, image and audio input 8. These are the developers’ own descriptions and benchmarks; how well a particular task works on a particular phone still needs testing.

Supplying the right text helps with what a small model doesn’t know. Retrieval-augmented generation adds passages fetched at question time, so the model works from sources rather than memory. Small models are also predictable to run: there is no per-question server cost, no queue and no dependence on a connection after the download.

## Where do small models fall short?

Small models hold less knowledge, are weaker at long or multi-step reasoning and handle long contexts less well than larger models, and like all language models they can state false things fluently.

The pattern shows within a single family. On Google’s model card, Gemma 4 E2B scores 60.0% on the MMLU Pro knowledge and reasoning benchmark and E4B 69.4%, against 85.2% for the 31-billion-parameter Gemma 4 model; on GPQA Diamond, a set of graduate-level science questions, the scores are 43.4%, 58.6% and 84.3% 8. These are Google’s results for its models.

Small models can also get stuck repeating themselves. Qwen’s model card warns that greedy decoding “can lead to performance degradation and endless repetitions” 7.

## Why do small models copy the shape of their instructions?

Small models tend to imitate the form of a prompt as well as follow its content, so a long, structured system prompt can come back as structure in every answer.

A prompt that told a model to “lead with the direct answer, then useful detail” came back as answers with bold labels reading “Direct Answer:” and “Useful Detail:”. The instruction was meant as advice about content; the model treated it as a template.

## How do thinking modes work in small models?

A thinking mode makes the model write out intermediate reasoning before its final answer, spending more time and tokens for more careful results on multi-step problems. Qwen3 and Gemma 4 both offer it as a switch within one model rather than as a separate reasoning model.

The idea comes from chain-of-thought prompting. Newer small models are trained to reason, in Qwen3’s case partly by learning from its larger models.

The switch is part of the prompt. Qwen3 1.7B thinks by default and writes its reasoning inside a `<think>…</think>` block; developers turn thinking off with an `enable_thinking` setting, and users can add `/think` or `/no_think` to a message to change mode from turn to turn 7. Gemma 4 enables thinking when its system prompt starts with a `<|think|>` token 8.

Thinking costs time and context, because reasoning tokens are part of the model’s output and share its window. The reasoning appears before the answer when a model is set up to expose it, but behavior depends on the model and the application.

## How do Qwen3 1.7B, Gemma 4 E2B and Gemma 4 E4B compare?

The three differ in size, inputs and context length. Qwen3 1.7B has 1.7 billion parameters and takes text only; Google lists Gemma 4 E2B and E4B at 2.3 and 4.5 billion effective parameters, with text, image and audio input. All three are released under Apache 2.0.

|  | Qwen3 1.7B | Gemma 4 E2B | Gemma 4 E4B |
|---|---|---|---|
| Developer | Alibaba (Qwen team) | Google DeepMind | Google DeepMind |
| Parameters | 1.7B, of which 1.4B outside the embeddings | 2.3B effective, 5.1B with embeddings | 4.5B effective, 8B with embeddings |
| Layers | 28 | 35 | 42 |
| Context length | 32,768 tokens | 128K tokens | 128K tokens |
| Input | Text | Text, image, audio | Text, image, audio |
| Thinking mode | Yes, on by default | Yes | Yes |
| Licence | Apache 2.0 | Apache 2.0 | Apache 2.0 |

Table 1. Specifications as published in Qwen’s Qwen3-1.7B model card and Google’s Gemma 4 model card, checked 5 October 2026 7 8.



## References

7. Qwen. “Qwen3-1.7B.” Model card and configuration, Hugging Face. Accessed 5 October 2026. [huggingface.co/Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B)
8. Google DeepMind. “Gemma 4 model card.” Google AI for Developers. Accessed 5 October 2026. [ai.google.dev/gemma/docs/core/model_card_4](https://ai.google.dev/gemma/docs/core/model_card_4)

## Put this guide to work with Ailoot

Connect AI models to your applications, agents and automations through one Ailoot API. Choose the model that fits your task and follow the [Ailoot documentation](https://ailoot.ai/docs/) to get started.

## Ailoot · The company and its project portfolio

Ailoot combines established operating experience, a large client base, stable service and ongoing KPI control.

The company and its project portfolio: millions of customers; monthly sales in the billions.

Ailoot brings that operating experience to API access for AI features in customer products, agents, applications and automations.
