tomaarsen/Qwen3-Embedding-0.6B-18-layers

SentenceTransformer based on Qwen/Qwen3-Embedding-0.6B

This is a sentence-transformers model finetuned from Qwen/Qwen3-Embedding-0.6B on the nq dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

This is a first experiment attempting to distil the powerful Qwen/Qwen3-Embedding-0.6B model from 28 layers down to some lower layer count, in an attempt to speed up inference with minimal performance reductions.

To be specific, this early model falls from ~0.55 NDCG@10 average across NanoMSMARCO, NanoNFCorpus, and NanoNQ with the full 28 layers, to 0.5155 NDCG@10 on that selection with just 18 layers. Early tests indicate that using only 18 layers results in a 1.51x speedup compared to the full model.

This model was distilled using only 200k texts from one domain, reaching superior performance should be possible, especially with stronger distillation techniques like MarginMSE.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Qwen/Qwen3-Embedding-0.6B
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • nq
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: Qwen3Model 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': True, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("tomaarsen/Qwen3-Embedding-0.6B-18-layers")
# Run inference
sentences = [
    'The actress was thirteen when she was offered the role of Annie.',
    'Contrasting significantly from other soccer leagues in the U.S., WLS intends to be an open entry, promotion and relegation competition.',
    'Narsingh Temple is situated at the across of the village just across confluence of Magri State village.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Datasets: NanoMSMARCO, NanoNFCorpus and NanoNQ
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "query_prompt": "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:"
  }
  

Metric	NanoMSMARCO	NanoNFCorpus	NanoNQ
cosine_accuracy@1	0.42	0.38	0.4
cosine_accuracy@3	0.64	0.44	0.72
cosine_accuracy@5	0.76	0.52	0.76
cosine_accuracy@10	0.82	0.66	0.82
cosine_precision@1	0.42	0.38	0.4
cosine_precision@3	0.2133	0.3133	0.2467
cosine_precision@5	0.152	0.292	0.16
cosine_precision@10	0.082	0.254	0.088
cosine_recall@1	0.42	0.0413	0.39
cosine_recall@3	0.64	0.0687	0.69
cosine_recall@5	0.76	0.0852	0.73
cosine_recall@10	0.82	0.1141	0.79
cosine_ndcg@10	0.6209	0.3043	0.6214
cosine_mrr@10	0.5568	0.4416	0.571
cosine_map@100	0.5664	0.1255	0.5648

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with NanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ],
      "query_prompts": {
          "msmarco": "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:",
          "nfcorpus": "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:",
          "nq": "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:"
      }
  }
  

Metric	Value
cosine_accuracy@1	0.4
cosine_accuracy@3	0.6
cosine_accuracy@5	0.68
cosine_accuracy@10	0.7667
cosine_precision@1	0.4
cosine_precision@3	0.2578
cosine_precision@5	0.2013
cosine_precision@10	0.1413
cosine_recall@1	0.2838
cosine_recall@3	0.4662
cosine_recall@5	0.5251
cosine_recall@10	0.5747
cosine_ndcg@10	0.5155
cosine_mrr@10	0.5232
cosine_map@100	0.4189

Knowledge Distillation

• Evaluated with MSEEvaluator

Metric	Value
negative_mse	-0.0168

Training Details

Training Dataset

nq

• Dataset: nq at f9e894e
• Size: 197,462 training samples
• Columns: text and label
• Approximate statistics based on the first 1000 samples:

  	text	label
  type	string	list
  details	min: 27 tokens mean: 89.38 tokens max: 505 tokens	size: 1024 elements

• Samples:

  text	label
  Instruct: Given a web search query, retrieve relevant passages that answer the query Query:the movie bernie based on a true story	[-0.05126953125, -0.0020294189453125, 0.00152587890625, 0.060791015625, 0.022216796875, ...]
  College World Series The College World Series, or CWS, is an annual June baseball tournament held in Omaha, Nebraska. The CWS is the culmination of the National Collegiate Athletic Association (NCAA) Division I Baseball Championship tournament—featuring 64 teams in the first round—which determines the NCAA Division I college baseball champion. The eight participating teams are split into two, four-team, double-elimination brackets, with the winners of each bracket playing in a best-of-three championship series.	[0.033935546875, -0.0908203125, -0.010498046875, 0.0625, -0.01263427734375, ...]
  Instruct: Given a web search query, retrieve relevant passages that answer the query Query:does the femoral nerve turn into the saphenous nerve	[0.052978515625, -0.0028228759765625, -0.0022430419921875, 0.0732421875, 0.044677734375, ...]

• Loss: MSELoss

Evaluation Datasets

nq

• Dataset: nq at f9e894e
• Size: 3,000 evaluation samples
• Columns: text and label
• Approximate statistics based on the first 1000 samples:

  	text	label
  type	string	list
  details	min: 21 tokens mean: 87.24 tokens max: 410 tokens	size: 1024 elements

• Samples:

  text	label
  Instruct: Given a web search query, retrieve relevant passages that answer the query Query:who was the heir apparent of the austro-hungarian empire in 1914	[0.0262451171875, 0.0556640625, -0.0, -0.03076171875, -0.05712890625, ...]
  Instruct: Given a web search query, retrieve relevant passages that answer the query Query:who played tommy in coward of the county	[-0.00848388671875, -0.02294921875, -0.00182342529296875, 0.060546875, -0.021240234375, ...]
  Vertebra The vertebral arch is formed by pedicles and laminae. Two pedicles extend from the sides of the vertebral body to join the body to the arch. The pedicles are short thick processes that extend, one from each side, posteriorly, from the junctions of the posteriolateral surfaces of the centrum, on its upper surface. From each pedicle a broad plate, a lamina, projects backwards and medialwards to join and complete the vertebral arch and form the posterior border of the vertebral foramen, which completes the triangle of the vertebral foramen.[6] The upper surfaces of the laminae are rough to give attachment to the ligamenta flava. These ligaments connect the laminae of adjacent vertebra along the length of the spine from the level of the second cervical vertebra. Above and below the pedicles are shallow depressions called vertebral notches (superior and inferior). When the vertebrae articulate the notches align with those on adjacent vertebrae and these form the openings of the int...	[0.062255859375, -0.005706787109375, -0.009765625, 0.035400390625, -0.0125732421875, ...]

• Loss: MSELoss

gooaq

• Dataset: gooaq at b089f72
• Size: 3,000 evaluation samples
• Columns: text and label
• Approximate statistics based on the first 1000 samples:

  	text	label
  type	string	list
  details	min: 10 tokens mean: 43.88 tokens max: 117 tokens	size: 1024 elements

• Samples:

  text	label
  Instruct: Given a web search query, retrieve relevant passages that answer the query Query:what essential oils are soothing?	[-0.025146484375, 0.06591796875, -0.0025634765625, 0.0732421875, -0.046630859375, ...]
  Titles of books should be underlined or put in italics . (Titles of stories, essays and poems are in "quotation marks.") Refer to the text specifically as a novel, story, essay, memoir, or poem, depending on what it is.	[-0.006988525390625, -0.050537109375, -0.007476806640625, -0.07177734375, -0.049560546875, ...]
  Dakine Cyclone Wet/Dry 32L Backpack. Born from the legacy of our most iconic surf pack, the Cyclone Collection is a family of super-technical and durable wet/dry packs and bags.	[0.0016632080078125, 0.04150390625, -0.01324462890625, 0.0234375, 0.03173828125, ...]

• Loss: MSELoss

wikipedia

• Dataset: wikipedia at 4a0972d
• Size: 3,000 evaluation samples
• Columns: text and label
• Approximate statistics based on the first 1000 samples:

  	text	label
  type	string	list
  details	min: 5 tokens mean: 28.1 tokens max: 105 tokens	size: 1024 elements

• Samples:

  text	label
  The daughter of Vice-admiral George Davies and Julia Hume, she spent her younger years on board the ship he was stationed, the Griper.	[0.0361328125, 0.01904296875, -0.003662109375, 0.0247802734375, 0.0140380859375, ...]
  The impetus for the project began when Amalgamated Dynamics, hired to provide the practical effects for The Thing, a prequel to John Carpenter's 1982 classic film-renowned for its almost exclusive use of practical effects-became disillusioned upon discovering the theatrical release had the bulk of their effects digitally replaced with computer-generated imagery.	[-0.0106201171875, -0.0439453125, -0.01104736328125, 0.00946044921875, 0.0322265625, ...]
  Lost Angeles, his second feature film, starring Joelle Carter and Kelly Blatz, had its world premiere at the Oldenburg International Film Festival in 2012.	[0.0272216796875, 0.0263671875, -0.007110595703125, 0.0294189453125, 0.01129150390625, ...]

• Loss: MSELoss

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 0.0001
• num_train_epochs: 1
• warmup_ratio: 0.1
• bf16: True
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 0.0001
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	nq loss	gooaq loss	wikipedia loss	NanoMSMARCO_cosine_ndcg@10	NanoNFCorpus_cosine_ndcg@10	NanoNQ_cosine_ndcg@10	NanoBEIR_mean_cosine_ndcg@10	negative_mse
-1	-1	-	-	-	-	0.2033	0.0972	0.1638	0.1548	-0.0985
0.0162	200	0.0008	-	-	-	-	-	-	-	-
0.0324	400	0.0004	-	-	-	-	-	-	-	-
0.0486	600	0.0003	-	-	-	-	-	-	-	-
0.0648	800	0.0003	-	-	-	-	-	-	-	-
0.0810	1000	0.0002	0.0002	0.0003	0.0003	0.5482	0.2864	0.5995	0.4780	-0.0280
0.0972	1200	0.0002	-	-	-	-	-	-	-	-
0.1134	1400	0.0002	-	-	-	-	-	-	-	-
0.1296	1600	0.0002	-	-	-	-	-	-	-	-
0.1458	1800	0.0002	-	-	-	-	-	-	-	-
0.1620	2000	0.0002	0.0002	0.0003	0.0003	0.6136	0.2926	0.6028	0.5030	-0.0218
0.1783	2200	0.0002	-	-	-	-	-	-	-	-
0.1945	2400	0.0001	-	-	-	-	-	-	-	-
0.2107	2600	0.0001	-	-	-	-	-	-	-	-
0.2269	2800	0.0001	-	-	-	-	-	-	-	-
0.2431	3000	0.0001	0.0001	0.0002	0.0002	0.6169	0.2990	0.5781	0.4980	-0.0199
0.2593	3200	0.0001	-	-	-	-	-	-	-	-
0.2755	3400	0.0001	-	-	-	-	-	-	-	-
0.2917	3600	0.0001	-	-	-	-	-	-	-	-
0.3079	3800	0.0001	-	-	-	-	-	-	-	-
0.3241	4000	0.0001	0.0001	0.0002	0.0002	0.6137	0.3000	0.5987	0.5041	-0.0187
0.3403	4200	0.0001	-	-	-	-	-	-	-	-
0.3565	4400	0.0001	-	-	-	-	-	-	-	-
0.3727	4600	0.0001	-	-	-	-	-	-	-	-
0.3889	4800	0.0001	-	-	-	-	-	-	-	-
0.4051	5000	0.0001	0.0001	0.0002	0.0002	0.6235	0.2945	0.6105	0.5095	-0.0182
0.4213	5200	0.0001	-	-	-	-	-	-	-	-
0.4375	5400	0.0001	-	-	-	-	-	-	-	-
0.4537	5600	0.0001	-	-	-	-	-	-	-	-
0.4699	5800	0.0001	-	-	-	-	-	-	-	-
0.4861	6000	0.0001	0.0001	0.0002	0.0002	0.6183	0.2999	0.6141	0.5108	-0.0175
0.5023	6200	0.0001	-	-	-	-	-	-	-	-
0.5186	6400	0.0001	-	-	-	-	-	-	-	-
0.5348	6600	0.0001	-	-	-	-	-	-	-	-
0.5510	6800	0.0001	-	-	-	-	-	-	-	-
0.5672	7000	0.0001	0.0001	0.0002	0.0002	0.6129	0.3005	0.6201	0.5112	-0.0173
0.5834	7200	0.0001	-	-	-	-	-	-	-	-
0.5996	7400	0.0001	-	-	-	-	-	-	-	-
0.6158	7600	0.0001	-	-	-	-	-	-	-	-
0.6320	7800	0.0001	-	-	-	-	-	-	-	-
0.6482	8000	0.0001	0.0001	0.0002	0.0002	0.6258	0.3032	0.6099	0.5130	-0.0170
0.6644	8200	0.0001	-	-	-	-	-	-	-	-
0.6806	8400	0.0001	-	-	-	-	-	-	-	-
0.6968	8600	0.0001	-	-	-	-	-	-	-	-
0.7130	8800	0.0001	-	-	-	-	-	-	-	-
0.7292	9000	0.0001	0.0001	0.0002	0.0002	0.6209	0.3043	0.6214	0.5155	-0.0168
0.7454	9200	0.0001	-	-	-	-	-	-	-	-
0.7616	9400	0.0001	-	-	-	-	-	-	-	-
0.7778	9600	0.0001	-	-	-	-	-	-	-	-
0.7940	9800	0.0001	-	-	-	-	-	-	-	-
0.8102	10000	0.0001	0.0001	0.0002	0.0002	0.6224	0.3015	0.6183	0.5141	-0.0168
0.8264	10200	0.0001	-	-	-	-	-	-	-	-
0.8427	10400	0.0001	-	-	-	-	-	-	-	-
0.8589	10600	0.0001	-	-	-	-	-	-	-	-
0.8751	10800	0.0001	-	-	-	-	-	-	-	-
0.8913	11000	0.0001	0.0001	0.0002	0.0002	0.6224	0.3014	0.6155	0.5131	-0.0167
0.9075	11200	0.0001	-	-	-	-	-	-	-	-
0.9237	11400	0.0001	-	-	-	-	-	-	-	-
0.9399	11600	0.0001	-	-	-	-	-	-	-	-
0.9561	11800	0.0001	-	-	-	-	-	-	-	-
0.9723	12000	0.0001	0.0001	0.0002	0.0002	0.6247	0.3020	0.6133	0.5133	-0.0167
0.9885	12200	0.0001	-	-	-	-	-	-	-	-
-1	-1	-	-	-	-	0.6209	0.3043	0.6214	0.5155	-0.0168

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.10
• Sentence Transformers: 4.2.0.dev0
• Transformers: 4.51.2
• PyTorch: 2.5.1+cu124
• Accelerate: 1.5.2
• Datasets: 3.5.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MSELoss

@inproceedings{reimers-2020-multilingual-sentence-bert,
    title = "Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2020",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2004.09813",
}