--- language: - en tags: - sentence-transformers - sentence-similarity - feature-extraction - dense - generated_from_trainer - dataset_size:477792 - loss:CachedMultipleNegativesRankingLoss base_model: answerdotai/ModernBERT-base widget: - source_sentence: tachyphylaxis definition sentences: - 1 In some areas plumbers charge $45 -$75 an hour; in other regions the hourly rate can be $75 -$150. 2 Most plumbers charge a two-hour minimum or a service call fee of $75 -$150, and some plumbers bill a flat fee per job instead of an hourly rate.3 Either away, exact costs will depend on the complexity and type of work done. Plumbers' rates vary significantly by location. 2 In some areas plumbers charge $45 -$75 an hour; in other regions the hourly rate can be $75 -$150. - "Medical Definition of tachyphylaxis. plural. tachyphylaxes. \\-Ë\x8CsÄ\x93z\\\ play. : diminished response to later increments in a sequence of applications\ \ of a physiologically active substance (as the diminished pressor response that\ \ follows repeated injections of renin)" - Quick Answer. Injury to the phrenic nerve can paralyze the diaphragm and have a serious impact on the regulation of breathing, such as difficulty during inhalation, according to the UCLA Division of Plastic & Reconstructive Surgery. The phrenic nerve is responsible for the function of the diaphragm. Continue Reading. - source_sentence: where is st malo beach sentences: - Nausea is a sensation of discomfort in the upper abdomen, accompanied by an urge to vomit. Also known of as qualm, nausea may be a side effect associated with several medications or a symptom of disease or disorder. Sometimes large, fatty or sugary meals may also lead to a feeling of nausea. Nausea is a sensation of discomfort in the upper abdomen, accompanied by an urge to vomit. Also known of as qualm, nausea may be a side effect associated with several medications or a symptom of disease or disorder. - Location of Pennsylvania in the United States. Folsom is a census-designated place (CDP) in Delaware County, Pennsylvania, United States. It is part of Ridley Township. The population was 8,323 at the 2010 census. - "Saint Malo Beach Oceanside homes. Developed in the late 1920â\x80\x99s, the community\ \ of Saint Malo Beach is one of the highlights of beautiful Carlsbad, CA â\x80\ \x93 and one of its most secluded hideaways." - source_sentence: who invented cotton candy dr pepper and electric chair sentences: - "Skill positions in football are the positions that are most responsible for causing\ \ or preventing points from being scored. The skill positions are: Skill positions\ \ are often contrasted with linemen â\x80\x93 players who line up along the line\ \ of scrimmage. Skill position players are generally smaller than linemen, but\ \ they must also be faster and have other talents (such as the ability to throw\ \ or catch the ball, cover an opposing receiver, or to dodge opponents) that rely\ \ more on finesse than on brute force." - A WBS Dictionary is merely a supporting document, which provides the definitions for each component contained in the Work Breakdown Structure. This type of dictionary is often recommended as a reference resource material for task-oriented projects comprising several work phases. - 'Cotton Candy (1897): Cotton Candy was invented in 1897 by the American inventors William Morrison and John C. Wharton. Cotton Gin (1793): The Cotton Gin was invented in 1793 by the American inventor Eli Whitney during the Industrial Revolution.' - source_sentence: what county is boston, ma sentences: - 'There are two kinds of clauses: independent and dependent clauses. Most simply, an independent clause can form a complete sentence on its own and a dependent clause cannot (at least, not by itself). Think of it this way: an independent clause is like a cup of coffee, and a dependent clause is like a caffeine lover. Caffeine lovers are dependent on coffee, so the two can be joined (quite happily) to form a cohesive unit. Similarly, two cups of coffee, or two independent clauses, can be combined.' - Boston is in the county of Suffolk in Massachusetts. The population is about 722,023, and Boston is the largest city. - 'Pre-diabetes is diagnosed by any one of the following: 1 A fasting blood glucose in between 100-125 mg/dL. 2 An A1c between 5.7 - 6.4 percent. Any value between 140 mg/dL and 199 mg/dL during a two-hour 75g oral glucose tolerance test.' - source_sentence: product key windows 8.1 how to find sentences: - 'If Windows 8.1 came preinstalled on your computer, your Windows 8.1 product key should be on a sticker on your computer or with your documentation. The Windows 8.1 product key is a series of 25 letters and numbers and should look like this: xxxxx-xxxxx-xxxxx-xxxxx-xxxxx.' - Al Gore not divorced from wife Tipper, confirms relationship with longtime girlfriend. 1 Pucker up! Al Gore planted a wet one on wife Tipper in 2000, during his presidential campaign. Ten years later the couple separated after 40 years of marriage. - springer spaniel. n. 1. (Breeds) either of two breeds of large quick-moving spaniels bred to spring game, having a slightly domed head and ears of medium length. The English springer spaniel is the larger and can be of various colours; the Welsh springer spaniel is always a rich red and white. n. datasets: - sentence-transformers/msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1 pipeline_tag: sentence-similarity library_name: sentence-transformers metrics: - cosine_accuracy@1 - cosine_accuracy@3 - cosine_accuracy@5 - cosine_accuracy@10 - cosine_precision@1 - cosine_precision@3 - cosine_precision@5 - cosine_precision@10 - cosine_recall@1 - cosine_recall@3 - cosine_recall@5 - cosine_recall@10 - cosine_ndcg@10 - cosine_mrr@10 - cosine_map@100 model-index: - name: SentenceTransformer based on answerdotai/ModernBERT-base results: - task: type: information-retrieval name: Information Retrieval dataset: name: eval type: eval metrics: - type: cosine_accuracy@1 value: 0.7949656022587187 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.9252395912037221 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.9530759136278681 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9735157275221696 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.7949656022587187 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.3084131970679073 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.19061518272557365 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09735157275221698 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.7949656022587187 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.9252395912037221 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.9530759136278681 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9735157275221696 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.8908733180956838 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.8636181159543801 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.864765622765834 name: Cosine Map@100 --- # SentenceTransformer based on answerdotai/ModernBERT-base This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) on the [msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1](https://huggingface.co/datasets/sentence-transformers/msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) - **Maximum Sequence Length:** 512 tokens - **Output Dimensionality:** 768 dimensions - **Similarity Function:** Cosine Similarity - **Training Dataset:** - [msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1](https://huggingface.co/datasets/sentence-transformers/msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1) - **Language:** en ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'OptimizedModule'}) (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("modernbert-msmarco") # Run inference queries = [ "product key windows 8.1 how to find", ] documents = [ 'If Windows 8.1 came preinstalled on your computer, your Windows 8.1 product key should be on a sticker on your computer or with your documentation. The Windows 8.1 product key is a series of 25 letters and numbers and should look like this: xxxxx-xxxxx-xxxxx-xxxxx-xxxxx.', 'springer spaniel. n. 1. (Breeds) either of two breeds of large quick-moving spaniels bred to spring game, having a slightly domed head and ears of medium length. The English springer spaniel is the larger and can be of various colours; the Welsh springer spaniel is always a rich red and white. n.', 'Al Gore not divorced from wife Tipper, confirms relationship with longtime girlfriend. 1 Pucker up! Al Gore planted a wet one on wife Tipper in 2000, during his presidential campaign. Ten years later the couple separated after 40 years of marriage.', ] query_embeddings = model.encode_query(queries) document_embeddings = model.encode_document(documents) print(query_embeddings.shape, document_embeddings.shape) # [1, 768] [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(query_embeddings, document_embeddings) print(similarities) # tensor([[ 0.8319, -0.0147, -0.0184]]) ``` ## Evaluation ### Metrics #### Information Retrieval * Dataset: `eval` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.795 | | cosine_accuracy@3 | 0.9252 | | cosine_accuracy@5 | 0.9531 | | cosine_accuracy@10 | 0.9735 | | cosine_precision@1 | 0.795 | | cosine_precision@3 | 0.3084 | | cosine_precision@5 | 0.1906 | | cosine_precision@10 | 0.0974 | | cosine_recall@1 | 0.795 | | cosine_recall@3 | 0.9252 | | cosine_recall@5 | 0.9531 | | cosine_recall@10 | 0.9735 | | **cosine_ndcg@10** | **0.8909** | | cosine_mrr@10 | 0.8636 | | cosine_map@100 | 0.8648 | ## Training Details ### Training Dataset #### msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1 * Dataset: [msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1](https://huggingface.co/datasets/sentence-transformers/msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1) at [84ed2d3](https://huggingface.co/datasets/sentence-transformers/msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1/tree/84ed2d35626f617d890bd493b4d6db69a741e0e2) * Size: 477,792 training samples * Columns: query and positive * Approximate statistics based on the first 1000 samples: | | query | positive | |:--------|:---------------------------------------------------------------------------------|:------------------------------------------------------------------------------------| | type | string | string | | details | | | * Samples: | query | positive | |:-----------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | what is the farthest distance in the universe | Depends on what you mean by seeing. The particle horizon is just the furthest. distance light could have traveled to us since the universe began. That is 93 billion. light years in diameter, or 47 billion light years in any direction, but we can't actually. see anything at that distance. | | what county is laurel ms in | Laurel, MS. Online Offers. Laurel is a city located in Jones County in Mississippi, a state of the United States of America. As of the 2000 census, the city had a total population of 18,393 although a significant population increase has been reported following Hurricane Katrina. Located in southeast Mississippi, southeast of Jackson on Tallahala Creek, Laurel was founded in 1882 as a lumber town. An American Indian reservation is located in nearby Sandersville. Laurel is the principal city of the Laurel Micropolitan Statistical Area. | | how to use a beadloom | How to string your bead loom. To string a loom, attach your nymo thread to one of the small nails at the end of the loom. Run the thread over the metal bars (located on both ends of the loom) and wrap it around one of the small nails on the other end of your loom.ize 8 seed beads are normally to heavy to be used on a loom. The end result would be beadwork that sags in the middle. Every other slow on the metal bar was skipped to accomodate size 8 seed beads. You will not need to do this with seed beads sizes 10-15 that are the correct size beads to use on a bead loom. | * Loss: [CachedMultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativesrankingloss) with these parameters: ```json { "scale": 20.0, "similarity_fct": "cos_sim", "mini_batch_size": 64, "gather_across_devices": false, "directions": [ "query_to_doc" ], "partition_mode": "joint", "hardness_mode": null, "hardness_strength": 0.0 } ``` ### Evaluation Dataset #### msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1 * Dataset: [msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1](https://huggingface.co/datasets/sentence-transformers/msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1) at [84ed2d3](https://huggingface.co/datasets/sentence-transformers/msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1/tree/84ed2d35626f617d890bd493b4d6db69a741e0e2) * Size: 25,147 evaluation samples * Columns: query and positive * Approximate statistics based on the first 1000 samples: | | query | positive | |:--------|:---------------------------------------------------------------------------------|:------------------------------------------------------------------------------------| | type | string | string | | details | | | * Samples: | query | positive | |:--------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | how long it take for a pimple to burst | Start doing warm compress over the pimple, so that it gradually gets drained over 2 to 3 days. Do it gently, without giving yourself much pain. Also, apply mupirocin ointment over it twice a day for 3 to 4 days.Read above in detail about dealing with your infected pimple.tart doing warm compress over the pimple, so that it gradually gets drained over 2 to 3 days. Do it gently, without giving yourself much pain. Also, apply mupirocin ointment over it twice a day for 3 to 4 days. | | tularosa population | The Village of Tularosa had a population of 2,677 as of July 1, 2017. Tularosa ranks in the upper quartile for Population Density when compared to the other cities, towns and Census Designated Places (CDPs) in New Mexico. See peer rankings below. The primary coordinate point for Tularosa is located at latitude 33.075 and longitude -106.0173 in Otero County. | | do some people have their blood flowing in reverse direction | As a result, not enough blood flows through the valve. Some valves can have both stenosis and backflow problems. Atresia occurs if a heart valve lacks an opening for blood to pass through. Some people are born with heart valve disease, while others acquire it later in life. | * Loss: [CachedMultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativesrankingloss) with these parameters: ```json { "scale": 20.0, "similarity_fct": "cos_sim", "mini_batch_size": 64, "gather_across_devices": false, "directions": [ "query_to_doc" ], "partition_mode": "joint", "hardness_mode": null, "hardness_strength": 0.0 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `per_device_train_batch_size`: 1024 - `num_train_epochs`: 1 - `learning_rate`: 2e-05 - `warmup_steps`: 0.1 - `bf16`: True - `eval_strategy`: epoch - `per_device_eval_batch_size`: 1024 - `push_to_hub`: True - `hub_model_id`: modernbert-msmarco - `load_best_model_at_end`: True - `dataloader_num_workers`: 4 - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `per_device_train_batch_size`: 1024 - `num_train_epochs`: 1 - `max_steps`: -1 - `learning_rate`: 2e-05 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: None - `warmup_steps`: 0.1 - `optim`: adamw_torch_fused - `optim_args`: None - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `optim_target_modules`: None - `gradient_accumulation_steps`: 1 - `average_tokens_across_devices`: True - `max_grad_norm`: 1.0 - `label_smoothing_factor`: 0.0 - `bf16`: True - `fp16`: False - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `use_liger_kernel`: False - `liger_kernel_config`: None - `use_cache`: False - `neftune_noise_alpha`: None - `torch_empty_cache_steps`: None - `auto_find_batch_size`: False - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `include_num_input_tokens_seen`: no - `log_level`: passive - `log_level_replica`: warning - `disable_tqdm`: False - `project`: huggingface - `trackio_space_id`: trackio - `eval_strategy`: epoch - `per_device_eval_batch_size`: 1024 - `prediction_loss_only`: True - `eval_on_start`: False - `eval_do_concat_batches`: True - `eval_use_gather_object`: False - `eval_accumulation_steps`: None - `include_for_metrics`: [] - `batch_eval_metrics`: False - `save_only_model`: False - `save_on_each_node`: False - `enable_jit_checkpoint`: False - `push_to_hub`: True - `hub_private_repo`: None - `hub_model_id`: modernbert-msmarco - `hub_strategy`: every_save - `hub_always_push`: False - `hub_revision`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `restore_callback_states_from_checkpoint`: False - `full_determinism`: False - `seed`: 42 - `data_seed`: None - `use_cpu`: False - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `parallelism_config`: None - `dataloader_drop_last`: False - `dataloader_num_workers`: 4 - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `dataloader_prefetch_factor`: None - `remove_unused_columns`: True - `label_names`: None - `train_sampling_strategy`: random - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `ddp_backend`: None - `ddp_timeout`: 1800 - `fsdp`: [] - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `deepspeed`: None - `debug`: [] - `skip_memory_metrics`: True - `do_predict`: False - `resume_from_checkpoint`: None - `warmup_ratio`: None - `local_rank`: -1 - `prompts`: None - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional - `router_mapping`: {} - `learning_rate_mapping`: {}
### Training Logs | Epoch | Step | Training Loss | Validation Loss | eval_cosine_ndcg@10 | |:-------:|:-------:|:-------------:|:---------------:|:-------------------:| | 0.1071 | 50 | 4.1149 | - | - | | 0.2141 | 100 | 0.5296 | - | - | | 0.3212 | 150 | 0.3000 | - | - | | 0.4283 | 200 | 0.2463 | - | - | | 0.5353 | 250 | 0.2247 | - | - | | 0.6424 | 300 | 0.2032 | - | - | | 0.7495 | 350 | 0.1923 | - | - | | 0.8565 | 400 | 0.1900 | - | - | | 0.9636 | 450 | 0.1888 | - | - | | **1.0** | **467** | **-** | **0.1889** | **0.8768** | | 0.1071 | 50 | 0.1866 | - | - | | 0.2141 | 100 | 0.1560 | - | - | | 0.3212 | 150 | 0.1455 | - | - | | 0.4283 | 200 | 0.1377 | - | - | | 0.5353 | 250 | 0.1397 | - | - | | 0.6424 | 300 | 0.1351 | - | - | | 0.7495 | 350 | 0.1355 | - | - | | 0.8565 | 400 | 0.1417 | - | - | | 0.9636 | 450 | 0.1468 | - | - | | **1.0** | **467** | **-** | **0.1512** | **0.8909** | * The bold row denotes the saved checkpoint. ### Framework Versions - Python: 3.12.12 - Sentence Transformers: 5.3.0 - Transformers: 5.3.0 - PyTorch: 2.10.0+cu128 - Accelerate: 1.13.0 - Datasets: 4.7.0 - Tokenizers: 0.22.2 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### CachedMultipleNegativesRankingLoss ```bibtex @misc{gao2021scaling, title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup}, author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan}, year={2021}, eprint={2101.06983}, archivePrefix={arXiv}, primaryClass={cs.LG} } ```