tranhuudan-fullstack-ai-engineer/stage4_1

SentenceTransformer based on huudan123/stag_123_cp8000

This is a sentence-transformers model finetuned from huudan123/stag_123_cp8000. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: huudan123/stag_123_cp8000
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: RobertaModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("huudan123/stage4_1")
# Run inference
sentences = [
    'Một người đàn ông đang lắp ráp các bộ phận loa.',
    'Một người đàn ông đang đi bộ trên vỉa hè.',
    'Một người đàn ông phun nước từ vòi cho một người đàn ông khác.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Semantic Similarity

• Dataset: sts-evaluator
• Evaluated with EmbeddingSimilarityEvaluator

Metric	Value
pearson_cosine	0.3596
spearman_cosine	0.3357
pearson_manhattan	0.3644
spearman_manhattan	0.3441
pearson_euclidean	0.3668
spearman_euclidean	0.3479
pearson_dot	0.3312
spearman_dot	0.3066
pearson_max	0.3668
spearman_max	0.3479

Training Details

Training Hyperparameters

Non-Default Hyperparameters

• overwrite_output_dir: True
• eval_strategy: steps
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• learning_rate: 2e-05
• num_train_epochs: 15
• lr_scheduler_type: cosine_with_restarts
• warmup_ratio: 0.1
• fp16: True
• load_best_model_at_end: True
• gradient_checkpointing: True

All Hyperparameters

Click to expand

• overwrite_output_dir: True
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 15
• max_steps: -1
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	loss	sts-evaluator_spearman_max
0	0	-	-	0.7131
0.1321	100	0.0438	-	-
0.2642	200	0.0141	-	-
0.3963	300	0.0073	-	-
0.5284	400	0.0049	-	-
0.6605	500	0.0038	0.0120	0.6353
0.7926	600	0.0031	-	-
0.9247	700	0.0027	-	-
1.0568	800	0.0024	-	-
1.1889	900	0.0021	-	-
1.321	1000	0.0019	0.0126	0.5158
1.4531	1100	0.0018	-	-
1.5852	1200	0.0017	-	-
1.7173	1300	0.0019	-	-
1.8494	1400	0.0016	-	-
1.9815	1500	0.0014	0.0125	0.4359
2.1136	1600	0.0014	-	-
2.2457	1700	0.0013	-	-
2.3778	1800	0.0013	-	-
2.5099	1900	0.0012	-	-
2.6420	2000	0.0012	0.0144	0.4196
2.7741	2100	0.0012	-	-
2.9062	2200	0.0011	-	-
3.0383	2300	0.0012	-	-
3.1704	2400	0.0011	-	-
3.3025	2500	0.0011	0.0159	0.3717
3.4346	2600	0.0011	-	-
3.5667	2700	0.0011	-	-
3.6988	2800	0.001	-	-
3.8309	2900	0.001	-	-
3.9630	3000	0.001	0.0160	0.3479

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.0.1
• Transformers: 4.42.4
• PyTorch: 2.3.1+cu121
• Accelerate: 0.33.0
• Datasets: 2.20.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}