shikshalokam/shiksha-statement-tagger

SetFit with sentence-transformers/all-mpnet-base-v2

This is a SetFit model that can be used for Text Classification. This SetFit model uses sentence-transformers/all-mpnet-base-v2 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a Sentence Transformer with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

• Model Type: SetFit
• Sentence Transformer body: sentence-transformers/all-mpnet-base-v2
• Classification head: a LogisticRegression instance
• Maximum Sequence Length: 384 tokens
• Number of Classes: 3 classes

Model Sources

• Repository: SetFit on GitHub
• Paper: Efficient Few-Shot Learning Without Prompts
• Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
Challenge	'Child education is affected due to poverty.' 'The mother makes her children work in the fields to help with household chores' 'Being illiterate, parents are not paying attention to education.'
Other	'Education and prevention of child marriage' 'In the Shiksha Chaupal meeting a woman said that all the children here are careless Master Sahib the children of Garjiya Allah Parvatiya should only understand studying' 'Reason for Aadhar Card'
Solution or Action	'The villagers decided to take the disabled girl to a disability camp in the block to get her certificate made.' 'All the parents have decided together that we will get our children admitted to school.' 'It was discussed to call a meeting of S.D.M.C.'

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("setfit_model_id")
# Run inference
preds = model("Every child has to excel")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	2	17.4997	65

Label	Training Sample Count
Challenge	1708
Other	1275
Solution or Action	1876

Training Hyperparameters

• batch_size: (16, 16)
• num_epochs: (1, 1)
• max_steps: -1
• sampling_strategy: oversampling
• num_iterations: 20
• body_learning_rate: (2e-05, 1e-05)
• head_learning_rate: 0.01
• loss: CosineSimilarityLoss
• distance_metric: cosine_distance
• margin: 0.25
• end_to_end: False
• use_amp: False
• warmup_proportion: 0.1
• l2_weight: 0.01
• seed: 42
• eval_max_steps: -1
• load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0002	1	0.3656	-
0.0082	50	0.2898	-
0.0165	100	0.2478	-
0.0247	150	0.2164	-
0.0329	200	0.1672	-
0.0412	250	0.1574	-
0.0494	300	0.1483	-
0.0576	350	0.1364	-
0.0659	400	0.1304	-
0.0741	450	0.1162	-
0.0823	500	0.1202	-
0.0905	550	0.1109	-
0.0988	600	0.1005	-
0.1070	650	0.095	-
0.1152	700	0.0846	-
0.1235	750	0.0875	-
0.1317	800	0.0842	-
0.1399	850	0.0792	-
0.1482	900	0.0694	-
0.1564	950	0.0671	-
0.1646	1000	0.0667	-
0.1729	1050	0.0592	-
0.1811	1100	0.0608	-
0.1893	1150	0.0511	-
0.1976	1200	0.0575	-
0.2058	1250	0.0553	-
0.2140	1300	0.0458	-
0.2223	1350	0.0475	-
0.2305	1400	0.0413	-
0.2387	1450	0.0342	-
0.2470	1500	0.0298	-
0.2552	1550	0.031	-
0.2634	1600	0.0298	-
0.2716	1650	0.0305	-
0.2799	1700	0.0284	-
0.2881	1750	0.0324	-
0.2963	1800	0.0228	-
0.3046	1850	0.0242	-
0.3128	1900	0.0231	-
0.3210	1950	0.0175	-
0.3293	2000	0.0218	-
0.3375	2050	0.0148	-
0.3457	2100	0.0171	-
0.3540	2150	0.0167	-
0.3622	2200	0.0158	-
0.3704	2250	0.0195	-
0.3787	2300	0.0149	-
0.3869	2350	0.0182	-
0.3951	2400	0.0184	-
0.4034	2450	0.0191	-
0.4116	2500	0.0143	-
0.4198	2550	0.0081	-
0.4281	2600	0.0174	-
0.4363	2650	0.0112	-
0.4445	2700	0.0126	-
0.4527	2750	0.0162	-
0.4610	2800	0.0129	-
0.4692	2850	0.0158	-
0.4774	2900	0.0166	-
0.4857	2950	0.0108	-
0.4939	3000	0.0105	-
0.5021	3050	0.0115	-
0.5104	3100	0.0118	-
0.5186	3150	0.0106	-
0.5268	3200	0.0102	-
0.5351	3250	0.0095	-
0.5433	3300	0.0097	-
0.5515	3350	0.0054	-
0.5598	3400	0.0143	-
0.5680	3450	0.0079	-
0.5762	3500	0.0115	-
0.5845	3550	0.0061	-
0.5927	3600	0.0071	-
0.6009	3650	0.0074	-
0.6092	3700	0.006	-
0.6174	3750	0.0043	-
0.6256	3800	0.0069	-
0.6338	3850	0.0064	-
0.6421	3900	0.0041	-
0.6503	3950	0.0076	-
0.6585	4000	0.0083	-
0.6668	4050	0.0064	-
0.6750	4100	0.0049	-
0.6832	4150	0.0036	-
0.6915	4200	0.0068	-
0.6997	4250	0.0049	-
0.7079	4300	0.0056	-
0.7162	4350	0.0055	-
0.7244	4400	0.0035	-
0.7326	4450	0.0017	-
0.7409	4500	0.0022	-
0.7491	4550	0.0018	-
0.7573	4600	0.0051	-
0.7656	4650	0.0005	-
0.7738	4700	0.0017	-
0.7820	4750	0.0031	-
0.7903	4800	0.003	-
0.7985	4850	0.0023	-
0.8067	4900	0.0021	-
0.8149	4950	0.002	-
0.8232	5000	0.0017	-
0.8314	5050	0.0024	-
0.8396	5100	0.0033	-
0.8479	5150	0.0015	-
0.8561	5200	0.0029	-
0.8643	5250	0.0038	-
0.8726	5300	0.0015	-
0.8808	5350	0.0013	-
0.8890	5400	0.0021	-
0.8973	5450	0.0014	-
0.9055	5500	0.0006	-
0.9137	5550	0.0025	-
0.9220	5600	0.0017	-
0.9302	5650	0.0009	-
0.9384	5700	0.0012	-
0.9467	5750	0.0014	-
0.9549	5800	0.0001	-
0.9631	5850	0.0046	-
0.9714	5900	0.0013	-
0.9796	5950	0.0019	-
0.9878	6000	0.0028	-
0.9960	6050	0.0026	-

Framework Versions

• Python: 3.12.13
• SetFit: 1.1.0
• Sentence Transformers: 3.1.1
• Transformers: 4.45.2
• PyTorch: 2.10.0+cu128
• Datasets: 3.2.0
• Tokenizers: 0.20.3

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}