Same LR , 1.3B Pre-Trained models (4x-Chinchilla on C4, 100B+ tokens) from "LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition".
🤝 Open to Collab
Shikhar Srivastava
shikhar-srivastava
AI & ML interests
None yet
Recent Activity
upvoted a paper 10 days ago
LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition updated a collection 17 days ago
LayerRoPE 1.3B | LR Matched published a model 17 days ago
shikhar-srivastava/1b_lrmatched_pre_lr5e-4_seed1Organizations
None yet
models 452
shikhar-srivastava/1b_lrmatched_pre_lr5e-4_seed1
Updated • 53
shikhar-srivastava/1b_lrmatched_norm_after_lr5e-4_seed1
Updated • 43
shikhar-srivastava/1b_lrmatched_lns_lr5e-4_seed1
Updated • 47
shikhar-srivastava/1b_lrmatched_layerrope_lr5e-4_seed1
Updated • 48
shikhar-srivastava/1b_layerrope_lr3.2e-2_seed1
Updated • 5
shikhar-srivastava/1b_scaling_lns_lr8e-3_seed1
Updated • 7
shikhar-srivastava/1b_scaling_norm_after_lr8e-3_seed1
Updated • 7
shikhar-srivastava/1b_scaling_pre_lr2e-3_seed1
Updated • 8
shikhar-srivastava/mono_350m_pre_lr1e-3_fineweb2_urd_unigram_unscaled_65536
0.4B • Updated • 15
shikhar-srivastava/mono_350m_pre_lr1e-3_fineweb2_tha_unigram_unscaled_65536
0.4B • Updated • 13
datasets 0
None public yet