Krishnasri2027/qwen25-coder-1.5b-instruct-fine-tuned-merged

Model Card for Qwen2.5-Coder-1.5B Python 3.14 Fine-Tuned Merged Model

Model Details

Model Description

This model is a standalone fine-tuned version of Qwen/Qwen2.5-Coder-1.5B-Instruct, specialized for Python code modernization with a focus on updating Python code toward Python 3.14-compatible syntax and practices.

The model was fine-tuned using QLoRA with a 4-bit NF4 quantized base model and LoRA adapters. After training, the learned LoRA adapter weights were merged into the original base model to produce this standalone model.

Unlike the adapter repository, this repository contains the merged model weights and does not require a separate LoRA adapter at inference time.

• Base model: Qwen/Qwen2.5-Coder-1.5B-Instruct
• Model type: Standalone merged causal language model
• Task: Python code modernization
• Primary target: Python 3.14 modernization
• Training method: QLoRA + LoRA, followed by adapter merging
• Training-time base quantization: 4-bit NF4
• Training-time nested quantization: Enabled
• Training-time compute dtype: FP16
• Deployment format: FP16 merged model
• Adapter required for inference: No

Model Sources

• Base model: Qwen/Qwen2.5-Coder-1.5B-Instruct
• Adapter model: Krishnasri2027/qwen25-coder-1.5b-python3147-nf4-adapter
• Merged model repository: Krishnasri2027/qwen25-coder-1.5b-instruct-fine-tuned-merged

Uses

Direct Use

This model is intended to directly generate modernized Python source code, particularly when migrating or refactoring code toward Python 3.14.

Potential applications include:

• Modernizing legacy Python syntax
• Updating Python code to newer language features
• Refactoring older Python implementations
• Assisting developers with Python version migration
• Generating modernized Python code from older implementations
• Supporting automated Python modernization workflows

Because this is the merged model, no separate PEFT adapter needs to be loaded for inference.

Downstream Use

The model can be integrated into:

• Developer assistants
• IDE and code-editor tooling
• Python migration pipelines
• Automated refactoring systems
• Code review and modernization workflows
• Educational tools for Python modernization

The FP16 model can also be further quantized into a deployment format appropriate for the target hardware.

Out-of-Scope Use

This model is not intended to:

• Guarantee that generated code is fully compatible with every Python 3.14 environment
• Replace automated testing or human code review
• Perform security-critical code migration without validation
• Generate production-ready software without testing
• Serve as a general-purpose replacement for the original Qwen2.5-Coder model
• Be treated as a verified source-to-source compiler

Generated code should always be validated, tested, and reviewed before production use.

Bias, Risks, and Limitations

The model inherits limitations from the underlying Qwen2.5-Coder model and from the fine-tuning dataset.

The model may:

• Produce syntactically incorrect code in some situations
• Introduce behavioral changes during modernization
• Apply an inappropriate modernization depending on the context
• Fail to preserve edge-case behavior
• Generate code that appears valid but requires additional testing
• Miss modernization opportunities
• Perform an unnecessary or undesirable refactor
• Reflect biases or limitations present in the base model and training data

A low validation loss or high token-level accuracy does not guarantee semantic equivalence between the original and generated programs.

Recommendations

Generated code should be:

1. Reviewed by a developer.
2. Parsed or compiled using the target Python version.
3. Tested against the original implementation's expected behavior.
4. Validated with appropriate unit and integration tests.
5. Checked for unintended API, performance, or behavioral changes.

How to Get Started with the Model

This repository contains a standalone merged model, so the LoRA adapter is not required for inference.

import torch

from transformers import AutoModelForCausalLM, AutoTokenizer


MODEL_ID = "Krishnasri2027/qwen25-coder-1.5b-instruct-fine-tuned-merged"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True,
)

model.eval()

messages = [
    {
        "role": "user",
        "content": "Modernize the following Python code for Python 3.14:\n\n<your Python code here>",
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    outputs = model.generate(
        inputs,
        max_new_tokens=512,
        temperature=0.2,
        do_sample=False,
    )

response = tokenizer.decode(
    outputs[0][inputs.shape[-1]:],
    skip_special_tokens=True,
)

print(response)

Low-Memory Deployment

This repository contains the FP16 merged model. For systems with limited GPU memory, such as GPUs with approximately 2 GB VRAM, the FP16 model may not fit entirely in VRAM.

For low-memory deployment, the merged model can be further quantized using a suitable inference format and runtime. Quantization is a separate deployment step from the LoRA merge performed during model creation.

Training Details

Training Data

The model was fine-tuned using a chat-formatted training dataset designed for Python code modernization.

The training examples were tokenized using the Qwen2.5-Coder tokenizer and its chat template.

The configured sequence length was 256 tokens.

Token-length analysis of the training examples produced:

Statistic	Tokens
Minimum	115
Maximum	197
Mean	150.792
Median	147
95th percentile	187
99th percentile	197
Configured sequence length	256

The maximum observed sequence length of 197 tokens was below the configured 256-token sequence length, providing approximately 59 tokens of headroom.

Training Procedure

The model was fine-tuned using QLoRA, keeping the quantized base model frozen while training a small set of LoRA adapter parameters.

The base model was quantized to 4-bit NF4 using BitsAndBytes with nested/double quantization.

LoRA adapters were applied to:

• q_proj
• k_proj
• v_proj
• o_proj
• gate_proj
• up_proj
• down_proj

After the training run was completed, the trained LoRA adapter was merged into the base model using PEFT's merge functionality. The resulting model was saved as a standalone FP16 model.

The final repository therefore contains the merged model rather than only the LoRA adapter.

Training Hyperparameters

Parameter	Value
Base model	Qwen/Qwen2.5-Coder-1.5B-Instruct
Fine-tuning method	QLoRA
Quantization	4-bit NF4
Nested quantization	Enabled
Compute dtype	FP16
LoRA rank (r)	16
LoRA alpha	32
LoRA dropout	0.05
Batch size	1
Gradient accumulation steps	8
Effective batch size	8
Number of epochs	4
Learning rate	1e-4
LR scheduler	Cosine
Weight decay	0.01
Warmup steps	30
Evaluation frequency	Every 50 steps
Checkpoint save frequency	Every 50 steps
Logging frequency	Every 10 steps
Maximum sequence length	256
Gradient checkpointing	Enabled
FIM rate	0.0
FIM SPM rate	0.0
Random seed	42
FP16	Enabled
BF16	Disabled

Speeds, Sizes, Times

The completed training run reported:

Metric	Value
Global steps	500
Epochs	4
Training runtime	3342.5477 seconds
Training runtime	~55 minutes 43 seconds
Training samples / second	1.197
Training steps / second	0.150
Total FLOPs	4,835,834,903,592,960
Total tokens processed	607,168
Final logged training loss	0.037346
Aggregate Trainer training loss	0.2003363176

The training_loss value reported by Trainer (0.2003363176) represents the aggregate training loss reported for the complete run, whereas the training loss shown at step 500 (0.037346) is the most recent logged training loss.

Evaluation

Testing Data, Factors & Metrics

Testing Data

The evaluation dataset was held separately from the training data and was used to calculate validation loss during training.

Validation was performed every 50 training steps.

Factors

The primary evaluation objective was to determine whether the model could learn the expected Python modernization patterns while reducing validation loss throughout training.

The evaluation results below are the validation losses recorded during the training run.

Metrics

The primary reported metrics are:

• Training loss
• Validation loss
• Entropy
• Mean token accuracy
• Number of processed tokens

Results

Summary

The model showed a substantial reduction in both training and validation loss during fine-tuning.

Validation loss decreased from 0.252825 at step 50 to 0.037586 at step 500.

This represents an approximately 85.1% reduction in validation loss over the recorded training run.

The final reported mean token accuracy was 98.6229% at step 500.

The validation loss continued to decrease throughout the later checkpoints, reaching its lowest recorded value of 0.037586 at step 500.

Step	Training Loss	Validation Loss	Entropy	Mean Token Accuracy	Num Tokens
50	0.329822	0.252825	0.280223	0.939414	60,652
100	0.070807	0.064497	0.067091	0.981024	121,228
150	0.043590	0.048881	0.047465	0.984845	182,259
200	0.040075	0.042402	0.043094	0.985729	242,627
250	0.038932	0.040337	0.044742	0.985716	303,584
300	0.037993	0.038709	0.041848	0.985923	364,412
350	0.037793	0.038682	0.041355	0.986143	424,864
400	0.038162	0.038088	0.039560	0.986307	485,911
450	0.040015	0.037656	0.041152	0.986229	546,449
500	0.037346	0.037586	0.040890	0.986229	607,168

The final training output was:

TrainOutput(
    global_step=500,
    training_loss=0.20033631759881973,
    epoch=4.0
)

with the following reported training metrics:

train_runtime: 3342.5477
train_samples_per_second: 1.197
train_steps_per_second: 0.15
total_flos: 4835834903592960.0
train_loss: 0.20033631759881973
epoch: 4.0

These results indicate that the model successfully optimized the training objective and achieved high token-level accuracy on the evaluation data.

However, token-level accuracy and validation loss should not be interpreted as a guarantee that every generated modernization preserves the original program's behavior. Functional testing remains necessary.

Model Examination

The model should be examined primarily through qualitative code-generation and modernization tests.

Recommended examination procedures include:

• Comparing legacy and modernized Python implementations.
• Checking generated code with the Python 3.14 interpreter.
• Running unit tests before and after modernization.
• Checking whether program behavior is preserved.
• Testing edge cases and uncommon Python constructs.
• Evaluating whether deprecated or legacy syntax is correctly modernized.
• Measuring semantic equivalence between original and modernized programs where practical.

No additional qualitative examination results are reported in this model card unless separately documented.

Environmental Impact

The training run required approximately 55 minutes and 43 seconds of compute time.

No verified carbon-emissions measurement was recorded for this training run, so a specific carbon footprint is not reported.

Technical Specifications

Model Architecture and Objective

The underlying architecture is based on Qwen2.5-Coder-1.5B-Instruct.

The fine-tuning objective uses supervised instruction tuning with LoRA adapters.

QLoRA was used during training to reduce the memory requirements of fine-tuning by quantizing the frozen base model to 4-bit NF4 while keeping the trainable LoRA parameters separate.

The LoRA configuration was:

r = 16
alpha = 32
dropout = 0.05

Target modules:

q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj

Following training, the LoRA adapter weights were merged into the base model. This repository contains the resulting standalone model.

Important: Training Quantization vs. Model Format

The 4-bit NF4 configuration describes how the base model was loaded during QLoRA training.

It does not mean that this merged repository is an NF4 model.

The merged model was saved as an FP16 standalone model. The LoRA adapter is already incorporated into the model weights, so a separate adapter is not required for inference.

If lower memory usage is required, the FP16 merged model can be quantized separately for deployment.

Compute Infrastructure

Hardware

Training was performed using an NVIDIA Tesla T4 GPU environment.

Software

Component	Version
Python	3.13
PyTorch	2.11.0+cu128
CUDA	12.8
Transformers	5.16.1
TRL	1.12.0
PEFT	0.20.0
Accelerate	1.14.0
BitsAndBytes	0.50.2

Citation

If you use this model, please also cite the underlying Qwen2.5-Coder model according to the citation information provided by its authors.

Glossary

LoRA
Low-Rank Adaptation, a parameter-efficient fine-tuning method that trains small low-rank matrices instead of updating the entire model.

QLoRA
Quantized LoRA fine-tuning, where the frozen base model is loaded in low-bit precision while LoRA parameters are trained.

NF4
NormalFloat 4-bit, a 4-bit quantization data type designed for normally distributed neural-network weights.

Nested Quantization / Double Quantization
A technique that further quantizes the quantization constants to reduce memory usage.

FP16
16-bit floating-point representation used for computation during training and for the saved merged model.

Mean Token Accuracy
The proportion of target tokens correctly predicted by the model during evaluation.

Validation Loss
The loss calculated on the held-out evaluation dataset.

Merged Model
A standalone model in which the trained LoRA adapter weights have been incorporated into the base model weights, eliminating the need to load a separate adapter during inference.

More Information

This model is specifically intended for Python code modernization toward Python 3.14.

This repository contains the standalone FP16 merged model corresponding to the trained LoRA adapter.

The relationship between the two repositories is:

Qwen/Qwen2.5-Coder-1.5B-Instruct
                |
                |  QLoRA fine-tuning
                v
       LoRA Adapter
                |
                |  merge adapter into base
                v
       Standalone FP16 Model

The adapter-only version is available as:

Krishnasri2027/qwen25-coder-1.5b-python3147-nf4-adapter

The merged version in this repository does not require the adapter for inference.

For deployment on hardware with limited memory, the FP16 merged model can be further quantized using an appropriate inference format and runtime.

Model Card Authors

Krishnasri2027

Model Card Contact

For questions, issues, or suggestions regarding this model, please use the discussion and issue facilities available on the model repository.