Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,76 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: cc-by-nc-sa-4.0
|
| 3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: cc-by-nc-sa-4.0
|
| 3 |
+
tags:
|
| 4 |
+
- audio
|
| 5 |
+
- vocoder
|
| 6 |
+
- deepfake-enhancement
|
| 7 |
+
- synthetic-audio
|
| 8 |
+
- vocos
|
| 9 |
+
datasets:
|
| 10 |
+
- grotsoV/4TTSDeepfakeData
|
| 11 |
+
language:
|
| 12 |
+
- en
|
| 13 |
+
base_model:
|
| 14 |
+
- charactr/vocos-mel-24khz
|
| 15 |
+
---
|
| 16 |
+
|
| 17 |
+
# Vocos for Deepfake Audio Enhancement
|
| 18 |
+
|
| 19 |
+
## Model Description
|
| 20 |
+
This model is a custom-trained checkpoint of the [Vocos](https://github.com/charactr-platform/vocos) vocoder architecture, specifically fine-tuned to enhance the phase coherence and acoustic quality of synthetic speech (deepfakes).
|
| 21 |
+
|
| 22 |
+
It was trained on the [4TTSDeepfakeDataset](https://huggingface.co/datasets/grotsoV/4TTSDeepfakeData), which contains 1:1 parallel bona fide and zero-shot deepfake utterances generated by F5-TTS, MaskGCT, CosyVoice2, and Fish-S1mini.
|
| 23 |
+
|
| 24 |
+
* **Architecture:** Vocos (Mel-spectrogram to 24kHz waveform)
|
| 25 |
+
* **Sampling Rate:** 24,000 Hz
|
| 26 |
+
* **Primary Task:** Audio enhancement / Deepfake artifact reduction
|
| 27 |
+
|
| 28 |
+
## Intended Use & Limitations
|
| 29 |
+
This model is released strictly for **academic research, educational, and personal purposes**.
|
| 30 |
+
It is intended to be used in audio forensics, deepfake detection research, and synthetic audio analysis pipelines.
|
| 31 |
+
|
| 32 |
+
**Prohibited Uses:** Due to the CC BY-NC-SA 4.0 license, you may **not** use these weights for any commercial purposes, productization, or malicious voice cloning applications.
|
| 33 |
+
|
| 34 |
+
## How to Use
|
| 35 |
+
You can load these weights using the original `vocos` Python library. Because this is a fine-tune of the original 24kHz mel-spectrogram model, you instantiate the base model first and then load your downloaded weights.
|
| 36 |
+
|
| 37 |
+
```bash
|
| 38 |
+
pip install vocos
|
| 39 |
+
```
|
| 40 |
+
|
| 41 |
+
```python
|
| 42 |
+
import torch
|
| 43 |
+
import torchaudio
|
| 44 |
+
from vocos import Vocos
|
| 45 |
+
|
| 46 |
+
# 1. Initialize the base Vocos model
|
| 47 |
+
vocos = Vocos.from_pretrained("charactr/vocos-mel-24khz")
|
| 48 |
+
|
| 49 |
+
# 2. Load the custom deepfake enhancement weights
|
| 50 |
+
# (Replace 'model.pt' with the actual name of the file you upload here)
|
| 51 |
+
state_dict = torch.load("model.pt", map_location="cpu")
|
| 52 |
+
|
| 53 |
+
# Handle nested state dictionaries if necessary
|
| 54 |
+
if 'vocos_state_dict' in state_dict:
|
| 55 |
+
vocos.load_state_dict(state_dict['vocos_state_dict'])
|
| 56 |
+
else:
|
| 57 |
+
vocos.load_state_dict(state_dict)
|
| 58 |
+
|
| 59 |
+
vocos.eval()
|
| 60 |
+
|
| 61 |
+
# 3. Inference example
|
| 62 |
+
wav, sr = torchaudio.load("path/to/your/deepfake.wav")
|
| 63 |
+
if sr != 24000:
|
| 64 |
+
wav = torchaudio.transforms.Resample(sr, 24000)(wav)
|
| 65 |
+
|
| 66 |
+
with torch.no_grad():
|
| 67 |
+
mel = vocos.feature_extractor(wav)
|
| 68 |
+
x = vocos.backbone(mel)
|
| 69 |
+
enhanced_wav = vocos.head(x)
|
| 70 |
+
|
| 71 |
+
torchaudio.save("enhanced_output.wav", enhanced_wav.cpu(), 24000)
|
| 72 |
+
```
|
| 73 |
+
|
| 74 |
+
## Acknowledgements and Attribution
|
| 75 |
+
* **Architecture:** The original Vocos model was created by [Charactr](https://charactr.com/) (Licensed under MIT).
|
| 76 |
+
* **Training Data:** Trained on the [4TTSDeepfakeDataset](https://huggingface.co/datasets/grotsoV/4TTSDeepfakeData), which is derived from the Hi-Fi TTS Dataset (CC BY 4.0) and utilizes outputs from F5-TTS, MaskGCT, Fish-S1mini, and CosyVoice2.
|