## NOAI2025 Synthetic Speech Detector

### Reference time to solve: 1 hour – 1.5 hours

### I. Problem Description

This task is a deep learning project implemented in PyTorch using a ResNet architecture, aiming to detect synthetic speech from human recordings.

In real life, synthetic speech (i.e., AI-generated speech) has been widely used. Although this technology has made significant progress, it has also raised concerns about potential misuse, such as deepfakes and misinformation. The ability to distinguish synthetic speech from real human speech is crucial for many applications, including content verification, security, and ethical considerations in AI-generated media. The rapid development of generative models has made it increasingly difficult to tell synthetic recordings apart from real human recordings. This project aims to develop a model that can effectively distinguish between these two kinds of audio samples.

> Note: This project is for educational purposes only. The dataset used is publicly available and limited in size, making it suitable for quick training in this competition. The dataset was collected in 2019; compared with the latest models, detecting synthetic speech in this dataset is much easier. Synthetic speech detection in real-world applications is more challenging and requires more complex solutions. The performance of this model may not generalize well to real scenarios.

### II. Dataset

The original data used in this task are raw audio files of human speech and synthetic speech. However, since audio files cannot be used directly for training, they need to be converted into Mel spectrograms (Mel Spectrum) first. The generation process can be summarized as follows:

1. First, frame the audio and apply windowing;
2. Then perform Short-Time Fourier Transform (STFT) to obtain time-frequency domain information. Hint: you can use a large language model to briefly learn about Fourier transform;
3. Next, use a set of Mel filter banks to project the linear spectrum onto the Mel scale (simulating human perception of frequency);
4. Finally, take the logarithm. The unit after taking the logarithm is dB, yielding the final Mel spectrogram.

Specifically, the conversion between Mel frequency $m$ and real frequency $f$ (in Hz) is:

$$
m = 2595 \cdot \lg\left(1 + \frac{f}{700}\right).
$$

For example, suppose we have a real human speech sample with a duration of about 3 seconds and a sampling rate of 16 kHz. We convert it into a Mel spectrogram using the following common parameters:

- Sampling rate: 16000 Hz
- Number of Mel filters: 128
- Hop length: 512
- FFT window length: 1024

Then after conversion into a Mel spectrogram, this audio will produce a tensor of shape:
```
torch.Size([1, 128, 94])
```

where:

- `1` indicates the audio is mono (single channel);
- `128` indicates there are 128 Mel bands along the frequency axis;
- `94` indicates the time axis is divided into 94 frames.

Visually, it is a 2D image, where the horizontal axis is time frames and the vertical axis is Mel frequency.

![Mel Spectrogram](https://dp-public.oss-cn-beijing.aliyuncs.com/community/NOAI2026%20mock%20competition/mel_spectrum.png)

Because this processing is relatively complicated, the dataset provided in this task consists of Mel spectrograms converted from the original audio files, rather than the raw audio. The corresponding spectrograms are saved as tensors in `.pt` format. The training set is stored at [training set link](). Files with `bonafide` in the filename are spectrograms of real human recordings, while the `spoof` folder stores all spectrograms of synthetic speech.

Besides the folders mentioned above, the `dataset` directory also contains a script `spectrogram_dataset.py`. This is a script that helps load spectrograms and provides the `Dataset` interface in PyTorch for training models. It will traverse each subdirectory of the dataset and help assign labels (`bonafide` as 0, `spoof` as 1). Its `__getitem__` magic method returns a dictionary in the form `{ 'spectrogram': Tensor, 'label': Tensor, 'path': str }`, where `spectrogram` is the spectrogram, `label` is the label, and `path` is the path of the spectrogram. In actual use, you need to import it with: `from dataset.spectrogram_dataset import SpectrogramDataset`, and then you can use it normally, e.g., `train_dataset = SpectrogramDataset('data/training_set')`. Specific usage can also be referenced in [baseline.ipynb](). 

> Note: When writing code, do not modify the script above; otherwise, it may cause dataset loading errors.

The validation set and test set cannot be accessed directly by contestants. They can only be accessed via encrypted environment variables. The test set and validation set contain only images and no labels. The access method can be referenced in [baseline.ipynb]().

### III. Task

This project aims to build a model that can distinguish between synthetic (AI-generated) speech and real human recordings. It uses spectrograms generated from audio samples as the input to a ResNet-based neural network. You can load a ResNet18 model pretrained on ImageNet and its weights via `from torchvision.models import resnet18, ResNet18_Weights`. You may also choose other versions of ResNet or other vision models provided by torchvision. The specific method can be referenced in [baseline.ipynb](). 

Your task is to fine-tune on this basis or modify the model structure (add or remove parts) and train on the training set, in order to achieve good detection performance for synthetic speech.

**Hint: If you choose a vision model larger than ResNet18, you need to control the number of training epochs; otherwise, training may not finish within the specified time. You can also treat this task purely as a computer vision task and solve it using a CNN model you implement yourself. Do not get stuck on the physical implementation details of the Mel Spectrum, as it helps little for solving this task.**

### IV. Submission

Contestants must submit a notebook file named `submission.ipynb`. In the file, you may submit only the trained model and omit the model training process, so that results can be obtained quickly. The notebook should be able to output a zip file containing prediction results, where the zip contains two files:

- `submissionA.csv`: the predicted labels of the model on the validation set, one 0 or 1 per line, no header;
- `submissionB.csv`: the predicted labels of the model on the test set, one 0 or 1 per line, no header.

The system will read `submission.zip` and compute the A-leaderboard and B-leaderboard scores based on the predictions and the ground-truth labels. The A-leaderboard score is the validation score, mainly helping contestants debug code and is visible during the competition; the B-leaderboard score is the test score and is used as the basis for ranking. The submitted files must strictly follow the above format and naming, otherwise the system will not be able to read them correctly. The code for the submission process can also be referenced in [baseline.ipynb](https://www.bohrium.com/en/notebooks/87645469178).

### V. Scoring

The scoring rule is to compare the submitted CSV files with the correct answers in `ground_truth_labels.csv`.

The specific metric is F1-score, which measures the overall performance of the model on a binary classification task. It is defined as:
$$
\text{F1-score} = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}
$$
where Precision is the proportion of samples predicted as positive that are actually positive, and Recall is the proportion of actually positive samples that are correctly predicted as positive by the model. The final score will be between $0$ and $1$; the closer to $1$, the better the model performance.