# NOAI 2026: User Intent Recognition in Zhihu Scenarios

Note: This problem is evaluated in the `noai:2026v1.1` image environment, which does not include the Qwen3-4B-Instruct model. Participants may use the `noai:2026v1` image (which includes this model) during development, but submitted Notebooks must not depend on Qwen3-4B-Instruct at runtime; otherwise, execution will fail in the evaluation environment.



## 1. Task Description

Intent Recognition is a core task in AI search products. Its goal is to determine the user's intent from their input and route the query to the appropriate downstream module, such as local services, health Q&A, or novel search.

This problem is set against the background of a real AI search product scenario, which includes two types of user input: single-turn direct search and multi-turn human-machine dialogue. Participants must classify each input text into one of the following 16 predefined intent categories.

### 1.1 Intent Categories

| Intent           | Description                                                  |
| ---------------- | ------------------------------------------------------------ |
| 游戏技巧         | Queries about game walkthroughs, cheats, operational techniques, or methods to clear levels |
| 游戏角色信息     | Queries about character builds, skins, acquisition methods, personality settings, or secondary creations |
| 周边饭馆         | Queries about restaurants near attractions, communities, residences, or other locations |
| 查找高端酒店     | Queries about four-star, five-star, or highly rated hotels   |
| 查找地址         | Queries about the specific address of restaurants, hotels, parks, schools, or other locations |
| 健康知识         | Queries about healthcare, wellness, and healthy lifestyle-related knowledge |
| 查找医疗信息     | Queries about hospitals, doctors, diseases, or symptoms      |
| 美容化妆技巧     | Queries about beauty, cosmetics, aesthetic medicine, or plastic surgery knowledge and techniques |
| 美食烹饪技巧     | Queries about recipes, cooking methods, steps, or ingredient handling |
| 软件开发问题     | Consulting on programming languages, development frameworks, program performance, or optimization |
| 软件使用问题     | Consulting on software installation, configuration, or usage problems |
| 查找小说剧情     | Queries about novel content, plot points, or story developments |
| 查找小说角色信息 | Queries about character information in novels, including background, personality, and notable events |
| 查找营业时间     | Queries about operating hours of banks, shops, hotels, restaurants, or other service venues |
| 法律法条解释     | Queries about the meaning and interpretation of laws, regulations, or policy provisions |
| 法律问题咨询     | Legal consultation for specific issues or cases              |

### 1.2 Input Data Format

The `input` field of each sample may take one of the following two forms:

1. **Single-turn user input**: A single query text entered directly by the user, for example:

   ```
   越南春卷的做法
   ```

2. **Multi-turn human-machine dialogue**: A sequence of messages in chronological order, each prefixed with `usr:` (user message) or `sys:` (system reply), for example:

   ```
   usr: 您好，我想找一家评分 4.5 分以上的餐馆
   sys: 为您推荐 XX 餐厅
   usr: 这家店周边有什么酒店吗？
   ```

**For multi-turn dialogues, the intent label is determined by the intent expressed in the last `usr:` message; earlier dialogue history serves only as contextual reference.** The [baseline](https://www.bohrium.com/notebooks/44754271216) code provides preprocessing functions such as `extract_user_utterance` and `preprocess_input_text` as references; participants may also design their own text preprocessing methods.

---

## 2. Dataset

| Split                        | # Samples | Description                                                  |
| ---------------------------- | --------- | ------------------------------------------------------------ |
| Training Set (Train)         | 16        | Contains 16 labeled samples covering all 16 intent categories, with only 1 example per category. |
| Leaderboard A Validation Set | 2,500     | Unlabeled                                                    |
| Leaderboard B Test Set       | 2,500     | Unlabeled                                                    |

Each training sample contains two fields: `input` (input text) and `label` (intent label); the test set contains only the `input` field.

Training set sample example:

```json
{"input": "越南春卷的做法", "label": "美食烹饪技巧"}
```

During the development phase, participants can only directly access the training set. The validation set and test set are only available in the official evaluation environment; participants must read their storage paths via specified environment variables. Please refer to the [baseline](https://www.bohrium.com/notebooks/44754271216) Notebook for details.



## 3. Models Pre-installed in the Image and Training Set

### 3.1 LLM Models Pre-installed in the Image

The `noai:2026v1` image includes the Qwen3-4B-Instruct model, which participants may call during the development phase. For usage details, refer to the [baseline](https://www.bohrium.com/notebooks/44754271216) Notebook.

**Note**: Qwen3-4B-Instruct is only available in the development node and cannot be called in the evaluation environment. Participants may use this model to process training samples, category definitions, or self-constructed texts, and save the static artifacts generated during development to an additional dataset. The submitted Notebook must not call or depend on this model at runtime. Please refer to the *NOAI 2026 Bohrium User Guide* for instructions.

### 3.2 bert-base-chinese Model Pre-installed in the Training Set

A pre-trained bert-base-chinese model is provided in the training set and can be fine-tuned. Since the evaluation machine has access to the training set, the pre-trained bert-base-chinese model in the training set is accessible during both the training and prediction phases after submission. For usage details, refer to the [baseline](https://www.bohrium.com/notebooks/44754271216) Notebook.



## 4. Task

Train an intent recognition system to classify the intent of each input in the test set.



## 5. Submission

Participants must submit a Notebook named `submission.ipynb`. At most one additional `dataset` may be attached. The `dataset` must not exceed 5 MB.

### 5.1 Input and Output

- **Input**: Access and retrieval details for the training set, validation set, and test set are provided in the [baseline](https://www.bohrium.com/notebooks/44754271216) code.
- **Output**: Upon completion of the Notebook run, a `submission.zip` file must be generated in the current directory, containing:
  - `submission_val.jsonl` — validation set prediction results
  - `submission_test.jsonl` — test set prediction results

### 5.2 File Structure

Please refer to the complete file structure in the baseline Notebook.

### 5.3 JSONL Format Generated by the Notebook

Each file contains one JSON object per line, **which must be strictly aligned with the corresponding test set sample order**:

```json
{"label": "美食烹饪技巧"}
```

---

## 6. Scoring

### 6.1 Metric: Weighted F1 Score

For each category $i$, the F1 score is calculated from precision and recall:

$$F1(i) = \frac{2 \times \text{Precision}(i) \times \text{Recall}(i)}{\text{Precision}(i) + \text{Recall}(i)}$$

Assuming there are $N$ categories in total, the number of samples in category $i$ is $w_i$, and the total number of samples is $W = \sum_{i=1}^{N} w_i$:

$$\text{Weighted F1} = \sum_{i=1}^{N} \left( \frac{w_i}{W} \times F1(i) \right)$$

### 6.2 Public Leaderboard and Private Leaderboard

- **Public Leaderboard (A)**: Calculated based on the Validation Set;
- **Private Leaderboard (B)**: Calculated based on the Test Set, and published after the competition ends.



## 7. Constraints

- The evaluation machine for this problem uses the `noai:2026v1.1` image, which does not include the Qwen3-4B-Instruct model. During problem-solving, the `noai:2026v1` image may be used, which pre-includes Qwen3-4B-Instruct; therefore, if the submitted Notebook retains code that calls Qwen3-4B-Instruct, it will definitely produce an error.
- Using external large language model APIs beyond this problem (such as GPT or Claude) for prediction, feature generation, data annotation, or model ensembling is not allowed;
- The evaluation environment does not provide internet access; participant programs must not perform any network operations, nor install additional dependencies via `pip install`. Participants may only use packages pre-installed in the specified image;
- This problem uses a **GPU** for training and evaluation; the total time for training + inference must not exceed 25 minutes.



## 8. Baseline Score and Reference Score

- **Leaderboard B Baseline Score ([baseline](https://www.bohrium.com/notebooks/44754271216))**: 0.1133
- **Scientific Committee Reference Solution B Leaderboard Score (Reference Result)**: 0.8154



## 9. Acknowledgements

- Thanks to the technical experts at Zhihu for providing this problem.