Force Qwen not to use Chinese

Force Qwen not to use Chinese

As you might imagine, this post is closely related to my previous post: Force a language model not to use the letter ‘e’. In that post, we forbade the language model from generating any token containing the letter ‘e’. While this may have some literary interest perhaps it does not hold much practical value. We now want to forbid the language model from generating any of the tokens that contain Chinese characters. This has some practical interest especially in the case of Qwen models where the language models tend to generate Chinese characters when it’s not desired.

Similar to my previous post, we first want to find all the tokens containing Chinese characters and then forbid them by using a logits processor.

We start by downloading a model and its tokenizer.

import unicodedata
from typing import List

import torch
from IPython.display import Markdown
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation import LogitsProcessor

# Auto select device (CUDA > MPS > CPU)
if torch.cuda.is_available():
    device = torch.device("cuda")
elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
    device = torch.device("mps")
else:
    device = torch.device("cpu")

model_id = "Qwen/Qwen3-0.6B"
model = AutoModelForCausalLM.from_pretrained(
    model_id, cache_dir="/big_storage/llms/hf_models/"
).to(device)
tokenizer = AutoTokenizer.from_pretrained(
    model_id, cache_dir="/big_storage/llms/hf_models/"
)

We then create a basic generate response function and check it is working.

def generate_response(user_input, logits_processor=[], enable_thinking=False):
    messages = [{"role": "user", "content": user_input}]
    inputs = tokenizer.apply_chat_template(
        messages,
        add_generation_prompt=True,
        enable_thinking=enable_thinking,
        return_tensors="pt",
        return_dict=True,
    ).to(model.device)
    prompt_length = inputs["input_ids"].shape[-1]
    generated = model.generate(
        **inputs, temperature=0.1, logits_processor=logits_processor, 
        max_new_tokens=1000
    )
    return tokenizer.decode(generated[0][prompt_length:-1])

generate_response("Hi!")

'Hello! How can I assist you today?'

We now create a function to check if a single character is Chinese and check that it’s working.

import unicodedata

def is_chinese_char(char):
    """Check if a single character is Chinese"""
    return "CJK" in unicodedata.name(char, "")

is_chinese_char("国"), is_chinese_char("E")

(True, False)

We create a function to check if a given token contains any Chinese characters and check that it’s working.

def has_chinese_chars(token):
    """Check if token contains any Chinese characters"""
    return any(is_chinese_char(char) for char in token)

has_chinese_chars("中国"), has_chinese_chars("Country 国"), has_chinese_chars("China")

(True, True, False)

The list of tokens containing Chinese characters is given by:

tokens_containing_chinese_characters = [
    token_id
    for token_id in range(tokenizer.vocab_size)
    if has_chinese_chars(tokenizer.decode(token_id))
]

There are 26,071 tokens containing Chinese characters which corresponds to 17.2% of all the tokens.

We probably also want to remove replacement characters in unicode which are used to represent unknown, unrecognized, or invalid characters. We follow the same procedure.

def is_replacement_char(char):
    """Check if a single character is a replacement character"""
    return 'REPLACEMENT CHARACTER' == unicodedata.name(char, "")

def has_replacement_chars(token):
    return any(is_replacement_char(char) for char in token)

tokens_containing_replacement_characters = [
    token_id
    for token_id in range(tokenizer.vocab_size)
    if has_replacement_chars(tokenizer.decode(token_id))
]

There are 1,457 tokens containing replacement characters which corresponds to 1% of all the tokens.

We are ready to create our logits processor. The logits processor will receive a list of forbidden tokens and sets their raw scores to negative infinity.

class ForbidTokensLogitsProcessor(LogitsProcessor):
    def __init__(self, forbidden_tokens: List[int]):
        self.forbidden_tokens = forbidden_tokens

    def __call__(
        self, input_ids: torch.LongTensor, scores: torch.FloatTensor
    ) -> torch.FloatTensor:
        scores[:, self.forbidden_tokens] = -torch.inf

        return scores

We instantiate the logits processor with the forbidden tokens we found previously.

logits_processors = [
    ForbidTokensLogitsProcessor(
        forbidden_tokens=tokens_containing_chinese_characters
        + tokens_containing_replacement_characters,
    )
]

To understand the impact of the logits processor, let’s begin by running the model without it as a baseline.

user_input = "Translate to Chinese the following text:\n I love China"
generate_response(user_input)

'我爱中国'

Now, let’s use our logits processor and see if it’s working.

generate_response(user_input, logits_processor=logits_processors)

'I love China.'

Great, the model is unable to generate Chinese characters.

Next, let’s run a second test with reasoning traces enabled.

user_input = "Give me an answer in Chinese to the following question:\n What's 2+2?"
print(generate_response(user_input, enable_thinking=True))

<think>
嗯，用户问“2+2?”，我需要先确认这个问题。2加2在中文里通常会被直接回答，但可能用户有其他意图。首先，中文里加法运算通常用汉字“加”来表示，所以答案应该是“四”。不过，用户可能希望得到更详细的解释，或者有其他隐藏的需求。比如，用户可能是在学习数学，或者有其他问题需要回答。也有可能用户只是想确认答案，不需要额外信息。我需要确保回答简洁明了，同时确认正确性。另外，用户可能希望答案用中文呈现，所以直接给出“四”即可。不需要添加其他内容，保持回答简洁。
</think>

2+2=4

Let’s now use our logits processor.

print(generate_response(user_input, enable_thinking=True, logits_processor=logits_processors))

<think>
Okay, the user is asking, "What's 2+2?" and wants an answer in Chinese. Let me think. The question is straightforward, right? It's a simple math problem. In Chinese, the answer would be "4". But wait, maybe there's more to it? Like, could there be a trick or something? No, 2+2 is just 4. So the answer should be 4. I need to make sure there's no hidden meaning or cultural context that I'm missing. But since it's a basic math question, the answer is clear. Just confirm that the user is asking for the sum, not a different question. Alright, the answer is 4.
</think>

2+2=4

Perfect!