[Question]: can TriMix be used in large models, such as deepseek, qwen3-235b?

[xwpaul3]

Describe the issue

I tried to use Trimix on qwen-235b, with different params such as starting layer=48, but still get an unrecognizable output, i.e messy code

[hzy46]

Hi xwpaul3. Thanks for the interest. Currently we haven't tested trimix on qwen-235b. I notice it has 94 layers and you may start with less aggressive settings such as starting_layer=64/72/80. I will carry out a gradient analysis on this model.

[xwpaul3]

Hi xwpaul3. Thanks for the interest. Currently we haven't tested trimix on qwen-235b. I notice it has 94 layers and you may start with less aggressive settings such as starting_layer=64/72/80. I will carry out a gradient analysis on this model.

I have tried different params, it does not work. However, use smaller starting_layer such as 20 looks better.

[xwpaul3]

As far as I concerned, trimix may be not adaptive to large models.