Teams that want one small model to do the work of several specialists face a familiar trade-off between two methods, and a new preprint tries to get the best of both. It matters to anyone folding specialist models into a single cheaper one, a common need once a company has several fine-tuned models and wants to serve them as one.
The two existing approaches pull in opposite directions. Multi-teacher on-policy distillation trains a student on outputs it generates itself, supervised by several expert models, which is flexible but slow. Task arithmetic simply adds model parameters together in fixed proportions with no training at all, which is fast but capped: whatever mixing weights you pick, the result can only express what the frozen student and experts already contain together. One buys quality with time, the other buys speed with a ceiling.
The paper, titled P²-MOPD, treats the weighted blend of the student's and experts' parameters as a single network that can itself be trained. It then learns both the mixing weights and the student's own weights under the same on-policy distillation objective. The key difference from task arithmetic is that the student keeps training, so the expert weights act as a starting guide, a parameter prior, rather than a hard boundary on what the model can learn. In effect it borrows the head start of model merging without inheriting its ceiling.
The reported numbers are encouraging. Across four domains, with models of 1.7 billion and 4 billion parameters, the method recovered expert performance faster and more evenly than standard multi-teacher distillation, beating that method's final average recovery in half the training steps. At the end of training the 1.7 billion parameter student reached 94.7% average recovery against 80.9% for the standard approach. If those hold up, half the steps for a higher final score is a real saving for teams running distillation repeatedly.
The caveats are the usual ones for early research, and they are worth stating. This is an arXiv preprint that has not been peer reviewed, so the claims rest on the authors' own runs. The tests stop at 4 billion parameters, which says little about whether the gains survive at the tens- or hundreds-of-billions scale where most production models sit, and the paper reports an average recovery figure for the 1.7 billion model but not the equivalent for 4 billion or a per-domain breakdown. It is a sensible idea with promising early results, best treated as a method to watch rather than a settled technique.