Add regression tests for evaluating init-time eval datasets after training with precomputed reference log-probs

[albertvillanova]

This PR adds regression tests to DPO and KTO verifying that an eval_dataset set at initialization keeps producing correct evaluation metrics after training when precompute_ref_log_probs=True.

Motivation

Follow-up to the eval-time reference log-prob guard added for DPO and KTO:

• Raise when DPO/KTO cannot precompute correct reference log-probs at eval time #6443

That guard raises when a new dataset is passed to evaluate() after training in full fine-tuning setups, where the reference model is no longer available. A concern was raised that the guard might also break evaluation of datasets set at initialization. It does not: those datasets have their reference log-probs precomputed once, against the untrained reference, and stored on the dataset, so evaluation reuses the stored values instead of recomputing them against the trained policy. These tests lock in that behavior so it cannot silently regress.

Solution

Add a parametrized test to each trainer that sets an eval_dataset at initialization, covering a plain Dataset, a DatasetDict, and a dict of datasets, then trains and calls evaluate() with no arguments, asserting it returns per-split metrics without raising.

Changes

• Add a parametrized regression test to the DPO trainer test suite
• Add the same parametrized regression test to the KTO trainer test suite, kept consistent with the DPO version

---

Note

Low Risk
Test-only changes with no production code; lowest risk for runtime behavior.

Overview
Adds parametrized regression tests to test_dpo_trainer.py and test_kto_trainer.py so init-time eval_dataset still works after training when precompute_ref_log_probs=True.

Each test builds a trainer with eval_dataset at construction (plain Dataset, DatasetDict, or dict of datasets), runs a short train step, then calls evaluate() with no dataset argument. It asserts eval loss metrics are returned and nothing raises—complementing the existing test that does raise when a new dataset is passed to evaluate() after training in full fine-tuning without a separate reference model.

No trainer or library code changes; test-only follow-up to the eval-time reference log-prob guard (#6443).

^{Reviewed by Cursor Bugbot for commit bf962ae. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.