[Tunix] Enforce strict formatting rewards and stop instruction in GSM8K demo - #2116
Open
andytwigg wants to merge 1 commit into
Open
[Tunix] Enforce strict formatting rewards and stop instruction in GSM8K demo#2116andytwigg wants to merge 1 commit into
andytwigg wants to merge 1 commit into
Conversation
…8K demo - Update GSM8K_PROMPT_TEMPLATE with an explicit instruction to immediately end the response and stop generating after the closing </answer> tag. - Update score_gsm8k_completion to require strict formatting: remove partial credit (0.5) when not format_correct and answer_correct, awarding 0.0 to avoid incentivizing rambling completions. - Update gsm8k_test.py to verify prompt stop instruction and strict formatting reward behavior.
andytwigg
requested review from
abheesht17,
hgao327,
jiangyangmu,
lc5211,
s-noghabi,
sizhit2,
tianshub and
wang2yn84
as code owners
September 4, 2026 01:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Improves trajectory completion and eliminates partial reward incentives for rambling generations past the numerical answer in the distributed GSM8K GRPO demo.
Details
Updated
GSM8K_PROMPT_TEMPLATEto explicitly instruct the model:"After the closing </answer> tag, immediately end your response and stop generating."Updated
score_gsm8k_completionto assign0.0(instead of0.5) whennot format_correct and answer_correct. Without strict formatting enforcement, the model receives partial credit even when generating run-on tokens or failing to properly terminate at</answer>.Updated
tests/experimental/examples/math_gsm8k_dist/gsm8k_test.pyto verify the stop instruction inbuild_promptand assert the strict formatting reward behavior (0.0for malformed outputs).Verification
python3 -m unittest tests/experimental/examples/math_gsm8k_dist/gsm8k_test.py: all 9 unit tests passed.