Skip to content

add a fallback for more f16 intrinsics #159175

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rust-bors merged 2 commits into from
Jul 18, 2026
+76 −14
Merged

add a fallback for more f16 intrinsics #159175

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7ce9176
add a fallback for more `f16` intrinsics
folkertdev Jul 12, 2026
da925bb
add float intrinsic with fallbacks to `replaced_intrinsics`
folkertdev Jul 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 26 additions & 2 deletions compiler/rustc_codegen_llvm/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -324,9 +324,33 @@ impl CodegenBackend for LlvmCodegenBackend {
target_config(sess)
}

/// Intrinsics whose fallback body will not be used by the LLVM backend.
fn replaced_intrinsics(&self) -> Vec<Symbol> {
let mut will_not_use_fallback =
vec![sym::unchecked_funnel_shl, sym::unchecked_funnel_shr, sym::carrying_mul_add];
#[rustfmt::skip]
let mut will_not_use_fallback = vec![
// These are mapped to LLVM intrinsics instead.
sym::unchecked_funnel_shl,
sym::unchecked_funnel_shr,
sym::carrying_mul_add,

// Fallback via libm, but the LLVM intrinsic is used instead.
sym::sinf16, sym::sinf32, sym::sinf64,
sym::cosf16, sym::cosf32, sym::cosf64,
sym::powf16, sym::powf32, sym::powf64,
sym::expf16, sym::expf32, sym::expf64,
sym::exp2f16, sym::exp2f32, sym::exp2f64,
sym::logf16, sym::logf32, sym::logf64,
sym::log10f16, sym::log10f32, sym::log10f64,
sym::log2f16, sym::log2f32, sym::log2f64,

// Fallback via f32 or f64, but the LLVM intrinsic is used instead.
sym::floorf16, sym::ceilf16, sym::truncf16,
sym::round_ties_even_f16, sym::roundf16,
Comment thread
bjorn3 marked this conversation as resolved.
sym::sqrtf16, sym::powif16,
sym::fmaf16,

sym::copysignf16, sym::copysignf32, sym::copysignf64, sym::copysignf128,
];

if llvm_util::get_version() >= (22, 0, 0) {
will_not_use_fallback.push(sym::carryless_mul);
Expand Down
62 changes: 50 additions & 12 deletions library/core/src/intrinsics/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1026,9 +1026,12 @@ pub unsafe fn unaligned_volatile_store<T>(dst: *mut T, val: T);
///
/// The stabilized version of this intrinsic is
/// [`f16::sqrt`](../../std/primitive.f16.html#method.sqrt)
#[inline]
#[rustc_intrinsic]
#[rustc_nounwind]
pub fn sqrtf16(x: f16) -> f16;
pub fn sqrtf16(x: f16) -> f16 {
sqrtf32(x as f32) as f16
}
/// Returns the square root of an `f32`
///
/// The stabilized version of this intrinsic is
Expand All @@ -1055,9 +1058,12 @@ pub fn sqrtf128(x: f128) -> f128;
///
/// The stabilized version of this intrinsic is
/// [`f16::powi`](../../std/primitive.f16.html#method.powi)
#[inline]
#[rustc_intrinsic]
#[rustc_nounwind]
pub fn powif16(a: f16, x: i32) -> f16;
pub fn powif16(a: f16, x: i32) -> f16 {
powif32(a as f32, x) as f16
}
/// Raises an `f32` to an integer power.
///
/// The stabilized version of this intrinsic is
Expand Down Expand Up @@ -1437,9 +1443,14 @@ pub fn log2f128(x: f128) -> f128 {
/// The stabilized version of this intrinsic is
/// [`f16::mul_add`](../../std/primitive.f16.html#method.mul_add)
#[rustc_intrinsic_const_stable_indirect]
#[inline]
#[rustc_intrinsic]
#[rustc_nounwind]
pub const fn fmaf16(a: f16, b: f16, c: f16) -> f16;
pub const fn fmaf16(a: f16, b: f16, c: f16) -> f16 {
// NOTE: f32 does not have sufficient precision, so use f64 instead.
// see also https://github.com/llvm/llvm-project/issues/128450#issuecomment-2727540179.
fmaf64(a as f64, b as f64, c as f64) as f16

@tgross35 tgross35 Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a comment that f32 isn't enough precision so we need f64? The difference isn't obvious if you don't know the context.

@RalfJung RalfJung Jul 17, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you verified that f64 gives enough precision? I am not aware of a theorem for this case.

@folkertdev folkertdev Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dug it up, by @beetrees of course llvm/llvm-project#128450 (comment).

@RalfJung RalfJung Jul 17, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I can't really follow but good to know they looked into it. :D

This might be something one can just exhaustively test though?

@tgross35 tgross35 Jul 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be something one can just exhaustively test though?

I did a quick estimate by changing https://github.com/rust-lang/compiler-builtins/blob/5af7a65c0342d3f5ac1d250c7902aaabbfc2bd49/libm/src/math/fmaf16.rs to:

extern crate std;
(x as f64).mul_add(y as f64, z as f64) as f16

And running LIBM_EXTENSIVE_TESTS=fmaf16 LIBM_EXTENSIVE_ITERATIONS=max cargo test --features build-mpfr --test z_extensive --profile release-checked --no-default-features --features unstable-float. On my 11 core laptop it estimates half a year to complete :D. Maybe on something like a threadripper that's more feasible, or against hardware fma if available.

@RalfJung RalfJung Jul 18, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm fair, 48 bits is too big. Oh well.

@RalfJung RalfJung Jul 18, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't llvm/llvm-project#131531 say that f64 is not enough?

@folkertdev folkertdev Jul 18, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is for bf16 (bfloat in LLVM)

@RalfJung RalfJung Jul 18, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I'm mixing up the various 16-bit floats....

}
/// Returns `a * b + c` without rounding the intermediate result for `f32` values.
///
/// The stabilized version of this intrinsic is
Expand Down Expand Up @@ -1535,9 +1546,12 @@ pub const fn fmuladdf128(a: f128, b: f128, c: f128) -> f128 {
/// The stabilized version of this intrinsic is
/// [`f16::floor`](../../std/primitive.f16.html#method.floor)
#[rustc_intrinsic_const_stable_indirect]
#[inline]
#[rustc_intrinsic]
#[rustc_nounwind]
pub const fn floorf16(x: f16) -> f16;
pub const fn floorf16(x: f16) -> f16 {
floorf32(x as f32) as f16
}
/// Returns the largest integer less than or equal to an `f32`.
///
/// The stabilized version of this intrinsic is
Expand Down Expand Up @@ -1568,9 +1582,12 @@ pub const fn floorf128(x: f128) -> f128;
/// The stabilized version of this intrinsic is
/// [`f16::ceil`](../../std/primitive.f16.html#method.ceil)
#[rustc_intrinsic_const_stable_indirect]
#[inline]
#[rustc_intrinsic]
#[rustc_nounwind]
pub const fn ceilf16(x: f16) -> f16;
pub const fn ceilf16(x: f16) -> f16 {
ceilf32(x as f32) as f16
}
/// Returns the smallest integer greater than or equal to an `f32`.
///
/// The stabilized version of this intrinsic is
Expand Down Expand Up @@ -1601,9 +1618,12 @@ pub const fn ceilf128(x: f128) -> f128;
/// The stabilized version of this intrinsic is
/// [`f16::trunc`](../../std/primitive.f16.html#method.trunc)
#[rustc_intrinsic_const_stable_indirect]
#[inline]
#[rustc_intrinsic]
#[rustc_nounwind]
pub const fn truncf16(x: f16) -> f16;
pub const fn truncf16(x: f16) -> f16 {
truncf32(x as f32) as f16
}
/// Returns the integer part of an `f32`.
///
/// The stabilized version of this intrinsic is
Expand Down Expand Up @@ -1635,9 +1655,12 @@ pub const fn truncf128(x: f128) -> f128;
/// The stabilized version of this intrinsic is
/// [`f16::round_ties_even`](../../std/primitive.f16.html#method.round_ties_even)
#[rustc_intrinsic_const_stable_indirect]
#[inline]
#[rustc_intrinsic]
#[rustc_nounwind]
pub const fn round_ties_even_f16(x: f16) -> f16;
pub const fn round_ties_even_f16(x: f16) -> f16 {
round_ties_even_f32(x as f32) as f16
}

/// Returns the nearest integer to an `f32`. Rounds half-way cases to the number with an even
/// least significant digit.
Expand Down Expand Up @@ -1674,9 +1697,12 @@ pub const fn round_ties_even_f128(x: f128) -> f128;
/// The stabilized version of this intrinsic is
/// [`f16::round`](../../std/primitive.f16.html#method.round)
#[rustc_intrinsic_const_stable_indirect]
#[inline]
#[rustc_intrinsic]
#[rustc_nounwind]
pub const fn roundf16(x: f16) -> f16;
pub const fn roundf16(x: f16) -> f16 {
roundf32(x as f32) as f16
}
/// Returns the nearest integer to an `f32`. Rounds half-way cases away from zero.
///
/// The stabilized version of this intrinsic is
Expand Down Expand Up @@ -3610,34 +3636,46 @@ pub const fn fabs<T: bounds::FloatPrimitive>(x: T) -> T;
///
/// The stabilized version of this intrinsic is
/// [`f16::copysign`](../../std/primitive.f16.html#method.copysign)
#[inline]
#[rustc_nounwind]
#[rustc_intrinsic]
pub const fn copysignf16(x: f16, y: f16) -> f16;
pub const fn copysignf16(x: f16, y: f16) -> f16 {
f16::from_bits((x.to_bits() & !f16::SIGN_MASK) | (y.to_bits() & f16::SIGN_MASK))
}

/// Copies the sign from `y` to `x` for `f32` values.
///
/// The stabilized version of this intrinsic is
/// [`f32::copysign`](../../std/primitive.f32.html#method.copysign)
#[inline]
#[rustc_nounwind]
#[rustc_intrinsic_const_stable_indirect]
#[rustc_intrinsic]
pub const fn copysignf32(x: f32, y: f32) -> f32;
pub const fn copysignf32(x: f32, y: f32) -> f32 {
f32::from_bits((x.to_bits() & !f32::SIGN_MASK) | (y.to_bits() & f32::SIGN_MASK))
}

@folkertdev folkertdev Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RalfJung should I remove the custom const-eval implementation (and add miri::intrinsic_fallback_is_spec) when adding a const fallback? const-eval already duplicates minimumf*, and I think it makes sense to split out sym::fabs into a variant for each float too (all backends need to retrieve the width anyway, if you look at its usage).

@RalfJung RalfJung Jul 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fabs was merged a few months ago by @N1ark and I think she planned to merge the others as well. Just from the perspective of not having to duplicate the signatures and doc comments four times in this file, it'd be nice to merge more intrinsics... but either way please coordinate with her. I am very much not a fan of how the float intrinsics are turning this file into a mess where every change has to be done 4 times, so I think merging the different type variants is quite important.

Merging makes the fallback bodies more tricky of course. For these bitwise operations maybe a simple trait is enough to implement them generically?

Minimum has non-determinism and other complications, so that's not duplicated in const-eval, it's a genuinely different implementation. For these here... not sure, they seem sufficiently primitive to me that I think it's worth keeping the existing const-eval implementations.

@N1ark N1ark Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes sorry i started doing it in #153934 and never got around to finishing it 🫠 ill pick it back up next week, @folkertdev once im done maybe rebasing this PR on that will make your life easier?

@folkertdev folkertdev Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really, combining the intrinsics means it's impossible to add fallbacks for just f16 (by deferring to f32). Adding those fallbacks removes nasty logic from the backends, and if the price is a to copy-paste a comment, that seems acceptable to me? Especially for simple intrinsics without complex safety/soundness requirements.

For abs and copysign we can use some trait machinery to provide a fallback for all four float types, but for functions like powf or sqrt that won't work.

@RalfJung RalfJung Jul 17, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have edited the various minimum intrinsics' docs quite a bit, so those certainly do not qualify as "simple". So I feel very strongly that we should deduplicate these comments -- but we can do that without merging the functions, by writings the docs once on the f32 version and making all the others a link.

@folkertdev folkertdev Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, fair. So then I think next steps are

  • combine the docs of some of the trickier functions
  • write a generic fallback for e.g. abs and copysign where we can reasonably provide a fallback for all 4 float types.
  • simplify the backends because they won't have to handle many libcall cases any more (I made a start with that in use core fallbacks for more f16/f128 operations rustc_codegen_gcc#910), with this PR we can do more, and cranelift can also remove some logic. It's probably best to do this in their respective repos for improved CI coverage.

@N1ark are you interested in any of that? I'll be doing some of the cranelift stuff to remove another .patch there but otherwise I'm happy to leave it to you.

@RalfJung RalfJung Jul 18, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To add a bit of context -- the reason why merging the intrinsics is nice is that every single tool that consumes MIR (such as the verification tool @N1ark is working on) needs to deal with all those intrinsics, and they can typically deal with them uniformly across all types, so merging them avoids a bunch of code duplication.

Now, when in doubt we'll prioritize in-tree MIR consumers over out-of-tree MIR consumers. But it is a shame that we need to make a choice either way at all. So I wonder if we can somehow have our cake and eat it too. Is the problem with sqrt and pow and the others that they have a fallback for some types but not for others? Or could we have a fallback for them always, and what's annoying is just dispatching on the type? The latter could be solved...

/// Copies the sign from `y` to `x` for `f64` values.
///
/// The stabilized version of this intrinsic is
/// [`f64::copysign`](../../std/primitive.f64.html#method.copysign)
#[inline]
#[rustc_nounwind]
#[rustc_intrinsic_const_stable_indirect]
#[rustc_intrinsic]
pub const fn copysignf64(x: f64, y: f64) -> f64;
pub const fn copysignf64(x: f64, y: f64) -> f64 {
f64::from_bits((x.to_bits() & !f64::SIGN_MASK) | (y.to_bits() & f64::SIGN_MASK))
}

/// Copies the sign from `y` to `x` for `f128` values.
///
/// The stabilized version of this intrinsic is
/// [`f128::copysign`](../../std/primitive.f128.html#method.copysign)
#[inline]
#[rustc_nounwind]
#[rustc_intrinsic]
pub const fn copysignf128(x: f128, y: f128) -> f128;
pub const fn copysignf128(x: f128, y: f128) -> f128 {
f128::from_bits((x.to_bits() & !f128::SIGN_MASK) | (y.to_bits() & f128::SIGN_MASK))
}

/// Generates the LLVM body for the automatic differentiation of `f` using Enzyme,
/// with `df` as the derivative function and `args` as its arguments.
Expand Down
Loading