FP16 support

[SadiinsoSnowfall]

EVE FP16 support proposal

Introduction

We propose to implement support for the IEEE-754 half-precision floating-point format (aka FP16) in the EVE library in three steps.

There is no standard fp16 type in C++ as of yet, the only way to represent it is to either use platform-specific intrinsics types, such as _Float16 on x86 and float16_t on ARM, or use a custom 16-bit type on which specialized operations are implemented.

To unify the representation of fp16 across architectures and compilers, we propose to implement it as a struct guarenteed to be exactly 16 bits wide.

struct fp16_t {
    using arch_type = /* arch dependant type (_Float16, float16_t, etc...) */

    arch_type storage;

    operator arch_type () const noexcept { .... }
};

static_assert(sizeof(fp16_t) == 2 && alignof(fp16_t) == 2);

This will allow us to implement the necessary operators on this type to allow interop with the platform-specific fp16 types.

Step 1. Transparent types

This step consists of making eve::wide support transparent types (see definitions below). The goal of this step is to prepare for the end goal: being able to express eve::wide<fp16_t>.

template <>
struct transparent_trait<fp16_t> {
    using type = uint16_t;
};

This allows us to instantiate objects of type eve::wide<fp16_t>, and to use many of the eve::wide methods on them, such as get, set, operator==, etc.

As as side-effect, this will also effectively resolve the issue #1573.

Step 2. Internal function rework

This step consists of reworking internal functions such as as_register (as layed-out in issue #1844) to allow for users to extends the list of supported types, for example for transparent types:

// example for x86
template <typename Size>
constexpr auto as_register(as<wide<fp16_t>>, Size, x86_abi) {
    constexpr auto width = sizeof(Type) * Size::value;

    if constexpr (current_api >= avx512fp16) {
        if      constexpr (size <= 16) return __m128h{}
        else if constexpr (size == 32) return __m256h{}
        else if constexpr (size == 64) return __m512h{}
    } else {
        if      constexpr (size <= 16) return __m128i{}
    }
}

Other impacted functions may include make, combine, slice.
Note that reworking these function may also reduce the overwall compile time and make it easier to extend them later for new types.

Step 3. FP16 Support in EVE

Finally, once the groundwork is laid, we can then add the fp16 type to EVE, implement the necessary operators for interop with platform-specific fp16 types and implement the necessary intrinsics in the backends.

Open questions & research

FP16 codegen

Consider the following C++ code targeting a F16C capable x86 CPU:

#include <immintrin.h>

_Float16 test(_Float16 a, _Float16 b) {
    return a + b;
}

The codegen when compiling using Clang 18 is the following:

vpextrw     eax, xmm0, 0
vpextrw     ecx, xmm1, 0
vmovd       xmm0, ecx
vmovd       xmm1, eax
vcvtph2ps   xmm0, xmm0 
vcvtph2ps   xmm1, xmm1
vaddss      xmm0, xmm1, xmm0
vcvtps2ph   xmm0, xmm0, 4
vmovd       eax, xmm0 
vpinsrw     xmm0, xmm0, eax, 0
ret

We can observe that a lot of the instructions produced (mostly the function prelude and epilogue) consists of clearing the highest 112 bits of the xmm vectors. While apparently due to the ABI imposed upon the compiler, this indicates that, while usefull, the compilers' native support for fp16 scalar operations cannot be fully relied upon codegen-wise when optimizing for maximum performance. As such, it is possible that a hand-written implementation for fp16 support in EVE could be beneficial in the long term, although optional.

Definitions & Name bikeshedding

Transparent

From #[repr(transparent)] from Rust, also used in C++ (see transparent comparators). Refers to an enum, or to a struct which has a single member of non-zero size, and for which the layout in memory is guaranteed to be the same as that of that member.

Exemple:

Enums are considered transparent types by default:

enum class E: int { };

static_assert(transparent_value<E>);

Structs which can satisfy the definition of a transparent type can implement the transparent trait:

struct S { int v; }

template <> struct transparent_trait<S> { using type = int; }

For any transparent type T, this assertion must hold (ie. the memory layout of T and transparent_trait<T>::type must be the same):

static_assert((sizeof(T) == sizeof(transparent_trait<T>::type)) && (alignof(T) == alignof(transparent_trait<T>::type)));

[DenisYaroshevskiy]

fp16

Objections!

In C++23 there is a float type: https://en.cppreference.com/w/cpp/types/floating-point
We are not on 23 but if 23 is supported - that's what we should be using.

On clang/gcc _Float16 is a thing - that's what we should be using

Otherwise - OK - have a custom struct.

I think spy should give us sonething helpful here, @jfalcou?

Let's have

namespace eve {
using float16_t = 
}

where it's equal to std::float16_t, when that's avaliable.

[SadiinsoSnowfall]

There may be a std::float16_t type in C++23, but EVE is still based on C++20. Our design will be compatible with this new type, just as it is planned to be compatible with _Float16 on x86, but it won't be dependent on it.

where it's equal to std::float16_t, when that's avaliable.

We don't necessary need to use std::float_16 as the backing storage as long as we provide sufficient interop with, for ex., user-defined conversion operators.

[DenisYaroshevskiy]

I don't like this

Ppl would want eve::widestd::float16_t and there is zero reasons not to support it

[DenisYaroshevskiy]

Step 1. Transparent types

That's an absolutely separate issue, as you have already encountered.

1. You already saw the equality problem. Equality for enums is completly different.
2. None of the functions should have specializations for equivalent types. Here you will have a ton. Like element_wise_callable should just make them dissapear.
3. logical is a questionable thing. We already don't do that for bundle.
   There maybe some logical_for typedef in our future, but we don't have it now.

[DenisYaroshevskiy]

This step consists of reworking internal functions such as as_register (as layed-out in issue #1844) to allow for users to extends the list of supported types, for example for transparent types:

I don't understand what are you trying to do. There is no way to support arbitrary types in eve

[SadiinsoSnowfall]

Ok. After some implementation tests with @jfalcou, it turns out you were right, we need to split the enum support and the fp16 support into two separated PRs, as we found out that we could not easily implement the actual fp16 backend based on a transparent type.

As such, we propose to keep the work on transparent types but reduce its scope to enums and single-member structs, and implement the fp16 support in another separated PR, which will need more discussion.

Let's discuss the details on Wednesday.

[DenisYaroshevskiy]

That would be amazing if you put in work into enums but please be mindful of the scope creep.
Probably prioritizing f16 if that's what you actually need to succeed - that's already massive.