One thing that stuck out to me is that deals with a lot more data formats, in particular, low-precision formats like FP4, FP6 and FP8. Manipulating those formats can take a lot of annoying effort; in general, x86 (until AVX-512, at least) has unconvincing support for so-called "lane-crossing" instructions that move data across 16-byte boundaries within a vector. So you can imagine unpacking, e.g., tightly packed 7-bit data to 8-bit data is a real slog.
I can already immediately think of a use case for vunpackb in some of the stuff I'm working on, where we'd like to efficiently unpack weights from the high half of a vector.
Separately, adding all signed–unsigned variants of the VNNI dot product instructions is a welcome (albeit niche) change. There was an annoying divergence here between major ISAs: x86 added vpdpbusd which computed a dot product between u8 and i8, while ARM added vdotq, which computes a dot product either between u8 and u8 elements, or i8 and i8. So for broad compatibility, you generally had to restrict one of your inputs to [0,127]. This difference shows in the design of (for example) WASM relaxed SIMD, where the result of wasm.dot.i8x16.i7x16.add.signed is implementation-defined if you exceed the [0,127] range. ARM later added mixed-sign variants, and now x86 consummates it.
This is not a vector extension (like Intel AVX/AVX-512 or Arm SVE), but a matrix extension (like Intel AMX or Arm SME or the "tensor" operations of NVIDIA GPUs).
Some of the latest generations of Intel server CPUs with P-cores already have the AMX matrix extension, which can be used to implement fast AI inference.
AMD has not implemented AMX yet, and probably they will not implement it, because this new "AI Compute Extension", which has been defined by Intel and AMD together, is an alternative/extension to AMX.
Matrix extensions are more efficient for AI inference than vector extensions, because they reduce the ratio between memory accesses and computation operations.
The fraction of the installed base of x86 CPUs that support AMX is very small (i.e. only a part of the recent Intel server CPUs support AMX, while the other Intel server CPUs, all Intel consumer CPUs and all AMD CPUs do not support AMX).
CPUs with ACE will in most cases replace CPUs that did not support AMX, so all the registers specified by ACE, but not by AVX10 a.k.a. AVX-512 are new.
Please define new. Also, I think AMD uses very similar cores in server and client. So, disabling AVX512 may be an Intel thing (my guess is that so they can easily move threads between E & P cores).
They didn't disable it at first on their client CPUs, and it resulted in code randomly crashing depending on whether ifunc resolvers first ran on a big core or a little core.
It's pretty surprising that multiple CPU vendors have run into issues like this (some more than once, fucking Samsung), when it's pretty much the first thing that anyone on the toolchain side of thing asks when they hear about heterogenous cores on a CPU.
So how does this differ from available sse / avx instructions already in most x64 machines?
One thing that stuck out to me is that deals with a lot more data formats, in particular, low-precision formats like FP4, FP6 and FP8. Manipulating those formats can take a lot of annoying effort; in general, x86 (until AVX-512, at least) has unconvincing support for so-called "lane-crossing" instructions that move data across 16-byte boundaries within a vector. So you can imagine unpacking, e.g., tightly packed 7-bit data to 8-bit data is a real slog.
I can already immediately think of a use case for vunpackb in some of the stuff I'm working on, where we'd like to efficiently unpack weights from the high half of a vector.
Separately, adding all signed–unsigned variants of the VNNI dot product instructions is a welcome (albeit niche) change. There was an annoying divergence here between major ISAs: x86 added vpdpbusd which computed a dot product between u8 and i8, while ARM added vdotq, which computes a dot product either between u8 and u8 elements, or i8 and i8. So for broad compatibility, you generally had to restrict one of your inputs to [0,127]. This difference shows in the design of (for example) WASM relaxed SIMD, where the result of wasm.dot.i8x16.i7x16.add.signed is implementation-defined if you exceed the [0,127] range. ARM later added mixed-sign variants, and now x86 consummates it.
This is not a vector extension (like Intel AVX/AVX-512 or Arm SVE), but a matrix extension (like Intel AMX or Arm SME or the "tensor" operations of NVIDIA GPUs).
Some of the latest generations of Intel server CPUs with P-cores already have the AMX matrix extension, which can be used to implement fast AI inference.
AMD has not implemented AMX yet, and probably they will not implement it, because this new "AI Compute Extension", which has been defined by Intel and AMD together, is an alternative/extension to AMX.
Matrix extensions are more efficient for AI inference than vector extensions, because they reduce the ratio between memory accesses and computation operations.
this also adds new registers to operate on (more state) - 1KB more state at least (512b x 16)
It reuses AMX registers, so I think the only new state is the block scale register (1024 bits)?
The fraction of the installed base of x86 CPUs that support AMX is very small (i.e. only a part of the recent Intel server CPUs support AMX, while the other Intel server CPUs, all Intel consumer CPUs and all AMD CPUs do not support AMX).
CPUs with ACE will in most cases replace CPUs that did not support AMX, so all the registers specified by ACE, but not by AVX10 a.k.a. AVX-512 are new.
Thank $ALL_DIETIES that the TCG wasn’t involved!
AVX512 isn't available on most new CPUs, I'm guessing ACE will only be available on server CPUs for at least a couple of years at launch?
> AVX512 isn't available on most new CPUs
Please define new. Also, I think AMD uses very similar cores in server and client. So, disabling AVX512 may be an Intel thing (my guess is that so they can easily move threads between E & P cores).
They didn't disable it at first on their client CPUs, and it resulted in code randomly crashing depending on whether ifunc resolvers first ran on a big core or a little core.
It's pretty surprising that multiple CPU vendors have run into issues like this (some more than once, fucking Samsung), when it's pretty much the first thing that anyone on the toolchain side of thing asks when they hear about heterogenous cores on a CPU.