Investigate moving the types in `System.Numerics.Vectors` to use Hardware Intrinsics

[tannergooding]

Today, hardware acceleration of certain types in the System.Numerics.Vectors project is achieved via [Intrinsic] attributes and corresponding runtime support. There are a few downsides to this approach:

• Minor tweaks to the backing implementation require shipping a new runtime
• It is not obvious that the code has a hardware accelerated path (outside reading documentation)
• Many of the types (such as the Matrix types) are not directly hardware accelerated

In netcoreapp30, the new Hardware Intrinsics feature is supposed to ship. This feature also allows hardware acceleration but at a much more fine-grained level (APIs typically have a 1-to-1 mapping with underlying instructions).

We should investigate moving the types in System.Numerics.Vectors to use hardware intrinsics as this has multiple potential benefits:

• The hardware acceleration is still tied to the runtime, but minor tweaks can be made without having to ship a new runtime
• The code having a hardware accelerated path becomes obvious as does the code that will be generated for a given platform/cpu
• It will become much easier to add hardware acceleration support to types currently missing it (such as Matrix4x4)

[tannergooding]

Related to https://github.com/dotnet/corefx/issues/17214

[tannergooding]

I had done some basic/preliminary investigation some time back and the numbers looked promising: https://github.com/dotnet/corefx/issues/25386#issuecomment-361137982

[tannergooding]

I am marking this up for grabs in case someone from the community wants to work on this. If you want to take a stab, feel free to say so and I can assign the issue out.

The "Microsoft DirectX Math" library is open-source, MIT-licensed, and has existing HWIntrinsic accelerated code for many of these APIs (but for Native code): https://github.com/microsoft/directxmath. This would likely be a good starting point as the algorithms have been around for a long time, have been in active use in production code, and are known to generally be correct and performant.

• NOTE: In some cases the C# algorithm and the DirectX algorithm differ slightly, we should make note of these cases and determine the best course of action.

[tannergooding]

FYI. @eerhardt, @danmosemsft, @CarolEidt, @fiigii

[tannergooding]

Also CC. @benaadams, who I know has looked at vectorizing/improving some of these types previously

[danmoseley]

@hughbe in case he is interested in a meaty task.

[fiigii]

@tannergooding Thanks for opening this issue. I agree that porting S.N.Vectors to managed code in HW intrinsic is much better to maintain. But before that, we may need to address some differences between Vector128/256<T> and Vector2/3/4/<T>. For example,

1. We should have an internal API to convert Vector2/3/4/<T> from/to Vector128/256<T>
2. Vector128/256<T> are immutable types, but Vector3/4 are mutable (fields of Vector3/4 can be assigned), how can we solve this difference? Or remain the implementation of some operations in the runtime?

[fiigii]

The "Microsoft DirectX Math" library is open-source,

@tannergooding Would you like to propose a "VectorMath" library for Vector128/256<T>?

[4creators]

IMHO we should take the opportunity when rewriting Vector<T> implementation to:

1. Use almost complete implementation of Intel HW Intrinsics up to AVX2 to expand Vector API surface with several missing operations which were requested by community or became possible due to HW intrinsics implementation (this would require API proposal, discussion and approval)

2. Provide Arm64 HW intrinsics support where possible

cc @CarolEidt @danmosemsft @eerhardt @fiigii @tannergooding

[tannergooding]

We should have an internal API to convert Vector2/3/4/ from/to Vector128/256

I don't believe this is needed. You should be able to load a Vector2/3/4 into a Vector128<T>, perform all operations, and then convert back to the return type trivially and without much (if any) overhead.

Users will also have their own vector/numeric types, so we should focus on making sure that getting a Vector128<T> from a user defined struct is fast/efficient in general.

Vector128/256 are immutable types, but Vector3/4 are mutable (fields of Vector3/4 can be assigned), how can we solve this difference? Or remain the implementation of some operations in the runtime?

I'm not sure why you think this is an issue. Could you elaborate?

[tannergooding]

@tannergooding Would you like to propose a "VectorMath" library for Vector128/256?

There will likely need to be some minimal helper library that operates solely on Vector128<T> that the Vector2/3/4 code would call into. This would also be useful for other places in the framework that could take advantage of this.

It would need to be internal at first, and discussions on making it public could happen later.

[tannergooding]

IMHO we should take the opportunity when rewriting Vector implementation to:

Any port should be a two/three step process:

1. Do a clean port of the existing code and make sure it is equally performant
2. Do a rewrite of the existing APIs to make improvements, where possible
3. Add new APIs, as required, and implement accordingly

Doing them together makes it harder to review the code and track the changes.

ARM/ARM64 support should definitely happen, but only after the APIs have been reviewed/approved/implemented.

[fiigii]

should be able to load a Vector2/3/4 into a Vector128

That would be great if the runtime can eliminate the round trip. If not, the memory load/store may have considerable overhead when we have __vectorcall in the future.

Users will also have their own vector/numeric types, so we should focus on making sure that getting a Vector128 from a user defined struct is fast/efficient in general.

Agree! Related to https://github.com/dotnet/coreclr/issues/19116

[tannergooding]

That would be great if the runtime can eliminate the round trip.

I would certainly hope that we can.

From my view, anything that would prevent us from moving the System.Runtime.Vector project to use exclusively HWIntrinsics (outside the dynamic Vector<T> sizing feature) is a potential perf problem for users wanting to use HWIntrinsics themselves. So we want to get those types of things flagged, investigated, fixed, etc.

[saucecontrol]

anything that would prevent us from moving the System.Runtime.Vector project to use exclusively HWIntrinsics (outside the dynamic Vector sizing feature) is a potential perf problem for users wanting to use HWIntrinsics themselves

This seems like a good enough excuse for me to take this on if nobody else is interested 😉

I'm at a bit of a loss how to start, though. Even something as trivial as operator + on Vector2 is twisting my brain. As an intrinsic, the JIT knows whether the struct is currently in an xmm register and can emit just a addps. If it's not in a register, it loads it with movsd before emitting the addps. Assuming I pin this and take its address so I can explicitly load it with Sse2.LoadScalarVector128(double*), is the JIT containment logic smart enough to know it's already in a xmm register and skip that part? And would it do the same with Vector3 where the load process is movsd + movss + insertps?

The constructors are even more confusing to me since they may be a setup for further SIMD operations or the fields may be immediately accessed. The JIT seems to know what to do about that now, but I can't imagine how managed code could get that right.

As far as the porting process, would the goal be to replace all [intrinsic] methods first and then SIMD-ize the things that aren't using intrinsics today?