Fix System.Numerics xunit tests and add them to CI (#5006)

Fix System.Numerics xunit tests and add them to CI (#5006)

* Fix System.Numerics xunit tests and add them to CI

Xunit.Extensions.NetCore.dll is available as a net46 version now
so we can just use it instead of building it from our buildtools fork.
This also allows us to remove the fork from our submodules.

We were missing running the System.Numerics xunit tests in CI,
this adds them so we catch breaks like these earlier.

* Skip tests on Windows, they seem to fail there