When cross compiling, it adds the compile flags of the host machine incorrectly

[ChipKerchner]

cmake .. \
        -G Ninja \
        -DCMAKE_SYSTEM_NAME=Linux \
        -DCMAKE_SYSTEM_PROCESSOR=riscv64 \
        -DCMAKE_BUILD_TYPE=Release \
        -DCMAKE_C_COMPILER=/proj_sw/user_dev/ckerchner/tmp/tt-riscv-toolchain-20250709/bin/riscv64-unknown-linux-gnu-gcc \
        -DCMAKE_CXX_COMPILER=/proj_sw/user_dev/ckerchner/tmp/tt-riscv-toolchain-20250709/bin/riscv64-unknown-linux-gnu-g++ \
        -DCMAKE_STRIP=/proj_sw/user_dev/ckerchner/tmp/tt-riscv-toolchain-20250709/bin/riscv64-unknown-linux-gnu-strip \
        -DCMAKE_C_FLAGS="-O3 -mcpu=tt-ascalon-d8" \
        -DCMAKE_CXX_FLAGS="-O3 -mcpu=tt-ascalon-d8" \
        -DLLAMA_CURL=OFF \
        -DGGML_NATIVE=OFF \
        -DGGML_RVV=ON \
        -DGGML_RV_ZFH=ON \
        -DGGML_RV_ZICBOP=ON \
        -DGGML_OPENMP=OFF

ninja
[2/108] Building C object tests/CMakeFiles/test-pool.dir/test-pool.c.o
FAILED: tests/CMakeFiles/test-pool.dir/test-pool.c.o
ccache /proj_sw/user_dev/ckerchner/tmp/tt-riscv-toolchain-20250709/bin/riscv64-unknown-linux-gnu-gcc -DGGML_BACKEND_SHARED -DGGML_SHARED -DGGML_USE_CPU -I/proj_sw/user_dev/ckerchner/ggml/src/../include -O3 -mcpu=tt-ascalon-d8 -mavx -mavx2 -mfma -mf16c -msse3 -O3 -DNDEBUG   -UNDEBUG -std=gnu11 -MD -MT tests/CMakeFiles/test-pool.dir/test-pool.c.o -MF tests/CMakeFiles/test-pool.dir/test-pool.c.o.d -o tests/CMakeFiles/test-pool.dir/test-pool.c.o -c /proj_sw/user_dev/ckerchner/ggml/tests/test-pool.c
riscv64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-mavx'
riscv64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-mavx2'
riscv64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-mfma'
riscv64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-mf16c'
riscv64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-msse3'

[ChipKerchner]

I have to manually edit build.ninja and remove those compile flags in order to get it to work.