Add AVX2 implementation of dequantize_row_q4_0

[slaren]

I couldn't notice a big performance improvement, more testing necessary

[slaren]

A quick performance test shows significant improvement in the function itself (with k=4096):

Running ./test-dq
Run on (16 X 3600 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x8)
  L1 Instruction 32 KiB (x8)
  L2 Unified 256 KiB (x8)
  L3 Unified 16384 KiB (x1)
Load Average: 1.56, 1.26, 1.46
----------------------------------------------------------------------
Benchmark                            Time             CPU   Iterations
----------------------------------------------------------------------
BM_dequantize_row_q4_0           10351 ns        10351 ns        66698
BM_dequantize_row_q4_0_avx2       1384 ns         1384 ns       509491

[slaren]

The first chunks of the perplexity computation show the same values. I didn't run the full test but I have no reason to believe that it would produce different values.
[1]4.5690,[2]5.2058,[3]6.0526,

[Green-Sky]

@ggerganov we need some sort of benchmarking suite for ggml.

@slaren how complex is the ./test-dq ? can you provide the code, does it require the model files or is it standalone? (should be easy to create synthetic data)

[slaren]

It's a standalone test using the google benchmark library. Here is the code: https://gist.github.com/slaren/ba732ed08abd0ba148129eab3335dfb7
To do that, I split the avx and scalar implementations into dequantize_row_q4_0_avx2 and dequantize_row_q4_0 beforehand.

[ggerganov]

The first chunks of the perplexity computation show the same values. I didn't run the full test but I have no reason to believe that it would produce different values. [1]4.5690,[2]5.2058,[3]6.0526,

The dequantize functions are only used if you link against BLAS and use -b 32 or bigger:

make clean
LLAMA_OPENBLAS=1 make

Otherwise, they will never be called.

[slaren]

@ggerganov that's not what I am seeing, here is a stack trace for example:

#2  0x00005555555660d4 in dequantize_row_q4_0 (x=0x7ffedc43a0d0, y=0x7ffe585b81a0, k=k@entry=4096) at ggml.c:767
#3  0x000055555556b1e7 in ggml_compute_forward_get_rows_q4_0 (params=<optimized out>, params=<optimized out>,
    dst=0x7ffe585b8100, src1=0x7ffe585b8030, src0=0x7ffedc43a030) at ggml.c:7249
#4  ggml_compute_forward_get_rows (dst=0x7ffe585b8100, src1=0x7ffe585b8030, src0=0x7ffedc43a030, params=<optimized out>)
    at ggml.c:7345
#5  ggml_compute_forward (params=<optimized out>, tensor=0x7ffe585b8100) at ggml.c:9027
#6  0x0000555555571435 in ggml_graph_compute (ctx=<optimized out>, cgraph=0x7ffffffe4d90) at ggml.c:9911
#7  0x00005555555793f5 in llama_eval_internal (lctx=..., tokens=<optimized out>, n_tokens=4, n_past=0, n_threads=<optimized out>)
    at llama.cpp:822
#8  0x000055555557976d in llama_eval (ctx=<optimized out>, tokens=<optimized out>, n_tokens=<optimized out>,
    n_past=<optimized out>, n_threads=<optimized out>) at llama.cpp:1493
#9  0x000055555555c396 in main (argc=<optimized out>, argv=<optimized out>) at main.cpp:224

[ggerganov]

Ah yes - there is one exception -- the ggml_get_rows at the start of the inference. It is a very lightweight call so I don't expect it to take a measurable amount of time.

[slaren]

Ah I see. I am running some tests with BLAS now, will report back when I have some results. Unfortunately it seems to be much slower, probably need to find a better BLAS library than just using the libopenblas-dev package from ubuntu..

[slaren]

@ggerganov When building with BLAS, -b 32 and a long enough prompt I only get garbage generation (not just bad, but random tokens). This happens on master too. Is it possible that BLAS support is broken at the moment?

[ggerganov]

Yes, it is broken. Weird ..

[ggerganov]

Ok, BLAS has been fixed and for large prompts and batch size ( > 256) there is significant benefit to enable BLAS.
Tested on M1 so far, but I expect the same results for x86

[slaren]

I am seeing a very significant improvement with x86 as well, for instance the perplexity computation went from ~8 hours to ~5 hours.