Added RISC-V Vector Support for K-Quants and improved the existing intrinsics

[Tameem-10xE]

Hi,

In #2929, we have added the RISC-V intrinsics for the dot product functions in GGML, this PR improves these existing dot product functions in ggml.c and also adds the new risc-v vector intrinsics for k_quants and row quantize (Q8_0 and Q8_1) functions. Now LLaMa.cpp fully supports to run on RISC-V vector processor with GGUF.

In future, this will enable GGML and LLaMa.cpp to run efficiently on RISC-V hardware with vector support and also open a way to compare its performance with other vector processors like Intel AVX and Arm Neon.

Update: Got access to RISC-V vector board with 8 cores and 4GB RAM, the performance boost is 6-7 times against the scalar version on the same board.

Runining llama.cpp AI model on RVV1.0 vs RISC-V Scalar

The RISC-V Vector intrinsics support is added for the following K_quants functions with both QKK = 256 and QKK = 64 block size

   ggml_vec_dot_q2_K_q8_K
   ggml_vec_dot_q3_K_q8_K
   ggml_vec_dot_q4_K_q8_K
   ggml_vec_dot_q5_K_q8_K
   ggml_vec_dot_q6_K_q8_K

The RVV intrinsics is also added for the following Q8 quantize row functions

    quantize_row_q8_0
    quantize_row_q8_1

The following dot product functions have also been optimized by using fractional LMUL (i.e. 1/2) instead of LMUL = 1. I am a little skeptical of this since it works correctly but I have noticed some decrease in inference accuracy, which I think could be a problem with my system or weights. Although I prefer to stick with it since it utilizes a much less number of vector registers after product

    ggml_vec_dot_q4_0_q8_0
    ggml_vec_dot_q4_1_q8_1
    ggml_vec_dot_q5_0_q8_0
    ggml_vec_dot_q5_1_q8_1

And finally, the vector initialization in Q5 by the temporary array is also replaced by the vid_v intrinsics

[Compilation]
Ubuntu: 22.10
riscv-toolchain: 2023.07.05 riscv64 linux glibc

To compile it for RISC-V run,

$   make   llama-cli                   # For RISC-V CPU

$   make clean
$   make   RISCV_CROSS_COMPILE=1       # For Cross Compilation only

[Directly on RISC-V CPU]

$   ./llama-cli -m ./path/to/model.gguf -p "Anything" -n 50

[QEMU]

$   qemu-riscv64 -L /path/to/sysroot/  -cpu rv64,v=true,vlen=256,elen=64,vext_spec=v1.0 ./llama-cli -m ./path/to/model.gguf -p "Anything" -n 50

Note: Running on qemu emulator could be very slow and may take 2-5 minutes per token

Any feedback is welcome, if you have any suggestions or improvements, especially for fractional LMUL change, please share.

Thanks!

[grigohas]

hello, i am doing what you suggested and i have results. I have 2 questions, when i want to run it without vector proccesor in qemu, what comand do i have to run? also , how can i check that those 2 runs are different and the one with vector proccesor working like i wanted to ? sorry, i am new to this

[Tameem-10xE]

Hi, for running on CPU (scalar) provide the path to risc-v toolchain and then use qemu

make llama-cli CC="riscv64-unknown-linux-gnu-gcc -march=rv64gc -mabi=lp64d" CXX="riscv64-unknown-linux-gnu-g++ -march=rv64gc -mabi=lp64d"

qemu-riscv64 -L /path/to/sysroot/  -cpu rv64 ./llama-cli -m ./path/to/model.gguf -p "Anything" -n 100

You can set the seed to get the same results i.e; llama-cli -s (some_seed number) ...

More details: RVV article
Also, this is old, and many things have change, like main -> llama-cli etc.

Thank you

[grigohas]

Hi, for running on CPU (scalar) provide the path to risc-v toolchain and then use qemu

make llama-cli CC="riscv64-unknown-linux-gnu-gcc -march=rv64gc -mabi=lp64d" CXX="riscv64-unknown-linux-gnu-g++ -march=rv64gc -mabi=lp64d"

qemu-riscv64 -L /path/to/sysroot/  -cpu rv64 ./llama-cli -m ./path/to/model.gguf -p "Anything" -n 100

You can set the seed to get the same results i.e; llama-cli -s (some_seed number) ...

More details: RVV article Also, this is old, and many things have change, like main -> llama-cli etc.

Thank you

yeah i read this article but when i do the make you provided , i get an error for "march=native" error and from what i searched on makefile, i have to do RISCV_CROSS_COMPILE=1 RISCV=1

[Tameem-10xE]

Sorry yes, I just identified the makefile has been reorder and RISCV=1 is required in current version

[Tameem-10xE]

After line 432 in makefile, update the flags for vector version with scalar, i.e

MK_CFLAGS += -march=rv64gc -mabi=lp64d
MK_CXXFLAGS += -march=rv64gc -mabi=lp64d

and then run qemu with,

make llama-cli RISCV=1 CC="riscv64-unknown-linux-gnu-gcc" CXX="riscv64-unknown-linux-gnu-g++"

[grigohas]

okay okay, one last question, i use the same seed and i have results for both with vector and without, but the only difference in log print is the print time . with vector is 2-2.5X more than without. Is it correct?

[Tameem-10xE]

Yes, on qemu the vector emulation time is much slower (the actual reason is not known to me, could be due to qemu has to additionally emulate vector processor with the scalar one or parallel processing issues, and also the log use the real-time for comparison), but this should not be the case with actual RISC-V vector board

[grigohas]

hello again, i am running llama with vector extension on gem5 but since there isnt something on log to check if vector extension is enabled , how do i know ?

[Tameem-10xE]

Hi, I’ve submitted a PR (#9442) which will print RISCV_VECT=1 on the terminal, if the vector processor is found. Also, I slightly changed Makefile so it no longer requires a flag for RISC-V vector boards—only RISCV_CROSS_COMPILATION=1 is needed for the emulator (i.e. QEMU).

The following is the output from the RISC-V BPI-F3 board with vector support,
...

...

...

...

[grigohas]

Hello, i have a question, why the load time is less when rvv is enabled ? load time is the loading time of the model right ? how the rvv affects it ?

[Tameem-10xE]

Hi, Actually that was due to the quantization happening while loading the weights into memory (Not all weights are quantized, some are done while loading them into memory; specifically check the function quantize_row_q8_0_reference in GGML quants.c file or quantize_row_q4_K), also enabling auto-vectorization can affect the load time.

[grigohas]

okay but why the quantization affects the loadimg time ? can you explain me what changes with the rvv enabled ?

[Tameem-10xE]

During model's initialization phase, there could be several reasons, such as auto-vectorization of memory operations by compiler, recalculation, decompression, and data alignment. However, during benchmarking, I noticed the most change due to the quantize_row function, since it was performing some computations before the weights were fully loaded into memory.

[grigohas]

okay so when the rvv is enabled , whats happening with the quantize_row function and the load time is less ? also if the model is already quantized is there any change ?

[grigohas]

when you enable the rvv , the quantize_row function that you mentioned that is perforforming computations doesnt execute ?
@Tameem-10xE

[Tameem-10xE]

Sorry, I last worked on this project a year ago and did not go deeper into how it works except for what I could quantify. I might be mistaken or unclear about which functions are affecting the load time (also many things has been change). I think you should ask in the GitHub discussion or Discord if they have one. I also noticed a reduction in load time on x86 with vector (~5 times), and I think the most probable cause must be compiler auto-vectorization. If not, they may be able to provide the exact reason.

[grigohas]

Sorry, I last worked on this project a year ago and did not go deeper into how it works except for what I could quantify. I might be mistaken or unclear about which functions are affecting the load time (also many things has been change). I think you should ask in the GitHub discussion or Discord if they have one. I also noticed a reduction in load time on x86 with vector (~5 times), and I think the most probable cause must be compiler auto-vectorization. If not, they may be able to provide the exact reason.

Okay, thank you very much. are the weights quantized differently in scalar than with rvv on ? I mean, does it make sense to compare the results of scalar with the rvv on or do they follow a different loading procedure ?

[Tameem-10xE]

Welcome!
No, weights are quantized the same for scalar and RVV (lllama-qunatize script is independent of this), I think it does not matter much since loading is a one-time process, but the more important metric for comparison would be inference time i.e how much tokens are being generated with RVV on against the scalar time.

[grigohas]

Hello again, i am simulating through gem5 a riscv environment and i am running llama-cli with a llama 4b model. When i change the vlen and elen of the rvv to greater than 256bit and 64 bit , the generated phrase i get , it doesnt make sense. Is the llama implementation working only on rvv with vlen=256 and elen=64 bit ?

[Tameem-10xE]

Hi, What Output did you got?
Some functions were using dynamic vl i.e (size_t vl = __riscv_vsetvl_e8m1(qk/2);)
But to avoid duplicating instructions to load qh and smaller size arrays, I did limit the vl size to 256, but smaller VL should work without any issues.

[grigohas]

For rvv with vlen=512 and elen=64 i got this

[Tameem-10xE]

..., this could be a bug, I missed something or the second half of the register could be interfering with the output causing undefined or junk text. Sorry, due to other tasks, I will not be able to look into this for now, but meanwhile you can file an issue or ask further it about in RISC-V intrinsic repo.

[grigohas]

Now that we build llama.cpp with cmake instead of make command , what file do i have to change to build it for riscv but without vector extension ? i changed the -march on makefile but it was built again with the vector extension

[Tameem-10xE]

You can use this flag -DGGML_RVV=OFF i.e;

mkdir build
cd build

cmake -DGGML_RVV=OFF [... other flags ...]

[grigohas]

..., this could be a bug, I missed something or the second half of the register could be interfering with the output causing undefined or junk text. Sorry, due to other tasks, I will not be able to look into this for now, but meanwhile you can file an issue or ask further it about in RISC-V intrinsic repo.

Hello again. What do you think i can change to fix this bug and the exe can run with higher vlen with correct results?

[Tameem-10xE]

Usually, this could be due to the vl variable, outer or inner for loop (for (int i = 0; i < nb; i++) not completed or missed edge elements) or vsetvli instruction. Also, I am unsure about GEMM5 simulation, which could be different from the QEMU or RV board I tested on (with 256-bit VLEN). Sorry, I have this in my mind, but I will try to fix this next week.

[grigohas]

Usually, this could be due to the vl variable, outer or inner for loop (for (int i = 0; i < nb; i++) not completed or missed edge elements) or vsetvli instruction. Also, I am unsure about GEMM5 simulation, which could be different from the QEMU or RV board I tested on (with 256-bit VLEN). Sorry, I have this in my mind, but I will try to fix this next week.

Any news ?

[Tameem-10xE]

Really sorry for the late response. I was inactive last week.
Actually, I have only access to 256-bit RV Hardware, so my only option was to check it on an emulator (qemu), and it is working correctly for 512 VLEN with Q4_K. I also go through code but was unable to find any issue. It may be issue with GEMM5? But can't say for sure.

[grigohas]

Really sorry for the late response. I was inactive last week. Actually, I have only access to 256-bit RV Hardware, so my only option was to check it on an emulator (qemu), and it is working correctly for 512 VLEN with Q4_K. I also go through code but was unable to find any issue. It may be issue with GEMM5? But can't say for sure.

Can you explain how do you run it on qemu ? I just run llama-cli for vlen=512 and i got this generated phrase with q4_k model
"" Anything�stagemblụ shelterirasichenForKeyronessageHinttexttåk Weltkrieg ""

[Tameem-10xE]

Sorry, I plugged the wrong weights above (need to change my naming convention...).

Identified the issue in the function named void ggml_vec_dot_q4_K_q8_K in ggml-cpu-quant.c file. Unable to test this right now, but try to change this variable value of size_t vl=8 in this function to either 16 or 32. (Only for 4-bit q4_k)

[grigohas]

Sorry, I plugged the wrong weights above (need to change my naming convention...).

Identified the issue in the function named void ggml_vec_dot_q4_K_q8_K in ggml-cpu-quant.c file. Unable to test this right now, but try to change this variable value of size_t vl=8 in this function to either 16 or 32. (Only for 4-bit q4_k)

I changed the vl to 16 and 32 and it still doesnt work correctly. The same happens for other models too. I tried a q2_k model for 512 and the generated phrase is not correct