Skip to content

Add experimental ggml-hexagon backend for the Hexagon NPU #16547

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 37 commits into from
Oct 22, 2025
+13,530 −0
Merged

Add experimental ggml-hexagon backend for the Hexagon NPU #16547

Changes from 1 commit
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
0625382
model: add support for extra bufs for all devices
max-krasnyansky Jul 16, 2025
80dc8e8
hexagon: add experimental ggml-hexagon backend for the Hexagon NPU
max-krasnyansky Jun 23, 2025
ec4436f
hexagon: fix format checker errors
max-krasnyansky Oct 13, 2025
aa65f21
hexagon: update readme and cmake presets
max-krasnyansky Oct 14, 2025
647fa3d
ci: add android-ndk-build jobs that build plain ARM64 and Snapdragon …
max-krasnyansky Oct 15, 2025
da7caac
hexagon: add simple graph optimizer for stacking MUL_MAT ops with the…
max-krasnyansky Oct 14, 2025
bbbc8ea
hexagon: move ADB helper scripts into scripts/snapdragon/adb
max-krasnyansky Oct 15, 2025
cc7dbd4
hexagon: replace all f/printfs with GGML_LOG_...
max-krasnyansky Oct 16, 2025
69a8047
readme: add hexagon to the list supported backends
max-krasnyansky Oct 16, 2025
debdb3b
hexagon: stack malmuts with quantized inputs only
max-krasnyansky Oct 16, 2025
3475e29
hexagon: add TODO for fixing issues in hexagon_graph_optimize
max-krasnyansky Oct 17, 2025
1e750df
hexagon: update to hex-sdk 6.4.0 and add scripts for running on QDC
max-krasnyansky Oct 17, 2025
8e7d8b5
scripts: fix lint errors
max-krasnyansky Oct 17, 2025
20aa689
scripts: update qdc pytest script to make linter happy
max-krasnyansky Oct 17, 2025
03e2b9c
hexagon: add reduce sum in fp32
max-krasnyansky Oct 18, 2025
384164d
hexagon: reduce number of vector stores in matmul output
max-krasnyansky Oct 18, 2025
a314eb6
hexagon: remove the need for vdelta in reduce-multiply-x8
max-krasnyansky Oct 18, 2025
7f2d00b
hexagon: consistent use of reduce_sum_fp32 for row_sums
max-krasnyansky Oct 19, 2025
5de19f8
hexagon: some more matmul optimizations and comments
max-krasnyansky Oct 19, 2025
cf0242e
hexagon: update cmake presets
max-krasnyansky Oct 21, 2025
250e3a6
hexagon: add OPMASK support for run-bench.sh wrapper
max-krasnyansky Oct 21, 2025
08a97e6
hexagon: update to use GGML_BACKEND_API
max-krasnyansky Oct 21, 2025
6d2d0bd
hexagon: remove unused logic for setting tensor flags for the views
max-krasnyansky Oct 21, 2025
18d7d20
hexagon: add asserts to set/get_tensor to make sure we handle complet…
max-krasnyansky Oct 21, 2025
26a90a0
hexagon: use cpy_tensor slow path for non-host buffers
max-krasnyansky Oct 21, 2025
a8e5ad8
hexagon: error checks in the buffer allocator
max-krasnyansky Oct 21, 2025
dc001b9
cmake: move include(extProj) under ggml-hexagon
max-krasnyansky Oct 21, 2025
c749b86
hexagon: don't forget to delete the backend on free
max-krasnyansky Oct 22, 2025
0c01229
hexagon: set/get_tensor size assert apply only to quantized tensors
max-krasnyansky Oct 22, 2025
62ef4eb
hexagon: reintroduce HEX_VERBOSE wrapper for GGML_LOG_DEBUG for now
max-krasnyansky Oct 22, 2025
19041f7
docs: typos in hexagon developer docs (libggm-...)
max-krasnyansky Oct 22, 2025
3e4ff73
hexagon: overhaul error handling in the session/device allocation
max-krasnyansky Oct 22, 2025
6acc285
hexagon: update cmake presets to enable fp16 vectors
max-krasnyansky Oct 22, 2025
dda466c
hexagon: remove unused time_usec function
max-krasnyansky Oct 22, 2025
b0e5beb
hexagon: don't forget to release buffer contexts
max-krasnyansky Oct 22, 2025
3049de5
hexagon: fixed indents in hvx-utils (missed clang-format auto-format …
max-krasnyansky Oct 22, 2025
f7d7411
hexagon: remove custom can_repeat function and use ggml_can_repeat
max-krasnyansky Oct 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
hexagon: add OPMASK support for run-bench.sh wrapper
  • Loading branch information
@max-krasnyansky
max-krasnyansky committed Oct 22, 2025
commit 250e3a66252a8dc18abfcee6872d60113509ffb6
5 changes: 4 additions & 1 deletion scripts/snapdragon/adb/run-bench.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@ device="HTP0"
verbose=""
[ "$V" != "" ] && verbose="$V"

opmask=
[ "$OPMASK" != "" ] && opmask="GGML_HEXAGON_OPMASK=$OPMASK"

nhvx=
[ "$NHVX" != "" ] && nhvx="GGML_HEXAGON_NHVX=$NHVX"

Expand All @@ -31,6 +34,6 @@ adb $adbserial shell " \
cd $basedir; \
LD_LIBRARY_PATH=$basedir/$branch/lib \
ADSP_LIBRARY_PATH=$basedir/$branch/lib \
$ndev $nhvx ./$branch/bin/llama-bench --device $device --mmap 0 -m $basedir/../gguf/$model \
$ndev $nhvx $opmask ./$branch/bin/llama-bench --device $device --mmap 0 -m $basedir/../gguf/$model \
-t 4 --batch-size 128 -ngl 99 $@ \
"