whisper-cpp: add ROCm and Vulkan support, rename from openai-whisper-cpp #365033

aviallon · 2024-12-13T21:51:11Z

Rewrite to use CMake build
Add aviallon to whisper-cpp's maintainers

Fixes #357736
Fixes #349669
Closes #358174

Things done

Add a 👍 reaction to pull requests you find important.

aviallon · 2024-12-14T11:45:27Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review

`x86_64-linux`

✅ 2 packages built:

whisper-cpp
whisper-cpp-vulkan

aviallon · 2024-12-14T11:46:34Z

If someone could test on Darwin, it would be awesome.

PapayaJackal · 2024-12-15T01:40:15Z

Great job! I tested it on Darwin, but unfortunately, it didn't work:

dyld[98714]: Library not loaded: libwhisper.1.dylib
  Referenced from: <snip> /nix/store/kxk6zw08rfq61apq09mr5n8z6lndngv8-whisper-cpp-1.7.2/bin/whisper-cpp
  Reason: tried: 'libwhisper.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibwhisper.1.dylib' (no such file), 'libwhisper.1.dylib' (no such file), '/usr/local/lib/libwhisper.1.dylib' (no such file), '/usr/lib/libwhisper.1.dylib' (no such file, not in dyld cache), '/Users/<snip>/libwhisper.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/<snip>/libwhisper.1.dylib' (no such file), '/Users/<snip>/libwhisper.1.dylib' (no such file), '/usr/local/lib/libwhisper.1.dylib' (no such file), '/usr/lib/libwhisper.1.dylib' (no such file, not in dyld cache)
[1]    98714 abort      whisper-cpp

aviallon · 2024-12-15T10:30:57Z

Great job! I tested it on Darwin, but unfortunately, it didn't work:

dyld[98714]: Library not loaded: libwhisper.1.dylib
  Referenced from: <snip> /nix/store/kxk6zw08rfq61apq09mr5n8z6lndngv8-whisper-cpp-1.7.2/bin/whisper-cpp
  Reason: tried: 'libwhisper.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibwhisper.1.dylib' (no such file), 'libwhisper.1.dylib' (no such file), '/usr/local/lib/libwhisper.1.dylib' (no such file), '/usr/lib/libwhisper.1.dylib' (no such file, not in dyld cache), '/Users/<snip>/libwhisper.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/<snip>/libwhisper.1.dylib' (no such file), '/Users/<snip>/libwhisper.1.dylib' (no such file), '/usr/local/lib/libwhisper.1.dylib' (no such file), '/usr/lib/libwhisper.1.dylib' (no such file, not in dyld cache)
[1]    98714 abort      whisper-cpp

It looks like the RPATH is not set correctly.
It could be because of the fact the binaries are not installation targets, and may need specific patching.
Can you try adding an explicit patchelf rpath fix in postInstall?

I'll also see if I can create a KVM-OSX VM to test the derivation directly.

sersorrel

tested Vulkan support with:

$ nixpkgs-review pr 365033
$ mkdir /tmp/whisper
$ ./results/whisper-cpp-vulkan-x86_64-linux/bin/whisper-cpp-download-ggml-model base.en /tmp/whisper/
$ ./results/whisper-cpp-vulkan-x86_64-linux/bin/whisper-cpp -m /tmp/whisper/ggml-base.en.bin -m /tmp/whisper/ggml-base.en.bin ~/Downloads/jfk.wav

and it seems to work ok:

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XTX (RADV NAVI31) ((null)) | uma: 1 | fp16: 1986290034 | warp size: 0

with rocmSupport = true, it seems to work fine as well (though it takes ~25 minutes to build locally):

ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon RX 7900 XTX, compute capability 11.0, VMM: no

pkgs/by-name/wh/whisper-cpp/package.nix

hughobrien · 2024-12-16T16:12:22Z

The vanilla build works (and fixes the SIMD issues thank you)

[nix-shell:~/.cache/nixpkgs-review/pr-365033]$ ./results/whisper-cpp-x86_64-linux/bin/whisper-cpp-bench --model "$model"
whisper_init_from_file_with_params_no_state: loading model from '/home/hugh/.local/share/whisper/ggml-medium.en.bin'
whisper_init_with_params_no_state: use gpu    = 1
whisper_init_with_params_no_state: flash attn = 0
whisper_init_with_params_no_state: gpu_device = 0
whisper_init_with_params_no_state: dtw        = 0
whisper_init_with_params_no_state: backends   = 1
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51864
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 1024
whisper_model_load: n_audio_head  = 16
whisper_model_load: n_audio_layer = 24
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 1024
whisper_model_load: n_text_head   = 16
whisper_model_load: n_text_layer  = 24
whisper_model_load: n_mels        = 80
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 4 (medium)
whisper_model_load: adding 1607 extra tokens
whisper_model_load: n_langs       = 99
whisper_model_load:      CPU total size =  1533.14 MB
whisper_model_load: model size    = 1533.14 MB
whisper_init_state: kv self size  =   50.33 MB
whisper_init_state: kv cross size =  150.99 MB
whisper_init_state: kv pad  size  =    6.29 MB
whisper_init_state: compute buffer (conv)   =   28.55 MB
whisper_init_state: compute buffer (encode) =  170.15 MB
whisper_init_state: compute buffer (cross)  =    7.72 MB
whisper_init_state: compute buffer (decode) =   98.18 MB

system_info: n_threads = 4 / 16 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | CUDA = 0 | COREML = 0 | OPENVINO = 0 | CANN = 0

whisper_print_timings:     load time =  1724.71 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =     0.00 ms
whisper_print_timings:   sample time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:   encode time = 18957.00 ms /     1 runs (18957.00 ms per run)
whisper_print_timings:   decode time =  7771.20 ms /   256 runs (   30.36 ms per run)
whisper_print_timings:   batchd time =  5600.82 ms /   320 runs (   17.50 ms per run)
whisper_print_timings:   prompt time = 62897.61 ms /  4096 runs (   15.36 ms per run)
whisper_print_timings:    total time = 95227.88 ms

However the vulkan build did not:

[nix-shell:~/.cache/nixpkgs-review/pr-365033]$ ./results/whisper-cpp-vulkan-x86_64-linux/bin/whisper-cpp-bench --model "$model"
whisper_init_from_file_with_params_no_state: loading model from '/home/hugh/.local/share/whisper/ggml-medium.en.bin'
whisper_init_with_params_no_state: use gpu    = 1
whisper_init_with_params_no_state: flash attn = 0
whisper_init_with_params_no_state: gpu_device = 0
whisper_init_with_params_no_state: dtw        = 0
ggml_vulkan: Found 1 Vulkan devices:
Segmentation fault (core dumped)

[nix-shell:~/.cache/nixpkgs-review/pr-365033]$ ./results/whisper-cpp-vulkan-x86_64-linux/bin/whisper-cpp --model "$model" /tmp/out.wav 
whisper_init_from_file_with_params_no_state: loading model from '/home/hugh/.local/share/whisper/ggml-medium.en.bin'
whisper_init_with_params_no_state: use gpu    = 1
whisper_init_with_params_no_state: flash attn = 0
whisper_init_with_params_no_state: gpu_device = 0
whisper_init_with_params_no_state: dtw        = 0
ggml_vulkan: Found 1 Vulkan devices:
Segmentation fault (core dumped)

PapayaJackal · 2024-12-16T16:15:14Z

Great job! I tested it on Darwin, but unfortunately, it didn't work:
dyld[98714]: Library not loaded: libwhisper.1.dylib
  Referenced from: <snip> /nix/store/kxk6zw08rfq61apq09mr5n8z6lndngv8-whisper-cpp-1.7.2/bin/whisper-cpp
  Reason: tried: 'libwhisper.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibwhisper.1.dylib' (no such file), 'libwhisper.1.dylib' (no such file), '/usr/local/lib/libwhisper.1.dylib' (no such file), '/usr/lib/libwhisper.1.dylib' (no such file, not in dyld cache), '/Users/<snip>/libwhisper.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/<snip>/libwhisper.1.dylib' (no such file), '/Users/<snip>/libwhisper.1.dylib' (no such file), '/usr/local/lib/libwhisper.1.dylib' (no such file), '/usr/lib/libwhisper.1.dylib' (no such file, not in dyld cache)
[1]    98714 abort      whisper-cpp
It looks like the RPATH is not set correctly. It could be because of the fact the binaries are not installation targets, and may need specific patching. Can you try adding an explicit patchelf rpath fix in postInstall?

I'll also see if I can create a KVM-OSX VM to test the derivation directly.

I'm a bit swamped with other stuff right now, so I won't be able to dive into debugging this further. Sorry!

aviallon · 2024-12-16T23:07:30Z

@PapayaJackal I tried adding addAutoPatchelfSearchPath, to see how it goes on macOS.

rodfersou · 2024-12-20T02:56:06Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 365033

`aarch64-darwin`

❌ 2 packages failed to build:

whisper-cpp
whisper-cpp-vulkan

rodfersou · 2024-12-20T03:02:13Z

$ nix build --file /nix/store/w1hgvm5rb3rblya8if90cdx5lvwyf4wc-nixpkgs-review-2.12.0/lib/python3.12/site-packages/nixpkgs_review/nix/review-shell.nix --nix-path 'nixpkgs=/Users/rodrigo/.cache/nixpkgs-review/pr-365033-1/nixpkgs nixpkgs-overlays=/var/folders/by/9xt1lyys0qvds8zzbx60c58m0000gn/T/tmppcq_ecwm' --extra-experimental-features 'nix-command no-url-literals' --no-link --keep-going --no-allow-import-from-derivation --argstr local-system aarch64-darwin --argstr nixpkgs-path /Users/rodrigo/.cache/nixpkgs-review/pr-365033-1/nixpkgs --argstr nixpkgs-config-path /var/folders/by/9xt1lyys0qvds8zzbx60c58m0000gn/T/tmp0qmt4adr.nix --argstr attrs-path /Users/rodrigo/.cache/nixpkgs-review/pr-365033-1/attrs.nix
error: builder for '/nix/store/dinzb2x13v9ssxvp36p7m76fk35dc13a-whisper-cpp-1.7.2.drv' failed with exit code 1;
       last 10 log lines:
       >           ^^^^^^^^^^^^^^^
       >   File "/nix/store/whmwqp5qa1lpadz0ik6wfjgjzbbgjh1z-python3-3.12.7-env/lib/python3.12/site-packages/elftools/elf/elffile.py", line 84, in __init__
       >     self._identify_file()
       >   File "/nix/store/whmwqp5qa1lpadz0ik6wfjgjzbbgjh1z-python3-3.12.7-env/lib/python3.12/site-packages/elftools/elf/elffile.py", line 570, in _identify_file
       >     elf_assert(magic == b'\x7fELF', 'Magic number does not match')
       >   File "/nix/store/whmwqp5qa1lpadz0ik6wfjgjzbbgjh1z-python3-3.12.7-env/lib/python3.12/site-packages/elftools/common/utils.py", line 80, in elf_assert
       >     _assert_with_exception(cond, msg, ELFError)
       >   File "/nix/store/whmwqp5qa1lpadz0ik6wfjgjzbbgjh1z-python3-3.12.7-env/lib/python3.12/site-packages/elftools/common/utils.py", line 143, in _assert_with_exception
       >     raise exception_type(msg)
       > elftools.common.exceptions.ELFError: Magic number does not match
       For full logs, run 'nix-store -l /nix/store/dinzb2x13v9ssxvp36p7m76fk35dc13a-whisper-cpp-1.7.2.drv'.
error: builder for '/nix/store/gdcg3r5pwz8m8w0vbf6jk7cy6w76jylc-whisper-cpp-1.7.2.drv' failed with exit code 1;
       last 10 log lines:
       >           ^^^^^^^^^^^^^^^
       >   File "/nix/store/whmwqp5qa1lpadz0ik6wfjgjzbbgjh1z-python3-3.12.7-env/lib/python3.12/site-packages/elftools/elf/elffile.py", line 84, in __init__
       >     self._identify_file()
       >   File "/nix/store/whmwqp5qa1lpadz0ik6wfjgjzbbgjh1z-python3-3.12.7-env/lib/python3.12/site-packages/elftools/elf/elffile.py", line 570, in _identify_file
       >     elf_assert(magic == b'\x7fELF', 'Magic number does not match')
       >   File "/nix/store/whmwqp5qa1lpadz0ik6wfjgjzbbgjh1z-python3-3.12.7-env/lib/python3.12/site-packages/elftools/common/utils.py", line 80, in elf_assert
       >     _assert_with_exception(cond, msg, ELFError)
       >   File "/nix/store/whmwqp5qa1lpadz0ik6wfjgjzbbgjh1z-python3-3.12.7-env/lib/python3.12/site-packages/elftools/common/utils.py", line 143, in _assert_with_exception
       >     raise exception_type(msg)
       > elftools.common.exceptions.ELFError: Magic number does not match
       For full logs, run 'nix-store -l /nix/store/gdcg3r5pwz8m8w0vbf6jk7cy6w76jylc-whisper-cpp-1.7.2.drv'.
error: 2 dependencies of derivation '/nix/store/dasbp7ay07cbiw4l4rp56bxfk5ink0jp-review-shell.drv' failed to build

rodfersou · 2024-12-20T03:17:21Z

@aviallon, this is the first time I've run those commands, and my environment might also be broken.

Please let me know if there is anything more I can do to help

aviallon · 2024-12-20T19:51:51Z

@rodfersou thank you very much.
I think I'll set up a Hackintosh VM in order to debug it.

rodfersou · 2024-12-21T04:34:53Z

@aviallon I run the command for full logs looks like the target macos (12) is different from the one I have (11)

note: 'whisper_encoder_impl' has been marked as being introduced in macOS 12.0 here, but the deployment target is macOS 11.0.0

[2/55] Building CXX object src/CMakeFiles/whisper.coreml.dir/coreml/whisper-encoder.mm.o
/tmp/nix-build-whisper-cpp-1.7.2.drv-1/source/src/coreml/whisper-encoder.mm:31:44: warning: 'whisper_encoder_impl' is only available on macOS 12.0 or newer [-
Wunguarded-availability-new]
    const void * data = CFBridgingRetain([[whisper_encoder_impl alloc] initWithContentsOfURL:url_model configuration:config error:nil]);
                                           ^~~~~~~~~~~~~~~~~~~~
/tmp/nix-build-whisper-cpp-1.7.2.drv-1/source/src/coreml/whisper-encoder-impl.h:41:12: note: 'whisper_encoder_impl' has been marked as being introduced in macOS 12.0 here, but the deployment target is macOS 11.0.0
@interface whisper_encoder_impl : NSObject
           ^
/tmp/nix-build-whisper-cpp-1.7.2.drv-1/source/src/coreml/whisper-encoder.mm:31:44: note: enclose 'whisper_encoder_impl' in an @available check to silence this warning
    const void * data = CFBridgingRetain([[whisper_encoder_impl alloc] initWithContentsOfURL:url_model configuration:config error:nil]);
                                           ^~~~~~~~~~~~~~~~~~~~
/tmp/nix-build-whisper-cpp-1.7.2.drv-1/source/src/coreml/whisper-encoder.mm:65:9: warning: 'whisper_encoder_implOutput' is only available on macOS 12.0 or newer [-Wunguarded-availability-new]
        whisper_encoder_implOutput * outCoreML = [(__bridge id) ctx->data predictionFromLogmel_data:inMultiArray error:nil];
        ^~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/nix-build-whisper-cpp-1.7.2.drv-1/source/src/coreml/whisper-encoder-impl.h:29:12: note: 'whisper_encoder_implOutput' has been marked as being introduced in macOS 12.0 here, but the deployment target is macOS 11.0.0
@interface whisper_encoder_implOutput : NSObject<MLFeatureProvider>
           ^
/tmp/nix-build-whisper-cpp-1.7.2.drv-1/source/src/coreml/whisper-encoder.mm:65:9: note: enclose 'whisper_encoder_implOutput' in an @available check to silence this warning
        whisper_encoder_implOutput * outCoreML = [(__bridge id) ctx->data predictionFromLogmel_data:inMultiArray error:nil];
        ^~~~~~~~~~~~~~~~~~~~~~~~~~

rodfersou · 2024-12-22T03:25:51Z

@aviallon can we try with SDK12 instead?

rodfersou · 2025-01-21T15:14:36Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 365033

rodfersou · 2025-01-21T15:16:07Z

$ nixpkgs-review pr --post-result 365033
$ git -c fetch.prune=false fetch --no-tags --force https://github.com/NixOS/nixpkgs master:refs/nixpkgs-review/0 pull/365033/head:refs/nixpkgs-review/1
$ git worktree add /Users/rodrigo/.cache/nixpkgs-review/pr-365033-5/nixpkgs 489c04e40ca17f831e45cefc4a72e8884bf7a51a
Preparing worktree (detached HEAD 489c04e40ca1)
Updating files: 100% (45779/45779), done.
HEAD is now at 489c04e40ca1 miniflux: 2.2.4 -> 2.2.5 (#375556)
$ git merge --no-commit --no-ff d97db76924bd0185e1f9aebb1ecd3520cd1acea2
Auto-merging pkgs/top-level/aliases.nix
Auto-merging pkgs/top-level/all-packages.nix
Automatic merge went well; stopped before committing as requested
Nothing to be built.

Link to currently reviewing PR:
https://github.com/NixOS/nixpkgs/pull/365033

Posting result comment on https://github.com/NixOS/nixpkgs/pull/365033

$ /nix/store/c4kw7qpffsq7vd6brwgkfnglvj85ivqk-nix-2.24.11/bin/nix-shell --argstr local-system aarch64-darwin --argstr nixpkgs-path /Users/rodrigo/.cache/nixpkgs-review/pr-365033-5/nixpkgs --argstr nixpkgs-config-path /var/folders/by/9xt1lyys0qvds8zzbx60c58m0000gn/T/tmpn9m2a4bs.nix --argstr attrs-path /Users/rodrigo/.cache/nixpkgs-review/pr-365033-5/attrs.nix --nix-path 'nixpkgs=/Users/rodrigo/.cache/nixpkgs-review/pr-365033-5/nixpkgs nixpkgs-overlays=/var/folders/by/9xt1lyys0qvds8zzbx60c58m0000gn/T/tmpc3n6y285' /nix/store/w1hgvm5rb3rblya8if90cdx5lvwyf4wc-nixpkgs-review-2.12.0/lib/python3.12/site-packages/nixpkgs_review/nix/review-shell.nix

[nix-shell:~/.cache/nixpkgs-review/pr-365033-5]$

rodfersou · 2025-01-21T15:17:22Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 365033

rodfersou · 2025-01-21T15:18:17Z

@aviallon I'm confused, not sure if it's working

rodfersou · 2025-01-21T18:22:07Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 365033

Rewrite to use CMake build Add aviallon to whisper-cpp's maintainers

aviallon · 2025-01-22T12:29:48Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 365033

`x86_64-linux`

✅ 2 packages built:

whisper-cpp
whisper-cpp-vulkan

aviallon · 2025-01-22T12:31:27Z

@rodfersou there was a very weird bug there, I re-created my branch on top of master, which should fix it.

rodfersou · 2025-01-23T01:23:59Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 365033

rodfersou · 2025-01-23T01:36:43Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 365033

aviallon · 2025-01-23T08:01:27Z

@rodfersou Your nixpkgs-review built nothing at all?!
Perhaps you should specify it which package you want to have built, explicitly.

Also, I tried to rework the PR for whisper.cpp 1.7.4, but there is a weird link issue I have to solve.

rodfersou · 2025-01-23T08:45:48Z

can you guide me how to run the command?

before the last changes was at least trying to build, now it's not anymore

lucasew · 2025-01-23T13:17:06Z

can you guide me how to run the command?

before the last changes was at least trying to build, now it's not anymore

Try with --eval local

rodfersou · 2025-01-23T15:56:34Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 365033

`aarch64-darwin`

✅ 2 packages built:

whisper-cpp
whisper-cpp-vulkan

rodfersou · 2025-01-23T15:57:47Z

@lucasew it works!!! thanks

aviallon · 2025-01-23T18:50:27Z

@rodfersou Hurra!!!

aviallon · 2025-01-23T18:52:26Z

Quite frankly, if it works with aarch64-darwin, x86_64-linux and aarch64-linux, there is a 97.9% chance it'll work on x86_64-darwin.
If someone with write access could review and merge, it'd be welcomed!

hughobrien · 2025-01-23T21:10:10Z

[nix-shell:~/.cache/nixpkgs-review/pr-365033-2/results]$ hyperfine "./whisper-cpp-x86_64-linux/bin/whisper-cpp --threads $(nproc) --language en --model ~/.local/share/whisper/ggml-medium.en.bin --file /data/whisper-test.wav"

Benchmark 1: ./whisper-cpp-x86_64-linux/bin/whisper-cpp --threads 16 --language en --model ~/.local/share/whisper/ggml-medium.en.bin --file /data/whisper-test.wav
  Time (mean ± σ):     122.031 s ±  0.253 s    [User: 1545.379 s, System: 51.247 s]
  Range (min … max):   121.624 s … 122.399 s    10 runs

The previous version with force-enabled CPU vector extensions.

hyperfine "whisper-cpp --threads $(nproc) --language en --model ~/.local/share/whisper/ggml-medium.en.bin --file /data/whisper-test.wav"
Benchmark 1: whisper-cpp --threads 16 --language en --model ~/.local/share/whisper/ggml-medium.en.bin --file /data/whisper-test.wav
  Time (mean ± σ):     136.110 s ±  4.581 s    [User: 2066.905 s, System: 6.383 s]
  Range (min … max):   129.642 s … 146.108 s    10 runs

hughobrien · 2025-01-23T21:13:57Z

However the vulkan build is still segfaulting:

[nix-shell:~/.cache/nixpkgs-review/pr-365033-2/results]$ ./whisper-cpp-x86_64-linux/bin/whisper-cpp-bench --model ~/.local/share/whisper/ggml-medium.en.bin
whisper_init_from_file_with_params_no_state: loading model from '/home/hugh/.local/share/whisper/ggml-medium.en.bin'
whisper_init_with_params_no_state: use gpu    = 1
whisper_init_with_params_no_state: flash attn = 0
whisper_init_with_params_no_state: gpu_device = 0
whisper_init_with_params_no_state: dtw        = 0
whisper_init_with_params_no_state: backends   = 1
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51864
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 1024
whisper_model_load: n_audio_head  = 16
whisper_model_load: n_audio_layer = 24
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 1024
whisper_model_load: n_text_head   = 16
whisper_model_load: n_text_layer  = 24
whisper_model_load: n_mels        = 80
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 4 (medium)
whisper_model_load: adding 1607 extra tokens
whisper_model_load: n_langs       = 99
whisper_model_load:      CPU total size =  1533.14 MB
whisper_model_load: model size    = 1533.14 MB
whisper_init_state: kv self size  =   50.33 MB
whisper_init_state: kv cross size =  150.99 MB
whisper_init_state: kv pad  size  =    6.29 MB
whisper_init_state: compute buffer (conv)   =   28.55 MB
whisper_init_state: compute buffer (encode) =  170.15 MB
whisper_init_state: compute buffer (cross)  =    7.72 MB
whisper_init_state: compute buffer (decode) =   98.18 MB

system_info: n_threads = 4 / 16 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | CUDA = 0 | COREML = 0 | OPENVINO = 0 | CANN = 0

whisper_print_timings:     load time =  1027.43 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =     0.00 ms
whisper_print_timings:   sample time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:   encode time = 19500.96 ms /     1 runs (19500.96 ms per run)
whisper_print_timings:   decode time = 10304.86 ms /   256 runs (   40.25 ms per run)
whisper_print_timings:   batchd time =  6413.53 ms /   320 runs (   20.04 ms per run)
whisper_print_timings:   prompt time = 61788.45 ms /  4096 runs (   15.09 ms per run)
whisper_print_timings:    total time = 98009.29 ms

[nix-shell:~/.cache/nixpkgs-review/pr-365033-2/results]$ ./whisper-cpp-vulkan-x86_64-linux/bin/whisper-cpp-bench --model ~/.local/share/whisper/ggml-medium.en.bin
whisper_init_from_file_with_params_no_state: loading model from '/home/hugh/.local/share/whisper/ggml-medium.en.bin'
whisper_init_with_params_no_state: use gpu    = 1
whisper_init_with_params_no_state: flash attn = 0
whisper_init_with_params_no_state: gpu_device = 0
whisper_init_with_params_no_state: dtw        = 0
ggml_vulkan: Found 1 Vulkan devices:
Segmentation fault (core dumped)

aviallon · 2025-01-23T21:45:03Z

@hughobrien your issue is probably a hardware or a driver bug.
Perhaps you could try to debug it, get a good stacktrace, and eventually post it here or on mesa.

aviallon mentioned this pull request Dec 13, 2024

Package request: openai-whisper-cpp with VULKAN support #349669

Open

aviallon force-pushed the feature/rework-whisper-cpp branch from a67bf75 to 7246230 Compare December 13, 2024 22:55

aviallon requested a review from sersorrel December 13, 2024 22:56

aviallon force-pushed the feature/rework-whisper-cpp branch 2 times, most recently from 9058dfa to d05678a Compare December 14, 2024 11:03

github-actions bot added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Dec 14, 2024

aviallon requested review from PapayaJackal and Artturin December 15, 2024 00:06

ofborg bot added 8.has: clean-up 8.has: package (new) This PR adds a new package labels Dec 15, 2024

ofborg bot requested review from dit7ya and hughobrien December 15, 2024 00:25

ofborg bot added the 11.by: package-maintainer This PR was created by the maintainer of the package it changes label Dec 15, 2024

sersorrel reviewed Dec 15, 2024

View reviewed changes

kurnevsky reviewed Dec 15, 2024

View reviewed changes

pkgs/by-name/wh/whisper-cpp/package.nix Outdated Show resolved Hide resolved

aviallon force-pushed the feature/rework-whisper-cpp branch from d05678a to bfc4257 Compare December 16, 2024 23:06

aviallon force-pushed the feature/rework-whisper-cpp branch from f72f772 to d97db76 Compare January 21, 2025 13:41

aviallon force-pushed the feature/rework-whisper-cpp branch from d97db76 to 27a56b6 Compare January 21, 2025 17:21

This comment was marked as outdated.

Sign in to view

aviallon force-pushed the feature/rework-whisper-cpp branch from 27a56b6 to 6a0cc18 Compare January 22, 2025 12:22

aviallon added 2 commits January 22, 2025 13:23

whisper-cpp: add ROCm and Vulkan support, rename from openai-whisper-cpp

2d9a69a

Rewrite to use CMake build Add aviallon to whisper-cpp's maintainers

whisper-cpp: convert to by-name

89de5a9

aviallon force-pushed the feature/rework-whisper-cpp branch from 6a0cc18 to 89de5a9 Compare January 22, 2025 12:23

rodfersou approved these changes Jan 23, 2025

View reviewed changes

wegank added the 12.approvals: 1 This PR was reviewed and approved by one reputable person label Jan 23, 2025

lucasew requested a review from GaetanLepage January 23, 2025 23:01

whisper-cpp: add ROCm and Vulkan support, rename from openai-whisper-cpp #365033

Are you sure you want to change the base?

whisper-cpp: add ROCm and Vulkan support, rename from openai-whisper-cpp #365033

Conversation

aviallon commented Dec 13, 2024 • edited Loading

Things done

aviallon commented Dec 14, 2024

nixpkgs-review result

x86_64-linux

aviallon commented Dec 14, 2024

PapayaJackal commented Dec 15, 2024

aviallon commented Dec 15, 2024

sersorrel left a comment

Choose a reason for hiding this comment

hughobrien commented Dec 16, 2024

PapayaJackal commented Dec 16, 2024

aviallon commented Dec 16, 2024

rodfersou commented Dec 20, 2024

nixpkgs-review result

aarch64-darwin

rodfersou commented Dec 20, 2024

rodfersou commented Dec 20, 2024

aviallon commented Dec 20, 2024

rodfersou commented Dec 21, 2024

rodfersou commented Dec 22, 2024

rodfersou commented Jan 21, 2025

nixpkgs-review result

rodfersou commented Jan 21, 2025

rodfersou commented Jan 21, 2025

nixpkgs-review result

rodfersou commented Jan 21, 2025

rodfersou commented Jan 21, 2025

nixpkgs-review result

This comment was marked as outdated.

This comment was marked as outdated.

aviallon commented Jan 22, 2025

nixpkgs-review result

x86_64-linux

aviallon commented Jan 22, 2025

rodfersou commented Jan 23, 2025

nixpkgs-review result

rodfersou commented Jan 23, 2025

nixpkgs-review result

aviallon commented Jan 23, 2025 • edited Loading

rodfersou commented Jan 23, 2025

lucasew commented Jan 23, 2025

rodfersou commented Jan 23, 2025

nixpkgs-review result

aarch64-darwin

rodfersou commented Jan 23, 2025

aviallon commented Jan 23, 2025

aviallon commented Jan 23, 2025 • edited Loading

hughobrien commented Jan 23, 2025

hughobrien commented Jan 23, 2025

aviallon commented Jan 23, 2025

aviallon commented Dec 13, 2024 •

edited

Loading

`nixpkgs-review` result

`x86_64-linux`

`nixpkgs-review` result

`aarch64-darwin`

`nixpkgs-review` result

`nixpkgs-review` result

`nixpkgs-review` result

`nixpkgs-review` result

`x86_64-linux`

`nixpkgs-review` result

`nixpkgs-review` result

aviallon commented Jan 23, 2025 •

edited

Loading

`nixpkgs-review` result

`aarch64-darwin`

aviallon commented Jan 23, 2025 •

edited

Loading