NPU convertation

[ha-seungwon]

Hello,

How do I have a conversation with NPU?

rknn-llm might use rkllm-runtime/Linux/librkllm_api/aarch64/librkllmrt.so to npu kernel setting.

But is there any open-source code to test npu calculation?

[waydong]

Hi, this is a demo, I'm not sure if it's what you need：
https://github.com/airockchip/rknn-llm/tree/main/examples/rkllm_api_demo

[ha-seungwon]

Thanks for the reply :)

But my question is slightly different.

https://github.com/airockchip/rknn-llm/tree/main/examples/rkllm_api_demo demo using the "rkllm.h" to initialize the new setting or memory setting for calculation and also include the calculation part.

So, my question is where can I find the actual running code in rkllm-runtime/Linux/librkllm_api/include/rkllm.h in this header file?

[ha-seungwon]

How can I find or set the npu core??

Test code

RKLLM doc

ERROR

[waydong]

How can I find or set the npu core??

Test code

RKLLM doc ERROR

num_npu_core is set during the model conversion process; however, the documentation reflecting this configuration has not been updated yet.

[ha-seungwon]

How does the NPU kernel perform matrix multiplication across multiple cores, and what optimizations are necessary to ensure that matrix multiplication efficiently utilizes all available cores on a multicore NPU?

[c0zaut]

@ha-seungwon - all of my models are converted with multicore support, which I set during the conversion process as @waydong points out. To monitor the npu core usage, run the following command as root:

watch -n 1 'cat /sys/kernel/debug/rknpu/load'

Running that command shows NPU cores running at top performance.

Models are here in a variety of quants of optimizations: https://huggingface.co/c01zaut

[ha-seungwon]

@c0zaut Yes, I know how to monitor the NPU core usage.

My question is whether there is another way to monitor or detect NPU utilization during real-time inference after converting to an RKLLM model file, such as through memory mapping or matrix calculations