Project#15: GGUF Reader in OpenVINO for direct GGUF Execution (Prototype)

[shahkarnav115-beep]

Summary

The PR introduces the new gguf reader which is a temporary work-around (prototype)

related information can be found in below given links:

Proposed Plan:
Drafft-proposal.pdf

Link to thirdparty module changes (llama.cpp): ravi9/llama.cpp#55

log for testing tinnyllama.gguf:

OpenVINO: using device CPU
llama_context: OPENVINO0_HOST  output buffer size =     0.12 MiB
llama_kv_cache: layer   0: dev = OPENVINO0
llama_kv_cache: layer   1: dev = OPENVINO0
llama_kv_cache: layer   2: dev = OPENVINO0
llama_kv_cache: layer   3: dev = OPENVINO0
llama_kv_cache: layer   4: dev = OPENVINO0
llama_kv_cache: layer   5: dev = OPENVINO0
llama_kv_cache: layer   6: dev = OPENVINO0
llama_kv_cache: layer   7: dev = OPENVINO0
llama_kv_cache: layer   8: dev = OPENVINO0
llama_kv_cache: layer   9: dev = OPENVINO0
llama_kv_cache: layer  10: dev = OPENVINO0
llama_kv_cache: layer  11: dev = OPENVINO0
llama_kv_cache: layer  12: dev = OPENVINO0
llama_kv_cache: layer  13: dev = OPENVINO0
llama_kv_cache: layer  14: dev = OPENVINO0
llama_kv_cache: layer  15: dev = OPENVINO0
llama_kv_cache: layer  16: dev = OPENVINO0
llama_kv_cache: layer  17: dev = OPENVINO0
llama_kv_cache: layer  18: dev = OPENVINO0
llama_kv_cache: layer  19: dev = OPENVINO0
llama_kv_cache: layer  20: dev = OPENVINO0
llama_kv_cache: layer  21: dev = OPENVINO0
llama_kv_cache:  OPENVINO0 KV buffer size =     5.50 MiB
llama_kv_cache: size =    5.50 MiB (   256 cells,  22 layers,  1/1 seqs), K (f16):    2.75 MiB, V (f16):    2.75 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 2
sched_reserve: reserving ...
sched_reserve: max_nodes = 1608
sched_reserve: reserving full memory module
sched_reserve: worst-case: n_tokens = 1, n_seqs = 1, n_outputs = 1
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
sched_reserve: Flash Attention was auto, set to enabled
sched_reserve: resolving fused Gated Delta Net support:
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
sched_reserve: fused Gated Delta Net (autoregressive) enabled
graph_reserve: reserving a graph for ubatch with n_tokens =   16, n_seqs =  1, n_outputs =   16
sched_reserve: fused Gated Delta Net (chunked) enabled
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
sched_reserve:  OPENVINO0 compute buffer size =     0.13 MiB
sched_reserve: OPENVINO0_HOST compute buffer size =     0.01 MiB
sched_reserve: graph nodes  = 689
sched_reserve: graph splits = 1
sched_reserve: reserve took 3.67 ms, sched copies = 1
~llama_context:  OPENVINO0 compute buffer size is   0.1299 MiB, matches expectation of   0.1299 MiB
~llama_context: OPENVINO0_HOST compute buffer size is   0.0089 MiB, matches expectation of   0.0089 MiB
[       OK ] GGUFReaderV2Test.MathEquivalenceValidation (6325 ms)
[----------] 1 test from GGUFReaderV2Test (6326 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (6327 ms total)
[  PASSED  ] 1 test.

Checklist:

• This PR follows GenAI Contributing guidelines.
• Tests have been updated or added to cover the new code.
• This PR fully addresses the ticket.
• I have made corresponding changes to the documentation.

[shahkarnav115-beep]

@ravi9 some of the work for the build is left to be done. Will push the commit shortly.

[as-suvorov]

@shahkarnav115-beep Converting PR to draft as description states it's WIP

[shahkarnav115-beep]

This is the complete integration of llama.cpp ("dev_backend_openvino" - branch) in openvino.genai as a submodule (static library).

I have implemented the minimal architecture (skeleton) of GGUFReaderV2 in gguf_reader_v2.cpp and its .hpp. To verify that it successfully calls the APIs from the submodule, I have also added a minimal C++ regression test in gguf_reader_v2_test.cpp.

Locally, I have built all the required components and tested them, and this minimal scaffold is compiling and working perfectly on my system. As part of my GSoC 2026 preparation, I would be extremely grateful to get your reviews on this initial architectural integration.

cc: @ravi9, @cavusmustafa

[shahkarnav115-beep]

Link to headers file for submodule's (dev_backend_openvino branch): ravi9/llama.cpp#55

[SearchSavior]

@shahkarnav115-beep hi Karnav!

I read your proposal and am very excited to see your work!!

I run OpenArc, where your changes in this pr will be most welcome! https://github.com/SearchSavior/OpenArc

I encourage you to join our discord- we can help you test when you need feedback, other machines etc. Plus it's great to have support. Thanks for your work so far!! (And of course, your mentors as well!)

[shahkarnav115-beep]

By bypassing some of the input tensors and applying the correct causal mask, the math execution is now working. TinyLlama-1.1B-Chat-v1.0-f16.gguf is successfully running through the translated OpenVINO graph, and the output logits are matching the native llama.cpp execution within standard floating-point tolerances.

model download link: https://huggingface.co/andrijdavid/TinyLlama-1.1B-Chat-v1.0-GGUF?utm_source=chatgpt.com&show_file_info=TinyLlama-1.1B-Chat-v1.0-f16.gguf

[sgonorov]

Should be removed

[sgonorov]

Why have you removed this line?

[sgonorov]

Is this behavior ok? Maybe add some warning or avoid calling this method completely if HAS_LLAMA_CPP is enabled.

[shahkarnav115-beep]

Hii @sgonorov,

Thanks for looking into the prototype. The main aim of the prototype was to extract ggml_cgraph from llama.cpp. So, in order to make it possible, some changes were made in certain files. As this was prototypal work, it will only be used as a reference while implementing the actual parsing and translation of models from llama.cpp to OpenVINO.

The changes in the code have some known flaws which will need proper implementation during GSoC's coding period — until then it is just a proof of work. Though, to answer some doubts:

• The empty load_arrays is left intentionally so that the dummy pipeline can compile and pass through it without actually loading tensor weights, since in the V2 path, weight loading is delegated to llama.cpp internally (gguf.hpp: line 43). During actual implementation, this will either be guarded with #ifndef HAS_LLAMA_CPP or removed entirely.

• The target_compile_definitions(${TARGET_NAME_OBJ} PRIVATE ENABLE_GGUF) was removed because during debugging I found that ENABLE_GGUF was never actually reaching the C++ code as a preprocessor macro — the V2 code path was always dead. Instead of fixing that, I switched the V2 code guards to use #ifdef HAS_LLAMA_CPP, which is properly defined via CMake (line 273 of CMakeLists.txt). However, I'm aware that existing code like lora/adapter.cpp still checks #ifdef ENABLE_GGUF, so during the actual implementation this will need to be reconciled — either by restoring the definition or migrating those guards to HAS_LLAMA_CPP.

Again, Thanks for reviewing!!

[as-suvorov]

Hii @sgonorov,

Thanks for looking into the prototype. The main aim of the prototype was to extract ggml_cgraph from llama.cpp. So, in order to make it possible, some changes were made in certain files. As this was prototypal work, it will only be used as a reference while implementing the actual parsing and translation of models from llama.cpp to OpenVINO.

The changes in the code have some known flaws which will need proper implementation during GSoC's coding period — until then it is just a proof of work. Though, to answer some doubts:

• The empty load_arrays is left intentionally so that the dummy pipeline can compile and pass through it without actually loading tensor weights, since in the V2 path, weight loading is delegated to llama.cpp internally (gguf.hpp: line 43). During actual implementation, this will either be guarded with #ifndef HAS_LLAMA_CPP or removed entirely.
• The target_compile_definitions(${TARGET_NAME_OBJ} PRIVATE ENABLE_GGUF) was removed because during debugging I found that ENABLE_GGUF was never actually reaching the C++ code as a preprocessor macro — the V2 code path was always dead. Instead of fixing that, I switched the V2 code guards to use #ifdef HAS_LLAMA_CPP, which is properly defined via CMake (line 273 of CMakeLists.txt). However, I'm aware that existing code like lora/adapter.cpp still checks #ifdef ENABLE_GGUF, so during the actual implementation this will need to be reconciled — either by restoring the definition or migrating those guards to HAS_LLAMA_CPP.

Again, Thanks for reviewing!!

As it's a prototype converting pr to draft

[github-actions]

This PR will be closed in a week because of 2 weeks of no activity.

[github-actions]

This PR will be closed in a week because of 2 weeks of no activity.

[github-actions]

Closing due to inactivity. Feel free to reopen if you plan to continue working on this.