Skip to content

Expose string metadata fetching methods #700

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 3, 2025

Conversation

AsbjornOlling
Copy link
Contributor

@AsbjornOlling AsbjornOlling commented Mar 26, 2025

Specifically, it implements these functions (excerpt from llama.h):

    // Functions to access the model's GGUF metadata scalar values
    // - The functions return the length of the string on success, or -1 on failure
    // - The output string is always null-terminated and cleared on failure
    // - When retrieving a string, an extra byte must be allocated to account for the null terminator
    // - GGUF array values are not supported by these functions

    // Get metadata value as a string by key name
    LLAMA_API int32_t llama_model_meta_val_str(const struct llama_model * model, const char * key, char * buf, size_t buf_size);

    // Get the number of metadata key/value pairs
    LLAMA_API int32_t llama_model_meta_count(const struct llama_model * model);

    // Get metadata key name by index
    LLAMA_API int32_t llama_model_meta_key_by_index(const struct llama_model * model, int32_t i, char * buf, size_t buf_size);

    // Get metadata value as a string by index
    LLAMA_API int32_t llama_model_meta_val_str_by_index(const struct llama_model * model, int32_t i, char * buf, size_t buf_size);

I tested it using a quick script that looks like this:

use llama_cpp_2::llama_backend::LlamaBackend;
use llama_cpp_2::model::LlamaModel;
use llama_cpp_2::model::params::LlamaModelParams;

fn main() {
    let args: Vec<String> = std::env::args().collect();
    if args.len() < 2 {
        println!("Remember to pass a model path, you doofus!");
    }
    let model_path = &args[1];

    let backend = LlamaBackend::init().expect("Failed to initialize llama backend");
    let params = LlamaModelParams::default();
    let model =
        LlamaModel::load_from_file(&backend, model_path, &params).expect("Failed loading model");

    for i in 0..model.meta_count() {
        let key = model
            .meta_key_by_index(i)
            .expect("Could not find key at index");

        if key.starts_with("tokenizer.chat_template") {
            println!("--- {key} ---");
            let template = model
                .meta_val_str_by_index(i)
                .expect("Could not find value at index");
            let test_templ = model
                .meta_val_str(&key)
                .expect("Could not find value at key");
            assert_eq!(template, test_templ);
            println!("{template}");
        }
    }
}

@MarcusDunn
Copy link
Contributor

MarcusDunn commented Mar 29, 2025

@AsbjornOlling I will merge once the conflicts are resolved and checks (build on mac and test on linux) pass

@AsbjornOlling AsbjornOlling force-pushed the model-meta-val-stuff branch from 71ba4ba to c33b4a5 Compare April 3, 2025 12:15
@AsbjornOlling
Copy link
Contributor Author

AsbjornOlling commented Apr 3, 2025

@MarcusDunn I rebased on main.

It builds, passes tests, and runs fine on my linux machine. There are some warnings, but none that are introduced by this PR.

@MarcusDunn MarcusDunn merged commit ca28629 into utilityai:main Apr 3, 2025
2 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants