Integration with Unreal Engine 5.5.4

[gomidas]

Hello,

I had an idea to put this into unreal engine. I do not know if this is possible.

1 ]
Here is my Cmake command to make it work for Visual Studio 2022 and Unreal Engine (I also had to compile libcurl to make it compile for VS):

cmake .. -G "Visual Studio 17 2022" -A x64 -T version=14.38.33130 ^
 -DCMAKE_BUILD_TYPE=Release ^
 -DBUILD_SHARED_LIBS=OFF ^
 -DCURL_INCLUDE_DIR="...FULL_PATH_TO_LIBCURL.../include/" ^
 -DCMAKE_CXX_FLAGS_RELEASE="/MD /O2 /DNDEBUG" ^
 -DCMAKE_C_FLAGS_RELEASE="/MD" ^
  -DCMAKE_BUILD_TYPE=Release ^
  -DBUILD_SHARED_LIBS=OFF ^
  -DLLAMA_BUILD_TESTS=OFF ^
  -DLLAMA_BUILD_EXAMPLES=OFF ^
  -DLLAMA_BUILD_COMMON=OFF ^
  -DLLAMA_BUILD_SERVER=OFF ^
  -DCURL_USE_SCHANNEL=ON ^
  -DCURL_USE_OPENSSL=OFF ^
  -DCURL_STATICLIB=ON ^
  -DCURL_LIBRARY="...FULL_PATH_TO_LIBCURL.../libcurl.lib" 

2]

I've made a plugin and put ".lib" files and llama.cpp-master files there. Build.cs looks like this:
        if (Target.Platform == UnrealTargetPlatform.Win64)
        {
            string PluginPath = Path.GetFullPath(ModuleDirectory);
            string SourceFile_LLAMA_LIB_X64 = Path.Combine(PluginPath, "..", "ThirdParty", "Libraries", "windows", "x64", "llama.lib");

            string SourceFile_GGML_LIB_X64 = Path.Combine(PluginPath, "..", "ThirdParty", "Libraries", "windows", "x64", "ggml.lib");

            string SourceFile_GGML_BASE_LIB_X64 = Path.Combine(PluginPath, "..", "ThirdParty", "Libraries", "windows", "x64", "ggml-base.lib");
            string SourceFile_GGML_CPU_LIB_X64 = Path.Combine(PluginPath, "..", "ThirdParty", "Libraries", "windows", "x64", "ggml-cpu.lib");

            string INCLUDE_PATH_LLAMA = Path.Combine(PluginPath, "..", "ThirdParty", "llama.cpp-master", "include");
            string INCLUDE_PATH_GGML = Path.Combine(PluginPath, "..", "ThirdParty", "llama.cpp-master", "ggml", "include");

            string SRC_PATH_LLAMA = Path.Combine(PluginPath, "..", "ThirdParty", "llama.cpp-master", "src"); 
            string SRC_PATH_GGML = Path.Combine(PluginPath, "..", "ThirdParty", "llama.cpp-master", "ggml", "src"); 

            PublicIncludePaths.AddRange(new string[] { INCLUDE_PATH_LLAMA,INCLUDE_PATH_GGML,SRC_PATH_LLAMA,SRC_PATH_GGML });

            // PublicAdditionalLibraries.Add(INCLUDE_PATH_LLAMA);
            // PublicAdditionalLibraries.Add(INCLUDE_PATH_GGML);
            // (Opsiyonel) Kütüphanenin bulunduğu dizini ekleyebilirsiniz.
            //PublicIncludePaths.Add(SourceFile_LLAMA_LIB_X64);

            //LLAMACPP SETTINGS
            // Derleme seçenekleri
            //PublicDefinitions.Add("GGML_NO_SIMD"); // if needed
            OptimizeCode = CodeOptimization.Never;
            PublicDefinitions.Add("NDEBUG");
            PublicDefinitions.Add("WITH_LLAMA=1");
            //PublicDefinitions.Add("GGML_NO_ACCEL=1");// if needed
            //PublicDefinitions.Add("GGML_STATIC=1");
            //PublicDefinitions.Add("LLAMA_STATIC=1");
            bEnableExceptions = false;
            bUseRTTI = false;
            //LLAMACPP SETTINGS
            //PublicDefinitions.Add("__STDC_VERSION__=201112L");

            if (File.Exists(SourceFile_LLAMA_LIB_X64))
            {
                PublicAdditionalLibraries.Add(SourceFile_LLAMA_LIB_X64);
                System.Console.WriteLine("Found");
            }
            else
            {
                System.Console.WriteLine("Not Found");
            }

            if (File.Exists(SourceFile_GGML_LIB_X64))
            {
                PublicAdditionalLibraries.Add(SourceFile_GGML_LIB_X64);
                System.Console.WriteLine("Found");
            }
            else
            {
                System.Console.WriteLine("Not Found");
            }

            if (File.Exists(SourceFile_GGML_BASE_LIB_X64))
            {
                PublicAdditionalLibraries.Add(SourceFile_GGML_BASE_LIB_X64);
                System.Console.WriteLine("Found");
            }
            else
            {
                System.Console.WriteLine("Not Found");
            }

            if (File.Exists(SourceFile_GGML_CPU_LIB_X64))
            {
                PublicAdditionalLibraries.Add(SourceFile_GGML_CPU_LIB_X64);
                System.Console.WriteLine("Found");
            }
            else
            {
                System.Console.WriteLine("Not Found");
            }

        }

3]
After all inside plugin I tried this:

	llama_model_params model_params = llama_model_default_params();
	const char* path = "S:\\...FULL-PATH...\\phi-2.Q8_0.gguf";
	llama_model* model = llama_model_load_from_file(path, model_params);
	if (model== NULL)
	{ return "error"; }

llama_model_load_from_file(...) does not load any .gguf file that returns NULL.

I suspected that I need to call ggml_backend_load_all(); However, when I check it with code:

	try
	{
		
		ggml_backend_load_all();
	}
	catch (const std::exception& ex)
	{
		UE_LOG(LogTemp, Error, TEXT("Exception: %s"), *FString(ex.what()));
	}

I see:

LogTemp: Error: GGML_BACKEND_PATH environment variable is not set! // From module file check --> 	FString GGMLBackendPath = FPlatformMisc::GetEnvironmentVariable(TEXT("GGML_BACKEND_PATH"));
LogTemp: Error: Exception: Failed to retrieve GGML_BACKEND_PATH environment variable.

What I did wrong here?

[gomidas]

Later I used compiled project in different project and I get the problem. Here: #13132