happyz
/
llama.cpp
mirror of https://github.com/ggerganov/llama.cpp.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
llama.cpp/docs/backend/GGML-VirtGPU/configuration.md

6.7 KiB
Raw Blame History

GGML-VirtGPU Backend Configuration

This document describes the environment variables used by the ggml-virtgpu backend system, covering both the frontend (guest-side) and backend (host-side) components.

Environment Variables Overview

The ggml-virtgpu backend uses environment variables for configuration across three main components:

  • Frontend (Guest): GGML applications running in VMs
  • Hypervisor: Virglrenderer/APIR system
  • Backend (Host): Host-side GGML backend integration

Frontend (Guest-side) Configuration

GGML_REMOTING_USE_APIR_CAPSET

  • Location: ggml/src/ggml-virtgpu/virtgpu.cpp
  • Type: Boolean flag (presence-based)
  • Purpose: Controls which virtio-gpu capability set to use for communication
  • Values:
    • Set (any value): Use the APIR capset (long-term setup)
    • Unset: Use the Venus capset (easier for testing with an unmodified hypervisor)
  • Default: Unset (Venus capset)
  • Usage: 
    export GGML_REMOTING_USE_APIR_CAPSET=1  # Use APIR capset
# or leave unset for Venus capset

Hypervisor (Virglrenderer/APIR) Configuration

These environment variables are used during the transition phase for running with an unmodified hypervisor (not supporting the VirglRenderer APIR component). They will be removed in the future, and the hypervisor will instead configure VirglRenderer with the APIR Configuration Key.

VIRGL_APIR_BACKEND_LIBRARY

  • Location: virglrenderer/src/apir/apir-context.c
  • Configuration Key: apir.load_library.path
  • Type: File path string
  • Purpose: Path to the APIR backend library that virglrenderer should dynamically load
  • Required: Yes
  • Example: 
    export VIRGL_APIR_BACKEND_LIBRARY="/path/to/libggml-remotingbackend.so"

VIRGL_ROUTE_VENUS_TO_APIR

  • Location: virglrenderer/src/apir/apir-renderer.h
  • Type: Boolean flag (presence-based)
  • Purpose: Temporary workaround to route Venus capset calls to APIR during hypervisor transition period
  • Status: will be removed once hypervisors support APIR natively
  • Warning: Breaks normal Vulkan/Venus functionality
  • Usage: 
    export VIRGL_ROUTE_VENUS_TO_APIR=1  # For testing with an unmodified hypervisor

VIRGL_APIR_LOG_TO_FILE

  • Location: virglrenderer/src/apir/apir-renderer.c
  • Environment Variable: VIRGL_APIR_LOG_TO_FILE
  • Type: File path string
  • Purpose: Enable debug logging from the VirglRenderer APIR component to specified file
  • Required: No (optional debugging)
  • Default: Logging to stderr
  • Usage: 
    export VIRGL_APIR_LOG_TO_FILE="/tmp/apir-debug.log"

Backend (Host-side) Configuration

These environment variables are used during the transition phase for running with an unmodified hypervisor (not supporting the VirglRenderer APIR component). They will be removed in the future, and the hypervisor will instead configure VirglRenderer with the APIR Configuration Key.

APIR_LLAMA_CPP_GGML_LIBRARY_PATH

  • Location: ggml/src/ggml-virtgpu/backend/backend.cpp
  • Environment Variable: APIR_LLAMA_CPP_GGML_LIBRARY_PATH
  • Configuration Key: ggml.library.path
  • Type: File path string
  • Purpose: Path to the actual GGML backend library (Metal, CUDA, Vulkan, etc.)
  • Required: Yes - backend initialization fails without this
  • Examples: 
    # macOS with Metal backend
export APIR_LLAMA_CPP_GGML_LIBRARY_PATH="/opt/llama.cpp/lib/libggml-metal.dylib"

# Linux with CUDA backend
export APIR_LLAMA_CPP_GGML_LIBRARY_PATH="/opt/llama.cpp/lib/libggml-cuda.so"

# macOS or Linux with Vulkan backend
export APIR_LLAMA_CPP_GGML_LIBRARY_PATH="/opt/llama.cpp/lib/libggml-vulkan.so"

APIR_LLAMA_CPP_GGML_LIBRARY_REG

  • Location: ggml/src/ggml-virtgpu/backend/backend.cpp
  • Environment Variable: APIR_LLAMA_CPP_GGML_LIBRARY_REG
  • Configuration Key: ggml.library.reg
  • Type: Function symbol name string
  • Purpose: Name of the backend registration function to call after loading the library
  • Required: No (defaults to ggml_backend_init)
  • Default: ggml_backend_init
  • Examples: 
    # Metal backend
export APIR_LLAMA_CPP_GGML_LIBRARY_REG="ggml_backend_metal_reg"

# CUDA backend
export APIR_LLAMA_CPP_GGML_LIBRARY_REG="ggml_backend_cuda_reg"

# Vulkan backend
export APIR_LLAMA_CPP_GGML_LIBRARY_REG="ggml_backend_vulkan_reg"

# Generic fallback (default)
# export APIR_LLAMA_CPP_GGML_LIBRARY_REG="ggml_backend_init"

APIR_LLAMA_CPP_LOG_TO_FILE

  • Location: ggml/src/ggml-virtgpu/backend/backend.cpp:62
  • Environment Variable: APIR_LLAMA_CPP_LOG_TO_FILE
  • Type: File path string
  • Purpose: Enable debug logging from the GGML backend to specified file
  • Required: No (optional debugging)
  • Usage: 
    export APIR_LLAMA_CPP_LOG_TO_FILE="/tmp/ggml-backend-debug.log"

Configuration Flow

The configuration system works as follows:

  1. Hypervisor Setup: Virglrenderer loads the APIR backend library specified by VIRGL_APIR_BACKEND_LIBRARY

  2. Context Creation: When an APIR context is created, it populates a configuration table with environment variables:

    • apir.load_library.path ← VIRGL_APIR_BACKEND_LIBRARY
    • ggml.library.path ← APIR_LLAMA_CPP_GGML_LIBRARY_PATH
    • ggml.library.reg ← APIR_LLAMA_CPP_GGML_LIBRARY_REG
    • ggml.library.init ← APIR_LLAMA_CPP_GGML_LIBRARY_INIT
    • this step will eventually be performed by the hypervisor itself, with command-line arguments instead of environment variables.

  3. Backend Initialization: The backend queries the configuration via callbacks:

    • virgl_cbs->get_config(ctx_id, "ggml.library.path") returns the library path
    • virgl_cbs->get_config(ctx_id, "ggml.library.reg") returns the registration function

  4. Library Loading: The backend dynamically loads and initializes the specified GGML library

Error Messages

Common error scenarios and their messages:

  • Missing library path: "cannot open the GGML library: env var 'APIR_LLAMA_CPP_GGML_LIBRARY_PATH' not defined"
  • Missing registration function: "cannot register the GGML library: env var 'APIR_LLAMA_CPP_GGML_LIBRARY_REG' not defined"

Example Complete Configuration

Here's an example configuration for a macOS host with Metal backend:

# Hypervisor environment
export VIRGL_APIR_BACKEND_LIBRARY="/opt/llama.cpp/lib/libggml-virtgpu-backend.dylib"

# Backend configuration
export APIR_LLAMA_CPP_GGML_LIBRARY_PATH="/opt/llama.cpp/lib/libggml-metal.dylib"
export APIR_LLAMA_CPP_GGML_LIBRARY_REG="ggml_backend_metal_reg"

# Optional logging
export VIRGL_APIR_LOG_TO_FILE="/tmp/apir.log"
export APIR_LLAMA_CPP_LOG_TO_FILE="/tmp/ggml.log"

# Guest configuration
export GGML_REMOTING_USE_APIR_CAPSET=1