LLAMACTL_DATABASE_MAX_IDLE_CONNECTIONS - Maximum idle database connectionsLLAMACTL_DATABASE_CONN_MAX_LIFETIME - Connection max lifetime (e.g., "5m", "1h")
llamactl supports two types of authentication:
+auth:
require_inference_auth: true # Require API key for OpenAI endpoints (default: true)
- inference_keys: [] # List of valid inference API keys
- require_management_auth: true # Require API key for management endpoints (default: true)
- management_keys: [] # List of valid management API keys
+ require_management_auth: true # Require API key for management endpoints (default: true)
+ management_keys: [] # List of valid management API keys
Managing Inference API Keys:
+Inference API keys are managed through the web UI or management API and stored in the database. To create and manage inference keys:
+Environment Variables:
- LLAMACTL_REQUIRE_INFERENCE_AUTH - Require auth for OpenAI endpoints (true/false)
-- LLAMACTL_INFERENCE_KEYS - Comma-separated inference API keys
- LLAMACTL_REQUIRE_MANAGEMENT_AUTH - Require auth for management endpoints (true/false)
- LLAMACTL_MANAGEMENT_KEYS - Comma-separated management API keys
Llamactl uses two types of API keys:
/v1/chat/completions, /v1/completions, etc.). /v1/chat/completions, /v1/completions, etc.). These are created and managed via the web UI. By default, authentication is required. If you don't configure these keys in your configuration file, llamactl will auto-generate them and print them to the terminal on startup. You can also configure custom keys or disable authentication entirely in the Configuration guide.
+By default, authentication is required for both management and inference endpoints. You can configure custom management keys or disable authentication in the Configuration guide.
Start the Llamactl server:
llamactl
@@ -789,22 +798,15 @@
sk-management-...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-⚠️ INFERENCE AUTHENTICATION REQUIRED
+⚠️ IMPORTANT
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-🔑 Generated Inference API Key:
-
- sk-inference-...
-
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-⚠️ IMPORTANT
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-• These keys are auto-generated and will change on restart
-• For production, add explicit keys to your configuration
-• Copy these keys before they disappear from the terminal
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-Llamactl server listening on 0.0.0.0:8080
+• This key is auto-generated and will change on restart
+• For production, add explicit management_keys to your configuration
+• Copy this key before it disappears from the terminal
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+Llamactl server listening on 0.0.0.0:8080
Copy the Management and Inference API Keys from the terminal - you'll need them to access the web UI and make inference requests.
+Copy the Management API Key from the terminal - you'll need it to access the web UI.
By default, Llamactl will start on http://localhost:8080.
Open your web browser and navigate to:
@@ -826,7 +828,7 @@Auto-Assignment
-Llamactl automatically assigns ports from the configured port range (default: 8000-9000) and generates API keys if authentication is enabled. You typically don't need to manually specify these values.
+Llamactl automatically assigns ports from the configured port range (default: 8000-9000) and manages API keys if authentication is enabled. You typically don't need to manually specify these values.
Remote Node Deployment
@@ -845,6 +847,21 @@To make inference requests to your instances, you'll need an inference API key:
+The key will look like: llamactl-...
You can create multiple inference keys with different permissions for different use cases (e.g., one for development, one for production, or keys limited to specific instances).
Here are basic example configurations for each backend:
llama.cpp backend:
@@ -966,7 +983,7 @@
API Key
-If you disabled authentication in your config, you can use any value for api_key (e.g., "not-needed"). Otherwise, use the inference API key shown in the terminal output on startup.
If you disabled authentication in your config, you can use any value for api_key (e.g., "not-needed"). Otherwise, use the inference API key you created via the web UI (Settings → API Keys).
Get a list of running instances (models) in OpenAI-compatible format:
@@ -998,7 +1015,7 @@ - October 26, 2025 + December 8, 2025 diff --git a/dev/search/search_index.json b/dev/search/search_index.json index d7d5e0b..0ac7796 100644 --- a/dev/search/search_index.json +++ b/dev/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Llamactl Documentation","text":"Welcome to the Llamactl documentation!
"},{"location":"#what-is-llamactl","title":"What is Llamactl?","text":"
Unified management and routing for llama.cpp, MLX and vLLM models with web dashboard.
"},{"location":"#features","title":"Features","text":"\ud83d\ude80 Easy Model Management - Multiple Models Simultaneously: Run different models at the same time (7B for speed, 70B for quality) - Smart Resource Management: Automatic idle timeout, LRU eviction, and configurable instance limits - Web Dashboard: Modern React UI for managing instances, monitoring health, and viewing logs
\ud83d\udd17 Flexible Integration - OpenAI API Compatible: Drop-in replacement - route requests to different models by instance name - Multi-Backend Support: Native support for llama.cpp, MLX (Apple Silicon optimized), and vLLM - Docker Ready: Run backends in containers with full GPU support
\ud83c\udf10 Distributed Deployment - Remote Instances: Deploy instances on remote hosts - Central Management: Manage everything from a single dashboard with automatic routing
"},{"location":"#quick-links","title":"Quick Links","text":"If you need help or have questions:
MIT License - see the LICENSE file.
"},{"location":"api-reference/","title":"API Reference","text":""},{"location":"api-reference/#llamactl-api-10","title":"llamactl API 1.0","text":"llamactl is a control server for managing Llama Server instances.
License: MIT License"},{"location":"api-reference/#keys","title":"Keys","text":""},{"location":"api-reference/#get-apiv1authkeys","title":"GET /api/v1/auth/keys","text":"List all API keys
DescriptionReturns a list of all API keys for the system user (excludes key hash and plain-text key)
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 500 Internal Server Error
"},{"location":"api-reference/#post-apiv1authkeys","title":"POST /api/v1/auth/keys","text":"Create a new API key
DescriptionCreates a new API key with the specified permissions and returns the plain- text key (only shown once)
Input parameters
Parameter In Type Default Nullable Descriptionkey body None No API key configuration Response 201 Created
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#delete-apiv1authkeysid","title":"DELETE /api/v1/auth/keys/{id}","text":"Delete an API key
DescriptionDeletes an API key by ID
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No id path None No Key ID Response 204 No Content
Response 400 Bad Request
Response 404 Not Found
Response 500 Internal Server Error
"},{"location":"api-reference/#get-apiv1authkeysid","title":"GET /api/v1/auth/keys/{id}","text":"Get details of a specific API key
DescriptionReturns details for a specific API key by ID (excludes key hash and plain- text key)
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No id path None No Key ID Response 200 OK
Response 400 Bad Request
Response 404 Not Found
Response 500 Internal Server Error
"},{"location":"api-reference/#get-apiv1authkeysidpermissions","title":"GET /api/v1/auth/keys/{id}/permissions","text":"Get API key permissions
DescriptionReturns the instance-level permissions for a specific API key (includes instance names)
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No id path None No Key ID Response 200 OK
Response 400 Bad Request
Response 404 Not Found
Response 500 Internal Server Error
"},{"location":"api-reference/#backends","title":"Backends","text":""},{"location":"api-reference/#get-apiv1backendsllama-cppdevices","title":"GET /api/v1/backends/llama-cpp/devices","text":"List available devices for llama server
DescriptionReturns a list of available devices for the llama server
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 500 Internal Server Error
"},{"location":"api-reference/#get-apiv1backendsllama-cpphelp","title":"GET /api/v1/backends/llama-cpp/help","text":"Get help for llama server
DescriptionReturns the help text for the llama server command
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 500 Internal Server Error
"},{"location":"api-reference/#post-apiv1backendsllama-cppparse-command","title":"POST /api/v1/backends/llama-cpp/parse-command","text":"Parse llama-server command
DescriptionParses a llama-server command string into instance options
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No request body None No Command to parse Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#get-apiv1backendsllama-cppversion","title":"GET /api/v1/backends/llama-cpp/version","text":"Get version of llama server
DescriptionReturns the version of the llama server command
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 500 Internal Server Error
"},{"location":"api-reference/#post-apiv1backendsmlxparse-command","title":"POST /api/v1/backends/mlx/parse-command","text":"Parse mlx_lm.server command
DescriptionParses MLX-LM server command string into instance options
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No request body None No Command to parse Response 200 OK
Response 400 Bad Request
"},{"location":"api-reference/#post-apiv1backendsvllmparse-command","title":"POST /api/v1/backends/vllm/parse-command","text":"Parse vllm serve command
DescriptionParses a vLLM serve command string into instance options
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No request body None No Command to parse Response 200 OK
Response 400 Bad Request
"},{"location":"api-reference/#system","title":"System","text":""},{"location":"api-reference/#get-apiv1config","title":"GET /api/v1/config","text":"Get server configuration
DescriptionReturns the current server configuration (sanitized)
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 500 Internal Server Error
"},{"location":"api-reference/#get-apiv1version","title":"GET /api/v1/version","text":"Get llamactl version
DescriptionReturns the version of the llamactl command
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 500 Internal Server Error
"},{"location":"api-reference/#instances","title":"Instances","text":""},{"location":"api-reference/#get-apiv1instances","title":"GET /api/v1/instances","text":"List all instances
DescriptionReturns a list of all instances managed by the server
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 500 Internal Server Error
"},{"location":"api-reference/#delete-apiv1instancesname","title":"DELETE /api/v1/instances/{name}","text":"Delete an instance
DescriptionStops and removes a specific instance by name
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 204 No Content
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#get-apiv1instancesname","title":"GET /api/v1/instances/{name}","text":"Get details of a specific instance
DescriptionReturns the details of a specific instance by name
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-apiv1instancesname","title":"POST /api/v1/instances/{name}","text":"Create and start a new instance
DescriptionCreates a new instance with the provided configuration options
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name options body None No Instance configuration options Response 201 Created
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#put-apiv1instancesname","title":"PUT /api/v1/instances/{name}","text":"Update an instance's configuration
DescriptionUpdates the configuration of a specific instance by name
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name options body None No Instance configuration options Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#get-apiv1instancesnamelogs","title":"GET /api/v1/instances/{name}/logs","text":"Get logs from a specific instance
DescriptionReturns the logs from a specific instance by name with optional line limit
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No lines query None No Number of lines to retrieve (default: all lines) name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#get-apiv1instancesnameproxy","title":"GET /api/v1/instances/{name}/proxy","text":"Proxy requests to a specific instance, does not autostart instance if stopped
DescriptionForwards HTTP requests to the llama-server instance running on a specific port
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
Response 503 Service Unavailable
"},{"location":"api-reference/#post-apiv1instancesnameproxy","title":"POST /api/v1/instances/{name}/proxy","text":"Proxy requests to a specific instance, does not autostart instance if stopped
DescriptionForwards HTTP requests to the llama-server instance running on a specific port
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
Response 503 Service Unavailable
"},{"location":"api-reference/#post-apiv1instancesnamerestart","title":"POST /api/v1/instances/{name}/restart","text":"Restart a running instance
DescriptionRestarts a specific instance by name
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-apiv1instancesnamestart","title":"POST /api/v1/instances/{name}/start","text":"Start a stopped instance
DescriptionStarts a specific instance by name
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-apiv1instancesnamestop","title":"POST /api/v1/instances/{name}/stop","text":"Stop a running instance
DescriptionStops a specific instance by name
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#nodes","title":"Nodes","text":""},{"location":"api-reference/#get-apiv1nodes","title":"GET /api/v1/nodes","text":"List all configured nodes
DescriptionReturns a map of all nodes configured in the server (node name -> node config)
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 500 Internal Server Error
"},{"location":"api-reference/#get-apiv1nodesname","title":"GET /api/v1/nodes/{name}","text":"Get details of a specific node
DescriptionReturns the details of a specific node by name
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Node Name Response 200 OK
Response 400 Bad Request
Response 404 Not Found
Response 500 Internal Server Error
"},{"location":"api-reference/#llamacpp","title":"Llama.cpp","text":""},{"location":"api-reference/#get-llama-cppname","title":"GET /llama-cpp/{name}/","text":"Proxy requests to llama.cpp UI for the instance
DescriptionProxies requests to the llama.cpp UI for the specified instance
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name query None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-llama-cppnameapply-template","title":"POST /llama-cpp/{name}/apply-template","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-llama-cppnamecompletion","title":"POST /llama-cpp/{name}/completion","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-llama-cppnamedetokenize","title":"POST /llama-cpp/{name}/detokenize","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-llama-cppnameembeddings","title":"POST /llama-cpp/{name}/embeddings","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-llama-cppnameinfill","title":"POST /llama-cpp/{name}/infill","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-llama-cppnamemetrics","title":"POST /llama-cpp/{name}/metrics","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#get-llama-cppnameprops","title":"GET /llama-cpp/{name}/props","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-llama-cppnameprops","title":"POST /llama-cpp/{name}/props","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-llama-cppnamereranking","title":"POST /llama-cpp/{name}/reranking","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#get-llama-cppnameslots","title":"GET /llama-cpp/{name}/slots","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-llama-cppnametokenize","title":"POST /llama-cpp/{name}/tokenize","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#openai","title":"OpenAI","text":""},{"location":"api-reference/#post-v1","title":"POST /v1/","text":"OpenAI-compatible proxy endpoint
DescriptionHandles all POST requests to /v1/*, routing to the appropriate instance based on the request body. Requires API key authentication via the Authorization header.
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#get-v1models","title":"GET /v1/models","text":"List instances in OpenAI-compatible format
DescriptionReturns a list of instances in a format compatible with OpenAI API
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 500 Internal Server Error
"},{"location":"configuration/","title":"Configuration","text":"llamactl can be configured via configuration files or environment variables. Configuration is loaded in the following order of precedence:
Defaults < Configuration file < Environment variables\nllamactl works out of the box with sensible defaults, but you can customize the behavior to suit your needs.
"},{"location":"configuration/#default-configuration","title":"Default Configuration","text":"Here's the default configuration with all available options:
server:\n host: \"0.0.0.0\" # Server host to bind to\n port: 8080 # Server port to bind to\n allowed_origins: [\"*\"] # Allowed CORS origins (default: all)\n allowed_headers: [\"*\"] # Allowed CORS headers (default: all)\n enable_swagger: false # Enable Swagger UI for API docs\n\nbackends:\n llama-cpp:\n command: \"llama-server\"\n args: []\n environment: {} # Environment variables for the backend process\n docker:\n enabled: false\n image: \"ghcr.io/ggml-org/llama.cpp:server\"\n args: [\"run\", \"--rm\", \"--network\", \"host\", \"--gpus\", \"all\"]\n environment: {}\n response_headers: {} # Additional response headers to send with responses\n\n vllm:\n command: \"vllm\"\n args: [\"serve\"]\n environment: {} # Environment variables for the backend process\n docker:\n enabled: false\n image: \"vllm/vllm-openai:latest\"\n args: [\"run\", \"--rm\", \"--network\", \"host\", \"--gpus\", \"all\", \"--shm-size\", \"1g\"]\n environment: {}\n response_headers: {} # Additional response headers to send with responses\n\n mlx:\n command: \"mlx_lm.server\"\n args: []\n environment: {} # Environment variables for the backend process\n response_headers: {} # Additional response headers to send with responses\n\ndata_dir: ~/.local/share/llamactl # Main data directory (database, instances, logs), default varies by OS\n\ninstances:\n port_range: [8000, 9000] # Port range for instances\n configs_dir: data_dir/instances # Instance configs directory\n logs_dir: data_dir/logs # Logs directory\n auto_create_dirs: true # Auto-create data/config/logs dirs if missing\n max_instances: -1 # Max instances (-1 = unlimited)\n max_running_instances: -1 # Max running instances (-1 = unlimited)\n enable_lru_eviction: true # Enable LRU eviction for idle instances\n default_auto_restart: true # Auto-restart new instances by default\n default_max_restarts: 3 # Max restarts for new instances\n default_restart_delay: 5 # Restart delay (seconds) for new instances\n default_on_demand_start: true # Default on-demand start setting\n on_demand_start_timeout: 120 # Default on-demand start timeout in seconds\n timeout_check_interval: 5 # Idle instance timeout check in minutes\n\ndatabase:\n path: data_dir/llamactl.db # Database file path\n max_open_connections: 25 # Maximum open database connections\n max_idle_connections: 5 # Maximum idle database connections\n connection_max_lifetime: 5m # Connection max lifetime\n\nauth:\n require_inference_auth: true # Require auth for inference endpoints\n inference_keys: [] # Keys for inference endpoints\n require_management_auth: true # Require auth for management endpoints\n management_keys: [] # Keys for management endpoints\n\nlocal_node: \"main\" # Name of the local node (default: \"main\")\nnodes: # Node configuration for multi-node deployment\n main: # Default local node (empty config)\nConfiguration files are searched in the following locations (in order of precedence, first found is used):
Linux: - ./llamactl.yaml or ./config.yaml (current directory) - $HOME/.config/llamactl/config.yaml - /etc/llamactl/config.yaml
macOS: - ./llamactl.yaml or ./config.yaml (current directory) - $HOME/Library/Application Support/llamactl/config.yaml - /Library/Application Support/llamactl/config.yaml
Windows: - ./llamactl.yaml or ./config.yaml (current directory) - %APPDATA%\\llamactl\\config.yaml - %USERPROFILE%\\llamactl\\config.yaml - %PROGRAMDATA%\\llamactl\\config.yaml
You can specify the path to config file with LLAMACTL_CONFIG_PATH environment variable.
server:\n host: \"0.0.0.0\" # Server host to bind to (default: \"0.0.0.0\")\n port: 8080 # Server port to bind to (default: 8080)\n allowed_origins: [\"*\"] # CORS allowed origins (default: [\"*\"])\n allowed_headers: [\"*\"] # CORS allowed headers (default: [\"*\"])\n enable_swagger: false # Enable Swagger UI (default: false)\nEnvironment Variables: - LLAMACTL_HOST - Server host - LLAMACTL_PORT - Server port - LLAMACTL_ALLOWED_ORIGINS - Comma-separated CORS origins - LLAMACTL_ENABLE_SWAGGER - Enable Swagger UI (true/false)
backends:\n llama-cpp:\n command: \"llama-server\"\n args: []\n environment: {} # Environment variables for the backend process\n docker:\n enabled: false # Enable Docker runtime (default: false)\n image: \"ghcr.io/ggml-org/llama.cpp:server\"\n args: [\"run\", \"--rm\", \"--network\", \"host\", \"--gpus\", \"all\"]\n environment: {}\n response_headers: {} # Additional response headers to send with responses\n\n vllm:\n command: \"vllm\"\n args: [\"serve\"]\n environment: {} # Environment variables for the backend process\n docker:\n enabled: false # Enable Docker runtime (default: false)\n image: \"vllm/vllm-openai:latest\"\n args: [\"run\", \"--rm\", \"--network\", \"host\", \"--gpus\", \"all\", \"--shm-size\", \"1g\"]\n environment: {}\n response_headers: {} # Additional response headers to send with responses\n\n mlx:\n command: \"mlx_lm.server\"\n args: []\n environment: {} # Environment variables for the backend process\n # MLX does not support Docker\n response_headers: {} # Additional response headers to send with responses\nBackend Configuration Fields: - command: Executable name/path for the backend - args: Default arguments prepended to all instances - environment: Environment variables for the backend process (optional) - response_headers: Additional response headers to send with responses (optional) - docker: Docker-specific configuration (optional) - enabled: Boolean flag to enable Docker runtime - image: Docker image to use - args: Additional arguments passed to docker run - environment: Environment variables for the container (optional)
If llamactl is behind an NGINX proxy, X-Accel-Buffering: no response header may be required for NGINX to properly stream the responses without buffering.
Environment Variables:
LlamaCpp Backend: - LLAMACTL_LLAMACPP_COMMAND - LlamaCpp executable command - LLAMACTL_LLAMACPP_ARGS - Space-separated default arguments - LLAMACTL_LLAMACPP_ENV - Environment variables in format \"KEY1=value1,KEY2=value2\" - LLAMACTL_LLAMACPP_DOCKER_ENABLED - Enable Docker runtime (true/false) - LLAMACTL_LLAMACPP_DOCKER_IMAGE - Docker image to use - LLAMACTL_LLAMACPP_DOCKER_ARGS - Space-separated Docker arguments - LLAMACTL_LLAMACPP_DOCKER_ENV - Docker environment variables in format \"KEY1=value1,KEY2=value2\" - LLAMACTL_LLAMACPP_RESPONSE_HEADERS - Response headers in format \"KEY1=value1;KEY2=value2\"
VLLM Backend: - LLAMACTL_VLLM_COMMAND - VLLM executable command - LLAMACTL_VLLM_ARGS - Space-separated default arguments - LLAMACTL_VLLM_ENV - Environment variables in format \"KEY1=value1,KEY2=value2\" - LLAMACTL_VLLM_DOCKER_ENABLED - Enable Docker runtime (true/false) - LLAMACTL_VLLM_DOCKER_IMAGE - Docker image to use - LLAMACTL_VLLM_DOCKER_ARGS - Space-separated Docker arguments - LLAMACTL_VLLM_DOCKER_ENV - Docker environment variables in format \"KEY1=value1,KEY2=value2\" - LLAMACTL_VLLM_RESPONSE_HEADERS - Response headers in format \"KEY1=value1;KEY2=value2\"
MLX Backend: - LLAMACTL_MLX_COMMAND - MLX executable command - LLAMACTL_MLX_ARGS - Space-separated default arguments - LLAMACTL_MLX_ENV - Environment variables in format \"KEY1=value1,KEY2=value2\" - LLAMACTL_MLX_RESPONSE_HEADERS - Response headers in format \"KEY1=value1;KEY2=value2\"
data_dir: \"~/.local/share/llamactl\" # Main data directory for database, instances, and logs (default varies by OS)\nEnvironment Variables: - LLAMACTL_DATA_DIRECTORY - Main data directory path
Default Data Directory by Platform: - Linux: ~/.local/share/llamactl - macOS: ~/Library/Application Support/llamactl - Windows: %LOCALAPPDATA%\\llamactl or %PROGRAMDATA%\\llamactl
instances:\n port_range: [8000, 9000] # Port range for instances (default: [8000, 9000])\n configs_dir: \"instances\" # Directory for instance configs, default: data_dir/instances\n logs_dir: \"logs\" # Directory for instance logs, default: data_dir/logs\n auto_create_dirs: true # Automatically create data/config/logs directories (default: true)\n max_instances: -1 # Maximum instances (-1 = unlimited)\n max_running_instances: -1 # Maximum running instances (-1 = unlimited)\n enable_lru_eviction: true # Enable LRU eviction for idle instances\n default_auto_restart: true # Default auto-restart setting\n default_max_restarts: 3 # Default maximum restart attempts\n default_restart_delay: 5 # Default restart delay in seconds\n default_on_demand_start: true # Default on-demand start setting\n on_demand_start_timeout: 120 # Default on-demand start timeout in seconds\n timeout_check_interval: 5 # Default instance timeout check interval in minutes\nEnvironment Variables: - LLAMACTL_INSTANCE_PORT_RANGE - Port range (format: \"8000-9000\" or \"8000,9000\") - LLAMACTL_INSTANCES_DIR - Instance configs directory path - LLAMACTL_LOGS_DIR - Log directory path - LLAMACTL_AUTO_CREATE_DATA_DIR - Auto-create data/config/logs directories (true/false) - LLAMACTL_MAX_INSTANCES - Maximum number of instances - LLAMACTL_MAX_RUNNING_INSTANCES - Maximum number of running instances - LLAMACTL_ENABLE_LRU_EVICTION - Enable LRU eviction for idle instances - LLAMACTL_DEFAULT_AUTO_RESTART - Default auto-restart setting (true/false) - LLAMACTL_DEFAULT_MAX_RESTARTS - Default maximum restarts - LLAMACTL_DEFAULT_RESTART_DELAY - Default restart delay in seconds - LLAMACTL_DEFAULT_ON_DEMAND_START - Default on-demand start setting (true/false) - LLAMACTL_ON_DEMAND_START_TIMEOUT - Default on-demand start timeout in seconds - LLAMACTL_TIMEOUT_CHECK_INTERVAL - Default instance timeout check interval in minutes
database:\n path: \"llamactl.db\" # Database file path, default: data_dir/llamactl.db\n max_open_connections: 25 # Maximum open database connections (default: 25)\n max_idle_connections: 5 # Maximum idle database connections (default: 5)\n connection_max_lifetime: 5m # Connection max lifetime (default: 5m)\nEnvironment Variables: - LLAMACTL_DATABASE_PATH - Database file path (relative to data_dir or absolute) - LLAMACTL_DATABASE_MAX_OPEN_CONNECTIONS - Maximum open database connections - LLAMACTL_DATABASE_MAX_IDLE_CONNECTIONS - Maximum idle database connections - LLAMACTL_DATABASE_CONN_MAX_LIFETIME - Connection max lifetime (e.g., \"5m\", \"1h\")
auth:\n require_inference_auth: true # Require API key for OpenAI endpoints (default: true)\n inference_keys: [] # List of valid inference API keys\n require_management_auth: true # Require API key for management endpoints (default: true)\n management_keys: [] # List of valid management API keys\nEnvironment Variables: - LLAMACTL_REQUIRE_INFERENCE_AUTH - Require auth for OpenAI endpoints (true/false) - LLAMACTL_INFERENCE_KEYS - Comma-separated inference API keys - LLAMACTL_REQUIRE_MANAGEMENT_AUTH - Require auth for management endpoints (true/false) - LLAMACTL_MANAGEMENT_KEYS - Comma-separated management API keys
llamactl supports remote node deployments. Configure remote nodes to deploy instances on remote hosts and manage them centrally.
local_node: \"main\" # Name of the local node (default: \"main\")\nnodes: # Node configuration map\n main: # Local node (empty address means local)\n address: \"\" # Not used for local node\n api_key: \"\" # Not used for local node\n worker1: # Remote worker node\n address: \"http://192.168.1.10:8080\"\n api_key: \"worker1-api-key\" # Management API key for authentication\nNode Configuration Fields: - local_node: Specifies which node in the nodes map represents the local node. Must match exactly what other nodes call this node. - nodes: Map of node configurations - address: HTTP/HTTPS URL of the remote node (empty for local node) - api_key: Management API key for authenticating with the remote node
Environment Variables: - LLAMACTL_LOCAL_NODE - Name of the local node
This guide will walk you through installing Llamactl on your system.
"},{"location":"installation/#prerequisites","title":"Prerequisites","text":""},{"location":"installation/#backend-dependencies","title":"Backend Dependencies","text":"llamactl supports multiple backends. Install at least one:
For llama.cpp backend (all platforms):
You need llama-server from llama.cpp installed:
# Homebrew (macOS/Linux)\nbrew install llama.cpp\n# Winget (Windows)\nwinget install llama.cpp\nOr build from source - see llama.cpp docs
For MLX backend (macOS only):
MLX provides optimized inference on Apple Silicon. Install MLX-LM:
# Install via pip (requires Python 3.8+)\npip install mlx-lm\n\n# Or in a virtual environment (recommended)\npython -m venv mlx-env\nsource mlx-env/bin/activate\npip install mlx-lm\nNote: MLX backend is only available on macOS with Apple Silicon (M1, M2, M3, etc.)
For vLLM backend:
vLLM provides high-throughput distributed serving for LLMs. Install vLLM:
# Install in a virtual environment\npython -m venv vllm-env\nsource vllm-env/bin/activate\npip install vllm\nDownload the latest release from the GitHub releases page:
# Linux/macOS - Get latest version and download\nLATEST_VERSION=$(curl -s https://api.github.com/repos/lordmathis/llamactl/releases/latest | grep '\"tag_name\":' | sed -E 's/.*\"([^\"]+)\".*/\\1/')\ncurl -L https://github.com/lordmathis/llamactl/releases/download/${LATEST_VERSION}/llamactl-${LATEST_VERSION}-$(uname -s | tr '[:upper:]' '[:lower:]')-$(uname -m).tar.gz | tar -xz\nsudo mv llamactl /usr/local/bin/\n\n# Or download manually from:\n# https://github.com/lordmathis/llamactl/releases/latest\n\n# Windows - Download from releases page\nllamactl provides Dockerfiles for creating Docker images with backends pre-installed. The resulting images include the latest llamactl release with the respective backend.
Available Dockerfiles (CUDA): - llamactl with llama.cpp CUDA: docker/Dockerfile.llamacpp (based on ghcr.io/ggml-org/llama.cpp:server-cuda) - llamactl with vLLM CUDA: docker/Dockerfile.vllm (based on vllm/vllm-openai:latest) - llamactl built from source: docker/Dockerfile.source (multi-stage build with webui)
Note: These Dockerfiles are configured for CUDA. For other platforms (CPU, ROCm, Vulkan, etc.), adapt the base image. For llama.cpp, see available tags at llama.cpp Docker docs. For vLLM, check vLLM docs.
Using Docker Compose
# Clone the repository\ngit clone https://github.com/lordmathis/llamactl.git\ncd llamactl\n\n# Create directories for data and models\nmkdir -p data/llamacpp data/vllm models\n\n# Start llamactl with llama.cpp backend\ndocker-compose -f docker/docker-compose.yml up llamactl-llamacpp -d\n\n# Or start llamactl with vLLM backend\ndocker-compose -f docker/docker-compose.yml up llamactl-vllm -d\nAccess the dashboard at: - llamactl with llama.cpp: http://localhost:8080 - llamactl with vLLM: http://localhost:8081
Using Docker Build and Run
llamactl with llama.cpp CUDA:
docker build -f docker/Dockerfile.llamacpp -t llamactl:llamacpp-cuda .\ndocker run -d \\\n --name llamactl-llamacpp \\\n --gpus all \\\n -p 8080:8080 \\\n -v ~/.cache/llama.cpp:/root/.cache/llama.cpp \\\n llamactl:llamacpp-cuda\nllamactl with vLLM CUDA:
docker build -f docker/Dockerfile.vllm -t llamactl:vllm-cuda .\ndocker run -d \\\n --name llamactl-vllm \\\n --gpus all \\\n -p 8080:8080 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n llamactl:vllm-cuda\nllamactl built from source:
docker build -f docker/Dockerfile.source -t llamactl:source .\ndocker run -d \\\n --name llamactl \\\n -p 8080:8080 \\\n llamactl:source\nRequirements: - Go 1.24 or later - Node.js 22 or later - Git
If you prefer to build from source:
# Clone the repository\ngit clone https://github.com/lordmathis/llamactl.git\ncd llamactl\n\n# Build the web UI\ncd webui && npm ci && npm run build && cd ..\n\n# Build the application\ngo build -o llamactl ./cmd/server\nFor deployments with remote nodes: - Install llamactl on each node using any of the methods above - Configure API keys for authentication between nodes - Ensure node names are consistent across all configurations
"},{"location":"installation/#verification","title":"Verification","text":"Verify your installation by checking the version:
llamactl --version\nNow that Llamactl is installed, continue to the Quick Start guide to get your first instance running!
For remote node deployments, see the Configuration Guide for node setup instructions.
"},{"location":"managing-instances/","title":"Managing Instances","text":"Learn how to effectively manage your llama.cpp, MLX, and vLLM instances with Llamactl through both the Web UI and API.
"},{"location":"managing-instances/#overview","title":"Overview","text":"Llamactl provides two ways to manage instances:
http://localhost:8080 with an intuitive dashboard "},{"location":"managing-instances/#authentication","title":"Authentication","text":"
Llamactl uses a Management API Key to authenticate requests to the management API (creating, starting, stopping instances). All curl examples below use <token> as a placeholder - replace this with your actual Management API Key.
By default, authentication is required. If you don't configure a management API key in your configuration file, llamactl will auto-generate one and print it to the terminal on startup. See the Configuration guide for details.
For Web UI access: 1. Navigate to the web UI 2. Enter your Management API Key 3. Bearer token is stored for the session
"},{"location":"managing-instances/#theme-support","title":"Theme Support","text":"Each instance is displayed as a card showing:
Via Web UI
Instance Settings:
Backend Configuration:
Auto-Assignment
Llamactl automatically assigns ports from the configured port range (default: 8000-9000) and generates API keys if authentication is enabled. You typically don't need to manually specify these values.
Via API
# Create llama.cpp instance with local model file\ncurl -X POST http://localhost:8080/api/v1/instances/my-llama-instance \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer <token>\" \\\n -d '{\n \"backend_type\": \"llama_cpp\",\n \"backend_options\": {\n \"model\": \"/path/to/model.gguf\",\n \"threads\": 8,\n \"ctx_size\": 4096,\n \"gpu_layers\": 32,\n \"flash_attn\": \"on\"\n },\n \"auto_restart\": true,\n \"max_restarts\": 3,\n \"docker_enabled\": false,\n \"command_override\": \"/opt/llama-server-dev\",\n \"nodes\": [\"main\"]\n }'\n\n# Create vLLM instance with environment variables\ncurl -X POST http://localhost:8080/api/v1/instances/my-vllm-instance \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer <token>\" \\\n -d '{\n \"backend_type\": \"vllm\",\n \"backend_options\": {\n \"model\": \"microsoft/DialoGPT-medium\",\n \"tensor_parallel_size\": 2,\n \"gpu_memory_utilization\": 0.9\n },\n \"on_demand_start\": true,\n \"environment\": {\n \"CUDA_VISIBLE_DEVICES\": \"0,1\"\n },\n \"nodes\": [\"worker1\", \"worker2\"]\n }'\n\n# Create MLX instance (macOS only)\ncurl -X POST http://localhost:8080/api/v1/instances/my-mlx-instance \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer <token>\" \\\n -d '{\n \"backend_type\": \"mlx_lm\",\n \"backend_options\": {\n \"model\": \"mlx-community/Mistral-7B-Instruct-v0.3-4bit\",\n \"temp\": 0.7,\n \"max_tokens\": 2048\n },\n \"nodes\": [\"main\"]\n }'\nVia Web UI 1. Click the \"Start\" button on an instance card 2. Watch the status change to \"Unknown\" 3. Monitor progress in the logs 4. Instance status changes to \"Ready\" when ready
Via API
curl -X POST http://localhost:8080/api/v1/instances/{name}/start \\\n -H \"Authorization: Bearer <token>\"\nVia Web UI 1. Click the \"Stop\" button on an instance card 2. Instance gracefully shuts down
Via API
curl -X POST http://localhost:8080/api/v1/instances/{name}/stop \\\n -H \"Authorization: Bearer <token>\"\nVia Web UI 1. Click the \"Edit\" button on an instance card 2. Modify settings in the configuration dialog 3. Changes require instance restart to take effect 4. Click \"Update & Restart\" to apply changes
Via API Modify instance settings:
curl -X PUT http://localhost:8080/api/v1/instances/{name} \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer <token>\" \\\n -d '{\n \"backend_options\": {\n \"threads\": 8,\n \"context_size\": 4096\n }\n }'\nNote
Configuration changes require restarting the instance to take effect.
"},{"location":"managing-instances/#export-instance","title":"Export Instance","text":"Via Web UI 1. Click the \"More actions\" button (three dots) on an instance card 2. Click \"Export\" to download the instance configuration as a JSON file
"},{"location":"managing-instances/#view-logs","title":"View Logs","text":"Via Web UI
Via API Check instance status in real-time:
# Get instance logs\ncurl http://localhost:8080/api/v1/instances/{name}/logs \\\n -H \"Authorization: Bearer <token>\"\nVia Web UI 1. Click the \"Delete\" button on an instance card 2. Only stopped instances can be deleted 3. Confirm deletion in the dialog
Via API
curl -X DELETE http://localhost:8080/api/v1/instances/{name} \\\n -H \"Authorization: Bearer <token>\"\nLlamactl proxies all requests to the underlying backend instances (llama-server, MLX, or vLLM).
# Proxy requests to the instance\ncurl http://localhost:8080/api/v1/instances/{name}/proxy/ \\\n -H \"Authorization: Bearer <token>\"\nAll backends provide OpenAI-compatible endpoints. Check the respective documentation: - llama-server docs - MLX-LM docs - vLLM docs
"},{"location":"managing-instances/#instance-health","title":"Instance Health","text":"Via Web UI
Via API
Check the health status of your instances:
curl http://localhost:8080/api/v1/instances/{name}/proxy/health \\\n -H \"Authorization: Bearer <token>\"\nThis guide will help you get Llamactl up and running in just a few minutes.
Before you begin: Ensure you have at least one backend installed (llama.cpp, MLX, or vLLM). See the Installation Guide for backend setup.
"},{"location":"quick-start/#core-concepts","title":"Core Concepts","text":"Before you start, let's clarify a few key terms:
http://localhost:8080/v1/chat/completions), and it routes them to the appropriate backend instance. This means you don't need to track individual instance ports or endpoints. Llamactl uses two types of API keys:
/v1/chat/completions, /v1/completions, etc.). By default, authentication is required. If you don't configure these keys in your configuration file, llamactl will auto-generate them and print them to the terminal on startup. You can also configure custom keys or disable authentication entirely in the Configuration guide.
"},{"location":"quick-start/#start-llamactl","title":"Start Llamactl","text":"Start the Llamactl server:
llamactl\n\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\n\u26a0\ufe0f MANAGEMENT AUTHENTICATION REQUIRED\n\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\n\ud83d\udd11 Generated Management API Key:\n\n sk-management-...\n\n\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\n\u26a0\ufe0f INFERENCE AUTHENTICATION REQUIRED\n\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\n\ud83d\udd11 Generated Inference API Key:\n\n sk-inference-...\n\n\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\n\u26a0\ufe0f IMPORTANT\n\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\n\u2022 These keys are auto-generated and will change on restart\n\u2022 For production, add explicit keys to your configuration\n\u2022 Copy these keys before they disappear from the terminal\n\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\nLlamactl server listening on 0.0.0.0:8080\nCopy the Management and Inference API Keys from the terminal - you'll need them to access the web UI and make inference requests.
By default, Llamactl will start on http://localhost:8080.
Open your web browser and navigate to:
http://localhost:8080\nLogin with the management API key from the terminal output.
You should see the Llamactl web interface.
"},{"location":"quick-start/#create-your-first-instance","title":"Create Your First Instance","text":"Fill in the instance configuration:
Auto-Assignment
Llamactl automatically assigns ports from the configured port range (default: 8000-9000) and generates API keys if authentication is enabled. You typically don't need to manually specify these values.
Remote Node Deployment
If you have configured remote nodes in your configuration file, you can select which node to deploy the instance to. This allows you to distribute instances across multiple machines. See the Configuration guide for details on setting up remote nodes.
Click \"Create Instance\"
Once created, you can:
Here are basic example configurations for each backend:
llama.cpp backend:
{\n \"name\": \"llama2-7b\",\n \"backend_type\": \"llama_cpp\",\n \"backend_options\": {\n \"model\": \"/path/to/llama-2-7b-chat.gguf\",\n \"threads\": 4,\n \"ctx_size\": 2048,\n \"gpu_layers\": 32\n },\n \"nodes\": [\"main\"]\n}\nMLX backend (macOS only):
{\n \"name\": \"mistral-mlx\",\n \"backend_type\": \"mlx_lm\",\n \"backend_options\": {\n \"model\": \"mlx-community/Mistral-7B-Instruct-v0.3-4bit\",\n \"temp\": 0.7,\n \"max_tokens\": 2048\n },\n \"nodes\": [\"main\"]\n}\nvLLM backend:
{\n \"name\": \"dialogpt-vllm\",\n \"backend_type\": \"vllm\",\n \"backend_options\": {\n \"model\": \"microsoft/DialoGPT-medium\",\n \"tensor_parallel_size\": 2,\n \"gpu_memory_utilization\": 0.9\n },\n \"nodes\": [\"main\"]\n}\nRemote node deployment example:
{\n \"name\": \"distributed-model\",\n \"backend_type\": \"llama_cpp\",\n \"backend_options\": {\n \"model\": \"/path/to/model.gguf\",\n \"gpu_layers\": 32\n },\n \"nodes\": [\"worker1\"]\n}\nLlamactl can run backends in Docker containers. To enable Docker for a backend, add a docker section to that backend in your YAML configuration file (e.g. config.yaml) as shown below:
backends:\n vllm:\n command: \"vllm\"\n args: [\"serve\"]\n docker:\n enabled: true\n image: \"vllm/vllm-openai:latest\"\n args: [\"run\", \"--rm\", \"--network\", \"host\", \"--gpus\", \"all\", \"--shm-size\", \"1g\"]\nYou can also manage instances via the REST API:
# List all instances\ncurl http://localhost:8080/api/v1/instances\n\n# Create a new llama.cpp instance\ncurl -X POST http://localhost:8080/api/v1/instances/my-model \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"backend_type\": \"llama_cpp\",\n \"backend_options\": {\n \"model\": \"/path/to/model.gguf\"\n }\n }'\n\n# Start an instance\ncurl -X POST http://localhost:8080/api/v1/instances/my-model/start\nLlamactl provides OpenAI-compatible endpoints, making it easy to integrate with existing OpenAI client libraries and tools.
"},{"location":"quick-start/#chat-completions","title":"Chat Completions","text":"Once you have an instance running, you can use it with the OpenAI-compatible chat completions endpoint:
curl -X POST http://localhost:8080/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"my-model\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hello! Can you help me write a Python function?\"\n }\n ],\n \"max_tokens\": 150,\n \"temperature\": 0.7\n }'\nYou can also use the official OpenAI Python client:
from openai import OpenAI\n\n# Point the client to your Llamactl server\nclient = OpenAI(\n base_url=\"http://localhost:8080/v1\",\n api_key=\"your-inference-api-key\" # Use the inference API key from terminal or config\n)\n\n# Create a chat completion\nresponse = client.chat.completions.create(\n model=\"my-model\", # Use the name of your instance\n messages=[\n {\"role\": \"user\", \"content\": \"Explain quantum computing in simple terms\"}\n ],\n max_tokens=200,\n temperature=0.7\n)\n\nprint(response.choices[0].message.content)\nAPI Key
If you disabled authentication in your config, you can use any value for api_key (e.g., \"not-needed\"). Otherwise, use the inference API key shown in the terminal output on startup.
Get a list of running instances (models) in OpenAI-compatible format:
curl http://localhost:8080/v1/models\nIssues specific to Llamactl deployment and operation.
"},{"location":"troubleshooting/#configuration-issues","title":"Configuration Issues","text":""},{"location":"troubleshooting/#invalid-configuration","title":"Invalid Configuration","text":"Problem: Invalid configuration preventing startup
Solutions: 1. Use minimal configuration:
server:\n host: \"0.0.0.0\"\n port: 8080\ninstances:\n port_range: [8000, 9000]\n# Ensure data directory is writable (default: ~/.local/share/llamactl)\nmkdir -p ~/.local/share/llamactl/{instances,logs}\nProblem: Instance fails to start or immediately stops
Solutions:
Check instance logs to see the actual error:
curl http://localhost:8080/api/v1/instances/{name}/logs\n# Or check log files directly\ntail -f ~/.local/share/llamactl/logs/{instance-name}.log\nVerify backend is installed:
llama-server is in PATH mlx-lm Python package is installed vllm Python package is installed Check model path and format:
Verify backend command configuration:
command is correctly configured in the global config /path/to/venv/bin/mlx_lm.server) Problem: Model loading, memory, GPU, or performance issues
Most model-specific issues (memory, GPU configuration, performance tuning) are backend-specific and should be resolved by consulting the respective backend documentation:
llama.cpp: - llama.cpp GitHub - llama-server README
MLX: - MLX-LM GitHub - MLX-LM Server Guide
vLLM: - vLLM Documentation - OpenAI Compatible Server - vllm serve Command
Testing backends directly:
Testing your model and configuration directly with the backend helps determine if the issue is with llamactl or the backend itself:
# llama.cpp\nllama-server --model /path/to/model.gguf --port 8081\n\n# MLX\nmlx_lm.server --model mlx-community/Mistral-7B-Instruct-v0.3-4bit --port 8081\n\n# vLLM\nvllm serve microsoft/DialoGPT-medium --port 8081\nProblem: Web UI shows CORS errors in browser console
Solutions: 1. Configure allowed origins:
server:\n allowed_origins:\n - \"http://localhost:3000\"\n - \"https://yourdomain.com\"\nProblem: API requests failing with authentication errors
Solutions: 1. Disable authentication temporarily:
auth:\n require_management_auth: false\n require_inference_auth: false\nConfigure API keys:
auth:\n management_keys:\n - \"your-management-key\"\n inference_keys:\n - \"your-inference-key\"\nUse correct Authorization header:
curl -H \"Authorization: Bearer your-api-key\" \\\n http://localhost:8080/api/v1/instances\nProblem: Remote instances not appearing or cannot be managed
Solutions: 1. Verify node configuration:
local_node: \"main\" # Must match a key in nodes map\nnodes:\n main:\n address: \"\" # Empty for local node\n worker1:\n address: \"http://worker1.internal:8080\"\n api_key: \"secure-key\" # Must match worker1's management key\nlocal_node on each node must match what other nodes call it Node names are case-sensitive
Test remote node connectivity:
curl -H \"Authorization: Bearer remote-node-key\" \\\n http://remote-node:8080/api/v1/instances\n# Get instance logs via API\ncurl http://localhost:8080/api/v1/instances/{name}/logs\n\n# Or check log files directly\ntail -f ~/.local/share/llamactl/logs/{instance-name}.log\nexport LLAMACTL_LOG_LEVEL=debug\nllamactl\nWhen reporting issues, include:
System information:
llamactl --version\nConfiguration file (remove sensitive keys)
Relevant log output
Steps to reproduce the issue
Welcome to the Llamactl documentation!
"},{"location":"#what-is-llamactl","title":"What is Llamactl?","text":"
Unified management and routing for llama.cpp, MLX and vLLM models with web dashboard.
"},{"location":"#features","title":"Features","text":"\ud83d\ude80 Easy Model Management - Multiple Models Simultaneously: Run different models at the same time (7B for speed, 70B for quality) - Smart Resource Management: Automatic idle timeout, LRU eviction, and configurable instance limits - Web Dashboard: Modern React UI for managing instances, monitoring health, and viewing logs
\ud83d\udd17 Flexible Integration - OpenAI API Compatible: Drop-in replacement - route requests to different models by instance name - Multi-Backend Support: Native support for llama.cpp, MLX (Apple Silicon optimized), and vLLM - Docker Ready: Run backends in containers with full GPU support
\ud83c\udf10 Distributed Deployment - Remote Instances: Deploy instances on remote hosts - Central Management: Manage everything from a single dashboard with automatic routing
"},{"location":"#quick-links","title":"Quick Links","text":"If you need help or have questions:
MIT License - see the LICENSE file.
"},{"location":"api-reference/","title":"API Reference","text":""},{"location":"api-reference/#llamactl-api-10","title":"llamactl API 1.0","text":"llamactl is a control server for managing Llama Server instances.
License: MIT License"},{"location":"api-reference/#keys","title":"Keys","text":""},{"location":"api-reference/#get-apiv1authkeys","title":"GET /api/v1/auth/keys","text":"List all API keys
DescriptionReturns a list of all API keys for the system user (excludes key hash and plain-text key)
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 500 Internal Server Error
"},{"location":"api-reference/#post-apiv1authkeys","title":"POST /api/v1/auth/keys","text":"Create a new API key
DescriptionCreates a new API key with the specified permissions and returns the plain- text key (only shown once)
Input parameters
Parameter In Type Default Nullable Descriptionkey body None No API key configuration Response 201 Created
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#delete-apiv1authkeysid","title":"DELETE /api/v1/auth/keys/{id}","text":"Delete an API key
DescriptionDeletes an API key by ID
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No id path None No Key ID Response 204 No Content
Response 400 Bad Request
Response 404 Not Found
Response 500 Internal Server Error
"},{"location":"api-reference/#get-apiv1authkeysid","title":"GET /api/v1/auth/keys/{id}","text":"Get details of a specific API key
DescriptionReturns details for a specific API key by ID (excludes key hash and plain- text key)
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No id path None No Key ID Response 200 OK
Response 400 Bad Request
Response 404 Not Found
Response 500 Internal Server Error
"},{"location":"api-reference/#get-apiv1authkeysidpermissions","title":"GET /api/v1/auth/keys/{id}/permissions","text":"Get API key permissions
DescriptionReturns the instance-level permissions for a specific API key (includes instance names)
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No id path None No Key ID Response 200 OK
Response 400 Bad Request
Response 404 Not Found
Response 500 Internal Server Error
"},{"location":"api-reference/#backends","title":"Backends","text":""},{"location":"api-reference/#get-apiv1backendsllama-cppdevices","title":"GET /api/v1/backends/llama-cpp/devices","text":"List available devices for llama server
DescriptionReturns a list of available devices for the llama server
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 500 Internal Server Error
"},{"location":"api-reference/#get-apiv1backendsllama-cpphelp","title":"GET /api/v1/backends/llama-cpp/help","text":"Get help for llama server
DescriptionReturns the help text for the llama server command
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 500 Internal Server Error
"},{"location":"api-reference/#post-apiv1backendsllama-cppparse-command","title":"POST /api/v1/backends/llama-cpp/parse-command","text":"Parse llama-server command
DescriptionParses a llama-server command string into instance options
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No request body None No Command to parse Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#get-apiv1backendsllama-cppversion","title":"GET /api/v1/backends/llama-cpp/version","text":"Get version of llama server
DescriptionReturns the version of the llama server command
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 500 Internal Server Error
"},{"location":"api-reference/#post-apiv1backendsmlxparse-command","title":"POST /api/v1/backends/mlx/parse-command","text":"Parse mlx_lm.server command
DescriptionParses MLX-LM server command string into instance options
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No request body None No Command to parse Response 200 OK
Response 400 Bad Request
"},{"location":"api-reference/#post-apiv1backendsvllmparse-command","title":"POST /api/v1/backends/vllm/parse-command","text":"Parse vllm serve command
DescriptionParses a vLLM serve command string into instance options
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No request body None No Command to parse Response 200 OK
Response 400 Bad Request
"},{"location":"api-reference/#system","title":"System","text":""},{"location":"api-reference/#get-apiv1config","title":"GET /api/v1/config","text":"Get server configuration
DescriptionReturns the current server configuration (sanitized)
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 500 Internal Server Error
"},{"location":"api-reference/#get-apiv1version","title":"GET /api/v1/version","text":"Get llamactl version
DescriptionReturns the version of the llamactl command
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 500 Internal Server Error
"},{"location":"api-reference/#instances","title":"Instances","text":""},{"location":"api-reference/#get-apiv1instances","title":"GET /api/v1/instances","text":"List all instances
DescriptionReturns a list of all instances managed by the server
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 500 Internal Server Error
"},{"location":"api-reference/#delete-apiv1instancesname","title":"DELETE /api/v1/instances/{name}","text":"Delete an instance
DescriptionStops and removes a specific instance by name
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 204 No Content
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#get-apiv1instancesname","title":"GET /api/v1/instances/{name}","text":"Get details of a specific instance
DescriptionReturns the details of a specific instance by name
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-apiv1instancesname","title":"POST /api/v1/instances/{name}","text":"Create and start a new instance
DescriptionCreates a new instance with the provided configuration options
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name options body None No Instance configuration options Response 201 Created
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#put-apiv1instancesname","title":"PUT /api/v1/instances/{name}","text":"Update an instance's configuration
DescriptionUpdates the configuration of a specific instance by name
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name options body None No Instance configuration options Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#get-apiv1instancesnamelogs","title":"GET /api/v1/instances/{name}/logs","text":"Get logs from a specific instance
DescriptionReturns the logs from a specific instance by name with optional line limit
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No lines query None No Number of lines to retrieve (default: all lines) name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#get-apiv1instancesnameproxy","title":"GET /api/v1/instances/{name}/proxy","text":"Proxy requests to a specific instance, does not autostart instance if stopped
DescriptionForwards HTTP requests to the llama-server instance running on a specific port
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
Response 503 Service Unavailable
"},{"location":"api-reference/#post-apiv1instancesnameproxy","title":"POST /api/v1/instances/{name}/proxy","text":"Proxy requests to a specific instance, does not autostart instance if stopped
DescriptionForwards HTTP requests to the llama-server instance running on a specific port
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
Response 503 Service Unavailable
"},{"location":"api-reference/#post-apiv1instancesnamerestart","title":"POST /api/v1/instances/{name}/restart","text":"Restart a running instance
DescriptionRestarts a specific instance by name
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-apiv1instancesnamestart","title":"POST /api/v1/instances/{name}/start","text":"Start a stopped instance
DescriptionStarts a specific instance by name
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-apiv1instancesnamestop","title":"POST /api/v1/instances/{name}/stop","text":"Stop a running instance
DescriptionStops a specific instance by name
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#nodes","title":"Nodes","text":""},{"location":"api-reference/#get-apiv1nodes","title":"GET /api/v1/nodes","text":"List all configured nodes
DescriptionReturns a map of all nodes configured in the server (node name -> node config)
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 500 Internal Server Error
"},{"location":"api-reference/#get-apiv1nodesname","title":"GET /api/v1/nodes/{name}","text":"Get details of a specific node
DescriptionReturns the details of a specific node by name
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Node Name Response 200 OK
Response 400 Bad Request
Response 404 Not Found
Response 500 Internal Server Error
"},{"location":"api-reference/#llamacpp","title":"Llama.cpp","text":""},{"location":"api-reference/#get-llama-cppname","title":"GET /llama-cpp/{name}/","text":"Proxy requests to llama.cpp UI for the instance
DescriptionProxies requests to the llama.cpp UI for the specified instance
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name query None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-llama-cppnameapply-template","title":"POST /llama-cpp/{name}/apply-template","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-llama-cppnamecompletion","title":"POST /llama-cpp/{name}/completion","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-llama-cppnamedetokenize","title":"POST /llama-cpp/{name}/detokenize","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-llama-cppnameembeddings","title":"POST /llama-cpp/{name}/embeddings","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-llama-cppnameinfill","title":"POST /llama-cpp/{name}/infill","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-llama-cppnamemetrics","title":"POST /llama-cpp/{name}/metrics","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#get-llama-cppnameprops","title":"GET /llama-cpp/{name}/props","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-llama-cppnameprops","title":"POST /llama-cpp/{name}/props","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-llama-cppnamereranking","title":"POST /llama-cpp/{name}/reranking","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#get-llama-cppnameslots","title":"GET /llama-cpp/{name}/slots","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#post-llama-cppnametokenize","title":"POST /llama-cpp/{name}/tokenize","text":"Proxy requests to llama.cpp server instance
DescriptionProxies requests to the specified llama.cpp server instance, starting it on- demand if configured
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No name path None No Instance Name Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#openai","title":"OpenAI","text":""},{"location":"api-reference/#post-v1","title":"POST /v1/","text":"OpenAI-compatible proxy endpoint
DescriptionHandles all POST requests to /v1/*, routing to the appropriate instance based on the request body. Requires API key authentication via the Authorization header.
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 400 Bad Request
Response 500 Internal Server Error
"},{"location":"api-reference/#get-v1models","title":"GET /v1/models","text":"List instances in OpenAI-compatible format
DescriptionReturns a list of instances in a format compatible with OpenAI API
Input parameters
Parameter In Type Default Nullable DescriptionApiKeyAuth header string N/A No Response 200 OK
Response 500 Internal Server Error
"},{"location":"configuration/","title":"Configuration","text":"llamactl can be configured via configuration files or environment variables. Configuration is loaded in the following order of precedence:
Defaults < Configuration file < Environment variables\nllamactl works out of the box with sensible defaults, but you can customize the behavior to suit your needs.
"},{"location":"configuration/#default-configuration","title":"Default Configuration","text":"Here's the default configuration with all available options:
server:\n host: \"0.0.0.0\" # Server host to bind to\n port: 8080 # Server port to bind to\n allowed_origins: [\"*\"] # Allowed CORS origins (default: all)\n allowed_headers: [\"*\"] # Allowed CORS headers (default: all)\n enable_swagger: false # Enable Swagger UI for API docs\n\nbackends:\n llama-cpp:\n command: \"llama-server\"\n args: []\n environment: {} # Environment variables for the backend process\n docker:\n enabled: false\n image: \"ghcr.io/ggml-org/llama.cpp:server\"\n args: [\"run\", \"--rm\", \"--network\", \"host\", \"--gpus\", \"all\"]\n environment: {}\n response_headers: {} # Additional response headers to send with responses\n\n vllm:\n command: \"vllm\"\n args: [\"serve\"]\n environment: {} # Environment variables for the backend process\n docker:\n enabled: false\n image: \"vllm/vllm-openai:latest\"\n args: [\"run\", \"--rm\", \"--network\", \"host\", \"--gpus\", \"all\", \"--shm-size\", \"1g\"]\n environment: {}\n response_headers: {} # Additional response headers to send with responses\n\n mlx:\n command: \"mlx_lm.server\"\n args: []\n environment: {} # Environment variables for the backend process\n response_headers: {} # Additional response headers to send with responses\n\ndata_dir: ~/.local/share/llamactl # Main data directory (database, instances, logs), default varies by OS\n\ninstances:\n port_range: [8000, 9000] # Port range for instances\n configs_dir: data_dir/instances # Instance configs directory\n logs_dir: data_dir/logs # Logs directory\n auto_create_dirs: true # Auto-create data/config/logs dirs if missing\n max_instances: -1 # Max instances (-1 = unlimited)\n max_running_instances: -1 # Max running instances (-1 = unlimited)\n enable_lru_eviction: true # Enable LRU eviction for idle instances\n default_auto_restart: true # Auto-restart new instances by default\n default_max_restarts: 3 # Max restarts for new instances\n default_restart_delay: 5 # Restart delay (seconds) for new instances\n default_on_demand_start: true # Default on-demand start setting\n on_demand_start_timeout: 120 # Default on-demand start timeout in seconds\n timeout_check_interval: 5 # Idle instance timeout check in minutes\n\ndatabase:\n path: data_dir/llamactl.db # Database file path\n max_open_connections: 25 # Maximum open database connections\n max_idle_connections: 5 # Maximum idle database connections\n connection_max_lifetime: 5m # Connection max lifetime\n\nauth:\n require_inference_auth: true # Require auth for inference endpoints\n require_management_auth: true # Require auth for management endpoints\n management_keys: [] # Keys for management endpoints\n\nlocal_node: \"main\" # Name of the local node (default: \"main\")\nnodes: # Node configuration for multi-node deployment\n main: # Default local node (empty config)\nConfiguration files are searched in the following locations (in order of precedence, first found is used):
Linux: - ./llamactl.yaml or ./config.yaml (current directory) - $HOME/.config/llamactl/config.yaml - /etc/llamactl/config.yaml
macOS: - ./llamactl.yaml or ./config.yaml (current directory) - $HOME/Library/Application Support/llamactl/config.yaml - /Library/Application Support/llamactl/config.yaml
Windows: - ./llamactl.yaml or ./config.yaml (current directory) - %APPDATA%\\llamactl\\config.yaml - %USERPROFILE%\\llamactl\\config.yaml - %PROGRAMDATA%\\llamactl\\config.yaml
You can specify the path to config file with LLAMACTL_CONFIG_PATH environment variable.
server:\n host: \"0.0.0.0\" # Server host to bind to (default: \"0.0.0.0\")\n port: 8080 # Server port to bind to (default: 8080)\n allowed_origins: [\"*\"] # CORS allowed origins (default: [\"*\"])\n allowed_headers: [\"*\"] # CORS allowed headers (default: [\"*\"])\n enable_swagger: false # Enable Swagger UI (default: false)\nEnvironment Variables: - LLAMACTL_HOST - Server host - LLAMACTL_PORT - Server port - LLAMACTL_ALLOWED_ORIGINS - Comma-separated CORS origins - LLAMACTL_ENABLE_SWAGGER - Enable Swagger UI (true/false)
backends:\n llama-cpp:\n command: \"llama-server\"\n args: []\n environment: {} # Environment variables for the backend process\n docker:\n enabled: false # Enable Docker runtime (default: false)\n image: \"ghcr.io/ggml-org/llama.cpp:server\"\n args: [\"run\", \"--rm\", \"--network\", \"host\", \"--gpus\", \"all\"]\n environment: {}\n response_headers: {} # Additional response headers to send with responses\n\n vllm:\n command: \"vllm\"\n args: [\"serve\"]\n environment: {} # Environment variables for the backend process\n docker:\n enabled: false # Enable Docker runtime (default: false)\n image: \"vllm/vllm-openai:latest\"\n args: [\"run\", \"--rm\", \"--network\", \"host\", \"--gpus\", \"all\", \"--shm-size\", \"1g\"]\n environment: {}\n response_headers: {} # Additional response headers to send with responses\n\n mlx:\n command: \"mlx_lm.server\"\n args: []\n environment: {} # Environment variables for the backend process\n # MLX does not support Docker\n response_headers: {} # Additional response headers to send with responses\nBackend Configuration Fields: - command: Executable name/path for the backend - args: Default arguments prepended to all instances - environment: Environment variables for the backend process (optional) - response_headers: Additional response headers to send with responses (optional) - docker: Docker-specific configuration (optional) - enabled: Boolean flag to enable Docker runtime - image: Docker image to use - args: Additional arguments passed to docker run - environment: Environment variables for the container (optional)
If llamactl is behind an NGINX proxy, X-Accel-Buffering: no response header may be required for NGINX to properly stream the responses without buffering.
Environment Variables:
LlamaCpp Backend: - LLAMACTL_LLAMACPP_COMMAND - LlamaCpp executable command - LLAMACTL_LLAMACPP_ARGS - Space-separated default arguments - LLAMACTL_LLAMACPP_ENV - Environment variables in format \"KEY1=value1,KEY2=value2\" - LLAMACTL_LLAMACPP_DOCKER_ENABLED - Enable Docker runtime (true/false) - LLAMACTL_LLAMACPP_DOCKER_IMAGE - Docker image to use - LLAMACTL_LLAMACPP_DOCKER_ARGS - Space-separated Docker arguments - LLAMACTL_LLAMACPP_DOCKER_ENV - Docker environment variables in format \"KEY1=value1,KEY2=value2\" - LLAMACTL_LLAMACPP_RESPONSE_HEADERS - Response headers in format \"KEY1=value1;KEY2=value2\"
VLLM Backend: - LLAMACTL_VLLM_COMMAND - VLLM executable command - LLAMACTL_VLLM_ARGS - Space-separated default arguments - LLAMACTL_VLLM_ENV - Environment variables in format \"KEY1=value1,KEY2=value2\" - LLAMACTL_VLLM_DOCKER_ENABLED - Enable Docker runtime (true/false) - LLAMACTL_VLLM_DOCKER_IMAGE - Docker image to use - LLAMACTL_VLLM_DOCKER_ARGS - Space-separated Docker arguments - LLAMACTL_VLLM_DOCKER_ENV - Docker environment variables in format \"KEY1=value1,KEY2=value2\" - LLAMACTL_VLLM_RESPONSE_HEADERS - Response headers in format \"KEY1=value1;KEY2=value2\"
MLX Backend: - LLAMACTL_MLX_COMMAND - MLX executable command - LLAMACTL_MLX_ARGS - Space-separated default arguments - LLAMACTL_MLX_ENV - Environment variables in format \"KEY1=value1,KEY2=value2\" - LLAMACTL_MLX_RESPONSE_HEADERS - Response headers in format \"KEY1=value1;KEY2=value2\"
data_dir: \"~/.local/share/llamactl\" # Main data directory for database, instances, and logs (default varies by OS)\nEnvironment Variables: - LLAMACTL_DATA_DIRECTORY - Main data directory path
Default Data Directory by Platform: - Linux: ~/.local/share/llamactl - macOS: ~/Library/Application Support/llamactl - Windows: %LOCALAPPDATA%\\llamactl or %PROGRAMDATA%\\llamactl
instances:\n port_range: [8000, 9000] # Port range for instances (default: [8000, 9000])\n configs_dir: \"instances\" # Directory for instance configs, default: data_dir/instances\n logs_dir: \"logs\" # Directory for instance logs, default: data_dir/logs\n auto_create_dirs: true # Automatically create data/config/logs directories (default: true)\n max_instances: -1 # Maximum instances (-1 = unlimited)\n max_running_instances: -1 # Maximum running instances (-1 = unlimited)\n enable_lru_eviction: true # Enable LRU eviction for idle instances\n default_auto_restart: true # Default auto-restart setting\n default_max_restarts: 3 # Default maximum restart attempts\n default_restart_delay: 5 # Default restart delay in seconds\n default_on_demand_start: true # Default on-demand start setting\n on_demand_start_timeout: 120 # Default on-demand start timeout in seconds\n timeout_check_interval: 5 # Default instance timeout check interval in minutes\nEnvironment Variables: - LLAMACTL_INSTANCE_PORT_RANGE - Port range (format: \"8000-9000\" or \"8000,9000\") - LLAMACTL_INSTANCES_DIR - Instance configs directory path - LLAMACTL_LOGS_DIR - Log directory path - LLAMACTL_AUTO_CREATE_DATA_DIR - Auto-create data/config/logs directories (true/false) - LLAMACTL_MAX_INSTANCES - Maximum number of instances - LLAMACTL_MAX_RUNNING_INSTANCES - Maximum number of running instances - LLAMACTL_ENABLE_LRU_EVICTION - Enable LRU eviction for idle instances - LLAMACTL_DEFAULT_AUTO_RESTART - Default auto-restart setting (true/false) - LLAMACTL_DEFAULT_MAX_RESTARTS - Default maximum restarts - LLAMACTL_DEFAULT_RESTART_DELAY - Default restart delay in seconds - LLAMACTL_DEFAULT_ON_DEMAND_START - Default on-demand start setting (true/false) - LLAMACTL_ON_DEMAND_START_TIMEOUT - Default on-demand start timeout in seconds - LLAMACTL_TIMEOUT_CHECK_INTERVAL - Default instance timeout check interval in minutes
database:\n path: \"llamactl.db\" # Database file path, default: data_dir/llamactl.db\n max_open_connections: 25 # Maximum open database connections (default: 25)\n max_idle_connections: 5 # Maximum idle database connections (default: 5)\n connection_max_lifetime: 5m # Connection max lifetime (default: 5m)\nEnvironment Variables: - LLAMACTL_DATABASE_PATH - Database file path (relative to data_dir or absolute) - LLAMACTL_DATABASE_MAX_OPEN_CONNECTIONS - Maximum open database connections - LLAMACTL_DATABASE_MAX_IDLE_CONNECTIONS - Maximum idle database connections - LLAMACTL_DATABASE_CONN_MAX_LIFETIME - Connection max lifetime (e.g., \"5m\", \"1h\")
llamactl supports two types of authentication:
auth:\n require_inference_auth: true # Require API key for OpenAI endpoints (default: true)\n require_management_auth: true # Require API key for management endpoints (default: true)\n management_keys: [] # List of valid management API keys\nManaging Inference API Keys:
Inference API keys are managed through the web UI or management API and stored in the database. To create and manage inference keys:
Environment Variables: - LLAMACTL_REQUIRE_INFERENCE_AUTH - Require auth for OpenAI endpoints (true/false) - LLAMACTL_REQUIRE_MANAGEMENT_AUTH - Require auth for management endpoints (true/false) - LLAMACTL_MANAGEMENT_KEYS - Comma-separated management API keys
llamactl supports remote node deployments. Configure remote nodes to deploy instances on remote hosts and manage them centrally.
local_node: \"main\" # Name of the local node (default: \"main\")\nnodes: # Node configuration map\n main: # Local node (empty address means local)\n address: \"\" # Not used for local node\n api_key: \"\" # Not used for local node\n worker1: # Remote worker node\n address: \"http://192.168.1.10:8080\"\n api_key: \"worker1-api-key\" # Management API key for authentication\nNode Configuration Fields: - local_node: Specifies which node in the nodes map represents the local node. Must match exactly what other nodes call this node. - nodes: Map of node configurations - address: HTTP/HTTPS URL of the remote node (empty for local node) - api_key: Management API key for authenticating with the remote node
Environment Variables: - LLAMACTL_LOCAL_NODE - Name of the local node
This guide will walk you through installing Llamactl on your system.
"},{"location":"installation/#prerequisites","title":"Prerequisites","text":""},{"location":"installation/#backend-dependencies","title":"Backend Dependencies","text":"llamactl supports multiple backends. Install at least one:
For llama.cpp backend (all platforms):
You need llama-server from llama.cpp installed:
# Homebrew (macOS/Linux)\nbrew install llama.cpp\n# Winget (Windows)\nwinget install llama.cpp\nOr build from source - see llama.cpp docs
For MLX backend (macOS only):
MLX provides optimized inference on Apple Silicon. Install MLX-LM:
# Install via pip (requires Python 3.8+)\npip install mlx-lm\n\n# Or in a virtual environment (recommended)\npython -m venv mlx-env\nsource mlx-env/bin/activate\npip install mlx-lm\nNote: MLX backend is only available on macOS with Apple Silicon (M1, M2, M3, etc.)
For vLLM backend:
vLLM provides high-throughput distributed serving for LLMs. Install vLLM:
# Install in a virtual environment\npython -m venv vllm-env\nsource vllm-env/bin/activate\npip install vllm\nDownload the latest release from the GitHub releases page:
# Linux/macOS - Get latest version and download\nLATEST_VERSION=$(curl -s https://api.github.com/repos/lordmathis/llamactl/releases/latest | grep '\"tag_name\":' | sed -E 's/.*\"([^\"]+)\".*/\\1/')\ncurl -L https://github.com/lordmathis/llamactl/releases/download/${LATEST_VERSION}/llamactl-${LATEST_VERSION}-$(uname -s | tr '[:upper:]' '[:lower:]')-$(uname -m).tar.gz | tar -xz\nsudo mv llamactl /usr/local/bin/\n\n# Or download manually from:\n# https://github.com/lordmathis/llamactl/releases/latest\n\n# Windows - Download from releases page\nllamactl provides Dockerfiles for creating Docker images with backends pre-installed. The resulting images include the latest llamactl release with the respective backend.
Available Dockerfiles (CUDA): - llamactl with llama.cpp CUDA: docker/Dockerfile.llamacpp (based on ghcr.io/ggml-org/llama.cpp:server-cuda) - llamactl with vLLM CUDA: docker/Dockerfile.vllm (based on vllm/vllm-openai:latest) - llamactl built from source: docker/Dockerfile.source (multi-stage build with webui)
Note: These Dockerfiles are configured for CUDA. For other platforms (CPU, ROCm, Vulkan, etc.), adapt the base image. For llama.cpp, see available tags at llama.cpp Docker docs. For vLLM, check vLLM docs.
Using Docker Compose
# Clone the repository\ngit clone https://github.com/lordmathis/llamactl.git\ncd llamactl\n\n# Create directories for data and models\nmkdir -p data/llamacpp data/vllm models\n\n# Start llamactl with llama.cpp backend\ndocker-compose -f docker/docker-compose.yml up llamactl-llamacpp -d\n\n# Or start llamactl with vLLM backend\ndocker-compose -f docker/docker-compose.yml up llamactl-vllm -d\nAccess the dashboard at: - llamactl with llama.cpp: http://localhost:8080 - llamactl with vLLM: http://localhost:8081
Using Docker Build and Run
llamactl with llama.cpp CUDA:
docker build -f docker/Dockerfile.llamacpp -t llamactl:llamacpp-cuda .\ndocker run -d \\\n --name llamactl-llamacpp \\\n --gpus all \\\n -p 8080:8080 \\\n -v ~/.cache/llama.cpp:/root/.cache/llama.cpp \\\n llamactl:llamacpp-cuda\nllamactl with vLLM CUDA:
docker build -f docker/Dockerfile.vllm -t llamactl:vllm-cuda .\ndocker run -d \\\n --name llamactl-vllm \\\n --gpus all \\\n -p 8080:8080 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n llamactl:vllm-cuda\nllamactl built from source:
docker build -f docker/Dockerfile.source -t llamactl:source .\ndocker run -d \\\n --name llamactl \\\n -p 8080:8080 \\\n llamactl:source\nRequirements: - Go 1.24 or later - Node.js 22 or later - Git
If you prefer to build from source:
# Clone the repository\ngit clone https://github.com/lordmathis/llamactl.git\ncd llamactl\n\n# Build the web UI\ncd webui && npm ci && npm run build && cd ..\n\n# Build the application\ngo build -o llamactl ./cmd/server\nFor deployments with remote nodes: - Install llamactl on each node using any of the methods above - Configure API keys for authentication between nodes - Ensure node names are consistent across all configurations
"},{"location":"installation/#verification","title":"Verification","text":"Verify your installation by checking the version:
llamactl --version\nNow that Llamactl is installed, continue to the Quick Start guide to get your first instance running!
For remote node deployments, see the Configuration Guide for node setup instructions.
"},{"location":"managing-instances/","title":"Managing Instances","text":"Learn how to effectively manage your llama.cpp, MLX, and vLLM instances with Llamactl through both the Web UI and API.
"},{"location":"managing-instances/#overview","title":"Overview","text":"Llamactl provides two ways to manage instances:
http://localhost:8080 with an intuitive dashboard "},{"location":"managing-instances/#authentication","title":"Authentication","text":"
Llamactl uses a Management API Key to authenticate requests to the management API (creating, starting, stopping instances). All curl examples below use <token> as a placeholder - replace this with your actual Management API Key.
By default, authentication is required. If you don't configure a management API key in your configuration file, llamactl will auto-generate one and print it to the terminal on startup. See the Configuration guide for details.
For Web UI access: 1. Navigate to the web UI 2. Enter your Management API Key 3. Bearer token is stored for the session
"},{"location":"managing-instances/#theme-support","title":"Theme Support","text":"Each instance is displayed as a card showing:
Via Web UI
Instance Settings:
Backend Configuration:
Auto-Assignment
Llamactl automatically assigns ports from the configured port range (default: 8000-9000) and generates API keys if authentication is enabled. You typically don't need to manually specify these values.
Via API
# Create llama.cpp instance with local model file\ncurl -X POST http://localhost:8080/api/v1/instances/my-llama-instance \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer <token>\" \\\n -d '{\n \"backend_type\": \"llama_cpp\",\n \"backend_options\": {\n \"model\": \"/path/to/model.gguf\",\n \"threads\": 8,\n \"ctx_size\": 4096,\n \"gpu_layers\": 32,\n \"flash_attn\": \"on\"\n },\n \"auto_restart\": true,\n \"max_restarts\": 3,\n \"docker_enabled\": false,\n \"command_override\": \"/opt/llama-server-dev\",\n \"nodes\": [\"main\"]\n }'\n\n# Create vLLM instance with environment variables\ncurl -X POST http://localhost:8080/api/v1/instances/my-vllm-instance \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer <token>\" \\\n -d '{\n \"backend_type\": \"vllm\",\n \"backend_options\": {\n \"model\": \"microsoft/DialoGPT-medium\",\n \"tensor_parallel_size\": 2,\n \"gpu_memory_utilization\": 0.9\n },\n \"on_demand_start\": true,\n \"environment\": {\n \"CUDA_VISIBLE_DEVICES\": \"0,1\"\n },\n \"nodes\": [\"worker1\", \"worker2\"]\n }'\n\n# Create MLX instance (macOS only)\ncurl -X POST http://localhost:8080/api/v1/instances/my-mlx-instance \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer <token>\" \\\n -d '{\n \"backend_type\": \"mlx_lm\",\n \"backend_options\": {\n \"model\": \"mlx-community/Mistral-7B-Instruct-v0.3-4bit\",\n \"temp\": 0.7,\n \"max_tokens\": 2048\n },\n \"nodes\": [\"main\"]\n }'\nVia Web UI 1. Click the \"Start\" button on an instance card 2. Watch the status change to \"Unknown\" 3. Monitor progress in the logs 4. Instance status changes to \"Ready\" when ready
Via API
curl -X POST http://localhost:8080/api/v1/instances/{name}/start \\\n -H \"Authorization: Bearer <token>\"\nVia Web UI 1. Click the \"Stop\" button on an instance card 2. Instance gracefully shuts down
Via API
curl -X POST http://localhost:8080/api/v1/instances/{name}/stop \\\n -H \"Authorization: Bearer <token>\"\nVia Web UI 1. Click the \"Edit\" button on an instance card 2. Modify settings in the configuration dialog 3. Changes require instance restart to take effect 4. Click \"Update & Restart\" to apply changes
Via API Modify instance settings:
curl -X PUT http://localhost:8080/api/v1/instances/{name} \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer <token>\" \\\n -d '{\n \"backend_options\": {\n \"threads\": 8,\n \"context_size\": 4096\n }\n }'\nNote
Configuration changes require restarting the instance to take effect.
"},{"location":"managing-instances/#export-instance","title":"Export Instance","text":"Via Web UI 1. Click the \"More actions\" button (three dots) on an instance card 2. Click \"Export\" to download the instance configuration as a JSON file
"},{"location":"managing-instances/#view-logs","title":"View Logs","text":"Via Web UI
Via API Check instance status in real-time:
# Get instance logs\ncurl http://localhost:8080/api/v1/instances/{name}/logs \\\n -H \"Authorization: Bearer <token>\"\nVia Web UI 1. Click the \"Delete\" button on an instance card 2. Only stopped instances can be deleted 3. Confirm deletion in the dialog
Via API
curl -X DELETE http://localhost:8080/api/v1/instances/{name} \\\n -H \"Authorization: Bearer <token>\"\nLlamactl proxies all requests to the underlying backend instances (llama-server, MLX, or vLLM).
# Proxy requests to the instance\ncurl http://localhost:8080/api/v1/instances/{name}/proxy/ \\\n -H \"Authorization: Bearer <token>\"\nAll backends provide OpenAI-compatible endpoints. Check the respective documentation: - llama-server docs - MLX-LM docs - vLLM docs
"},{"location":"managing-instances/#instance-health","title":"Instance Health","text":"Via Web UI
Via API
Check the health status of your instances:
curl http://localhost:8080/api/v1/instances/{name}/proxy/health \\\n -H \"Authorization: Bearer <token>\"\nThis guide will help you get Llamactl up and running in just a few minutes.
Before you begin: Ensure you have at least one backend installed (llama.cpp, MLX, or vLLM). See the Installation Guide for backend setup.
"},{"location":"quick-start/#core-concepts","title":"Core Concepts","text":"Before you start, let's clarify a few key terms:
http://localhost:8080/v1/chat/completions), and it routes them to the appropriate backend instance. This means you don't need to track individual instance ports or endpoints. Llamactl uses two types of API keys:
/v1/chat/completions, /v1/completions, etc.). These are created and managed via the web UI. By default, authentication is required for both management and inference endpoints. You can configure custom management keys or disable authentication in the Configuration guide.
"},{"location":"quick-start/#start-llamactl","title":"Start Llamactl","text":"Start the Llamactl server:
llamactl\n\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\n\u26a0\ufe0f MANAGEMENT AUTHENTICATION REQUIRED\n\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\n\ud83d\udd11 Generated Management API Key:\n\n sk-management-...\n\n\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\n\u26a0\ufe0f IMPORTANT\n\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\n\u2022 This key is auto-generated and will change on restart\n\u2022 For production, add explicit management_keys to your configuration\n\u2022 Copy this key before it disappears from the terminal\n\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\nLlamactl server listening on 0.0.0.0:8080\nCopy the Management API Key from the terminal - you'll need it to access the web UI.
By default, Llamactl will start on http://localhost:8080.
Open your web browser and navigate to:
http://localhost:8080\nLogin with the management API key from the terminal output.
You should see the Llamactl web interface.
"},{"location":"quick-start/#create-your-first-instance","title":"Create Your First Instance","text":"Fill in the instance configuration:
Auto-Assignment
Llamactl automatically assigns ports from the configured port range (default: 8000-9000) and manages API keys if authentication is enabled. You typically don't need to manually specify these values.
Remote Node Deployment
If you have configured remote nodes in your configuration file, you can select which node to deploy the instance to. This allows you to distribute instances across multiple machines. See the Configuration guide for details on setting up remote nodes.
Click \"Create Instance\"
Once created, you can:
To make inference requests to your instances, you'll need an inference API key:
The key will look like: llamactl-...
You can create multiple inference keys with different permissions for different use cases (e.g., one for development, one for production, or keys limited to specific instances).
"},{"location":"quick-start/#example-configurations","title":"Example Configurations","text":"Here are basic example configurations for each backend:
llama.cpp backend:
{\n \"name\": \"llama2-7b\",\n \"backend_type\": \"llama_cpp\",\n \"backend_options\": {\n \"model\": \"/path/to/llama-2-7b-chat.gguf\",\n \"threads\": 4,\n \"ctx_size\": 2048,\n \"gpu_layers\": 32\n },\n \"nodes\": [\"main\"]\n}\nMLX backend (macOS only):
{\n \"name\": \"mistral-mlx\",\n \"backend_type\": \"mlx_lm\",\n \"backend_options\": {\n \"model\": \"mlx-community/Mistral-7B-Instruct-v0.3-4bit\",\n \"temp\": 0.7,\n \"max_tokens\": 2048\n },\n \"nodes\": [\"main\"]\n}\nvLLM backend:
{\n \"name\": \"dialogpt-vllm\",\n \"backend_type\": \"vllm\",\n \"backend_options\": {\n \"model\": \"microsoft/DialoGPT-medium\",\n \"tensor_parallel_size\": 2,\n \"gpu_memory_utilization\": 0.9\n },\n \"nodes\": [\"main\"]\n}\nRemote node deployment example:
{\n \"name\": \"distributed-model\",\n \"backend_type\": \"llama_cpp\",\n \"backend_options\": {\n \"model\": \"/path/to/model.gguf\",\n \"gpu_layers\": 32\n },\n \"nodes\": [\"worker1\"]\n}\nLlamactl can run backends in Docker containers. To enable Docker for a backend, add a docker section to that backend in your YAML configuration file (e.g. config.yaml) as shown below:
backends:\n vllm:\n command: \"vllm\"\n args: [\"serve\"]\n docker:\n enabled: true\n image: \"vllm/vllm-openai:latest\"\n args: [\"run\", \"--rm\", \"--network\", \"host\", \"--gpus\", \"all\", \"--shm-size\", \"1g\"]\nYou can also manage instances via the REST API:
# List all instances\ncurl http://localhost:8080/api/v1/instances\n\n# Create a new llama.cpp instance\ncurl -X POST http://localhost:8080/api/v1/instances/my-model \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"backend_type\": \"llama_cpp\",\n \"backend_options\": {\n \"model\": \"/path/to/model.gguf\"\n }\n }'\n\n# Start an instance\ncurl -X POST http://localhost:8080/api/v1/instances/my-model/start\nLlamactl provides OpenAI-compatible endpoints, making it easy to integrate with existing OpenAI client libraries and tools.
"},{"location":"quick-start/#chat-completions","title":"Chat Completions","text":"Once you have an instance running, you can use it with the OpenAI-compatible chat completions endpoint:
curl -X POST http://localhost:8080/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"my-model\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hello! Can you help me write a Python function?\"\n }\n ],\n \"max_tokens\": 150,\n \"temperature\": 0.7\n }'\nYou can also use the official OpenAI Python client:
from openai import OpenAI\n\n# Point the client to your Llamactl server\nclient = OpenAI(\n base_url=\"http://localhost:8080/v1\",\n api_key=\"your-inference-api-key\" # Use the inference API key from terminal or config\n)\n\n# Create a chat completion\nresponse = client.chat.completions.create(\n model=\"my-model\", # Use the name of your instance\n messages=[\n {\"role\": \"user\", \"content\": \"Explain quantum computing in simple terms\"}\n ],\n max_tokens=200,\n temperature=0.7\n)\n\nprint(response.choices[0].message.content)\nAPI Key
If you disabled authentication in your config, you can use any value for api_key (e.g., \"not-needed\"). Otherwise, use the inference API key you created via the web UI (Settings \u2192 API Keys).
Get a list of running instances (models) in OpenAI-compatible format:
curl http://localhost:8080/v1/models\nIssues specific to Llamactl deployment and operation.
"},{"location":"troubleshooting/#configuration-issues","title":"Configuration Issues","text":""},{"location":"troubleshooting/#invalid-configuration","title":"Invalid Configuration","text":"Problem: Invalid configuration preventing startup
Solutions: 1. Use minimal configuration:
server:\n host: \"0.0.0.0\"\n port: 8080\ninstances:\n port_range: [8000, 9000]\n# Ensure data directory is writable (default: ~/.local/share/llamactl)\nmkdir -p ~/.local/share/llamactl/{instances,logs}\nProblem: Instance fails to start or immediately stops
Solutions:
Check instance logs to see the actual error:
curl http://localhost:8080/api/v1/instances/{name}/logs\n# Or check log files directly\ntail -f ~/.local/share/llamactl/logs/{instance-name}.log\nVerify backend is installed:
llama-server is in PATH mlx-lm Python package is installed vllm Python package is installed Check model path and format:
Verify backend command configuration:
command is correctly configured in the global config /path/to/venv/bin/mlx_lm.server) Problem: Model loading, memory, GPU, or performance issues
Most model-specific issues (memory, GPU configuration, performance tuning) are backend-specific and should be resolved by consulting the respective backend documentation:
llama.cpp: - llama.cpp GitHub - llama-server README
MLX: - MLX-LM GitHub - MLX-LM Server Guide
vLLM: - vLLM Documentation - OpenAI Compatible Server - vllm serve Command
Testing backends directly:
Testing your model and configuration directly with the backend helps determine if the issue is with llamactl or the backend itself:
# llama.cpp\nllama-server --model /path/to/model.gguf --port 8081\n\n# MLX\nmlx_lm.server --model mlx-community/Mistral-7B-Instruct-v0.3-4bit --port 8081\n\n# vLLM\nvllm serve microsoft/DialoGPT-medium --port 8081\nProblem: Web UI shows CORS errors in browser console
Solutions: 1. Configure allowed origins:
server:\n allowed_origins:\n - \"http://localhost:3000\"\n - \"https://yourdomain.com\"\nProblem: API requests failing with authentication errors
Solutions: 1. Disable authentication temporarily:
auth:\n require_management_auth: false\n require_inference_auth: false\nauth:\n management_keys:\n - \"your-management-key\"\nFor inference API keys, create them via the web UI (Settings \u2192 API Keys) after logging in with your management key.
curl -H \"Authorization: Bearer your-api-key\" \\\n http://localhost:8080/api/v1/instances\nProblem: Remote instances not appearing or cannot be managed
Solutions: 1. Verify node configuration:
local_node: \"main\" # Must match a key in nodes map\nnodes:\n main:\n address: \"\" # Empty for local node\n worker1:\n address: \"http://worker1.internal:8080\"\n api_key: \"secure-key\" # Must match worker1's management key\nlocal_node on each node must match what other nodes call it Node names are case-sensitive
Test remote node connectivity:
curl -H \"Authorization: Bearer remote-node-key\" \\\n http://remote-node:8080/api/v1/instances\n# Get instance logs via API\ncurl http://localhost:8080/api/v1/instances/{name}/logs\n\n# Or check log files directly\ntail -f ~/.local/share/llamactl/logs/{instance-name}.log\nexport LLAMACTL_LOG_LEVEL=debug\nllamactl\nWhen reporting issues, include:
System information:
llamactl --version\nConfiguration file (remove sensitive keys)
Relevant log output
Steps to reproduce the issue
Configure API keys:
+
Use correct Authorization header:
+
For inference API keys, create them via the web UI (Settings → API Keys) after logging in with your management key.
+