Llamactl Documentation¶
-Welcome to the Llamactl documentation! Management server and proxy for multiple llama.cpp and MLX instances with OpenAI-compatible API routing.
+Welcome to the Llamactl documentation!
What is Llamactl?¶
-Llamactl is designed to simplify the deployment and management of llama-server and MLX instances. It provides a modern solution for running multiple large language models with centralized management and multi-backend support.
+Unified management and routing for llama.cpp, MLX and vLLM models with web dashboard.
Features¶
-🚀 Multiple Model Serving: Run different models simultaneously (7B for speed, 70B for quality) -🔗 OpenAI API Compatible: Drop-in replacement - route requests by model name -🍎 Multi-Backend Support: Native support for both llama.cpp and MLX (Apple Silicon optimized) -🌐 Web Dashboard: Modern React UI for visual management (unlike CLI-only tools) -🔐 API Key Authentication: Separate keys for management vs inference access -📊 Instance Monitoring: Health checks, auto-restart, log management -⚡ Smart Resource Management: Idle timeout, LRU eviction, and configurable instance limits -💡 On-Demand Instance Start: Automatically launch instances upon receiving OpenAI-compatible API requests -💾 State Persistence: Ensure instances remain intact across server restarts
+🚀 Easy Model Management¶
+-
+
- Multiple Model Serving: Run different models simultaneously (7B for speed, 70B for quality) +
- On-Demand Instance Start: Automatically launch instances upon receiving API requests +
- State Persistence: Ensure instances remain intact across server restarts +
🔗 Universal Compatibility¶
+-
+
- OpenAI API Compatible: Drop-in replacement - route requests by instance name +
- Multi-Backend Support: Native support for llama.cpp, MLX (Apple Silicon optimized), and vLLM +
🌐 User-Friendly Interface¶
+-
+
- Web Dashboard: Modern React UI for visual management (unlike CLI-only tools) +
- API Key Authentication: Separate keys for management vs inference access +
⚡ Smart Operations¶
+-
+
- Instance Monitoring: Health checks, auto-restart, log management +
- Smart Resource Management: Idle timeout, LRU eviction, and configurable instance limits +
Quick Links¶
- Installation Guide - Get Llamactl up and running @@ -791,7 +888,7 @@ - September 18, 2025 + September 22, 2025 diff --git a/dev/readme_sync.py b/dev/readme_sync.py new file mode 100644 index 0000000..df2a3f2 --- /dev/null +++ b/dev/readme_sync.py @@ -0,0 +1,62 @@ +""" +MkDocs hook to sync content from README.md to docs/index.md +""" +import re +import os + + +def on_page_markdown(markdown, page, config, **kwargs): + """Process markdown content before rendering""" + # Only process the index.md file + if page.file.src_path != 'index.md': + return markdown + + # Get the path to README.md (relative to mkdocs.yml) + readme_path = os.path.join(os.path.dirname(config['config_file_path']), 'README.md') + + if not os.path.exists(readme_path): + print(f"Warning: README.md not found at {readme_path}") + return markdown + + try: + with open(readme_path, 'r', encoding='utf-8') as f: + readme_content = f.read() + except Exception as e: + print(f"Error reading README.md: {e}") + return markdown + + # Extract headline (the text in bold after the title) + headline_match = re.search(r'\*\*(.*?)\*\*', readme_content) + headline = headline_match.group(1) if headline_match else 'Management server for llama.cpp and MLX instances' + + # Extract features section - everything between ## Features and the next ## heading + features_match = re.search(r'## Features\n(.*?)(?=\n## |\Z)', readme_content, re.DOTALL) + if features_match: + features_content = features_match.group(1).strip() + # Just add line breaks at the end of each line for proper MkDocs rendering + features_with_breaks = add_line_breaks(features_content) + else: + features_with_breaks = "Features content not found in README.md" + + # Replace placeholders in the markdown + markdown = markdown.replace('{{HEADLINE}}', headline) + markdown = markdown.replace('{{FEATURES}}', features_with_breaks) + + # Fix image paths: convert docs/images/ to images/ for MkDocs + markdown = re.sub(r'docs/images/', 'images/', markdown) + + return markdown + + +def add_line_breaks(content): + """Add two spaces at the end of each line for proper MkDocs line breaks""" + lines = content.split('\n') + processed_lines = [] + + for line in lines: + if line.strip(): # Only add spaces to non-empty lines + processed_lines.append(line.rstrip() + ' ') + else: + processed_lines.append(line) + + return '\n'.join(processed_lines) \ No newline at end of file diff --git a/dev/search/search_index.json b/dev/search/search_index.json index 33d6a9b..3a6b55a 100644 --- a/dev/search/search_index.json +++ b/dev/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Llamactl Documentation","text":"
- Installation Guide - Get Llamactl up and running
- Configuration Guide - Detailed configuration options
- Quick Start - Your first steps with Llamactl
- Managing Instances - Instance lifecycle management
- API Reference - Complete API documentation
- Check the Troubleshooting guide
- Visit the GitHub repository
- Review the Configuration Guide for advanced settings
Welcome to the Llamactl documentation! Management server and proxy for multiple llama.cpp and MLX instances with OpenAI-compatible API routing.
"},{"location":"#what-is-llamactl","title":"What is Llamactl?","text":"Llamactl is designed to simplify the deployment and management of llama-server and MLX instances. It provides a modern solution for running multiple large language models with centralized management and multi-backend support.
"},{"location":"#features","title":"Features","text":"\ud83d\ude80 Multiple Model Serving: Run different models simultaneously (7B for speed, 70B for quality) \ud83d\udd17 OpenAI API Compatible: Drop-in replacement - route requests by model name \ud83c\udf4e Multi-Backend Support: Native support for both llama.cpp and MLX (Apple Silicon optimized) \ud83c\udf10 Web Dashboard: Modern React UI for visual management (unlike CLI-only tools) \ud83d\udd10 API Key Authentication: Separate keys for management vs inference access \ud83d\udcca Instance Monitoring: Health checks, auto-restart, log management \u26a1 Smart Resource Management: Idle timeout, LRU eviction, and configurable instance limits \ud83d\udca1 On-Demand Instance Start: Automatically launch instances upon receiving OpenAI-compatible API requests \ud83d\udcbe State Persistence: Ensure instances remain intact across server restarts
"},{"location":"#quick-links","title":"Quick Links","text":"If you need help or have questions:
MIT License - see the LICENSE file.
"},{"location":"getting-started/configuration/","title":"Configuration","text":"llamactl can be configured via configuration files or environment variables. Configuration is loaded in the following order of precedence:
Defaults < Configuration file < Environment variables\nllamactl works out of the box with sensible defaults, but you can customize the behavior to suit your needs.
"},{"location":"getting-started/configuration/#default-configuration","title":"Default Configuration","text":"Here's the default configuration with all available options:
server:\n host: \"0.0.0.0\" # Server host to bind to\n port: 8080 # Server port to bind to\n allowed_origins: [\"*\"] # Allowed CORS origins (default: all)\n enable_swagger: false # Enable Swagger UI for API docs\n\nbackends:\n llama_executable: llama-server # Path to llama-server executable\n mlx_lm_executable: mlx_lm.server # Path to mlx_lm.server executable\n vllm_executable: vllm # Path to vllm executable\n\ninstances:\n port_range: [8000, 9000] # Port range for instances\n data_dir: ~/.local/share/llamactl # Data directory (platform-specific, see below)\n configs_dir: ~/.local/share/llamactl/instances # Instance configs directory\n logs_dir: ~/.local/share/llamactl/logs # Logs directory\n auto_create_dirs: true # Auto-create data/config/logs dirs if missing\n max_instances: -1 # Max instances (-1 = unlimited)\n max_running_instances: -1 # Max running instances (-1 = unlimited)\n enable_lru_eviction: true # Enable LRU eviction for idle instances\n default_auto_restart: true # Auto-restart new instances by default\n default_max_restarts: 3 # Max restarts for new instances\n default_restart_delay: 5 # Restart delay (seconds) for new instances\n default_on_demand_start: true # Default on-demand start setting\n on_demand_start_timeout: 120 # Default on-demand start timeout in seconds\n timeout_check_interval: 5 # Idle instance timeout check in minutes\n\nauth:\n require_inference_auth: true # Require auth for inference endpoints\n inference_keys: [] # Keys for inference endpoints\n require_management_auth: true # Require auth for management endpoints\n management_keys: [] # Keys for management endpoints\nConfiguration files are searched in the following locations (in order of precedence):
Linux: - ./llamactl.yaml or ./config.yaml (current directory) - $HOME/.config/llamactl/config.yaml - /etc/llamactl/config.yaml
macOS: - ./llamactl.yaml or ./config.yaml (current directory) - $HOME/Library/Application Support/llamactl/config.yaml - /Library/Application Support/llamactl/config.yaml
Windows: - ./llamactl.yaml or ./config.yaml (current directory) - %APPDATA%\\llamactl\\config.yaml - %USERPROFILE%\\llamactl\\config.yaml - %PROGRAMDATA%\\llamactl\\config.yaml
You can specify the path to config file with LLAMACTL_CONFIG_PATH environment variable.
server:\n host: \"0.0.0.0\" # Server host to bind to (default: \"0.0.0.0\")\n port: 8080 # Server port to bind to (default: 8080)\n allowed_origins: [\"*\"] # CORS allowed origins (default: [\"*\"])\n enable_swagger: false # Enable Swagger UI (default: false)\nEnvironment Variables: - LLAMACTL_HOST - Server host - LLAMACTL_PORT - Server port - LLAMACTL_ALLOWED_ORIGINS - Comma-separated CORS origins - LLAMACTL_ENABLE_SWAGGER - Enable Swagger UI (true/false)
backends:\n llama_executable: \"llama-server\" # Path to llama-server executable (default: \"llama-server\")\n mlx_lm_executable: \"mlx_lm.server\" # Path to mlx_lm.server executable (default: \"mlx_lm.server\")\n vllm_executable: \"vllm\" # Path to vllm executable (default: \"vllm\")\nEnvironment Variables: - LLAMACTL_LLAMA_EXECUTABLE - Path to llama-server executable - LLAMACTL_MLX_LM_EXECUTABLE - Path to mlx_lm.server executable - LLAMACTL_VLLM_EXECUTABLE - Path to vllm executable
instances:\n port_range: [8000, 9000] # Port range for instances (default: [8000, 9000])\n data_dir: \"~/.local/share/llamactl\" # Directory for all llamactl data (default varies by OS)\n configs_dir: \"~/.local/share/llamactl/instances\" # Directory for instance configs (default: data_dir/instances)\n logs_dir: \"~/.local/share/llamactl/logs\" # Directory for instance logs (default: data_dir/logs)\n auto_create_dirs: true # Automatically create data/config/logs directories (default: true)\n max_instances: -1 # Maximum instances (-1 = unlimited)\n max_running_instances: -1 # Maximum running instances (-1 = unlimited)\n enable_lru_eviction: true # Enable LRU eviction for idle instances\n default_auto_restart: true # Default auto-restart setting\n default_max_restarts: 3 # Default maximum restart attempts\n default_restart_delay: 5 # Default restart delay in seconds\n default_on_demand_start: true # Default on-demand start setting\n on_demand_start_timeout: 120 # Default on-demand start timeout in seconds\n timeout_check_interval: 5 # Default instance timeout check interval in minutes\nEnvironment Variables: - LLAMACTL_INSTANCE_PORT_RANGE - Port range (format: \"8000-9000\" or \"8000,9000\") - LLAMACTL_DATA_DIRECTORY - Data directory path - LLAMACTL_INSTANCES_DIR - Instance configs directory path - LLAMACTL_LOGS_DIR - Log directory path - LLAMACTL_AUTO_CREATE_DATA_DIR - Auto-create data/config/logs directories (true/false) - LLAMACTL_MAX_INSTANCES - Maximum number of instances - LLAMACTL_MAX_RUNNING_INSTANCES - Maximum number of running instances - LLAMACTL_ENABLE_LRU_EVICTION - Enable LRU eviction for idle instances - LLAMACTL_DEFAULT_AUTO_RESTART - Default auto-restart setting (true/false) - LLAMACTL_DEFAULT_MAX_RESTARTS - Default maximum restarts - LLAMACTL_DEFAULT_RESTART_DELAY - Default restart delay in seconds - LLAMACTL_DEFAULT_ON_DEMAND_START - Default on-demand start setting (true/false) - LLAMACTL_ON_DEMAND_START_TIMEOUT - Default on-demand start timeout in seconds - LLAMACTL_TIMEOUT_CHECK_INTERVAL - Default instance timeout check interval in minutes
auth:\n require_inference_auth: true # Require API key for OpenAI endpoints (default: true)\n inference_keys: [] # List of valid inference API keys\n require_management_auth: true # Require API key for management endpoints (default: true)\n management_keys: [] # List of valid management API keys\nEnvironment Variables: - LLAMACTL_REQUIRE_INFERENCE_AUTH - Require auth for OpenAI endpoints (true/false) - LLAMACTL_INFERENCE_KEYS - Comma-separated inference API keys - LLAMACTL_REQUIRE_MANAGEMENT_AUTH - Require auth for management endpoints (true/false) - LLAMACTL_MANAGEMENT_KEYS - Comma-separated management API keys
View all available command line options:
llamactl --help\nYou can also override configuration using command line flags when starting llamactl.
"},{"location":"getting-started/installation/","title":"Installation","text":"This guide will walk you through installing Llamactl on your system.
"},{"location":"getting-started/installation/#prerequisites","title":"Prerequisites","text":""},{"location":"getting-started/installation/#backend-dependencies","title":"Backend Dependencies","text":"llamactl supports multiple backends. Install at least one:
For llama.cpp backend (all platforms):
You need llama-server from llama.cpp installed:
# Homebrew (macOS/Linux)\nbrew install llama.cpp\n# Winget (Windows)\nwinget install llama.cpp\nOr build from source - see llama.cpp docs
For MLX backend (macOS only):
MLX provides optimized inference on Apple Silicon. Install MLX-LM:
# Install via pip (requires Python 3.8+)\npip install mlx-lm\n\n# Or in a virtual environment (recommended)\npython -m venv mlx-env\nsource mlx-env/bin/activate\npip install mlx-lm\nNote: MLX backend is only available on macOS with Apple Silicon (M1, M2, M3, etc.)
For vLLM backend:
vLLM provides high-throughput distributed serving for LLMs. Install vLLM:
# Install via pip (requires Python 3.8+, GPU required)\npip install vllm\n\n# Or in a virtual environment (recommended)\npython -m venv vllm-env\nsource vllm-env/bin/activate\npip install vllm\n\n# For production deployments, consider container-based installation\nDownload the latest release from the GitHub releases page:
# Linux/macOS - Get latest version and download\nLATEST_VERSION=$(curl -s https://api.github.com/repos/lordmathis/llamactl/releases/latest | grep '\"tag_name\":' | sed -E 's/.*\"([^\"]+)\".*/\\1/')\ncurl -L https://github.com/lordmathis/llamactl/releases/download/${LATEST_VERSION}/llamactl-${LATEST_VERSION}-$(uname -s | tr '[:upper:]' '[:lower:]')-$(uname -m).tar.gz | tar -xz\nsudo mv llamactl /usr/local/bin/\n\n# Or download manually from:\n# https://github.com/lordmathis/llamactl/releases/latest\n\n# Windows - Download from releases page\nRequirements: - Go 1.24 or later - Node.js 22 or later - Git
If you prefer to build from source:
# Clone the repository\ngit clone https://github.com/lordmathis/llamactl.git\ncd llamactl\n\n# Build the web UI\ncd webui && npm ci && npm run build && cd ..\n\n# Build the application\ngo build -o llamactl ./cmd/server\nVerify your installation by checking the version:
llamactl --version\nNow that Llamactl is installed, continue to the Quick Start guide to get your first instance running!
"},{"location":"getting-started/quick-start/","title":"Quick Start","text":"This guide will help you get Llamactl up and running in just a few minutes.
"},{"location":"getting-started/quick-start/#step-1-start-llamactl","title":"Step 1: Start Llamactl","text":"Start the Llamactl server:
llamactl\nBy default, Llamactl will start on http://localhost:8080.
Open your web browser and navigate to:
http://localhost:8080\nLogin with the management API key. By default it is generated during server startup. Copy it from the terminal output.
You should see the Llamactl web interface.
"},{"location":"getting-started/quick-start/#step-3-create-your-first-instance","title":"Step 3: Create Your First Instance","text":"- Click the \"Add Instance\" button
- Fill in the instance configuration:
- Name: Give your instance a descriptive name
- Backend Type: Choose from llama.cpp, MLX, or vLLM
- Model: Model path or identifier for your chosen backend
-
Additional Options: Backend-specific parameters
-
Click \"Create Instance\"
Once created, you can:
- Start the instance by clicking the start button
- Monitor its status in real-time
- View logs by clicking the logs button
- Stop the instance when needed
Here are basic example configurations for each backend:
llama.cpp backend:
{\n \"name\": \"llama2-7b\",\n \"backend_type\": \"llama_cpp\",\n \"backend_options\": {\n \"model\": \"/path/to/llama-2-7b-chat.gguf\",\n \"threads\": 4,\n \"ctx_size\": 2048,\n \"gpu_layers\": 32\n }\n}\nMLX backend (macOS only):
{\n \"name\": \"mistral-mlx\",\n \"backend_type\": \"mlx_lm\",\n \"backend_options\": {\n \"model\": \"mlx-community/Mistral-7B-Instruct-v0.3-4bit\",\n \"temp\": 0.7,\n \"max_tokens\": 2048\n }\n}\nvLLM backend:
{\n \"name\": \"dialogpt-vllm\",\n \"backend_type\": \"vllm\",\n \"backend_options\": {\n \"model\": \"microsoft/DialoGPT-medium\",\n \"tensor_parallel_size\": 2,\n \"gpu_memory_utilization\": 0.9\n }\n}\nYou can also manage instances via the REST API:
# List all instances\ncurl http://localhost:8080/api/instances\n\n# Create a new llama.cpp instance\ncurl -X POST http://localhost:8080/api/instances/my-model \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"backend_type\": \"llama_cpp\",\n \"backend_options\": {\n \"model\": \"/path/to/model.gguf\"\n }\n }'\n\n# Start an instance\ncurl -X POST http://localhost:8080/api/instances/my-model/start\nLlamactl provides OpenAI-compatible endpoints, making it easy to integrate with existing OpenAI client libraries and tools.
"},{"location":"getting-started/quick-start/#chat-completions","title":"Chat Completions","text":"Once you have an instance running, you can use it with the OpenAI-compatible chat completions endpoint:
curl -X POST http://localhost:8080/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"my-model\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hello! Can you help me write a Python function?\"\n }\n ],\n \"max_tokens\": 150,\n \"temperature\": 0.7\n }'\nYou can also use the official OpenAI Python client:
from openai import OpenAI\n\n# Point the client to your Llamactl server\nclient = OpenAI(\n base_url=\"http://localhost:8080/v1\",\n api_key=\"not-needed\" # Llamactl doesn't require API keys by default\n)\n\n# Create a chat completion\nresponse = client.chat.completions.create(\n model=\"my-model\", # Use the name of your instance\n messages=[\n {\"role\": \"user\", \"content\": \"Explain quantum computing in simple terms\"}\n ],\n max_tokens=200,\n temperature=0.7\n)\n\nprint(response.choices[0].message.content)\nGet a list of running instances (models) in OpenAI-compatible format:
curl http://localhost:8080/v1/models\n- Manage instances Managing Instances
- Explore the API Reference
- Configure advanced settings in the Configuration guide
Complete reference for the Llamactl REST API.
"},{"location":"user-guide/api-reference/#base-url","title":"Base URL","text":"All API endpoints are relative to the base URL:
http://localhost:8080/api/v1\nLlamactl supports API key authentication. If authentication is enabled, include the API key in the Authorization header:
curl -H \"Authorization: Bearer <your-api-key>\" \\\n http://localhost:8080/api/v1/instances\nThe server supports two types of API keys: - Management API Keys: Required for instance management operations (CRUD operations on instances) - Inference API Keys: Required for OpenAI-compatible inference endpoints
"},{"location":"user-guide/api-reference/#system-endpoints","title":"System Endpoints","text":""},{"location":"user-guide/api-reference/#get-llamactl-version","title":"Get Llamactl Version","text":"Get the version information of the llamactl server.
GET /api/v1/version\nResponse:
Version: 1.0.0\nCommit: abc123\nBuild Time: 2024-01-15T10:00:00Z\nGet help text for the llama-server command.
GET /api/v1/server/help\nResponse: Plain text help output from llama-server --help
Get version information of the llama-server binary.
GET /api/v1/server/version\nResponse: Plain text version output from llama-server --version
List available devices for llama-server.
GET /api/v1/server/devices\nResponse: Plain text device list from llama-server --list-devices
Get a list of all instances.
GET /api/v1/instances\nResponse:
[\n {\n \"name\": \"llama2-7b\",\n \"status\": \"running\",\n \"created\": 1705312200\n }\n]\nGet detailed information about a specific instance.
GET /api/v1/instances/{name}\nResponse:
{\n \"name\": \"llama2-7b\",\n \"status\": \"running\",\n \"created\": 1705312200\n}\nCreate and start a new instance.
POST /api/v1/instances/{name}\nRequest Body: JSON object with instance configuration. See Managing Instances for available configuration options.
Response:
{\n \"name\": \"llama2-7b\",\n \"status\": \"running\",\n \"created\": 1705312200\n}\nUpdate an existing instance configuration. See Managing Instances for available configuration options.
PUT /api/v1/instances/{name}\nRequest Body: JSON object with configuration fields to update.
Response:
{\n \"name\": \"llama2-7b\",\n \"status\": \"running\",\n \"created\": 1705312200\n}\nStop and remove an instance.
DELETE /api/v1/instances/{name}\nResponse: 204 No Content
Start a stopped instance.
POST /api/v1/instances/{name}/start\nResponse:
{\n \"name\": \"llama2-7b\",\n \"status\": \"running\",\n \"created\": 1705312200\n}\nError Responses: - 409 Conflict: Maximum number of running instances reached - 500 Internal Server Error: Failed to start instance
Stop a running instance.
POST /api/v1/instances/{name}/stop\nResponse:
{\n \"name\": \"llama2-7b\",\n \"status\": \"stopped\",\n \"created\": 1705312200\n}\nRestart an instance (stop then start).
POST /api/v1/instances/{name}/restart\nResponse:
{\n \"name\": \"llama2-7b\",\n \"status\": \"running\",\n \"created\": 1705312200\n}\nRetrieve instance logs.
GET /api/v1/instances/{name}/logs\nQuery Parameters: - lines: Number of lines to return (default: all lines, use -1 for all)
Response: Plain text log output
Example:
curl \"http://localhost:8080/api/v1/instances/my-instance/logs?lines=100\"\nProxy HTTP requests directly to the llama-server instance.
GET /api/v1/instances/{name}/proxy/*\nPOST /api/v1/instances/{name}/proxy/*\nThis endpoint forwards all requests to the underlying llama-server instance running on its configured port. The proxy strips the /api/v1/instances/{name}/proxy prefix and forwards the remaining path to the instance.
Example - Check Instance Health:
curl -H \"Authorization: Bearer your-api-key\" \\\n http://localhost:8080/api/v1/instances/my-model/proxy/health\nThis forwards the request to http://instance-host:instance-port/health on the actual llama-server instance.
Error Responses: - 503 Service Unavailable: Instance is not running
Llamactl provides OpenAI-compatible endpoints for inference operations.
"},{"location":"user-guide/api-reference/#list-models","title":"List Models","text":"List all instances in OpenAI-compatible format.
GET /v1/models\nResponse:
{\n \"object\": \"list\",\n \"data\": [\n {\n \"id\": \"llama2-7b\",\n \"object\": \"model\",\n \"created\": 1705312200,\n \"owned_by\": \"llamactl\"\n }\n ]\n}\nAll OpenAI-compatible inference endpoints are available:
POST /v1/chat/completions\nPOST /v1/completions\nPOST /v1/embeddings\nPOST /v1/rerank\nPOST /v1/reranking\nRequest Body: Standard OpenAI format with model field specifying the instance name
Example:
{\n \"model\": \"llama2-7b\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hello, how are you?\"\n }\n ]\n}\nThe server routes requests to the appropriate instance based on the model field in the request body. Instances with on-demand starting enabled will be automatically started if not running. For configuration details, see Managing Instances.
Error Responses: - 400 Bad Request: Invalid request body or missing model name - 503 Service Unavailable: Instance is not running and on-demand start is disabled - 409 Conflict: Cannot start instance due to maximum instances limit
Instances can have the following status values: - stopped: Instance is not running - running: Instance is running and ready to accept requests - failed: Instance failed to start or crashed
All endpoints may return error responses in the following format:
{\n \"error\": \"Error message description\"\n}\n200: Success201: Created204: No Content (successful deletion)400: Bad Request (invalid parameters or request body)401: Unauthorized (missing or invalid API key)403: Forbidden (insufficient permissions)404: Not Found (instance not found)409: Conflict (instance already exists, max instances reached)500: Internal Server Error503: Service Unavailable (instance not running)
# Create and start instance\ncurl -X POST http://localhost:8080/api/v1/instances/my-model \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer your-api-key\" \\\n -d '{\n \"model\": \"/models/llama-2-7b.gguf\"\n }'\n\n# Check instance status\ncurl -H \"Authorization: Bearer your-api-key\" \\\n http://localhost:8080/api/v1/instances/my-model\n\n# Get instance logs\ncurl -H \"Authorization: Bearer your-api-key\" \\\n \"http://localhost:8080/api/v1/instances/my-model/logs?lines=50\"\n\n# Use OpenAI-compatible chat completions\ncurl -X POST http://localhost:8080/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer your-inference-api-key\" \\\n -d '{\n \"model\": \"my-model\",\n \"messages\": [\n {\"role\": \"user\", \"content\": \"Hello!\"}\n ],\n \"max_tokens\": 100\n }'\n\n# Stop instance\ncurl -X POST -H \"Authorization: Bearer your-api-key\" \\\n http://localhost:8080/api/v1/instances/my-model/stop\n\n# Delete instance\ncurl -X DELETE -H \"Authorization: Bearer your-api-key\" \\\n http://localhost:8080/api/v1/instances/my-model\nYou can also directly proxy requests to the llama-server instance:
# Direct proxy to instance (bypasses OpenAI compatibility layer)\ncurl -X POST http://localhost:8080/api/v1/instances/my-model/proxy/completion \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer your-api-key\" \\\n -d '{\n \"prompt\": \"Hello, world!\",\n \"n_predict\": 50\n }'\nLlamactl provides endpoints to parse command strings from different backends into instance configuration options.
"},{"location":"user-guide/api-reference/#parse-llamacpp-command","title":"Parse Llama.cpp Command","text":"Parse a llama-server command string into instance options.
POST /api/v1/backends/llama-cpp/parse-command\nRequest Body:
{\n \"command\": \"llama-server -m /path/to/model.gguf -c 2048 --port 8080\"\n}\nResponse:
{\n \"backend_type\": \"llama_cpp\",\n \"llama_server_options\": {\n \"model\": \"/path/to/model.gguf\",\n \"ctx_size\": 2048,\n \"port\": 8080\n }\n}\nParse an MLX-LM server command string into instance options.
POST /api/v1/backends/mlx/parse-command\nRequest Body:
{\n \"command\": \"mlx_lm.server --model /path/to/model --port 8080\"\n}\nResponse:
{\n \"backend_type\": \"mlx_lm\",\n \"mlx_server_options\": {\n \"model\": \"/path/to/model\",\n \"port\": 8080\n }\n}\nParse a vLLM serve command string into instance options.
POST /api/v1/backends/vllm/parse-command\nRequest Body:
{\n \"command\": \"vllm serve /path/to/model --port 8080\"\n}\nResponse:
{\n \"backend_type\": \"vllm\",\n \"vllm_server_options\": {\n \"model\": \"/path/to/model\",\n \"port\": 8080\n }\n}\nError Responses for Parse Commands: - 400 Bad Request: Invalid request body, empty command, or parse error - 500 Internal Server Error: Encoding error
The API documentation is automatically generated from code annotations using Swagger/OpenAPI. To regenerate the documentation:
- Install the swag tool:
go install github.com/swaggo/swag/cmd/swag@latest - Generate docs:
swag init -g cmd/server/main.go -o apidocs
If swagger documentation is enabled in the server configuration, you can access the interactive API documentation at:
http://localhost:8080/swagger/\nThis provides a complete interactive interface for testing all API endpoints.
"},{"location":"user-guide/managing-instances/","title":"Managing Instances","text":"Learn how to effectively manage your llama.cpp, MLX, and vLLM instances with Llamactl through both the Web UI and API.
"},{"location":"user-guide/managing-instances/#overview","title":"Overview","text":"Llamactl provides two ways to manage instances:
- Web UI: Accessible at
http://localhost:8080with an intuitive dashboard - REST API: Programmatic access for automation and integration
If authentication is enabled: 1. Navigate to the web UI 2. Enter your credentials 3. Bearer token is stored for the session
"},{"location":"user-guide/managing-instances/#theme-support","title":"Theme Support","text":"- Switch between light and dark themes
- Setting is remembered across sessions
Each instance is displayed as a card showing:
- Instance name
- Health status badge (unknown, ready, error, failed)
- Action buttons (start, stop, edit, logs, delete)
- Click the \"Create Instance\" button on the dashboard
- Enter a unique Name for your instance (only required field)
- Choose Backend Type:
- llama.cpp: For GGUF models using llama-server
- MLX: For MLX-optimized models (macOS only)
- vLLM: For distributed serving and high-throughput inference
- Configure model source:
- For llama.cpp: GGUF model path or HuggingFace repo
- For MLX: MLX model path or identifier (e.g.,
mlx-community/Mistral-7B-Instruct-v0.3-4bit) - For vLLM: HuggingFace model identifier (e.g.,
microsoft/DialoGPT-medium)
- Configure optional instance management settings:
- Auto Restart: Automatically restart instance on failure
- Max Restarts: Maximum number of restart attempts
- Restart Delay: Delay in seconds between restart attempts
- On Demand Start: Start instance when receiving a request to the OpenAI compatible endpoint
- Idle Timeout: Minutes before stopping idle instance (set to 0 to disable)
- Configure backend-specific options:
- llama.cpp: Threads, context size, GPU layers, port, etc.
- MLX: Temperature, top-p, adapter path, Python environment, etc.
- vLLM: Tensor parallel size, GPU memory utilization, quantization, etc.
- Click \"Create\" to save the instance
# Create llama.cpp instance with local model file\ncurl -X POST http://localhost:8080/api/instances/my-llama-instance \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"backend_type\": \"llama_cpp\",\n \"backend_options\": {\n \"model\": \"/path/to/model.gguf\",\n \"threads\": 8,\n \"ctx_size\": 4096,\n \"gpu_layers\": 32\n }\n }'\n\n# Create MLX instance (macOS only)\ncurl -X POST http://localhost:8080/api/instances/my-mlx-instance \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"backend_type\": \"mlx_lm\",\n \"backend_options\": {\n \"model\": \"mlx-community/Mistral-7B-Instruct-v0.3-4bit\",\n \"temp\": 0.7,\n \"top_p\": 0.9,\n \"max_tokens\": 2048\n },\n \"auto_restart\": true,\n \"max_restarts\": 3\n }'\n\n# Create vLLM instance\ncurl -X POST http://localhost:8080/api/instances/my-vllm-instance \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"backend_type\": \"vllm\",\n \"backend_options\": {\n \"model\": \"microsoft/DialoGPT-medium\",\n \"tensor_parallel_size\": 2,\n \"gpu_memory_utilization\": 0.9\n },\n \"auto_restart\": true,\n \"on_demand_start\": true\n }'\n\n# Create llama.cpp instance with HuggingFace model\ncurl -X POST http://localhost:8080/api/instances/gemma-3-27b \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"backend_type\": \"llama_cpp\",\n \"backend_options\": {\n \"hf_repo\": \"unsloth/gemma-3-27b-it-GGUF\",\n \"hf_file\": \"gemma-3-27b-it-GGUF.gguf\",\n \"gpu_layers\": 32\n }\n }'\n- Click the \"Start\" button on an instance card
- Watch the status change to \"Unknown\"
- Monitor progress in the logs
- Instance status changes to \"Ready\" when ready
curl -X POST http://localhost:8080/api/instances/{name}/start\n- Click the \"Stop\" button on an instance card
- Instance gracefully shuts down
curl -X POST http://localhost:8080/api/instances/{name}/stop\n- Click the \"Edit\" button on an instance card
- Modify settings in the configuration dialog
- Changes require instance restart to take effect
- Click \"Update & Restart\" to apply changes
Modify instance settings:
curl -X PUT http://localhost:8080/api/instances/{name} \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"backend_options\": {\n \"threads\": 8,\n \"context_size\": 4096\n }\n }'\nNote
Configuration changes require restarting the instance to take effect.
"},{"location":"user-guide/managing-instances/#view-logs","title":"View Logs","text":""},{"location":"user-guide/managing-instances/#via-web-ui_4","title":"Via Web UI","text":"- Click the \"Logs\" button on any instance card
- Real-time log viewer opens
Check instance status in real-time:
# Get instance details\ncurl http://localhost:8080/api/instances/{name}/logs\n- Click the \"Delete\" button on an instance card
- Only stopped instances can be deleted
- Confirm deletion in the dialog
curl -X DELETE http://localhost:8080/api/instances/{name}\nLlamactl proxies all requests to the underlying backend instances (llama-server, MLX, or vLLM).
# Get instance details\ncurl http://localhost:8080/api/instances/{name}/proxy/\nAll backends provide OpenAI-compatible endpoints. Check the respective documentation: - llama-server docs - MLX-LM docs - vLLM docs
"},{"location":"user-guide/managing-instances/#instance-health","title":"Instance Health","text":""},{"location":"user-guide/managing-instances/#via-web-ui_6","title":"Via Web UI","text":"- The health status badge is displayed on each instance card
Check the health status of your instances:
curl http://localhost:8080/api/instances/{name}/proxy/health\nIssues specific to Llamactl deployment and operation.
"},{"location":"user-guide/troubleshooting/#configuration-issues","title":"Configuration Issues","text":""},{"location":"user-guide/troubleshooting/#invalid-configuration","title":"Invalid Configuration","text":"Problem: Invalid configuration preventing startup
Solutions: 1. Use minimal configuration:
server:\n host: \"0.0.0.0\"\n port: 8080\ninstances:\n port_range: [8000, 9000]\n- Check data directory permissions:
# Ensure data directory is writable (default: ~/.local/share/llamactl)\nmkdir -p ~/.local/share/llamactl/{instances,logs}\n
Problem: Instance fails to start with model loading errors
Common Solutions: - llama-server not found: Ensure llama-server binary is in PATH - Wrong model format: Ensure model is in GGUF format - Insufficient memory: Use smaller model or reduce context size - Path issues: Use absolute paths to model files
Problem: Out of memory errors or system becomes unresponsive
Solutions: 1. Reduce context size:
{\n \"n_ctx\": 1024\n}\n- Use quantized models:
- Try Q4_K_M instead of higher precision models
- Use smaller model variants (7B instead of 13B)
Problem: GPU not being used effectively
Solutions: 1. Configure GPU layers:
{\n \"n_gpu_layers\": 35\n}\nProblem: Complex model loading, performance, or compatibility issues
Since llamactl uses llama-server under the hood, many instance-related issues are actually llama.cpp issues. For advanced troubleshooting:
Resources: - llama.cpp Documentation: https://github.com/ggml/llama.cpp - llama.cpp Issues: https://github.com/ggml/llama.cpp/issues - llama.cpp Discussions: https://github.com/ggml/llama.cpp/discussions
Testing directly with llama-server:
# Test your model and parameters directly with llama-server\nllama-server --model /path/to/model.gguf --port 8081 --n-gpu-layers 35\nThis helps determine if the issue is with llamactl or with the underlying llama.cpp/llama-server.
"},{"location":"user-guide/troubleshooting/#api-and-network-issues","title":"API and Network Issues","text":""},{"location":"user-guide/troubleshooting/#cors-errors","title":"CORS Errors","text":"Problem: Web UI shows CORS errors in browser console
Solutions: 1. Configure allowed origins:
server:\n allowed_origins:\n - \"http://localhost:3000\"\n - \"https://yourdomain.com\"\nProblem: API requests failing with authentication errors
Solutions: 1. Disable authentication temporarily:
auth:\n require_management_auth: false\n require_inference_auth: false\n-
Configure API keys:
auth:\n management_keys:\n - \"your-management-key\"\n inference_keys:\n - \"your-inference-key\"\n -
Use correct Authorization header:
curl -H \"Authorization: Bearer your-api-key\" \\\n http://localhost:8080/api/v1/instances\n
# Get instance logs via API\ncurl http://localhost:8080/api/v1/instances/{name}/logs\n\n# Or check log files directly\ntail -f ~/.local/share/llamactl/logs/{instance-name}.log\nexport LLAMACTL_LOG_LEVEL=debug\nllamactl\nWhen reporting issues, include:
-
System information:
llamactl --version\n -
Configuration file (remove sensitive keys)
-
Relevant log output
-
Steps to reproduce the issue
Welcome to the Llamactl documentation!
"},{"location":"#what-is-llamactl","title":"What is Llamactl?","text":"Unified management and routing for llama.cpp, MLX and vLLM models with web dashboard.
"},{"location":"#features","title":"Features","text":""},{"location":"#easy-model-management","title":"\ud83d\ude80 Easy Model Management","text":"- Multiple Model Serving: Run different models simultaneously (7B for speed, 70B for quality)
- On-Demand Instance Start: Automatically launch instances upon receiving API requests
- State Persistence: Ensure instances remain intact across server restarts
- OpenAI API Compatible: Drop-in replacement - route requests by instance name
- Multi-Backend Support: Native support for llama.cpp, MLX (Apple Silicon optimized), and vLLM
- Web Dashboard: Modern React UI for visual management (unlike CLI-only tools)
- API Key Authentication: Separate keys for management vs inference access
- Instance Monitoring: Health checks, auto-restart, log management
- Smart Resource Management: Idle timeout, LRU eviction, and configurable instance limits
- Installation Guide - Get Llamactl up and running
- Configuration Guide - Detailed configuration options
- Quick Start - Your first steps with Llamactl
- Managing Instances - Instance lifecycle management
- API Reference - Complete API documentation
If you need help or have questions:
- Check the Troubleshooting guide
- Visit the GitHub repository
- Review the Configuration Guide for advanced settings
MIT License - see the LICENSE file.
"},{"location":"getting-started/configuration/","title":"Configuration","text":"llamactl can be configured via configuration files or environment variables. Configuration is loaded in the following order of precedence:
Defaults < Configuration file < Environment variables\nllamactl works out of the box with sensible defaults, but you can customize the behavior to suit your needs.
"},{"location":"getting-started/configuration/#default-configuration","title":"Default Configuration","text":"Here's the default configuration with all available options:
server:\n host: \"0.0.0.0\" # Server host to bind to\n port: 8080 # Server port to bind to\n allowed_origins: [\"*\"] # Allowed CORS origins (default: all)\n enable_swagger: false # Enable Swagger UI for API docs\n\nbackends:\n llama_executable: llama-server # Path to llama-server executable\n mlx_lm_executable: mlx_lm.server # Path to mlx_lm.server executable\n vllm_executable: vllm # Path to vllm executable\n\ninstances:\n port_range: [8000, 9000] # Port range for instances\n data_dir: ~/.local/share/llamactl # Data directory (platform-specific, see below)\n configs_dir: ~/.local/share/llamactl/instances # Instance configs directory\n logs_dir: ~/.local/share/llamactl/logs # Logs directory\n auto_create_dirs: true # Auto-create data/config/logs dirs if missing\n max_instances: -1 # Max instances (-1 = unlimited)\n max_running_instances: -1 # Max running instances (-1 = unlimited)\n enable_lru_eviction: true # Enable LRU eviction for idle instances\n default_auto_restart: true # Auto-restart new instances by default\n default_max_restarts: 3 # Max restarts for new instances\n default_restart_delay: 5 # Restart delay (seconds) for new instances\n default_on_demand_start: true # Default on-demand start setting\n on_demand_start_timeout: 120 # Default on-demand start timeout in seconds\n timeout_check_interval: 5 # Idle instance timeout check in minutes\n\nauth:\n require_inference_auth: true # Require auth for inference endpoints\n inference_keys: [] # Keys for inference endpoints\n require_management_auth: true # Require auth for management endpoints\n management_keys: [] # Keys for management endpoints\nConfiguration files are searched in the following locations (in order of precedence):
Linux: - ./llamactl.yaml or ./config.yaml (current directory) - $HOME/.config/llamactl/config.yaml - /etc/llamactl/config.yaml
macOS: - ./llamactl.yaml or ./config.yaml (current directory) - $HOME/Library/Application Support/llamactl/config.yaml - /Library/Application Support/llamactl/config.yaml
Windows: - ./llamactl.yaml or ./config.yaml (current directory) - %APPDATA%\\llamactl\\config.yaml - %USERPROFILE%\\llamactl\\config.yaml - %PROGRAMDATA%\\llamactl\\config.yaml
You can specify the path to config file with LLAMACTL_CONFIG_PATH environment variable.
server:\n host: \"0.0.0.0\" # Server host to bind to (default: \"0.0.0.0\")\n port: 8080 # Server port to bind to (default: 8080)\n allowed_origins: [\"*\"] # CORS allowed origins (default: [\"*\"])\n enable_swagger: false # Enable Swagger UI (default: false)\nEnvironment Variables: - LLAMACTL_HOST - Server host - LLAMACTL_PORT - Server port - LLAMACTL_ALLOWED_ORIGINS - Comma-separated CORS origins - LLAMACTL_ENABLE_SWAGGER - Enable Swagger UI (true/false)
backends:\n llama_executable: \"llama-server\" # Path to llama-server executable (default: \"llama-server\")\n mlx_lm_executable: \"mlx_lm.server\" # Path to mlx_lm.server executable (default: \"mlx_lm.server\")\n vllm_executable: \"vllm\" # Path to vllm executable (default: \"vllm\")\nEnvironment Variables: - LLAMACTL_LLAMA_EXECUTABLE - Path to llama-server executable - LLAMACTL_MLX_LM_EXECUTABLE - Path to mlx_lm.server executable - LLAMACTL_VLLM_EXECUTABLE - Path to vllm executable
instances:\n port_range: [8000, 9000] # Port range for instances (default: [8000, 9000])\n data_dir: \"~/.local/share/llamactl\" # Directory for all llamactl data (default varies by OS)\n configs_dir: \"~/.local/share/llamactl/instances\" # Directory for instance configs (default: data_dir/instances)\n logs_dir: \"~/.local/share/llamactl/logs\" # Directory for instance logs (default: data_dir/logs)\n auto_create_dirs: true # Automatically create data/config/logs directories (default: true)\n max_instances: -1 # Maximum instances (-1 = unlimited)\n max_running_instances: -1 # Maximum running instances (-1 = unlimited)\n enable_lru_eviction: true # Enable LRU eviction for idle instances\n default_auto_restart: true # Default auto-restart setting\n default_max_restarts: 3 # Default maximum restart attempts\n default_restart_delay: 5 # Default restart delay in seconds\n default_on_demand_start: true # Default on-demand start setting\n on_demand_start_timeout: 120 # Default on-demand start timeout in seconds\n timeout_check_interval: 5 # Default instance timeout check interval in minutes\nEnvironment Variables: - LLAMACTL_INSTANCE_PORT_RANGE - Port range (format: \"8000-9000\" or \"8000,9000\") - LLAMACTL_DATA_DIRECTORY - Data directory path - LLAMACTL_INSTANCES_DIR - Instance configs directory path - LLAMACTL_LOGS_DIR - Log directory path - LLAMACTL_AUTO_CREATE_DATA_DIR - Auto-create data/config/logs directories (true/false) - LLAMACTL_MAX_INSTANCES - Maximum number of instances - LLAMACTL_MAX_RUNNING_INSTANCES - Maximum number of running instances - LLAMACTL_ENABLE_LRU_EVICTION - Enable LRU eviction for idle instances - LLAMACTL_DEFAULT_AUTO_RESTART - Default auto-restart setting (true/false) - LLAMACTL_DEFAULT_MAX_RESTARTS - Default maximum restarts - LLAMACTL_DEFAULT_RESTART_DELAY - Default restart delay in seconds - LLAMACTL_DEFAULT_ON_DEMAND_START - Default on-demand start setting (true/false) - LLAMACTL_ON_DEMAND_START_TIMEOUT - Default on-demand start timeout in seconds - LLAMACTL_TIMEOUT_CHECK_INTERVAL - Default instance timeout check interval in minutes
auth:\n require_inference_auth: true # Require API key for OpenAI endpoints (default: true)\n inference_keys: [] # List of valid inference API keys\n require_management_auth: true # Require API key for management endpoints (default: true)\n management_keys: [] # List of valid management API keys\nEnvironment Variables: - LLAMACTL_REQUIRE_INFERENCE_AUTH - Require auth for OpenAI endpoints (true/false) - LLAMACTL_INFERENCE_KEYS - Comma-separated inference API keys - LLAMACTL_REQUIRE_MANAGEMENT_AUTH - Require auth for management endpoints (true/false) - LLAMACTL_MANAGEMENT_KEYS - Comma-separated management API keys
View all available command line options:
llamactl --help\nYou can also override configuration using command line flags when starting llamactl.
"},{"location":"getting-started/installation/","title":"Installation","text":"This guide will walk you through installing Llamactl on your system.
"},{"location":"getting-started/installation/#prerequisites","title":"Prerequisites","text":""},{"location":"getting-started/installation/#backend-dependencies","title":"Backend Dependencies","text":"llamactl supports multiple backends. Install at least one:
For llama.cpp backend (all platforms):
You need llama-server from llama.cpp installed:
# Homebrew (macOS/Linux)\nbrew install llama.cpp\n# Winget (Windows)\nwinget install llama.cpp\nOr build from source - see llama.cpp docs
For MLX backend (macOS only):
MLX provides optimized inference on Apple Silicon. Install MLX-LM:
# Install via pip (requires Python 3.8+)\npip install mlx-lm\n\n# Or in a virtual environment (recommended)\npython -m venv mlx-env\nsource mlx-env/bin/activate\npip install mlx-lm\nNote: MLX backend is only available on macOS with Apple Silicon (M1, M2, M3, etc.)
For vLLM backend:
vLLM provides high-throughput distributed serving for LLMs. Install vLLM:
# Install via pip (requires Python 3.8+, GPU required)\npip install vllm\n\n# Or in a virtual environment (recommended)\npython -m venv vllm-env\nsource vllm-env/bin/activate\npip install vllm\n\n# For production deployments, consider container-based installation\nDownload the latest release from the GitHub releases page:
# Linux/macOS - Get latest version and download\nLATEST_VERSION=$(curl -s https://api.github.com/repos/lordmathis/llamactl/releases/latest | grep '\"tag_name\":' | sed -E 's/.*\"([^\"]+)\".*/\\1/')\ncurl -L https://github.com/lordmathis/llamactl/releases/download/${LATEST_VERSION}/llamactl-${LATEST_VERSION}-$(uname -s | tr '[:upper:]' '[:lower:]')-$(uname -m).tar.gz | tar -xz\nsudo mv llamactl /usr/local/bin/\n\n# Or download manually from:\n# https://github.com/lordmathis/llamactl/releases/latest\n\n# Windows - Download from releases page\nRequirements: - Go 1.24 or later - Node.js 22 or later - Git
If you prefer to build from source:
# Clone the repository\ngit clone https://github.com/lordmathis/llamactl.git\ncd llamactl\n\n# Build the web UI\ncd webui && npm ci && npm run build && cd ..\n\n# Build the application\ngo build -o llamactl ./cmd/server\nVerify your installation by checking the version:
llamactl --version\nNow that Llamactl is installed, continue to the Quick Start guide to get your first instance running!
"},{"location":"getting-started/quick-start/","title":"Quick Start","text":"This guide will help you get Llamactl up and running in just a few minutes.
"},{"location":"getting-started/quick-start/#step-1-start-llamactl","title":"Step 1: Start Llamactl","text":"Start the Llamactl server:
llamactl\nBy default, Llamactl will start on http://localhost:8080.
Open your web browser and navigate to:
http://localhost:8080\nLogin with the management API key. By default it is generated during server startup. Copy it from the terminal output.
You should see the Llamactl web interface.
"},{"location":"getting-started/quick-start/#step-3-create-your-first-instance","title":"Step 3: Create Your First Instance","text":"- Click the \"Add Instance\" button
- Fill in the instance configuration:
- Name: Give your instance a descriptive name
- Backend Type: Choose from llama.cpp, MLX, or vLLM
- Model: Model path or identifier for your chosen backend
-
Additional Options: Backend-specific parameters
-
Click \"Create Instance\"
Once created, you can:
- Start the instance by clicking the start button
- Monitor its status in real-time
- View logs by clicking the logs button
- Stop the instance when needed
Here are basic example configurations for each backend:
llama.cpp backend:
{\n \"name\": \"llama2-7b\",\n \"backend_type\": \"llama_cpp\",\n \"backend_options\": {\n \"model\": \"/path/to/llama-2-7b-chat.gguf\",\n \"threads\": 4,\n \"ctx_size\": 2048,\n \"gpu_layers\": 32\n }\n}\nMLX backend (macOS only):
{\n \"name\": \"mistral-mlx\",\n \"backend_type\": \"mlx_lm\",\n \"backend_options\": {\n \"model\": \"mlx-community/Mistral-7B-Instruct-v0.3-4bit\",\n \"temp\": 0.7,\n \"max_tokens\": 2048\n }\n}\nvLLM backend:
{\n \"name\": \"dialogpt-vllm\",\n \"backend_type\": \"vllm\",\n \"backend_options\": {\n \"model\": \"microsoft/DialoGPT-medium\",\n \"tensor_parallel_size\": 2,\n \"gpu_memory_utilization\": 0.9\n }\n}\nYou can also manage instances via the REST API:
# List all instances\ncurl http://localhost:8080/api/instances\n\n# Create a new llama.cpp instance\ncurl -X POST http://localhost:8080/api/instances/my-model \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"backend_type\": \"llama_cpp\",\n \"backend_options\": {\n \"model\": \"/path/to/model.gguf\"\n }\n }'\n\n# Start an instance\ncurl -X POST http://localhost:8080/api/instances/my-model/start\nLlamactl provides OpenAI-compatible endpoints, making it easy to integrate with existing OpenAI client libraries and tools.
"},{"location":"getting-started/quick-start/#chat-completions","title":"Chat Completions","text":"Once you have an instance running, you can use it with the OpenAI-compatible chat completions endpoint:
curl -X POST http://localhost:8080/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"my-model\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hello! Can you help me write a Python function?\"\n }\n ],\n \"max_tokens\": 150,\n \"temperature\": 0.7\n }'\nYou can also use the official OpenAI Python client:
from openai import OpenAI\n\n# Point the client to your Llamactl server\nclient = OpenAI(\n base_url=\"http://localhost:8080/v1\",\n api_key=\"not-needed\" # Llamactl doesn't require API keys by default\n)\n\n# Create a chat completion\nresponse = client.chat.completions.create(\n model=\"my-model\", # Use the name of your instance\n messages=[\n {\"role\": \"user\", \"content\": \"Explain quantum computing in simple terms\"}\n ],\n max_tokens=200,\n temperature=0.7\n)\n\nprint(response.choices[0].message.content)\nGet a list of running instances (models) in OpenAI-compatible format:
curl http://localhost:8080/v1/models\n- Manage instances Managing Instances
- Explore the API Reference
- Configure advanced settings in the Configuration guide
Complete reference for the Llamactl REST API.
"},{"location":"user-guide/api-reference/#base-url","title":"Base URL","text":"All API endpoints are relative to the base URL:
http://localhost:8080/api/v1\nLlamactl supports API key authentication. If authentication is enabled, include the API key in the Authorization header:
curl -H \"Authorization: Bearer <your-api-key>\" \\\n http://localhost:8080/api/v1/instances\nThe server supports two types of API keys: - Management API Keys: Required for instance management operations (CRUD operations on instances) - Inference API Keys: Required for OpenAI-compatible inference endpoints
"},{"location":"user-guide/api-reference/#system-endpoints","title":"System Endpoints","text":""},{"location":"user-guide/api-reference/#get-llamactl-version","title":"Get Llamactl Version","text":"Get the version information of the llamactl server.
GET /api/v1/version\nResponse:
Version: 1.0.0\nCommit: abc123\nBuild Time: 2024-01-15T10:00:00Z\nGet help text for the llama-server command.
GET /api/v1/server/help\nResponse: Plain text help output from llama-server --help
Get version information of the llama-server binary.
GET /api/v1/server/version\nResponse: Plain text version output from llama-server --version
List available devices for llama-server.
GET /api/v1/server/devices\nResponse: Plain text device list from llama-server --list-devices
Get a list of all instances.
GET /api/v1/instances\nResponse:
[\n {\n \"name\": \"llama2-7b\",\n \"status\": \"running\",\n \"created\": 1705312200\n }\n]\nGet detailed information about a specific instance.
GET /api/v1/instances/{name}\nResponse:
{\n \"name\": \"llama2-7b\",\n \"status\": \"running\",\n \"created\": 1705312200\n}\nCreate and start a new instance.
POST /api/v1/instances/{name}\nRequest Body: JSON object with instance configuration. See Managing Instances for available configuration options.
Response:
{\n \"name\": \"llama2-7b\",\n \"status\": \"running\",\n \"created\": 1705312200\n}\nUpdate an existing instance configuration. See Managing Instances for available configuration options.
PUT /api/v1/instances/{name}\nRequest Body: JSON object with configuration fields to update.
Response:
{\n \"name\": \"llama2-7b\",\n \"status\": \"running\",\n \"created\": 1705312200\n}\nStop and remove an instance.
DELETE /api/v1/instances/{name}\nResponse: 204 No Content
Start a stopped instance.
POST /api/v1/instances/{name}/start\nResponse:
{\n \"name\": \"llama2-7b\",\n \"status\": \"running\",\n \"created\": 1705312200\n}\nError Responses: - 409 Conflict: Maximum number of running instances reached - 500 Internal Server Error: Failed to start instance
Stop a running instance.
POST /api/v1/instances/{name}/stop\nResponse:
{\n \"name\": \"llama2-7b\",\n \"status\": \"stopped\",\n \"created\": 1705312200\n}\nRestart an instance (stop then start).
POST /api/v1/instances/{name}/restart\nResponse:
{\n \"name\": \"llama2-7b\",\n \"status\": \"running\",\n \"created\": 1705312200\n}\nRetrieve instance logs.
GET /api/v1/instances/{name}/logs\nQuery Parameters: - lines: Number of lines to return (default: all lines, use -1 for all)
Response: Plain text log output
Example:
curl \"http://localhost:8080/api/v1/instances/my-instance/logs?lines=100\"\nProxy HTTP requests directly to the llama-server instance.
GET /api/v1/instances/{name}/proxy/*\nPOST /api/v1/instances/{name}/proxy/*\nThis endpoint forwards all requests to the underlying llama-server instance running on its configured port. The proxy strips the /api/v1/instances/{name}/proxy prefix and forwards the remaining path to the instance.
Example - Check Instance Health:
curl -H \"Authorization: Bearer your-api-key\" \\\n http://localhost:8080/api/v1/instances/my-model/proxy/health\nThis forwards the request to http://instance-host:instance-port/health on the actual llama-server instance.
Error Responses: - 503 Service Unavailable: Instance is not running
Llamactl provides OpenAI-compatible endpoints for inference operations.
"},{"location":"user-guide/api-reference/#list-models","title":"List Models","text":"List all instances in OpenAI-compatible format.
GET /v1/models\nResponse:
{\n \"object\": \"list\",\n \"data\": [\n {\n \"id\": \"llama2-7b\",\n \"object\": \"model\",\n \"created\": 1705312200,\n \"owned_by\": \"llamactl\"\n }\n ]\n}\nAll OpenAI-compatible inference endpoints are available:
POST /v1/chat/completions\nPOST /v1/completions\nPOST /v1/embeddings\nPOST /v1/rerank\nPOST /v1/reranking\nRequest Body: Standard OpenAI format with model field specifying the instance name
Example:
{\n \"model\": \"llama2-7b\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hello, how are you?\"\n }\n ]\n}\nThe server routes requests to the appropriate instance based on the model field in the request body. Instances with on-demand starting enabled will be automatically started if not running. For configuration details, see Managing Instances.
Error Responses: - 400 Bad Request: Invalid request body or missing instance name - 503 Service Unavailable: Instance is not running and on-demand start is disabled - 409 Conflict: Cannot start instance due to maximum instances limit
Instances can have the following status values: - stopped: Instance is not running - running: Instance is running and ready to accept requests - failed: Instance failed to start or crashed
All endpoints may return error responses in the following format:
{\n \"error\": \"Error message description\"\n}\n200: Success201: Created204: No Content (successful deletion)400: Bad Request (invalid parameters or request body)401: Unauthorized (missing or invalid API key)403: Forbidden (insufficient permissions)404: Not Found (instance not found)409: Conflict (instance already exists, max instances reached)500: Internal Server Error503: Service Unavailable (instance not running)
# Create and start instance\ncurl -X POST http://localhost:8080/api/v1/instances/my-model \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer your-api-key\" \\\n -d '{\n \"model\": \"/models/llama-2-7b.gguf\"\n }'\n\n# Check instance status\ncurl -H \"Authorization: Bearer your-api-key\" \\\n http://localhost:8080/api/v1/instances/my-model\n\n# Get instance logs\ncurl -H \"Authorization: Bearer your-api-key\" \\\n \"http://localhost:8080/api/v1/instances/my-model/logs?lines=50\"\n\n# Use OpenAI-compatible chat completions\ncurl -X POST http://localhost:8080/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer your-inference-api-key\" \\\n -d '{\n \"model\": \"my-model\",\n \"messages\": [\n {\"role\": \"user\", \"content\": \"Hello!\"}\n ],\n \"max_tokens\": 100\n }'\n\n# Stop instance\ncurl -X POST -H \"Authorization: Bearer your-api-key\" \\\n http://localhost:8080/api/v1/instances/my-model/stop\n\n# Delete instance\ncurl -X DELETE -H \"Authorization: Bearer your-api-key\" \\\n http://localhost:8080/api/v1/instances/my-model\nYou can also directly proxy requests to the llama-server instance:
# Direct proxy to instance (bypasses OpenAI compatibility layer)\ncurl -X POST http://localhost:8080/api/v1/instances/my-model/proxy/completion \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer your-api-key\" \\\n -d '{\n \"prompt\": \"Hello, world!\",\n \"n_predict\": 50\n }'\nLlamactl provides endpoints to parse command strings from different backends into instance configuration options.
"},{"location":"user-guide/api-reference/#parse-llamacpp-command","title":"Parse Llama.cpp Command","text":"Parse a llama-server command string into instance options.
POST /api/v1/backends/llama-cpp/parse-command\nRequest Body:
{\n \"command\": \"llama-server -m /path/to/model.gguf -c 2048 --port 8080\"\n}\nResponse:
{\n \"backend_type\": \"llama_cpp\",\n \"llama_server_options\": {\n \"model\": \"/path/to/model.gguf\",\n \"ctx_size\": 2048,\n \"port\": 8080\n }\n}\nParse an MLX-LM server command string into instance options.
POST /api/v1/backends/mlx/parse-command\nRequest Body:
{\n \"command\": \"mlx_lm.server --model /path/to/model --port 8080\"\n}\nResponse:
{\n \"backend_type\": \"mlx_lm\",\n \"mlx_server_options\": {\n \"model\": \"/path/to/model\",\n \"port\": 8080\n }\n}\nParse a vLLM serve command string into instance options.
POST /api/v1/backends/vllm/parse-command\nRequest Body:
{\n \"command\": \"vllm serve /path/to/model --port 8080\"\n}\nResponse:
{\n \"backend_type\": \"vllm\",\n \"vllm_server_options\": {\n \"model\": \"/path/to/model\",\n \"port\": 8080\n }\n}\nError Responses for Parse Commands: - 400 Bad Request: Invalid request body, empty command, or parse error - 500 Internal Server Error: Encoding error
The API documentation is automatically generated from code annotations using Swagger/OpenAPI. To regenerate the documentation:
- Install the swag tool:
go install github.com/swaggo/swag/cmd/swag@latest - Generate docs:
swag init -g cmd/server/main.go -o apidocs
If swagger documentation is enabled in the server configuration, you can access the interactive API documentation at:
http://localhost:8080/swagger/\nThis provides a complete interactive interface for testing all API endpoints.
"},{"location":"user-guide/managing-instances/","title":"Managing Instances","text":"Learn how to effectively manage your llama.cpp, MLX, and vLLM instances with Llamactl through both the Web UI and API.
"},{"location":"user-guide/managing-instances/#overview","title":"Overview","text":"Llamactl provides two ways to manage instances:
- Web UI: Accessible at
http://localhost:8080with an intuitive dashboard - REST API: Programmatic access for automation and integration
If authentication is enabled: 1. Navigate to the web UI 2. Enter your credentials 3. Bearer token is stored for the session
"},{"location":"user-guide/managing-instances/#theme-support","title":"Theme Support","text":"- Switch between light and dark themes
- Setting is remembered across sessions
Each instance is displayed as a card showing:
- Instance name
- Health status badge (unknown, ready, error, failed)
- Action buttons (start, stop, edit, logs, delete)
- Click the \"Create Instance\" button on the dashboard
- Enter a unique Name for your instance (only required field)
- Choose Backend Type:
- llama.cpp: For GGUF models using llama-server
- MLX: For MLX-optimized models (macOS only)
- vLLM: For distributed serving and high-throughput inference
- Configure model source:
- For llama.cpp: GGUF model path or HuggingFace repo
- For MLX: MLX model path or identifier (e.g.,
mlx-community/Mistral-7B-Instruct-v0.3-4bit) - For vLLM: HuggingFace model identifier (e.g.,
microsoft/DialoGPT-medium)
- Configure optional instance management settings:
- Auto Restart: Automatically restart instance on failure
- Max Restarts: Maximum number of restart attempts
- Restart Delay: Delay in seconds between restart attempts
- On Demand Start: Start instance when receiving a request to the OpenAI compatible endpoint
- Idle Timeout: Minutes before stopping idle instance (set to 0 to disable)
- Configure backend-specific options:
- llama.cpp: Threads, context size, GPU layers, port, etc.
- MLX: Temperature, top-p, adapter path, Python environment, etc.
- vLLM: Tensor parallel size, GPU memory utilization, quantization, etc.
- Click \"Create\" to save the instance
# Create llama.cpp instance with local model file\ncurl -X POST http://localhost:8080/api/instances/my-llama-instance \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"backend_type\": \"llama_cpp\",\n \"backend_options\": {\n \"model\": \"/path/to/model.gguf\",\n \"threads\": 8,\n \"ctx_size\": 4096,\n \"gpu_layers\": 32\n }\n }'\n\n# Create MLX instance (macOS only)\ncurl -X POST http://localhost:8080/api/instances/my-mlx-instance \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"backend_type\": \"mlx_lm\",\n \"backend_options\": {\n \"model\": \"mlx-community/Mistral-7B-Instruct-v0.3-4bit\",\n \"temp\": 0.7,\n \"top_p\": 0.9,\n \"max_tokens\": 2048\n },\n \"auto_restart\": true,\n \"max_restarts\": 3\n }'\n\n# Create vLLM instance\ncurl -X POST http://localhost:8080/api/instances/my-vllm-instance \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"backend_type\": \"vllm\",\n \"backend_options\": {\n \"model\": \"microsoft/DialoGPT-medium\",\n \"tensor_parallel_size\": 2,\n \"gpu_memory_utilization\": 0.9\n },\n \"auto_restart\": true,\n \"on_demand_start\": true\n }'\n\n# Create llama.cpp instance with HuggingFace model\ncurl -X POST http://localhost:8080/api/instances/gemma-3-27b \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"backend_type\": \"llama_cpp\",\n \"backend_options\": {\n \"hf_repo\": \"unsloth/gemma-3-27b-it-GGUF\",\n \"hf_file\": \"gemma-3-27b-it-GGUF.gguf\",\n \"gpu_layers\": 32\n }\n }'\n- Click the \"Start\" button on an instance card
- Watch the status change to \"Unknown\"
- Monitor progress in the logs
- Instance status changes to \"Ready\" when ready
curl -X POST http://localhost:8080/api/instances/{name}/start\n- Click the \"Stop\" button on an instance card
- Instance gracefully shuts down
curl -X POST http://localhost:8080/api/instances/{name}/stop\n- Click the \"Edit\" button on an instance card
- Modify settings in the configuration dialog
- Changes require instance restart to take effect
- Click \"Update & Restart\" to apply changes
Modify instance settings:
curl -X PUT http://localhost:8080/api/instances/{name} \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"backend_options\": {\n \"threads\": 8,\n \"context_size\": 4096\n }\n }'\nNote
Configuration changes require restarting the instance to take effect.
"},{"location":"user-guide/managing-instances/#view-logs","title":"View Logs","text":""},{"location":"user-guide/managing-instances/#via-web-ui_4","title":"Via Web UI","text":"- Click the \"Logs\" button on any instance card
- Real-time log viewer opens
Check instance status in real-time:
# Get instance details\ncurl http://localhost:8080/api/instances/{name}/logs\n- Click the \"Delete\" button on an instance card
- Only stopped instances can be deleted
- Confirm deletion in the dialog
curl -X DELETE http://localhost:8080/api/instances/{name}\nLlamactl proxies all requests to the underlying backend instances (llama-server, MLX, or vLLM).
# Get instance details\ncurl http://localhost:8080/api/instances/{name}/proxy/\nAll backends provide OpenAI-compatible endpoints. Check the respective documentation: - llama-server docs - MLX-LM docs - vLLM docs
"},{"location":"user-guide/managing-instances/#instance-health","title":"Instance Health","text":""},{"location":"user-guide/managing-instances/#via-web-ui_6","title":"Via Web UI","text":"- The health status badge is displayed on each instance card
Check the health status of your instances:
curl http://localhost:8080/api/instances/{name}/proxy/health\nIssues specific to Llamactl deployment and operation.
"},{"location":"user-guide/troubleshooting/#configuration-issues","title":"Configuration Issues","text":""},{"location":"user-guide/troubleshooting/#invalid-configuration","title":"Invalid Configuration","text":"Problem: Invalid configuration preventing startup
Solutions: 1. Use minimal configuration:
server:\n host: \"0.0.0.0\"\n port: 8080\ninstances:\n port_range: [8000, 9000]\n- Check data directory permissions:
# Ensure data directory is writable (default: ~/.local/share/llamactl)\nmkdir -p ~/.local/share/llamactl/{instances,logs}\n
Problem: Instance fails to start with model loading errors
Common Solutions: - llama-server not found: Ensure llama-server binary is in PATH - Wrong model format: Ensure model is in GGUF format - Insufficient memory: Use smaller model or reduce context size - Path issues: Use absolute paths to model files
Problem: Out of memory errors or system becomes unresponsive
Solutions: 1. Reduce context size:
{\n \"n_ctx\": 1024\n}\n- Use quantized models:
- Try Q4_K_M instead of higher precision models
- Use smaller model variants (7B instead of 13B)
Problem: GPU not being used effectively
Solutions: 1. Configure GPU layers:
{\n \"n_gpu_layers\": 35\n}\nProblem: Complex model loading, performance, or compatibility issues
Since llamactl uses llama-server under the hood, many instance-related issues are actually llama.cpp issues. For advanced troubleshooting:
Resources: - llama.cpp Documentation: https://github.com/ggml/llama.cpp - llama.cpp Issues: https://github.com/ggml/llama.cpp/issues - llama.cpp Discussions: https://github.com/ggml/llama.cpp/discussions
Testing directly with llama-server:
# Test your model and parameters directly with llama-server\nllama-server --model /path/to/model.gguf --port 8081 --n-gpu-layers 35\nThis helps determine if the issue is with llamactl or with the underlying llama.cpp/llama-server.
"},{"location":"user-guide/troubleshooting/#api-and-network-issues","title":"API and Network Issues","text":""},{"location":"user-guide/troubleshooting/#cors-errors","title":"CORS Errors","text":"Problem: Web UI shows CORS errors in browser console
Solutions: 1. Configure allowed origins:
server:\n allowed_origins:\n - \"http://localhost:3000\"\n - \"https://yourdomain.com\"\nProblem: API requests failing with authentication errors
Solutions: 1. Disable authentication temporarily:
auth:\n require_management_auth: false\n require_inference_auth: false\n-
Configure API keys:
auth:\n management_keys:\n - \"your-management-key\"\n inference_keys:\n - \"your-inference-key\"\n -
Use correct Authorization header:
curl -H \"Authorization: Bearer your-api-key\" \\\n http://localhost:8080/api/v1/instances\n
# Get instance logs via API\ncurl http://localhost:8080/api/v1/instances/{name}/logs\n\n# Or check log files directly\ntail -f ~/.local/share/llamactl/logs/{instance-name}.log\nexport LLAMACTL_LOG_LEVEL=debug\nllamactl\nWhen reporting issues, include:
-
System information:
llamactl --version\n -
Configuration file (remove sensitive keys)
-
Relevant log output
-
Steps to reproduce the issue
The server routes requests to the appropriate instance based on the model field in the request body. Instances with on-demand starting enabled will be automatically started if not running. For configuration details, see Managing Instances.
Error Responses:
-- 400 Bad Request: Invalid request body or missing model name
+- 400 Bad Request: Invalid request body or missing instance name
- 503 Service Unavailable: Instance is not running and on-demand start is disabled
- 409 Conflict: Cannot start instance due to maximum instances limit