Deployed cf20f30 to dev with MkDocs 1.5.3 and mike 2.0.0

This commit is contained in:
lordmathis
2025-10-09 21:28:27 +00:00
parent 88ce414cf5
commit 43ceed2d71
12 changed files with 392 additions and 332 deletions

View File

@@ -1396,50 +1396,50 @@
<h1 id="api-reference">API Reference<a class="headerlink" href="#api-reference" title="Permanent link">&para;</a></h1>
<p>Complete reference for the Llamactl REST API.</p>
<p>Complete reference for the Llamactl REST API. </p>
<h2 id="base-url">Base URL<a class="headerlink" href="#base-url" title="Permanent link">&para;</a></h2>
<p>All API endpoints are relative to the base URL:</p>
<p>All API endpoints are relative to the base URL: </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-0-1" name="__codelineno-0-1" href="#__codelineno-0-1"></a>http://localhost:8080/api/v1
</code></pre></div>
<h2 id="authentication">Authentication<a class="headerlink" href="#authentication" title="Permanent link">&para;</a></h2>
<p>Llamactl supports API key authentication. If authentication is enabled, include the API key in the Authorization header:</p>
<p>Llamactl supports API key authentication. If authentication is enabled, include the API key in the Authorization header: </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-1-1" name="__codelineno-1-1" href="#__codelineno-1-1"></a>curl<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Authorization: Bearer &lt;your-api-key&gt;&quot;</span><span class="w"> </span><span class="se">\</span>
<a id="__codelineno-1-2" name="__codelineno-1-2" href="#__codelineno-1-2"></a><span class="w"> </span>http://localhost:8080/api/v1/instances
</code></pre></div>
<p>The server supports two types of API keys:
- <strong>Management API Keys</strong>: Required for instance management operations (CRUD operations on instances)
- <strong>Inference API Keys</strong>: Required for OpenAI-compatible inference endpoints</p>
<p>The server supports two types of API keys:<br />
- <strong>Management API Keys</strong>: Required for instance management operations (CRUD operations on instances)<br />
- <strong>Inference API Keys</strong>: Required for OpenAI-compatible inference endpoints </p>
<h2 id="system-endpoints">System Endpoints<a class="headerlink" href="#system-endpoints" title="Permanent link">&para;</a></h2>
<h3 id="get-llamactl-version">Get Llamactl Version<a class="headerlink" href="#get-llamactl-version" title="Permanent link">&para;</a></h3>
<p>Get the version information of the llamactl server.</p>
<p>Get the version information of the llamactl server. </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-2-1" name="__codelineno-2-1" href="#__codelineno-2-1"></a><span class="err">GET /api/v1/version</span>
</code></pre></div>
<p><strong>Response:</strong>
<p><strong>Response:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-3-1" name="__codelineno-3-1" href="#__codelineno-3-1"></a>Version: 1.0.0
<a id="__codelineno-3-2" name="__codelineno-3-2" href="#__codelineno-3-2"></a>Commit: abc123
<a id="__codelineno-3-3" name="__codelineno-3-3" href="#__codelineno-3-3"></a>Build Time: 2024-01-15T10:00:00Z
</code></pre></div></p>
<h3 id="get-llama-server-help">Get Llama Server Help<a class="headerlink" href="#get-llama-server-help" title="Permanent link">&para;</a></h3>
<p>Get help text for the llama-server command.</p>
<p>Get help text for the llama-server command. </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-4-1" name="__codelineno-4-1" href="#__codelineno-4-1"></a><span class="err">GET /api/v1/server/help</span>
</code></pre></div>
<p><strong>Response:</strong> Plain text help output from <code>llama-server --help</code></p>
<p><strong>Response:</strong> Plain text help output from <code>llama-server --help</code> </p>
<h3 id="get-llama-server-version">Get Llama Server Version<a class="headerlink" href="#get-llama-server-version" title="Permanent link">&para;</a></h3>
<p>Get version information of the llama-server binary.</p>
<p>Get version information of the llama-server binary. </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-5-1" name="__codelineno-5-1" href="#__codelineno-5-1"></a><span class="err">GET /api/v1/server/version</span>
</code></pre></div>
<p><strong>Response:</strong> Plain text version output from <code>llama-server --version</code></p>
<p><strong>Response:</strong> Plain text version output from <code>llama-server --version</code> </p>
<h3 id="list-available-devices">List Available Devices<a class="headerlink" href="#list-available-devices" title="Permanent link">&para;</a></h3>
<p>List available devices for llama-server.</p>
<p>List available devices for llama-server. </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-6-1" name="__codelineno-6-1" href="#__codelineno-6-1"></a><span class="err">GET /api/v1/server/devices</span>
</code></pre></div>
<p><strong>Response:</strong> Plain text device list from <code>llama-server --list-devices</code></p>
<p><strong>Response:</strong> Plain text device list from <code>llama-server --list-devices</code> </p>
<h2 id="instances">Instances<a class="headerlink" href="#instances" title="Permanent link">&para;</a></h2>
<h3 id="list-all-instances">List All Instances<a class="headerlink" href="#list-all-instances" title="Permanent link">&para;</a></h3>
<p>Get a list of all instances.</p>
<p>Get a list of all instances. </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-7-1" name="__codelineno-7-1" href="#__codelineno-7-1"></a><span class="err">GET /api/v1/instances</span>
</code></pre></div>
<p><strong>Response:</strong>
<p><strong>Response:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-8-1" name="__codelineno-8-1" href="#__codelineno-8-1"></a><span class="p">[</span>
<a id="__codelineno-8-2" name="__codelineno-8-2" href="#__codelineno-8-2"></a><span class="w"> </span><span class="p">{</span>
<a id="__codelineno-8-3" name="__codelineno-8-3" href="#__codelineno-8-3"></a><span class="w"> </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;llama2-7b&quot;</span><span class="p">,</span>
@@ -1449,10 +1449,10 @@
<a id="__codelineno-8-7" name="__codelineno-8-7" href="#__codelineno-8-7"></a><span class="p">]</span>
</code></pre></div></p>
<h3 id="get-instance-details">Get Instance Details<a class="headerlink" href="#get-instance-details" title="Permanent link">&para;</a></h3>
<p>Get detailed information about a specific instance.</p>
<p>Get detailed information about a specific instance. </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-9-1" name="__codelineno-9-1" href="#__codelineno-9-1"></a><span class="err">GET /api/v1/instances/{name}</span>
</code></pre></div>
<p><strong>Response:</strong>
<p><strong>Response:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-10-1" name="__codelineno-10-1" href="#__codelineno-10-1"></a><span class="p">{</span>
<a id="__codelineno-10-2" name="__codelineno-10-2" href="#__codelineno-10-2"></a><span class="w"> </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;llama2-7b&quot;</span><span class="p">,</span>
<a id="__codelineno-10-3" name="__codelineno-10-3" href="#__codelineno-10-3"></a><span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;running&quot;</span><span class="p">,</span>
@@ -1460,23 +1460,23 @@
<a id="__codelineno-10-5" name="__codelineno-10-5" href="#__codelineno-10-5"></a><span class="p">}</span>
</code></pre></div></p>
<h3 id="create-instance">Create Instance<a class="headerlink" href="#create-instance" title="Permanent link">&para;</a></h3>
<p>Create and start a new instance.</p>
<p>Create and start a new instance. </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-11-1" name="__codelineno-11-1" href="#__codelineno-11-1"></a><span class="err">POST /api/v1/instances/{name}</span>
</code></pre></div>
<p><strong>Request Body:</strong> JSON object with instance configuration. Common fields include:</p>
<p><strong>Request Body:</strong> JSON object with instance configuration. Common fields include: </p>
<ul>
<li><code>backend_type</code>: Backend type (<code>llama_cpp</code>, <code>mlx_lm</code>, or <code>vllm</code>)</li>
<li><code>backend_options</code>: Backend-specific configuration</li>
<li><code>auto_restart</code>: Enable automatic restart on failure</li>
<li><code>max_restarts</code>: Maximum restart attempts</li>
<li><code>restart_delay</code>: Delay between restarts in seconds</li>
<li><code>on_demand_start</code>: Start instance when receiving requests</li>
<li><code>idle_timeout</code>: Idle timeout in minutes</li>
<li><code>environment</code>: Environment variables as key-value pairs</li>
<li><code>nodes</code>: Array with single node name to deploy the instance to (for remote deployments)</li>
<li><code>backend_type</code>: Backend type (<code>llama_cpp</code>, <code>mlx_lm</code>, or <code>vllm</code>) </li>
<li><code>backend_options</code>: Backend-specific configuration </li>
<li><code>auto_restart</code>: Enable automatic restart on failure </li>
<li><code>max_restarts</code>: Maximum restart attempts </li>
<li><code>restart_delay</code>: Delay between restarts in seconds </li>
<li><code>on_demand_start</code>: Start instance when receiving requests </li>
<li><code>idle_timeout</code>: Idle timeout in minutes </li>
<li><code>environment</code>: Environment variables as key-value pairs </li>
<li><code>nodes</code>: Array with single node name to deploy the instance to (for remote deployments) </li>
</ul>
<p>See <a href="../managing-instances/">Managing Instances</a> for complete configuration options.</p>
<p><strong>Response:</strong>
<p>See <a href="../managing-instances/">Managing Instances</a> for complete configuration options. </p>
<p><strong>Response:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-12-1" name="__codelineno-12-1" href="#__codelineno-12-1"></a><span class="p">{</span>
<a id="__codelineno-12-2" name="__codelineno-12-2" href="#__codelineno-12-2"></a><span class="w"> </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;llama2-7b&quot;</span><span class="p">,</span>
<a id="__codelineno-12-3" name="__codelineno-12-3" href="#__codelineno-12-3"></a><span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;running&quot;</span><span class="p">,</span>
@@ -1484,11 +1484,11 @@
<a id="__codelineno-12-5" name="__codelineno-12-5" href="#__codelineno-12-5"></a><span class="p">}</span>
</code></pre></div></p>
<h3 id="update-instance">Update Instance<a class="headerlink" href="#update-instance" title="Permanent link">&para;</a></h3>
<p>Update an existing instance configuration. See <a href="../managing-instances/">Managing Instances</a> for available configuration options.</p>
<p>Update an existing instance configuration. See <a href="../managing-instances/">Managing Instances</a> for available configuration options. </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-13-1" name="__codelineno-13-1" href="#__codelineno-13-1"></a><span class="err">PUT /api/v1/instances/{name}</span>
</code></pre></div>
<p><strong>Request Body:</strong> JSON object with configuration fields to update.</p>
<p><strong>Response:</strong>
<p><strong>Request Body:</strong> JSON object with configuration fields to update. </p>
<p><strong>Response:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-14-1" name="__codelineno-14-1" href="#__codelineno-14-1"></a><span class="p">{</span>
<a id="__codelineno-14-2" name="__codelineno-14-2" href="#__codelineno-14-2"></a><span class="w"> </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;llama2-7b&quot;</span><span class="p">,</span>
<a id="__codelineno-14-3" name="__codelineno-14-3" href="#__codelineno-14-3"></a><span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;running&quot;</span><span class="p">,</span>
@@ -1496,30 +1496,30 @@
<a id="__codelineno-14-5" name="__codelineno-14-5" href="#__codelineno-14-5"></a><span class="p">}</span>
</code></pre></div></p>
<h3 id="delete-instance">Delete Instance<a class="headerlink" href="#delete-instance" title="Permanent link">&para;</a></h3>
<p>Stop and remove an instance.</p>
<p>Stop and remove an instance. </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-15-1" name="__codelineno-15-1" href="#__codelineno-15-1"></a><span class="err">DELETE /api/v1/instances/{name}</span>
</code></pre></div>
<p><strong>Response:</strong> <code>204 No Content</code></p>
<p><strong>Response:</strong> <code>204 No Content</code> </p>
<h2 id="instance-operations">Instance Operations<a class="headerlink" href="#instance-operations" title="Permanent link">&para;</a></h2>
<h3 id="start-instance">Start Instance<a class="headerlink" href="#start-instance" title="Permanent link">&para;</a></h3>
<p>Start a stopped instance.</p>
<p>Start a stopped instance. </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-16-1" name="__codelineno-16-1" href="#__codelineno-16-1"></a><span class="err">POST /api/v1/instances/{name}/start</span>
</code></pre></div>
<p><strong>Response:</strong>
<p><strong>Response:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-17-1" name="__codelineno-17-1" href="#__codelineno-17-1"></a><span class="p">{</span>
<a id="__codelineno-17-2" name="__codelineno-17-2" href="#__codelineno-17-2"></a><span class="w"> </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;llama2-7b&quot;</span><span class="p">,</span>
<a id="__codelineno-17-3" name="__codelineno-17-3" href="#__codelineno-17-3"></a><span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;running&quot;</span><span class="p">,</span>
<a id="__codelineno-17-4" name="__codelineno-17-4" href="#__codelineno-17-4"></a><span class="w"> </span><span class="nt">&quot;created&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">1705312200</span>
<a id="__codelineno-17-5" name="__codelineno-17-5" href="#__codelineno-17-5"></a><span class="p">}</span>
</code></pre></div></p>
<p><strong>Error Responses:</strong>
- <code>409 Conflict</code>: Maximum number of running instances reached
- <code>500 Internal Server Error</code>: Failed to start instance</p>
<p><strong>Error Responses:</strong><br />
- <code>409 Conflict</code>: Maximum number of running instances reached<br />
- <code>500 Internal Server Error</code>: Failed to start instance </p>
<h3 id="stop-instance">Stop Instance<a class="headerlink" href="#stop-instance" title="Permanent link">&para;</a></h3>
<p>Stop a running instance.</p>
<p>Stop a running instance. </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-18-1" name="__codelineno-18-1" href="#__codelineno-18-1"></a><span class="err">POST /api/v1/instances/{name}/stop</span>
</code></pre></div>
<p><strong>Response:</strong>
<p><strong>Response:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-19-1" name="__codelineno-19-1" href="#__codelineno-19-1"></a><span class="p">{</span>
<a id="__codelineno-19-2" name="__codelineno-19-2" href="#__codelineno-19-2"></a><span class="w"> </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;llama2-7b&quot;</span><span class="p">,</span>
<a id="__codelineno-19-3" name="__codelineno-19-3" href="#__codelineno-19-3"></a><span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;stopped&quot;</span><span class="p">,</span>
@@ -1527,10 +1527,10 @@
<a id="__codelineno-19-5" name="__codelineno-19-5" href="#__codelineno-19-5"></a><span class="p">}</span>
</code></pre></div></p>
<h3 id="restart-instance">Restart Instance<a class="headerlink" href="#restart-instance" title="Permanent link">&para;</a></h3>
<p>Restart an instance (stop then start).</p>
<p>Restart an instance (stop then start). </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-20-1" name="__codelineno-20-1" href="#__codelineno-20-1"></a><span class="err">POST /api/v1/instances/{name}/restart</span>
</code></pre></div>
<p><strong>Response:</strong>
<p><strong>Response:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-21-1" name="__codelineno-21-1" href="#__codelineno-21-1"></a><span class="p">{</span>
<a id="__codelineno-21-2" name="__codelineno-21-2" href="#__codelineno-21-2"></a><span class="w"> </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;llama2-7b&quot;</span><span class="p">,</span>
<a id="__codelineno-21-3" name="__codelineno-21-3" href="#__codelineno-21-3"></a><span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;running&quot;</span><span class="p">,</span>
@@ -1538,35 +1538,35 @@
<a id="__codelineno-21-5" name="__codelineno-21-5" href="#__codelineno-21-5"></a><span class="p">}</span>
</code></pre></div></p>
<h3 id="get-instance-logs">Get Instance Logs<a class="headerlink" href="#get-instance-logs" title="Permanent link">&para;</a></h3>
<p>Retrieve instance logs.</p>
<p>Retrieve instance logs. </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-22-1" name="__codelineno-22-1" href="#__codelineno-22-1"></a><span class="err">GET /api/v1/instances/{name}/logs</span>
</code></pre></div>
<p><strong>Query Parameters:</strong>
- <code>lines</code>: Number of lines to return (default: all lines, use -1 for all)</p>
<p><strong>Response:</strong> Plain text log output</p>
<p><strong>Example:</strong>
<p><strong>Query Parameters:</strong><br />
- <code>lines</code>: Number of lines to return (default: all lines, use -1 for all) </p>
<p><strong>Response:</strong> Plain text log output </p>
<p><strong>Example:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-23-1" name="__codelineno-23-1" href="#__codelineno-23-1"></a>curl<span class="w"> </span><span class="s2">&quot;http://localhost:8080/api/v1/instances/my-instance/logs?lines=100&quot;</span>
</code></pre></div></p>
<h3 id="proxy-to-instance">Proxy to Instance<a class="headerlink" href="#proxy-to-instance" title="Permanent link">&para;</a></h3>
<p>Proxy HTTP requests directly to the llama-server instance.</p>
<p>Proxy HTTP requests directly to the llama-server instance. </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-24-1" name="__codelineno-24-1" href="#__codelineno-24-1"></a><span class="err">GET /api/v1/instances/{name}/proxy/*</span>
<a id="__codelineno-24-2" name="__codelineno-24-2" href="#__codelineno-24-2"></a><span class="err">POST /api/v1/instances/{name}/proxy/*</span>
</code></pre></div>
<p>This endpoint forwards all requests to the underlying llama-server instance running on its configured port. The proxy strips the <code>/api/v1/instances/{name}/proxy</code> prefix and forwards the remaining path to the instance.</p>
<p><strong>Example - Check Instance Health:</strong>
<p>This endpoint forwards all requests to the underlying llama-server instance running on its configured port. The proxy strips the <code>/api/v1/instances/{name}/proxy</code> prefix and forwards the remaining path to the instance. </p>
<p><strong>Example - Check Instance Health:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-25-1" name="__codelineno-25-1" href="#__codelineno-25-1"></a>curl<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Authorization: Bearer your-api-key&quot;</span><span class="w"> </span><span class="se">\</span>
<a id="__codelineno-25-2" name="__codelineno-25-2" href="#__codelineno-25-2"></a><span class="w"> </span>http://localhost:8080/api/v1/instances/my-model/proxy/health
</code></pre></div></p>
<p>This forwards the request to <code>http://instance-host:instance-port/health</code> on the actual llama-server instance.</p>
<p><strong>Error Responses:</strong>
- <code>503 Service Unavailable</code>: Instance is not running</p>
<p>This forwards the request to <code>http://instance-host:instance-port/health</code> on the actual llama-server instance. </p>
<p><strong>Error Responses:</strong><br />
- <code>503 Service Unavailable</code>: Instance is not running </p>
<h2 id="openai-compatible-api">OpenAI-Compatible API<a class="headerlink" href="#openai-compatible-api" title="Permanent link">&para;</a></h2>
<p>Llamactl provides OpenAI-compatible endpoints for inference operations.</p>
<p>Llamactl provides OpenAI-compatible endpoints for inference operations. </p>
<h3 id="list-models">List Models<a class="headerlink" href="#list-models" title="Permanent link">&para;</a></h3>
<p>List all instances in OpenAI-compatible format.</p>
<p>List all instances in OpenAI-compatible format. </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-26-1" name="__codelineno-26-1" href="#__codelineno-26-1"></a><span class="err">GET /v1/models</span>
</code></pre></div>
<p><strong>Response:</strong>
<p><strong>Response:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-27-1" name="__codelineno-27-1" href="#__codelineno-27-1"></a><span class="p">{</span>
<a id="__codelineno-27-2" name="__codelineno-27-2" href="#__codelineno-27-2"></a><span class="w"> </span><span class="nt">&quot;object&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;list&quot;</span><span class="p">,</span>
<a id="__codelineno-27-3" name="__codelineno-27-3" href="#__codelineno-27-3"></a><span class="w"> </span><span class="nt">&quot;data&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
@@ -1580,15 +1580,15 @@
<a id="__codelineno-27-11" name="__codelineno-27-11" href="#__codelineno-27-11"></a><span class="p">}</span>
</code></pre></div></p>
<h3 id="chat-completions-completions-embeddings">Chat Completions, Completions, Embeddings<a class="headerlink" href="#chat-completions-completions-embeddings" title="Permanent link">&para;</a></h3>
<p>All OpenAI-compatible inference endpoints are available:</p>
<p>All OpenAI-compatible inference endpoints are available: </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-28-1" name="__codelineno-28-1" href="#__codelineno-28-1"></a><span class="err">POST /v1/chat/completions</span>
<a id="__codelineno-28-2" name="__codelineno-28-2" href="#__codelineno-28-2"></a><span class="err">POST /v1/completions</span>
<a id="__codelineno-28-3" name="__codelineno-28-3" href="#__codelineno-28-3"></a><span class="err">POST /v1/embeddings</span>
<a id="__codelineno-28-4" name="__codelineno-28-4" href="#__codelineno-28-4"></a><span class="err">POST /v1/rerank</span>
<a id="__codelineno-28-5" name="__codelineno-28-5" href="#__codelineno-28-5"></a><span class="err">POST /v1/reranking</span>
</code></pre></div>
<p><strong>Request Body:</strong> Standard OpenAI format with <code>model</code> field specifying the instance name</p>
<p><strong>Example:</strong>
<p><strong>Request Body:</strong> Standard OpenAI format with <code>model</code> field specifying the instance name </p>
<p><strong>Example:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-29-1" name="__codelineno-29-1" href="#__codelineno-29-1"></a><span class="p">{</span>
<a id="__codelineno-29-2" name="__codelineno-29-2" href="#__codelineno-29-2"></a><span class="w"> </span><span class="nt">&quot;model&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;llama2-7b&quot;</span><span class="p">,</span>
<a id="__codelineno-29-3" name="__codelineno-29-3" href="#__codelineno-29-3"></a><span class="w"> </span><span class="nt">&quot;messages&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
@@ -1599,34 +1599,34 @@
<a id="__codelineno-29-8" name="__codelineno-29-8" href="#__codelineno-29-8"></a><span class="w"> </span><span class="p">]</span>
<a id="__codelineno-29-9" name="__codelineno-29-9" href="#__codelineno-29-9"></a><span class="p">}</span>
</code></pre></div></p>
<p>The server routes requests to the appropriate instance based on the <code>model</code> field in the request body. Instances with on-demand starting enabled will be automatically started if not running. For configuration details, see <a href="../managing-instances/">Managing Instances</a>.</p>
<p><strong>Error Responses:</strong>
- <code>400 Bad Request</code>: Invalid request body or missing instance name
- <code>503 Service Unavailable</code>: Instance is not running and on-demand start is disabled
- <code>409 Conflict</code>: Cannot start instance due to maximum instances limit</p>
<p>The server routes requests to the appropriate instance based on the <code>model</code> field in the request body. Instances with on-demand starting enabled will be automatically started if not running. For configuration details, see <a href="../managing-instances/">Managing Instances</a>. </p>
<p><strong>Error Responses:</strong><br />
- <code>400 Bad Request</code>: Invalid request body or missing instance name<br />
- <code>503 Service Unavailable</code>: Instance is not running and on-demand start is disabled<br />
- <code>409 Conflict</code>: Cannot start instance due to maximum instances limit </p>
<h2 id="instance-status-values">Instance Status Values<a class="headerlink" href="#instance-status-values" title="Permanent link">&para;</a></h2>
<p>Instances can have the following status values:
- <code>stopped</code>: Instance is not running
- <code>running</code>: Instance is running and ready to accept requests
<p>Instances can have the following status values:<br />
- <code>stopped</code>: Instance is not running<br />
- <code>running</code>: Instance is running and ready to accept requests<br />
- <code>failed</code>: Instance failed to start or crashed </p>
<h2 id="error-responses">Error Responses<a class="headerlink" href="#error-responses" title="Permanent link">&para;</a></h2>
<p>All endpoints may return error responses in the following format:</p>
<p>All endpoints may return error responses in the following format: </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-30-1" name="__codelineno-30-1" href="#__codelineno-30-1"></a><span class="p">{</span>
<a id="__codelineno-30-2" name="__codelineno-30-2" href="#__codelineno-30-2"></a><span class="w"> </span><span class="nt">&quot;error&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;Error message description&quot;</span>
<a id="__codelineno-30-3" name="__codelineno-30-3" href="#__codelineno-30-3"></a><span class="p">}</span>
</code></pre></div>
<h3 id="common-http-status-codes">Common HTTP Status Codes<a class="headerlink" href="#common-http-status-codes" title="Permanent link">&para;</a></h3>
<ul>
<li><code>200</code>: Success</li>
<li><code>201</code>: Created</li>
<li><code>204</code>: No Content (successful deletion)</li>
<li><code>400</code>: Bad Request (invalid parameters or request body)</li>
<li><code>401</code>: Unauthorized (missing or invalid API key)</li>
<li><code>403</code>: Forbidden (insufficient permissions)</li>
<li><code>404</code>: Not Found (instance not found)</li>
<li><code>409</code>: Conflict (instance already exists, max instances reached)</li>
<li><code>500</code>: Internal Server Error</li>
<li><code>503</code>: Service Unavailable (instance not running)</li>
<li><code>200</code>: Success </li>
<li><code>201</code>: Created </li>
<li><code>204</code>: No Content (successful deletion) </li>
<li><code>400</code>: Bad Request (invalid parameters or request body) </li>
<li><code>401</code>: Unauthorized (missing or invalid API key) </li>
<li><code>403</code>: Forbidden (insufficient permissions) </li>
<li><code>404</code>: Not Found (instance not found) </li>
<li><code>409</code>: Conflict (instance already exists, max instances reached) </li>
<li><code>500</code>: Internal Server Error </li>
<li><code>503</code>: Service Unavailable (instance not running) </li>
</ul>
<h2 id="examples">Examples<a class="headerlink" href="#examples" title="Permanent link">&para;</a></h2>
<h3 id="complete-instance-lifecycle">Complete Instance Lifecycle<a class="headerlink" href="#complete-instance-lifecycle" title="Permanent link">&para;</a></h3>
@@ -1704,7 +1704,7 @@
<a id="__codelineno-32-27" name="__codelineno-32-27" href="#__codelineno-32-27"></a><span class="s1"> }&#39;</span>
</code></pre></div>
<h3 id="using-the-proxy-endpoint">Using the Proxy Endpoint<a class="headerlink" href="#using-the-proxy-endpoint" title="Permanent link">&para;</a></h3>
<p>You can also directly proxy requests to the llama-server instance:</p>
<p>You can also directly proxy requests to the llama-server instance: </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-33-1" name="__codelineno-33-1" href="#__codelineno-33-1"></a><span class="c1"># Direct proxy to instance (bypasses OpenAI compatibility layer)</span>
<a id="__codelineno-33-2" name="__codelineno-33-2" href="#__codelineno-33-2"></a>curl<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span>http://localhost:8080/api/v1/instances/my-model/proxy/completion<span class="w"> </span><span class="se">\</span>
<a id="__codelineno-33-3" name="__codelineno-33-3" href="#__codelineno-33-3"></a><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
@@ -1716,17 +1716,17 @@
</code></pre></div>
<h2 id="backend-specific-endpoints">Backend-Specific Endpoints<a class="headerlink" href="#backend-specific-endpoints" title="Permanent link">&para;</a></h2>
<h3 id="parse-commands">Parse Commands<a class="headerlink" href="#parse-commands" title="Permanent link">&para;</a></h3>
<p>Llamactl provides endpoints to parse command strings from different backends into instance configuration options.</p>
<p>Llamactl provides endpoints to parse command strings from different backends into instance configuration options. </p>
<h4 id="parse-llamacpp-command">Parse Llama.cpp Command<a class="headerlink" href="#parse-llamacpp-command" title="Permanent link">&para;</a></h4>
<p>Parse a llama-server command string into instance options.</p>
<p>Parse a llama-server command string into instance options. </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-34-1" name="__codelineno-34-1" href="#__codelineno-34-1"></a><span class="err">POST /api/v1/backends/llama-cpp/parse-command</span>
</code></pre></div>
<p><strong>Request Body:</strong>
<p><strong>Request Body:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-35-1" name="__codelineno-35-1" href="#__codelineno-35-1"></a><span class="p">{</span>
<a id="__codelineno-35-2" name="__codelineno-35-2" href="#__codelineno-35-2"></a><span class="w"> </span><span class="nt">&quot;command&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;llama-server -m /path/to/model.gguf -c 2048 --port 8080&quot;</span>
<a id="__codelineno-35-3" name="__codelineno-35-3" href="#__codelineno-35-3"></a><span class="p">}</span>
</code></pre></div></p>
<p><strong>Response:</strong>
<p><strong>Response:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-36-1" name="__codelineno-36-1" href="#__codelineno-36-1"></a><span class="p">{</span>
<a id="__codelineno-36-2" name="__codelineno-36-2" href="#__codelineno-36-2"></a><span class="w"> </span><span class="nt">&quot;backend_type&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;llama_cpp&quot;</span><span class="p">,</span>
<a id="__codelineno-36-3" name="__codelineno-36-3" href="#__codelineno-36-3"></a><span class="w"> </span><span class="nt">&quot;llama_server_options&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
@@ -1737,15 +1737,15 @@
<a id="__codelineno-36-8" name="__codelineno-36-8" href="#__codelineno-36-8"></a><span class="p">}</span>
</code></pre></div></p>
<h4 id="parse-mlx-lm-command">Parse MLX-LM Command<a class="headerlink" href="#parse-mlx-lm-command" title="Permanent link">&para;</a></h4>
<p>Parse an MLX-LM server command string into instance options.</p>
<p>Parse an MLX-LM server command string into instance options. </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-37-1" name="__codelineno-37-1" href="#__codelineno-37-1"></a><span class="err">POST /api/v1/backends/mlx/parse-command</span>
</code></pre></div>
<p><strong>Request Body:</strong>
<p><strong>Request Body:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-38-1" name="__codelineno-38-1" href="#__codelineno-38-1"></a><span class="p">{</span>
<a id="__codelineno-38-2" name="__codelineno-38-2" href="#__codelineno-38-2"></a><span class="w"> </span><span class="nt">&quot;command&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;mlx_lm.server --model /path/to/model --port 8080&quot;</span>
<a id="__codelineno-38-3" name="__codelineno-38-3" href="#__codelineno-38-3"></a><span class="p">}</span>
</code></pre></div></p>
<p><strong>Response:</strong>
<p><strong>Response:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-39-1" name="__codelineno-39-1" href="#__codelineno-39-1"></a><span class="p">{</span>
<a id="__codelineno-39-2" name="__codelineno-39-2" href="#__codelineno-39-2"></a><span class="w"> </span><span class="nt">&quot;backend_type&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;mlx_lm&quot;</span><span class="p">,</span>
<a id="__codelineno-39-3" name="__codelineno-39-3" href="#__codelineno-39-3"></a><span class="w"> </span><span class="nt">&quot;mlx_server_options&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
@@ -1755,15 +1755,15 @@
<a id="__codelineno-39-7" name="__codelineno-39-7" href="#__codelineno-39-7"></a><span class="p">}</span>
</code></pre></div></p>
<h4 id="parse-vllm-command">Parse vLLM Command<a class="headerlink" href="#parse-vllm-command" title="Permanent link">&para;</a></h4>
<p>Parse a vLLM serve command string into instance options.</p>
<p>Parse a vLLM serve command string into instance options. </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-40-1" name="__codelineno-40-1" href="#__codelineno-40-1"></a><span class="err">POST /api/v1/backends/vllm/parse-command</span>
</code></pre></div>
<p><strong>Request Body:</strong>
<p><strong>Request Body:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-41-1" name="__codelineno-41-1" href="#__codelineno-41-1"></a><span class="p">{</span>
<a id="__codelineno-41-2" name="__codelineno-41-2" href="#__codelineno-41-2"></a><span class="w"> </span><span class="nt">&quot;command&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;vllm serve /path/to/model --port 8080&quot;</span>
<a id="__codelineno-41-3" name="__codelineno-41-3" href="#__codelineno-41-3"></a><span class="p">}</span>
</code></pre></div></p>
<p><strong>Response:</strong>
<p><strong>Response:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-42-1" name="__codelineno-42-1" href="#__codelineno-42-1"></a><span class="p">{</span>
<a id="__codelineno-42-2" name="__codelineno-42-2" href="#__codelineno-42-2"></a><span class="w"> </span><span class="nt">&quot;backend_type&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;vllm&quot;</span><span class="p">,</span>
<a id="__codelineno-42-3" name="__codelineno-42-3" href="#__codelineno-42-3"></a><span class="w"> </span><span class="nt">&quot;vllm_server_options&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
@@ -1772,20 +1772,20 @@
<a id="__codelineno-42-6" name="__codelineno-42-6" href="#__codelineno-42-6"></a><span class="w"> </span><span class="p">}</span>
<a id="__codelineno-42-7" name="__codelineno-42-7" href="#__codelineno-42-7"></a><span class="p">}</span>
</code></pre></div></p>
<p><strong>Error Responses for Parse Commands:</strong>
- <code>400 Bad Request</code>: Invalid request body, empty command, or parse error
- <code>500 Internal Server Error</code>: Encoding error</p>
<p><strong>Error Responses for Parse Commands:</strong><br />
- <code>400 Bad Request</code>: Invalid request body, empty command, or parse error<br />
- <code>500 Internal Server Error</code>: Encoding error </p>
<h2 id="auto-generated-documentation">Auto-Generated Documentation<a class="headerlink" href="#auto-generated-documentation" title="Permanent link">&para;</a></h2>
<p>The API documentation is automatically generated from code annotations using Swagger/OpenAPI. To regenerate the documentation:</p>
<p>The API documentation is automatically generated from code annotations using Swagger/OpenAPI. To regenerate the documentation: </p>
<ol>
<li>Install the swag tool: <code>go install github.com/swaggo/swag/cmd/swag@latest</code></li>
<li>Generate docs: <code>swag init -g cmd/server/main.go -o apidocs</code></li>
<li>Install the swag tool: <code>go install github.com/swaggo/swag/cmd/swag@latest</code> </li>
<li>Generate docs: <code>swag init -g cmd/server/main.go -o apidocs</code> </li>
</ol>
<h2 id="swagger-documentation">Swagger Documentation<a class="headerlink" href="#swagger-documentation" title="Permanent link">&para;</a></h2>
<p>If swagger documentation is enabled in the server configuration, you can access the interactive API documentation at:</p>
<p>If swagger documentation is enabled in the server configuration, you can access the interactive API documentation at: </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-43-1" name="__codelineno-43-1" href="#__codelineno-43-1"></a>http://localhost:8080/swagger/
</code></pre></div>
<p>This provides a complete interactive interface for testing all API endpoints.</p>
<p>This provides a complete interactive interface for testing all API endpoints. </p>

View File

@@ -1228,63 +1228,63 @@
<h1 id="managing-instances">Managing Instances<a class="headerlink" href="#managing-instances" title="Permanent link">&para;</a></h1>
<p>Learn how to effectively manage your llama.cpp, MLX, and vLLM instances with Llamactl through both the Web UI and API.</p>
<p>Learn how to effectively manage your llama.cpp, MLX, and vLLM instances with Llamactl through both the Web UI and API. </p>
<h2 id="overview">Overview<a class="headerlink" href="#overview" title="Permanent link">&para;</a></h2>
<p>Llamactl provides two ways to manage instances:</p>
<p>Llamactl provides two ways to manage instances: </p>
<ul>
<li><strong>Web UI</strong>: Accessible at <code>http://localhost:8080</code> with an intuitive dashboard</li>
<li><strong>REST API</strong>: Programmatic access for automation and integration</li>
<li><strong>Web UI</strong>: Accessible at <code>http://localhost:8080</code> with an intuitive dashboard </li>
<li><strong>REST API</strong>: Programmatic access for automation and integration </li>
</ul>
<p><img alt="Dashboard Screenshot" src="../../images/dashboard.png" /></p>
<p><img alt="Dashboard Screenshot" src="../../images/dashboard.png" /> </p>
<h3 id="authentication">Authentication<a class="headerlink" href="#authentication" title="Permanent link">&para;</a></h3>
<p>If authentication is enabled:
1. Navigate to the web UI
2. Enter your credentials
3. Bearer token is stored for the session</p>
<p>If authentication is enabled:<br />
1. Navigate to the web UI<br />
2. Enter your credentials<br />
3. Bearer token is stored for the session </p>
<h3 id="theme-support">Theme Support<a class="headerlink" href="#theme-support" title="Permanent link">&para;</a></h3>
<ul>
<li>Switch between light and dark themes</li>
<li>Setting is remembered across sessions</li>
<li>Switch between light and dark themes </li>
<li>Setting is remembered across sessions </li>
</ul>
<h2 id="instance-cards">Instance Cards<a class="headerlink" href="#instance-cards" title="Permanent link">&para;</a></h2>
<p>Each instance is displayed as a card showing:</p>
<p>Each instance is displayed as a card showing: </p>
<ul>
<li><strong>Instance name</strong></li>
<li><strong>Health status badge</strong> (unknown, ready, error, failed)</li>
<li><strong>Action buttons</strong> (start, stop, edit, logs, delete)</li>
<li><strong>Instance name</strong> </li>
<li><strong>Health status badge</strong> (unknown, ready, error, failed) </li>
<li><strong>Action buttons</strong> (start, stop, edit, logs, delete) </li>
</ul>
<h2 id="create-instance">Create Instance<a class="headerlink" href="#create-instance" title="Permanent link">&para;</a></h2>
<h3 id="via-web-ui">Via Web UI<a class="headerlink" href="#via-web-ui" title="Permanent link">&para;</a></h3>
<p><img alt="Create Instance Screenshot" src="../../images/create_instance.png" /></p>
<p><img alt="Create Instance Screenshot" src="../../images/create_instance.png" /> </p>
<ol>
<li>Click the <strong>"Create Instance"</strong> button on the dashboard</li>
<li>Enter a unique <strong>Name</strong> for your instance (only required field)</li>
<li><strong>Select Target Node</strong>: Choose which node to deploy the instance to from the dropdown</li>
<li><strong>Choose Backend Type</strong>:<ul>
<li><strong>llama.cpp</strong>: For GGUF models using llama-server</li>
<li><strong>MLX</strong>: For MLX-optimized models (macOS only)</li>
<li><strong>vLLM</strong>: For distributed serving and high-throughput inference</li>
<li>Click the <strong>"Create Instance"</strong> button on the dashboard </li>
<li>Enter a unique <strong>Name</strong> for your instance (only required field) </li>
<li><strong>Select Target Node</strong>: Choose which node to deploy the instance to from the dropdown </li>
<li><strong>Choose Backend Type</strong>: <ul>
<li><strong>llama.cpp</strong>: For GGUF models using llama-server </li>
<li><strong>MLX</strong>: For MLX-optimized models (macOS only) </li>
<li><strong>vLLM</strong>: For distributed serving and high-throughput inference </li>
</ul>
</li>
<li>Configure model source:<ul>
<li><strong>For llama.cpp</strong>: GGUF model path or HuggingFace repo</li>
<li><strong>For MLX</strong>: MLX model path or identifier (e.g., <code>mlx-community/Mistral-7B-Instruct-v0.3-4bit</code>)</li>
<li><strong>For vLLM</strong>: HuggingFace model identifier (e.g., <code>microsoft/DialoGPT-medium</code>)</li>
<li>Configure model source: <ul>
<li><strong>For llama.cpp</strong>: GGUF model path or HuggingFace repo </li>
<li><strong>For MLX</strong>: MLX model path or identifier (e.g., <code>mlx-community/Mistral-7B-Instruct-v0.3-4bit</code>) </li>
<li><strong>For vLLM</strong>: HuggingFace model identifier (e.g., <code>microsoft/DialoGPT-medium</code>) </li>
</ul>
</li>
<li>Configure optional instance management settings:<ul>
<li><strong>Auto Restart</strong>: Automatically restart instance on failure</li>
<li><strong>Max Restarts</strong>: Maximum number of restart attempts</li>
<li><strong>Restart Delay</strong>: Delay in seconds between restart attempts</li>
<li><strong>On Demand Start</strong>: Start instance when receiving a request to the OpenAI compatible endpoint</li>
<li><strong>Idle Timeout</strong>: Minutes before stopping idle instance (set to 0 to disable)</li>
<li><strong>Environment Variables</strong>: Set custom environment variables for the instance process</li>
<li>Configure optional instance management settings: <ul>
<li><strong>Auto Restart</strong>: Automatically restart instance on failure </li>
<li><strong>Max Restarts</strong>: Maximum number of restart attempts </li>
<li><strong>Restart Delay</strong>: Delay in seconds between restart attempts </li>
<li><strong>On Demand Start</strong>: Start instance when receiving a request to the OpenAI compatible endpoint </li>
<li><strong>Idle Timeout</strong>: Minutes before stopping idle instance (set to 0 to disable) </li>
<li><strong>Environment Variables</strong>: Set custom environment variables for the instance process </li>
</ul>
</li>
<li>Configure backend-specific options:<ul>
<li><strong>llama.cpp</strong>: Threads, context size, GPU layers, port, etc.</li>
<li><strong>MLX</strong>: Temperature, top-p, adapter path, Python environment, etc.</li>
<li><strong>vLLM</strong>: Tensor parallel size, GPU memory utilization, quantization, etc.</li>
<li>Configure backend-specific options: <ul>
<li><strong>llama.cpp</strong>: Threads, context size, GPU layers, port, etc. </li>
<li><strong>MLX</strong>: Temperature, top-p, adapter path, Python environment, etc. </li>
<li><strong>vLLM</strong>: Tensor parallel size, GPU memory utilization, quantization, etc. </li>
</ul>
</li>
<li>Click <strong>"Create"</strong> to save the instance </li>
@@ -1364,10 +1364,10 @@
<h2 id="start-instance">Start Instance<a class="headerlink" href="#start-instance" title="Permanent link">&para;</a></h2>
<h3 id="via-web-ui_1">Via Web UI<a class="headerlink" href="#via-web-ui_1" title="Permanent link">&para;</a></h3>
<ol>
<li>Click the <strong>"Start"</strong> button on an instance card</li>
<li>Watch the status change to "Unknown"</li>
<li>Monitor progress in the logs</li>
<li>Instance status changes to "Ready" when ready</li>
<li>Click the <strong>"Start"</strong> button on an instance card </li>
<li>Watch the status change to "Unknown" </li>
<li>Monitor progress in the logs </li>
<li>Instance status changes to "Ready" when ready </li>
</ol>
<h3 id="via-api_1">Via API<a class="headerlink" href="#via-api_1" title="Permanent link">&para;</a></h3>
<div class="highlight"><pre><span></span><code><a id="__codelineno-1-1" name="__codelineno-1-1" href="#__codelineno-1-1"></a>curl<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span>http://localhost:8080/api/instances/<span class="o">{</span>name<span class="o">}</span>/start
@@ -1375,8 +1375,8 @@
<h2 id="stop-instance">Stop Instance<a class="headerlink" href="#stop-instance" title="Permanent link">&para;</a></h2>
<h3 id="via-web-ui_2">Via Web UI<a class="headerlink" href="#via-web-ui_2" title="Permanent link">&para;</a></h3>
<ol>
<li>Click the <strong>"Stop"</strong> button on an instance card</li>
<li>Instance gracefully shuts down</li>
<li>Click the <strong>"Stop"</strong> button on an instance card </li>
<li>Instance gracefully shuts down </li>
</ol>
<h3 id="via-api_2">Via API<a class="headerlink" href="#via-api_2" title="Permanent link">&para;</a></h3>
<div class="highlight"><pre><span></span><code><a id="__codelineno-2-1" name="__codelineno-2-1" href="#__codelineno-2-1"></a>curl<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span>http://localhost:8080/api/instances/<span class="o">{</span>name<span class="o">}</span>/stop
@@ -1384,13 +1384,13 @@
<h2 id="edit-instance">Edit Instance<a class="headerlink" href="#edit-instance" title="Permanent link">&para;</a></h2>
<h3 id="via-web-ui_3">Via Web UI<a class="headerlink" href="#via-web-ui_3" title="Permanent link">&para;</a></h3>
<ol>
<li>Click the <strong>"Edit"</strong> button on an instance card</li>
<li>Modify settings in the configuration dialog</li>
<li>Changes require instance restart to take effect</li>
<li>Click <strong>"Update &amp; Restart"</strong> to apply changes</li>
<li>Click the <strong>"Edit"</strong> button on an instance card </li>
<li>Modify settings in the configuration dialog </li>
<li>Changes require instance restart to take effect </li>
<li>Click <strong>"Update &amp; Restart"</strong> to apply changes </li>
</ol>
<h3 id="via-api_3">Via API<a class="headerlink" href="#via-api_3" title="Permanent link">&para;</a></h3>
<p>Modify instance settings:</p>
<p>Modify instance settings: </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-3-1" name="__codelineno-3-1" href="#__codelineno-3-1"></a>curl<span class="w"> </span>-X<span class="w"> </span>PUT<span class="w"> </span>http://localhost:8080/api/instances/<span class="o">{</span>name<span class="o">}</span><span class="w"> </span><span class="se">\</span>
<a id="__codelineno-3-2" name="__codelineno-3-2" href="#__codelineno-3-2"></a><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<a id="__codelineno-3-3" name="__codelineno-3-3" href="#__codelineno-3-3"></a><span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{</span>
@@ -1402,45 +1402,45 @@
</code></pre></div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Configuration changes require restarting the instance to take effect.</p>
<p>Configuration changes require restarting the instance to take effect. </p>
</div>
<h2 id="view-logs">View Logs<a class="headerlink" href="#view-logs" title="Permanent link">&para;</a></h2>
<h3 id="via-web-ui_4">Via Web UI<a class="headerlink" href="#via-web-ui_4" title="Permanent link">&para;</a></h3>
<ol>
<li>Click the <strong>"Logs"</strong> button on any instance card</li>
<li>Real-time log viewer opens</li>
<li>Click the <strong>"Logs"</strong> button on any instance card </li>
<li>Real-time log viewer opens </li>
</ol>
<h3 id="via-api_4">Via API<a class="headerlink" href="#via-api_4" title="Permanent link">&para;</a></h3>
<p>Check instance status in real-time:</p>
<p>Check instance status in real-time: </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-4-1" name="__codelineno-4-1" href="#__codelineno-4-1"></a><span class="c1"># Get instance details</span>
<a id="__codelineno-4-2" name="__codelineno-4-2" href="#__codelineno-4-2"></a>curl<span class="w"> </span>http://localhost:8080/api/instances/<span class="o">{</span>name<span class="o">}</span>/logs
</code></pre></div>
<h2 id="delete-instance">Delete Instance<a class="headerlink" href="#delete-instance" title="Permanent link">&para;</a></h2>
<h3 id="via-web-ui_5">Via Web UI<a class="headerlink" href="#via-web-ui_5" title="Permanent link">&para;</a></h3>
<ol>
<li>Click the <strong>"Delete"</strong> button on an instance card</li>
<li>Only stopped instances can be deleted</li>
<li>Confirm deletion in the dialog</li>
<li>Click the <strong>"Delete"</strong> button on an instance card </li>
<li>Only stopped instances can be deleted </li>
<li>Confirm deletion in the dialog </li>
</ol>
<h3 id="via-api_5">Via API<a class="headerlink" href="#via-api_5" title="Permanent link">&para;</a></h3>
<div class="highlight"><pre><span></span><code><a id="__codelineno-5-1" name="__codelineno-5-1" href="#__codelineno-5-1"></a>curl<span class="w"> </span>-X<span class="w"> </span>DELETE<span class="w"> </span>http://localhost:8080/api/instances/<span class="o">{</span>name<span class="o">}</span>
</code></pre></div>
<h2 id="instance-proxy">Instance Proxy<a class="headerlink" href="#instance-proxy" title="Permanent link">&para;</a></h2>
<p>Llamactl proxies all requests to the underlying backend instances (llama-server, MLX, or vLLM).</p>
<p>Llamactl proxies all requests to the underlying backend instances (llama-server, MLX, or vLLM). </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-6-1" name="__codelineno-6-1" href="#__codelineno-6-1"></a><span class="c1"># Get instance details</span>
<a id="__codelineno-6-2" name="__codelineno-6-2" href="#__codelineno-6-2"></a>curl<span class="w"> </span>http://localhost:8080/api/instances/<span class="o">{</span>name<span class="o">}</span>/proxy/
</code></pre></div>
<p>All backends provide OpenAI-compatible endpoints. Check the respective documentation:
- <a href="https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md">llama-server docs</a>
- <a href="https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md">MLX-LM docs</a>
- <a href="https://docs.vllm.ai/en/latest/">vLLM docs</a></p>
<p>All backends provide OpenAI-compatible endpoints. Check the respective documentation:<br />
- <a href="https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md">llama-server docs</a><br />
- <a href="https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md">MLX-LM docs</a><br />
- <a href="https://docs.vllm.ai/en/latest/">vLLM docs</a> </p>
<h3 id="instance-health">Instance Health<a class="headerlink" href="#instance-health" title="Permanent link">&para;</a></h3>
<h4 id="via-web-ui_6">Via Web UI<a class="headerlink" href="#via-web-ui_6" title="Permanent link">&para;</a></h4>
<ol>
<li>The health status badge is displayed on each instance card</li>
<li>The health status badge is displayed on each instance card </li>
</ol>
<h4 id="via-api_6">Via API<a class="headerlink" href="#via-api_6" title="Permanent link">&para;</a></h4>
<p>Check the health status of your instances:</p>
<p>Check the health status of your instances: </p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-7-1" name="__codelineno-7-1" href="#__codelineno-7-1"></a>curl<span class="w"> </span>http://localhost:8080/api/instances/<span class="o">{</span>name<span class="o">}</span>/proxy/health
</code></pre></div>

View File

@@ -998,12 +998,12 @@
<h1 id="troubleshooting">Troubleshooting<a class="headerlink" href="#troubleshooting" title="Permanent link">&para;</a></h1>
<p>Issues specific to Llamactl deployment and operation.</p>
<p>Issues specific to Llamactl deployment and operation. </p>
<h2 id="configuration-issues">Configuration Issues<a class="headerlink" href="#configuration-issues" title="Permanent link">&para;</a></h2>
<h3 id="invalid-configuration">Invalid Configuration<a class="headerlink" href="#invalid-configuration" title="Permanent link">&para;</a></h3>
<p><strong>Problem:</strong> Invalid configuration preventing startup</p>
<p><strong>Solutions:</strong>
1. Use minimal configuration:
<p><strong>Problem:</strong> Invalid configuration preventing startup </p>
<p><strong>Solutions:</strong><br />
1. Use minimal configuration:<br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-0-1" name="__codelineno-0-1" href="#__codelineno-0-1"></a><span class="nt">server</span><span class="p">:</span>
<a id="__codelineno-0-2" name="__codelineno-0-2" href="#__codelineno-0-2"></a><span class="w"> </span><span class="nt">host</span><span class="p">:</span><span class="w"> </span><span class="s">&quot;0.0.0.0&quot;</span>
<a id="__codelineno-0-3" name="__codelineno-0-3" href="#__codelineno-0-3"></a><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">8080</span>
@@ -1011,23 +1011,23 @@
<a id="__codelineno-0-5" name="__codelineno-0-5" href="#__codelineno-0-5"></a><span class="w"> </span><span class="nt">port_range</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="nv">8000</span><span class="p p-Indicator">,</span><span class="w"> </span><span class="nv">9000</span><span class="p p-Indicator">]</span>
</code></pre></div></p>
<ol>
<li>Check data directory permissions:
<li>Check data directory permissions:<br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-1-1" name="__codelineno-1-1" href="#__codelineno-1-1"></a><span class="c1"># Ensure data directory is writable (default: ~/.local/share/llamactl)</span>
<a id="__codelineno-1-2" name="__codelineno-1-2" href="#__codelineno-1-2"></a>mkdir<span class="w"> </span>-p<span class="w"> </span>~/.local/share/llamactl/<span class="o">{</span>instances,logs<span class="o">}</span>
</code></pre></div></li>
</ol>
<h2 id="instance-management-issues">Instance Management Issues<a class="headerlink" href="#instance-management-issues" title="Permanent link">&para;</a></h2>
<h3 id="model-loading-failures">Model Loading Failures<a class="headerlink" href="#model-loading-failures" title="Permanent link">&para;</a></h3>
<p><strong>Problem:</strong> Instance fails to start with model loading errors</p>
<p><strong>Problem:</strong> Instance fails to start with model loading errors </p>
<p><strong>Common Solutions:</strong><br />
- <strong>llama-server not found:</strong> Ensure <code>llama-server</code> binary is in PATH<br />
- <strong>Wrong model format:</strong> Ensure model is in GGUF format<br />
- <strong>Insufficient memory:</strong> Use smaller model or reduce context size<br />
- <strong>Path issues:</strong> Use absolute paths to model files </p>
<h3 id="memory-issues">Memory Issues<a class="headerlink" href="#memory-issues" title="Permanent link">&para;</a></h3>
<p><strong>Problem:</strong> Out of memory errors or system becomes unresponsive</p>
<p><strong>Solutions:</strong>
1. <strong>Reduce context size:</strong>
<p><strong>Problem:</strong> Out of memory errors or system becomes unresponsive </p>
<p><strong>Solutions:</strong><br />
1. <strong>Reduce context size:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-2-1" name="__codelineno-2-1" href="#__codelineno-2-1"></a><span class="p">{</span>
<a id="__codelineno-2-2" name="__codelineno-2-2" href="#__codelineno-2-2"></a><span class="w"> </span><span class="nt">&quot;n_ctx&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">1024</span>
<a id="__codelineno-2-3" name="__codelineno-2-3" href="#__codelineno-2-3"></a><span class="p">}</span>
@@ -1038,16 +1038,16 @@
<li>Use smaller model variants (7B instead of 13B) </li>
</ol>
<h3 id="gpu-configuration">GPU Configuration<a class="headerlink" href="#gpu-configuration" title="Permanent link">&para;</a></h3>
<p><strong>Problem:</strong> GPU not being used effectively</p>
<p><strong>Solutions:</strong>
1. <strong>Configure GPU layers:</strong>
<p><strong>Problem:</strong> GPU not being used effectively </p>
<p><strong>Solutions:</strong><br />
1. <strong>Configure GPU layers:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-3-1" name="__codelineno-3-1" href="#__codelineno-3-1"></a><span class="p">{</span>
<a id="__codelineno-3-2" name="__codelineno-3-2" href="#__codelineno-3-2"></a><span class="w"> </span><span class="nt">&quot;n_gpu_layers&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">35</span>
<a id="__codelineno-3-3" name="__codelineno-3-3" href="#__codelineno-3-3"></a><span class="p">}</span>
</code></pre></div></p>
<h3 id="advanced-instance-issues">Advanced Instance Issues<a class="headerlink" href="#advanced-instance-issues" title="Permanent link">&para;</a></h3>
<p><strong>Problem:</strong> Complex model loading, performance, or compatibility issues</p>
<p>Since llamactl uses <code>llama-server</code> under the hood, many instance-related issues are actually llama.cpp issues. For advanced troubleshooting:</p>
<p><strong>Problem:</strong> Complex model loading, performance, or compatibility issues </p>
<p>Since llamactl uses <code>llama-server</code> under the hood, many instance-related issues are actually llama.cpp issues. For advanced troubleshooting: </p>
<p><strong>Resources:</strong><br />
- <strong>llama.cpp Documentation:</strong> <a href="https://github.com/ggml/llama.cpp">https://github.com/ggml/llama.cpp</a><br />
- <strong>llama.cpp Issues:</strong> <a href="https://github.com/ggml/llama.cpp/issues">https://github.com/ggml/llama.cpp/issues</a><br />
@@ -1056,28 +1056,28 @@
<div class="highlight"><pre><span></span><code><a id="__codelineno-4-1" name="__codelineno-4-1" href="#__codelineno-4-1"></a><span class="c1"># Test your model and parameters directly with llama-server</span>
<a id="__codelineno-4-2" name="__codelineno-4-2" href="#__codelineno-4-2"></a>llama-server<span class="w"> </span>--model<span class="w"> </span>/path/to/model.gguf<span class="w"> </span>--port<span class="w"> </span><span class="m">8081</span><span class="w"> </span>--n-gpu-layers<span class="w"> </span><span class="m">35</span>
</code></pre></div></p>
<p>This helps determine if the issue is with llamactl or with the underlying llama.cpp/llama-server.</p>
<p>This helps determine if the issue is with llamactl or with the underlying llama.cpp/llama-server. </p>
<h2 id="api-and-network-issues">API and Network Issues<a class="headerlink" href="#api-and-network-issues" title="Permanent link">&para;</a></h2>
<h3 id="cors-errors">CORS Errors<a class="headerlink" href="#cors-errors" title="Permanent link">&para;</a></h3>
<p><strong>Problem:</strong> Web UI shows CORS errors in browser console</p>
<p><strong>Solutions:</strong>
1. <strong>Configure allowed origins:</strong>
<p><strong>Problem:</strong> Web UI shows CORS errors in browser console </p>
<p><strong>Solutions:</strong><br />
1. <strong>Configure allowed origins:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-5-1" name="__codelineno-5-1" href="#__codelineno-5-1"></a><span class="nt">server</span><span class="p">:</span>
<a id="__codelineno-5-2" name="__codelineno-5-2" href="#__codelineno-5-2"></a><span class="w"> </span><span class="nt">allowed_origins</span><span class="p">:</span>
<a id="__codelineno-5-3" name="__codelineno-5-3" href="#__codelineno-5-3"></a><span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">&quot;http://localhost:3000&quot;</span>
<a id="__codelineno-5-4" name="__codelineno-5-4" href="#__codelineno-5-4"></a><span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">&quot;https://yourdomain.com&quot;</span>
</code></pre></div></p>
<h2 id="authentication-issues">Authentication Issues<a class="headerlink" href="#authentication-issues" title="Permanent link">&para;</a></h2>
<p><strong>Problem:</strong> API requests failing with authentication errors</p>
<p><strong>Solutions:</strong>
1. <strong>Disable authentication temporarily:</strong>
<p><strong>Problem:</strong> API requests failing with authentication errors </p>
<p><strong>Solutions:</strong><br />
1. <strong>Disable authentication temporarily:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-6-1" name="__codelineno-6-1" href="#__codelineno-6-1"></a><span class="nt">auth</span><span class="p">:</span>
<a id="__codelineno-6-2" name="__codelineno-6-2" href="#__codelineno-6-2"></a><span class="w"> </span><span class="nt">require_management_auth</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">false</span>
<a id="__codelineno-6-3" name="__codelineno-6-3" href="#__codelineno-6-3"></a><span class="w"> </span><span class="nt">require_inference_auth</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">false</span>
</code></pre></div></p>
<ol>
<li>
<p><strong>Configure API keys:</strong>
<p><strong>Configure API keys:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-7-1" name="__codelineno-7-1" href="#__codelineno-7-1"></a><span class="nt">auth</span><span class="p">:</span>
<a id="__codelineno-7-2" name="__codelineno-7-2" href="#__codelineno-7-2"></a><span class="w"> </span><span class="nt">management_keys</span><span class="p">:</span>
<a id="__codelineno-7-3" name="__codelineno-7-3" href="#__codelineno-7-3"></a><span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">&quot;your-management-key&quot;</span>
@@ -1086,7 +1086,7 @@
</code></pre></div></p>
</li>
<li>
<p><strong>Use correct Authorization header:</strong>
<p><strong>Use correct Authorization header:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-8-1" name="__codelineno-8-1" href="#__codelineno-8-1"></a>curl<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Authorization: Bearer your-api-key&quot;</span><span class="w"> </span><span class="se">\</span>
<a id="__codelineno-8-2" name="__codelineno-8-2" href="#__codelineno-8-2"></a><span class="w"> </span>http://localhost:8080/api/v1/instances
</code></pre></div></p>
@@ -1094,9 +1094,9 @@
</ol>
<h2 id="remote-node-issues">Remote Node Issues<a class="headerlink" href="#remote-node-issues" title="Permanent link">&para;</a></h2>
<h3 id="node-configuration">Node Configuration<a class="headerlink" href="#node-configuration" title="Permanent link">&para;</a></h3>
<p><strong>Problem:</strong> Remote instances not appearing or cannot be managed</p>
<p><strong>Solutions:</strong>
1. <strong>Verify node configuration:</strong>
<p><strong>Problem:</strong> Remote instances not appearing or cannot be managed </p>
<p><strong>Solutions:</strong><br />
1. <strong>Verify node configuration:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-9-1" name="__codelineno-9-1" href="#__codelineno-9-1"></a><span class="nt">local_node</span><span class="p">:</span><span class="w"> </span><span class="s">&quot;main&quot;</span><span class="w"> </span><span class="c1"># Must match a key in nodes map</span>
<a id="__codelineno-9-2" name="__codelineno-9-2" href="#__codelineno-9-2"></a><span class="nt">nodes</span><span class="p">:</span>
<a id="__codelineno-9-3" name="__codelineno-9-3" href="#__codelineno-9-3"></a><span class="w"> </span><span class="nt">main</span><span class="p">:</span>
@@ -1106,7 +1106,7 @@
<a id="__codelineno-9-7" name="__codelineno-9-7" href="#__codelineno-9-7"></a><span class="w"> </span><span class="nt">api_key</span><span class="p">:</span><span class="w"> </span><span class="s">&quot;secure-key&quot;</span><span class="w"> </span><span class="c1"># Must match worker1&#39;s management key</span>
</code></pre></div></p>
<ol>
<li><strong>Test remote node connectivity:</strong>
<li><strong>Test remote node connectivity:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-10-1" name="__codelineno-10-1" href="#__codelineno-10-1"></a>curl<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Authorization: Bearer remote-node-key&quot;</span><span class="w"> </span><span class="se">\</span>
<a id="__codelineno-10-2" name="__codelineno-10-2" href="#__codelineno-10-2"></a><span class="w"> </span>http://remote-node:8080/api/v1/instances
</code></pre></div></li>
@@ -1124,21 +1124,21 @@
<a id="__codelineno-12-2" name="__codelineno-12-2" href="#__codelineno-12-2"></a>llamactl
</code></pre></div>
<h2 id="getting-help">Getting Help<a class="headerlink" href="#getting-help" title="Permanent link">&para;</a></h2>
<p>When reporting issues, include:</p>
<p>When reporting issues, include: </p>
<ol>
<li>
<p><strong>System information:</strong>
<p><strong>System information:</strong><br />
<div class="highlight"><pre><span></span><code><a id="__codelineno-13-1" name="__codelineno-13-1" href="#__codelineno-13-1"></a>llamactl<span class="w"> </span>--version
</code></pre></div></p>
</li>
<li>
<p><strong>Configuration file</strong> (remove sensitive keys)</p>
<p><strong>Configuration file</strong> (remove sensitive keys) </p>
</li>
<li>
<p><strong>Relevant log output</strong></p>
<p><strong>Relevant log output</strong> </p>
</li>
<li>
<p><strong>Steps to reproduce the issue</strong></p>
<p><strong>Steps to reproduce the issue</strong> </p>
</li>
</ol>