llamactl/docs/user-guide/troubleshooting.md

Troubleshooting

Common issues and solutions for Llamactl deployment and operation.

Installation Issues

Binary Not Found

Problem: llamactl: command not found

Solutions:

1. Verify the binary is in your PATH:

   echo $PATH
   which llamactl
   

2. Add to PATH or use full path:

   export PATH=$PATH:/path/to/llamactl
   # or
   /full/path/to/llamactl
   

3. Check binary permissions:

   chmod +x llamactl
   

Permission Denied

Problem: Permission errors when starting Llamactl

Solutions:

1. Check file permissions:

   ls -la llamactl
   chmod +x llamactl
   

2. Verify directory permissions:

   # Check models directory
   ls -la /path/to/models/
   
   # Check logs directory
   sudo mkdir -p /var/log/llamactl
   sudo chown $USER:$USER /var/log/llamactl
   

3. Run with appropriate user:

   # Don't run as root unless necessary
   sudo -u llamactl ./llamactl
   

Startup Issues

Port Already in Use

Problem: bind: address already in use

Solutions:

1. Find process using the port:

   sudo netstat -tulpn | grep :8080
   # or
   sudo lsof -i :8080
   

2. Kill the conflicting process:

   sudo kill -9 <PID>
   

3. Use a different port:

   llamactl --port 8081
   

Configuration Errors

Problem: Invalid configuration preventing startup

Solutions:

1. Validate configuration file:

   llamactl --config /path/to/config.yaml --validate
   

2. Check YAML syntax:

   yamllint config.yaml
   

3. Use minimal configuration:

   server:
     host: "localhost"
     port: 8080
   

Instance Management Issues

Model Loading Failures

Problem: Instance fails to start with model loading errors

Diagnostic Steps:

1. Check model file exists:

   ls -la /path/to/model.gguf
   file /path/to/model.gguf
   

2. Verify model format:

   # Check if it's a valid GGUF file
   hexdump -C /path/to/model.gguf | head -5
   

3. Test with llama.cpp directly:

   llama-server --model /path/to/model.gguf --port 8081
   

Common Solutions:

• Corrupted model: Re-download the model file
• Wrong format: Ensure model is in GGUF format
• Insufficient memory: Reduce context size or use smaller model
• Path issues: Use absolute paths, check file permissions

Memory Issues

Problem: Out of memory errors or system becomes unresponsive

Diagnostic Steps:

1. Check system memory:

   free -h
   cat /proc/meminfo
   

2. Monitor memory usage:

   top -p $(pgrep llamactl)
   

3. Check instance memory requirements:

   curl http://localhost:8080/api/instances/{name}/stats
   

Solutions:

1. Reduce context size:

   {
     "options": {
       "context_size": 1024
     }
   }
   

2. Enable memory mapping:

   {
     "options": {
       "no_mmap": false
     }
   }
   

3. Use quantized models:

   • Try Q4_K_M instead of higher precision models
   • Use smaller model variants (7B instead of 13B)

GPU Issues

Problem: GPU not detected or not being used

Diagnostic Steps:

1. Check GPU availability:

   nvidia-smi
   

2. Verify CUDA installation:

   nvcc --version
   

3. Check llama.cpp GPU support:

   llama-server --help | grep -i gpu
   

Solutions:

1. Install CUDA drivers:

   sudo apt update
   sudo apt install nvidia-driver-470 nvidia-cuda-toolkit
   

2. Rebuild llama.cpp with GPU support:

   cmake -DLLAMA_CUBLAS=ON ..
   make
   

3. Configure GPU layers:

   {
     "options": {
       "gpu_layers": 35
     }
   }
   

Performance Issues

Slow Response Times

Problem: API responses are slow or timeouts occur

Diagnostic Steps:

1. Check API response times:

   time curl http://localhost:8080/api/instances
   

2. Monitor system resources:

   htop
   iotop
   

3. Check instance logs:

   curl http://localhost:8080/api/instances/{name}/logs
   

Solutions:

1. Optimize thread count:

   {
     "options": {
       "threads": 6
     }
   }
   

2. Adjust batch size:

   {
     "options": {
       "batch_size": 512
     }
   }
   

3. Enable GPU acceleration:

   {
     "options": {
       "gpu_layers": 35
     }
   }
   

High CPU Usage

Problem: Llamactl consuming excessive CPU

Diagnostic Steps:

1. Identify CPU-intensive processes:

   top -p $(pgrep -f llamactl)
   

2. Check thread allocation:

   curl http://localhost:8080/api/instances/{name}/config
   

Solutions:

1. Reduce thread count:

   {
     "options": {
       "threads": 4
     }
   }
   

2. Limit concurrent instances:

   limits:
     max_instances: 3
   

Network Issues

Connection Refused

Problem: Cannot connect to Llamactl web interface

Diagnostic Steps:

1. Check if service is running:

   ps aux | grep llamactl
   

2. Verify port binding:

   netstat -tulpn | grep :8080
   

3. Test local connectivity:

   curl http://localhost:8080/api/health
   

Solutions:

1. Check firewall settings:

   sudo ufw status
   sudo ufw allow 8080
   

2. Bind to correct interface:

   server:
     host: "0.0.0.0"  # Instead of "localhost"
     port: 8080
   

CORS Errors

Problem: Web UI shows CORS errors in browser console

Solutions:

1. Enable CORS in configuration:

   server:
     cors_enabled: true
     cors_origins:
       - "http://localhost:3000"
       - "https://yourdomain.com"
   

2. Use reverse proxy:

   server {
       listen 80;
       location / {
           proxy_pass http://localhost:8080;
           proxy_set_header Host $host;
           proxy_set_header X-Real-IP $remote_addr;
       }
   }
   

Database Issues

Startup Database Errors

Problem: Database connection failures on startup

Diagnostic Steps:

1. Check database service:

   systemctl status postgresql
   # or
   systemctl status mysql
   

2. Test database connectivity:

   psql -h localhost -U llamactl -d llamactl
   

Solutions:

1. Start database service:

   sudo systemctl start postgresql
   sudo systemctl enable postgresql
   

2. Create database and user:

   CREATE DATABASE llamactl;
   CREATE USER llamactl WITH PASSWORD 'password';
   GRANT ALL PRIVILEGES ON DATABASE llamactl TO llamactl;
   

Web UI Issues

Blank Page or Loading Issues

Problem: Web UI doesn't load or shows blank page

Diagnostic Steps:

1. Check browser console for errors (F12)

2. Verify API connectivity:

   curl http://localhost:8080/api/system/status
   

3. Check static file serving:

   curl http://localhost:8080/
   

Solutions:

1. Clear browser cache
2. Try different browser
3. Check for JavaScript errors in console
4. Verify API endpoint accessibility

Authentication Issues

Problem: Unable to login or authentication failures

Diagnostic Steps:

1. Check authentication configuration:

   curl http://localhost:8080/api/config | jq .auth
   

2. Verify user credentials:

   # Test login endpoint
   curl -X POST http://localhost:8080/api/auth/login \
     -H "Content-Type: application/json" \
     -d '{"username":"admin","password":"password"}'
   

Solutions:

1. Reset admin password:

   llamactl --reset-admin-password
   

2. Disable authentication temporarily:

   auth:
     enabled: false
   

Log Analysis

Enable Debug Logging

For detailed troubleshooting, enable debug logging:

logging:
  level: "debug"
  output: "/var/log/llamactl/debug.log"

Key Log Patterns

Look for these patterns in logs:

Startup issues:

ERRO Failed to start server
ERRO Database connection failed
ERRO Port binding failed

Instance issues:

ERRO Failed to start instance
ERRO Model loading failed
ERRO Process crashed

Performance issues:

WARN High memory usage detected
WARN Request timeout
WARN Resource limit exceeded

Getting Help

Collecting Information

When seeking help, provide:

1. System information:

   uname -a
   llamactl --version
   

2. Configuration:

   llamactl --config-dump
   

3. Logs:

   tail -100 /var/log/llamactl/app.log
   

4. Error details:

   • Exact error messages
   • Steps to reproduce
   • Environment details

Support Channels

• GitHub Issues: Report bugs and feature requests
• Documentation: Check this documentation first
• Community: Join discussions in GitHub Discussions

Preventive Measures

Health Monitoring

Set up monitoring to catch issues early:

# Regular health checks
*/5 * * * * curl -f http://localhost:8080/api/health || alert

Resource Monitoring

Monitor system resources:

# Disk space monitoring
df -h /var/log/llamactl/
df -h /path/to/models/

# Memory monitoring
free -h

Backup Configuration

Regular configuration backups:

# Backup configuration
cp ~/.llamactl/config.yaml ~/.llamactl/config.yaml.backup

# Backup instance configurations
curl http://localhost:8080/api/instances > instances-backup.json