API Documentation

Everything you need to integrate with labwatch. The API uses JSON for request/response bodies and standard HTTP status codes.

Overview

The labwatch API is organized around REST. All endpoints accept and return JSON (except where noted). The base URL for all API requests is your labwatch server instance.

Rate limits: agents can ingest once per minute per node. API queries are limited to 60 requests per minute per admin key.

Alpha notice: This API is in active development. Endpoints are stable but response shapes may gain new fields. We won't remove fields without a major version bump.

Authentication

labwatch uses two authentication methods depending on the endpoint:

JWT Token (Account Users)

Account users authenticate via JWT. After logging in at /api/v1/auth/login, include the token as a cookie or header:

# Cookie (set automatically by the login page) Cookie: lw_token=eyJhbGci... # Or Authorization header Authorization: Bearer eyJhbGci...

Tokens expire after 30 days. Most /api/v1/account/* endpoints use JWT auth.

Agent Bearer Token

Agents authenticate with a Bearer token received during registration. Include it in the Authorization header:

Authorization: Bearer lw_abc123...

Admin Secret

Admin endpoints require the X-Admin-Secret header. This is set during server setup and gives full access to all data and configuration.

X-Admin-Secret: your-secret-here

Quick Start

Get monitoring running in under 2 minutes:

1. Sign up for a free account

curl -X POST https://labwatch.zazastation.duckdns.org/api/v1/signup \ -H 'Content-Type: application/json' \ -d '{"email": "you@example.com", "hostname": "my-server"}'

You'll receive an admin_secret, lab_id, and token in the response.

2. Install the agent

curl -fsSL https://labwatch.zazastation.duckdns.org/install.sh | bash

3. Configure and start

# Edit /etc/labwatch/config.yaml with your server URL and token labwatch --register systemctl enable --now labwatch

Metrics will start flowing within 60 seconds. Open your dashboard to see them.

Create Account

POST /api/v1/auth/signup Create account with email & password public

Create a full account with email and password. Returns a JWT for immediate use. A verification email will be sent.

Request Body

FieldTypeRequiredDescription
email string required Your email address
password string required Password (minimum 8 characters)
name string optional Display name

Example

curl -X POST https://labwatch.zazastation.duckdns.org/api/v1/auth/signup \ -H 'Content-Type: application/json' \ -d '{"email": "you@example.com", "password": "secure-pass-123"}'

Response 200

{ "account_id": "acc_...", "email": "you@example.com", "plan": "free", "token": "eyJhbGci..." }

Login

POST /api/v1/auth/login Authenticate with email & password public

Login with email and password. Returns a JWT valid for 30 days. Rate limited: 5 failed attempts per email, 20 per IP in a 15-minute window.

Request Body

FieldTypeRequiredDescription
email string required Your email address
password string required Your password

Response 200

{ "account_id": "acc_...", "email": "you@example.com", "plan": "homelab", "token": "eyJhbGci..." }

Error 429

{ "detail": "Too many login attempts. Try again in 15 minutes." }

Forgot Password

POST /api/v1/auth/forgot-password Request a password reset link public

Sends a password reset link to the registered email. Always returns 200 regardless of whether the email exists (prevents email enumeration).

Request Body

FieldTypeRequiredDescription
email string required Account email address

Response 200

{ "message": "If that email is registered, a reset link was sent." }

Reset Password

POST /api/v1/auth/reset-password Set a new password with reset token public

Reset your password using a valid token from the reset email. Token is single-use and expires after 1 hour.

Request Body

FieldTypeRequiredDescription
token string required Reset token from email link
new_password string required New password (minimum 8 characters)

Response 200

{ "message": "Password reset successfully." }

Update Account

PUT /api/v1/account Update account display name jwt

Update your account's display name.

Request Body

FieldTypeRequiredDescription
name string optional New display name (max 100 characters)

Response 200

{ "message": "Account updated." }

Change Password

PUT /api/v1/account/password Change your password jwt

Change your password. Requires your current password for verification.

Request Body

FieldTypeRequiredDescription
current_password string required Your current password
new_password string required New password (minimum 8 characters)

Response 200

{ "message": "Password updated." }

Error 401

{ "detail": "Current password is incorrect" }

Delete Account

DELETE /api/v1/account Permanently delete your account jwt

Permanently deletes your account and all associated data: labs, metrics, alert rules, notification channels, status pages, and incidents. This cannot be undone.

Request Body

FieldTypeRequiredDescription
password string required Your password (confirmation)

Response 200

{ "message": "Account and all associated data deleted." }

Add Lab

POST /api/v1/account/labs Register a new lab under your account jwt (verified)

Add a new monitored node to your account. Returns the lab ID and agent token. Subject to plan limits on maximum nodes.

Request Body

FieldTypeRequiredDescription
hostname string optional Node hostname (default: "my-server")

Response 200

{ "lab_id": "lab_proxmox-01_x7k2", "token": "lw_...", "install_command": "curl -fsSL .../install.sh | sudo bash", "config_snippet": "api_endpoint: ...\ntoken: ...\nlab_id: ...\ninterval: 60s", "message": "Lab registered." }

Plan Limits

PlanMax Nodes
Free1
Homelab10
Pro50
BusinessUnlimited

Rename Lab

PUT /api/v1/account/labs/{lab_id} Rename a monitored node jwt

Update the hostname of one of your labs. Only alphanumeric characters, dots, hyphens, and underscores are allowed.

Request Body

FieldTypeRequiredDescription
hostname string required New hostname (max 64 characters)

Response 200

{ "message": "Lab updated." }

Regenerate Token

POST /api/v1/account/labs/{lab_id}/regenerate-token Rotate a lab's agent token jwt

Generates a new agent token for the lab. The old token is invalidated immediately. You'll need to update the agent's config file with the new token.

Response 200

{ "token": "lw_new_token...", "config_snippet": "api_endpoint: ...\ntoken: ...\nlab_id: ...\ninterval: 60s", "message": "Token regenerated. Update your agent config with the new token." }

Delete Lab (Account)

DELETE /api/v1/account/labs/{lab_id} Delete a lab and all its data jwt

Permanently removes a lab and all associated metrics, alerts, and digests. This cannot be undone.

Response 200

{ "message": "Lab and all associated data deleted." }

Signup

POST /api/v1/signup Self-service account creation public

Create a new free-tier account. Returns credentials for the admin dashboard and a pre-registered first node.

Request Body

FieldTypeRequiredDescription
email string required Your email address
hostname string optional Name for your first node (default: "my-server")

Example

curl -X POST https://your-server/api/v1/signup \ -H 'Content-Type: application/json' \ -d '{"email": "admin@lab.local", "hostname": "proxmox-01"}'

Response 200

{ "message": "Account created successfully", "admin_secret": "abc123...", "lab_id": "lab_proxmox-01_x7k2", "token": "lw_...", "dashboard_url": "/dashboard?secret=abc123..." }

Register Agent

POST /api/v1/register Register a new monitoring agent admin

Register a new node for monitoring. The agent calls this on first startup to get its credentials.

Headers

HeaderValue
X-Admin-SecretYour admin secret

Request Body

FieldTypeRequiredDescription
hostname string required Hostname of the node
os string required Operating system (e.g., "linux")
arch string required Architecture (e.g., "amd64", "arm64")
agent_version string required Agent version string

Response 200

{ "lab_id": "lab_proxmox-01_x7k2", "token": "lw_...", "message": "Registered successfully" }

Ingest Metrics

POST /api/v1/ingest Submit metrics from an agent bearer

Submit a metrics snapshot from a monitoring agent. The agent sends this every 60 seconds (configurable). Metrics are stored and processed for alerts, digests, and queries.

Headers

HeaderValue
AuthorizationBearer <token>

Request Body

FieldTypeRequiredDescription
lab_id string required Your lab identifier
timestamp string optional ISO 8601 timestamp (defaults to server time)
collectors object required Collector data keyed by type

Collector Types

The collectors object can contain the following keys:

KeyDescription
systemCPU, memory, disk, network, load, uptime, temperatures
dockerContainer list with health, restarts, CPU/memory per container
servicesHTTP/TCP health check results with response times
gpuGPU utilization, memory, temperature per device

Example

curl -X POST https://your-server/api/v1/ingest \ -H 'Authorization: Bearer lw_abc123...' \ -H 'Content-Type: application/json' \ -d '{ "lab_id": "lab_proxmox-01_x7k2", "collectors": { "system": { "data": { "cpu": {"total_percent": 23.5, "count": 8}, "memory": {"used_percent": 62.1, "total_bytes": 34359738368}, "disk": [{"mount": "/", "used_percent": 45.2}], "load_average": {"1m": 1.2, "5m": 0.8, "15m": 0.6}, "uptime_seconds": 864000 } }, "docker": { "data": { "containers": [ {"name": "caddy", "state": "running", "health": "healthy", "cpu_percent": 0.5, "memory_mb": 32} ] } } } }'

Response 200

{ "status": "ok", "alerts_triggered": 0 }

Agent Status

GET /api/v1/status/{lab_id} Get current node status bearer

Returns the current status of a monitored node, including latest metrics and active alerts.

Path Parameters

ParamTypeDescription
lab_idstringThe lab identifier

Response 200

{ "lab_id": "lab_proxmox-01_x7k2", "hostname": "proxmox-01", "last_seen": "2026-03-19T22:30:00", "cpu_percent": 23.5, "memory_percent": 62.1, "disk_percent": 45.2, "container_count": 12, "alerts": [] }

List Labs

GET /api/v1/admin/labs List all monitored nodes admin

Returns a list of all registered labs with their current online status and latest metrics summary.

Response 200

{ "labs": [ { "lab_id": "lab_proxmox-01_x7k2", "hostname": "proxmox-01", "last_seen": "2026-03-19T22:30:00", "online": true, "cpu_percent": 23.5, "memory_percent": 62.1, "disk_percent": 45.2 } ] }

Lab History

GET /api/v1/labs/{lab_id}/history Time-series metrics data admin

Returns time-series metrics for a specific lab, suitable for charting. Default window is 24 hours.

Query Parameters

ParamTypeDefaultDescription
hours int 24 Number of hours of history to return
secret string Admin secret (alternative to header)

Response 200

{ "lab_id": "lab_proxmox-01_x7k2", "hours": 24, "metrics": [ { "timestamp": "2026-03-19T21:00:00", "cpu_percent": 23.5, "memory_percent": 62.1, "disk_percent": 45.2, "load_1m": 1.2 } ] }

Export Data

GET /api/v1/admin/lab/{lab_id}/export Export all lab data as JSON admin

Exports all stored metrics for a lab as a downloadable JSON file. Useful for backups, migrations, or external analysis.

Response 200

Returns a JSON file download with all metrics, alerts, and configuration for the specified lab.

Delete Lab

DELETE /api/v1/admin/lab/{lab_id} Remove a monitored node admin

Permanently removes a lab and all its stored metrics, alerts, and digests. This cannot be undone.

Response 200

{ "message": "Lab deleted", "lab_id": "lab_proxmox-01_x7k2" }

Natural Language Query

POST /api/v1/query Ask questions in plain English admin

Ask questions about your infrastructure in natural language. The engine analyzes your metrics data and returns a plain-English answer with supporting data.

Request Body

FieldTypeRequiredDescription
question string required Your question in plain English

Example Questions

"How's my lab?" "Why is pve-storage slow?" "Is caddy running?" "Which server uses the most CPU?" "Am I running out of disk?" "What happened last night?"

Example

curl -X POST https://your-server/api/v1/query \ -H 'X-Admin-Secret: your-secret' \ -H 'Content-Type: application/json' \ -d '{"question": "Why is pve-storage slow?"}'

Response 200

{ "answer": "I/O pressure — load average 38.1 with only 2% CPU usage suggests a disk or network bottleneck, not a compute issue. Memory is fine at 62%. Check for heavy NFS traffic or failing drives.", "confidence": 0.85, "sources": ["system_metrics", "load_analysis"] }

Lab Digest

POST /api/v1/admin/digest/{lab_id} Generate a health digest for a node admin

Generates a plain-English intelligence digest for a specific lab, analyzing trends, anomalies, and health over the specified period.

Query Parameters

ParamTypeDefaultDescription
days int 7 Number of days to analyze

Response 200

{ "lab_id": "lab_proxmox-01_x7k2", "period_days": 7, "grade": "B+", "summary": "proxmox-01 had moderate activity over the last 7 days...", "metrics": { "cpu_avg": 15.2, "cpu_peak": 72.1, "memory_avg": 62.0, "disk_current": 78.3 }, "concerns": ["Disk at 78% — approaching warning threshold"], "generated_at": "2026-03-19T22:30:00" }
GET /api/v1/admin/digest/{lab_id} Retrieve stored digest admin

Returns the most recently generated digest for a lab without regenerating it.

Fleet Digest

POST /api/v1/admin/digest Generate fleet-wide intelligence briefing admin

Generates a comprehensive digest across all monitored nodes. Includes per-node grades, fleet-wide trends, anomalies, and recommendations.

Response 200

{ "fleet_size": 5, "period_days": 7, "overall_grade": "B", "summary": "Your fleet of 5 nodes is generally healthy...", "nodes": [ {"lab_id": "...", "hostname": "proxmox-01", "grade": "A"}, {"lab_id": "...", "hostname": "pve-storage", "grade": "C"} ], "recommendations": [ "pve-storage disk usage trending upward — consider cleanup" ] }

Uptime Checks

Monitor HTTP endpoints, TCP ports, and SSL certificates from the cloud. Checks run every 60 seconds by default.

GET /api/v1/account/uptime-checks List all uptime checks jwt

Returns all uptime checks configured for your account.

Response 200

{ "checks": [ { "id": "uc_...", "name": "Main API", "check_type": "http", "target": "https://api.example.com/health", "interval_seconds": 60, "timeout_seconds": 10, "expected_status": 200, "enabled": true, "last_status": "up", "last_checked_at": "2026-04-05T12:00:00" } ], "total": 1 }

Create Uptime Check

POST /api/v1/account/uptime-checks Create a new uptime check jwt (verified)

Create a new HTTP, TCP, or ping check. HTTPS targets automatically get SSL certificate monitoring. Subject to plan limits.

Request Body

FieldTypeRequiredDescription
name string required Display name for this check
check_type string optional http (default), tcp, or ping
target string required URL for HTTP, host:port for TCP
interval_seconds int optional Check interval (default: 60, min: 30)
timeout_seconds int optional Timeout per check (default: 10)
expected_status int optional Expected HTTP status code (default: 200)

Example

curl -X POST https://your-server/api/v1/account/uptime-checks \ -H 'Authorization: Bearer eyJhbGci...' \ -H 'Content-Type: application/json' \ -d '{"name": "API Health", "target": "https://api.example.com/health"}'

Plan Limits

PlanUptime Checks
Free3
Homelab10
Pro50
BusinessUnlimited

Check Results

GET /api/v1/account/uptime-checks/{check_id}/results Get check result history jwt

Returns the result history for an uptime check. Includes status, response time, HTTP status code, and SSL certificate info for HTTPS checks.

Query Parameters

ParamTypeDefaultDescription
hours int 24 Hours of history to return

Response 200

{ "results": [ { "status": "up", "response_time_ms": 142, "status_code": 200, "ssl_cert": { "valid": true, "expires": "2026-12-31T23:59:59", "days_remaining": 270, "issuer": "R3", "subject": "api.example.com" }, "checked_at": "2026-04-05T12:00:00" } ], "total": 1440, "hours": 24 }

Delete Uptime Check

DELETE /api/v1/account/uptime-checks/{check_id} Remove an uptime check jwt

Permanently removes an uptime check and all its stored results.

Response 200

{ "status": "deleted" }

Maintenance Windows

Schedule maintenance windows to suppress alerts and skip uptime checks during planned downtime.

GET /api/v1/account/maintenance List maintenance windows jwt

Returns all maintenance windows for your account, including past, active, and scheduled windows.

Response 200

{ "windows": [ { "id": "mw_...", "title": "Proxmox cluster upgrade", "start_time": "2026-04-06T02:00:00", "end_time": "2026-04-06T06:00:00", "lab_ids": ["lab_proxmox-01_x7k2"], "suppress_alerts": true, "created_at": "2026-04-05T12:00:00" } ], "total": 1 }

Schedule Maintenance

POST /api/v1/account/maintenance Schedule a maintenance window jwt

Schedule a maintenance window. During active windows, alerts are suppressed and uptime checks skip affected resources.

Request Body

FieldTypeRequiredDescription
title string required Description of the maintenance
start_time string required ISO 8601 start time
end_time string required ISO 8601 end time
lab_ids string[] optional Labs to include (empty = all labs)
suppress_alerts bool optional Suppress alerts during window (default: true)

Response 200

{ "id": "mw_...", "status": "created" }

Delete Maintenance Window

DELETE /api/v1/account/maintenance/{window_id} Remove a maintenance window jwt

Cancel or remove a scheduled maintenance window.

Response 200

{ "status": "deleted" }

List Notification Channels

GET /api/v1/admin/notifications List all notification channels admin

Returns all configured notification channels.

Response 200

{ "channels": [ { "id": 1, "type": "ntfy", "name": "Phone alerts", "config": {"topic": "labwatch-alerts"}, "enabled": true } ] }

Create Notification Channel

POST /api/v1/admin/notifications Create a new notification channel admin

Create a new notification channel. Supported types: webhook, discord, slack, telegram, ntfy, email.

Request Body

FieldTypeRequiredDescription
channel_type string required webhook, discord, slack, telegram, ntfy, or email
name string required Display name for the channel
config object required Channel-specific configuration (see below)
min_severity string optional Minimum severity: info, warning (default), or critical

Discord Config

{"webhook_url": "https://discord.com/api/webhooks/..."}

Slack Config

{"webhook_url": "https://hooks.slack.com/services/..."}

Telegram Config

{"bot_token": "123456:ABC-DEF...", "chat_id": "-1001234567890"}

ntfy Config

{"topic": "labwatch-alerts", "server": "https://ntfy.sh"}

Email Config

{"to_address": "alerts@example.com", "smtp_host": "smtp.gmail.com", "smtp_port": 587, "smtp_user": "...", "smtp_pass": "..."}

Generic Webhook Config

{"url": "https://your-endpoint.com/hook"}

Response 200

{ "id": 1, "message": "Channel created" }

Update Notification Channel

PUT /api/v1/admin/notifications/{channel_id} Update channel configuration admin

Update an existing notification channel's name, config, or enabled status.

Delete Notification Channel

DELETE /api/v1/admin/notifications/{channel_id} Remove a notification channel admin

Permanently removes a notification channel.

Test Notification Channel

POST /api/v1/admin/notifications/{channel_id}/test Send a test notification admin

Sends a test message through the specified channel to verify it's configured correctly.

Response 200

{ "message": "Test notification sent", "channel_id": 1 }

Alert Rules

Custom alert rules let you override the default thresholds for specific metrics, either account-wide or per-lab.

GET /api/v1/account/alert-rules List custom alert rules bearer

Returns all custom alert rules for your account.

Response 200

{ "rules": [ { "id": 1, "metric": "cpu", "warning_threshold": 80.0, "critical_threshold": 95.0, "lab_id": null, "enabled": true } ] }

Create Alert Rule

POST /api/v1/account/alert-rules Create or update a custom threshold bearer

Create a custom alert rule. If a rule already exists for the same metric and lab, it will be updated instead. Requires a paid plan (Homelab or higher).

Request Body

FieldTypeRequiredDescription
metric string required One of: cpu, memory, disk, load, gpu_util, gpu_mem, gpu_temp
warning_threshold number optional Warning threshold percentage
critical_threshold number optional Critical threshold percentage
lab_id string optional Apply to specific lab (omit for account-wide)

Priority Layering

Thresholds are merged with this priority: lab-specific > account-wide > system defaults. You only need to set the thresholds you want to override.

Effective Thresholds

GET /api/v1/account/alert-rules/effective Get merged thresholds for a lab bearer

Returns the effective thresholds after merging system defaults, account-wide rules, and lab-specific rules.

Query Parameters

ParamTypeDescription
lab_idstringLab to check (optional, omit for account-wide)

Response 200

{ "thresholds": { "cpu": {"warning": 80.0, "critical": 95.0}, "memory": {"warning": 85.0, "critical": 95.0}, "disk": {"warning": 80.0, "critical": 90.0}, "load": {"warning": null, "critical": null}, "gpu_util": {"warning": 90.0, "critical": null}, "gpu_mem": {"warning": 90.0, "critical": null}, "gpu_temp": {"warning": 85.0, "critical": 95.0} } }

Status Pages

Create public status pages to share your infrastructure uptime with users or teammates.

GET /api/v1/account/status-pages List your status pages bearer

Returns all status pages for your account.

Create Status Page

POST /api/v1/account/status-pages Create a public status page bearer

Create a public status page with a custom slug. The page will be accessible at /status/your-slug.

Request Body

FieldTypeRequiredDescription
slug string required URL slug (3-50 chars, lowercase alphanumeric + hyphens)
title string optional Page title (default: "System Status")
lab_ids string[] optional Labs to include (default: all your labs)

Plan Limits

PlanStatus Pages
Free1
Homelab3
Pro10
BusinessUnlimited

Incidents

GET /api/v1/account/incidents List incidents bearer

Returns all incidents for your account. By default only shows active (non-resolved) incidents.

Query Parameters

ParamTypeDefaultDescription
include_resolved bool false Include resolved incidents

Response 200

{ "incidents": [ { "id": "abc-123", "title": "Database connectivity issues", "status": "investigating", "severity": "major", "status_page_id": "def-456", "created_at": "2026-04-05T09:00:00", "updated_at": "2026-04-05T09:15:00", "resolved_at": null } ], "total": 1 }
POST /api/v1/account/incidents Create an incident bearer (verified)

Create a new incident. Optionally attach it to a status page to make it visible publicly. An initial timeline update is created automatically.

Body Parameters

FieldTypeRequiredDescription
title string Yes Short description (max 200 chars)
severity string No minor (default), major, or critical
status_page_id string No Attach to a status page
message string No Initial timeline message

Response 200

{ "id": "abc-123", "status": "created" }
PUT /api/v1/account/incidents/{incident_id} Update an incident bearer (verified)

Add a timeline update to an incident and change its status. Each update is appended to the incident's timeline, visible on the public status page.

Body Parameters

FieldTypeRequiredDescription
status string Yes investigating, identified, monitoring, or resolved
message string Yes Description of the update

Response 200

{ "status": "updated", "incident_status": "identified" }

Status Flow

investigating → identified → monitoring → resolved

Account Digest

Generate digests using JWT auth (for account users instead of admin secret).

POST /api/v1/account/digest Fleet digest for your account jwt

Generates a fleet-wide intelligence digest across all labs in your account. Requires Homelab plan or higher.

Query Parameters

ParamTypeDefaultDescription
hours int 168 Hours of data to analyze (default: 7 days)

Response 200

{ "summary": "**3 nodes** monitored. 2 healthy, 0 fair, 1 need attention.\n...", "nodes": [...], "node_count": 3, "concerns_count": 1 }
GET /api/v1/account/digest/{lab_id} Get stored digest for one of your labs jwt

Returns the most recently generated digest for a specific lab. Only returns data for labs you own.

Response 200

Returns the full digest object including grade, summary, metrics, and concerns.

Error 404

{ "detail": "No digest found. Generate one first." }

Chart Data

GET /api/v1/account/chart Time-series data for charting bearer

Returns downsampled CPU, memory, and disk time-series data for all labs in your account. Max ~200 points per lab.

Query Parameters

ParamTypeDefaultDescription
hours int 24 Time window (1-720 hours)
lab_id string Filter to a specific lab

Response 200

{ "series": { "proxmox-01": [ {"t": "2026-03-19T20:00:00", "cpu": 23.5, "mem": 62.1, "disk": 45.2}, {"t": "2026-03-19T20:05:00", "cpu": 18.2, "mem": 61.8, "disk": 45.2} ] }, "hours": 24, "lab_count": 3 }