> ## Documentation Index
> Fetch the complete documentation index at: https://hmis-docs.derrickmugabwa.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring and maintenance

> Monitor HMIS availability, background work, databases, storage, certificates, backups, and integrations.

Monitoring must detect user-visible failure before routine clinical work is materially affected. Send production alerts to a staffed channel with an escalation path.

## Required checks

| Check                        | Frequency             | Alert condition                                                |
| ---------------------------- | --------------------- | -------------------------------------------------------------- |
| `https://{{HMIS_DOMAIN}}/up` | Every minute          | Non-`200` for 3 consecutive checks                             |
| External login page and TLS  | Every 5 minutes       | Unreachable, invalid certificate, or unexpected content        |
| Container state/restarts     | Every minute          | Required service stopped or repeated restart                   |
| Queue failures               | Every 5 minutes       | Any new failed job or sustained queue growth                   |
| Scheduler                    | Daily after `00:01`   | `op-visits:close-expired` does not execute successfully        |
| PostgreSQL health            | Every minute          | Unhealthy, connection exhaustion, or storage error             |
| Disk usage                   | Every 5 minutes       | Warning at 80%, critical at 90%                                |
| Backup age                   | Hourly                | No verified production backup younger than 26 hours            |
| Certificate expiry           | Daily                 | Less than 30 days without successful renewal                   |
| OIE channels                 | Every 5 minutes       | Stopped channel, connector errors, or growing error queue      |
| Orthanc storage              | Every 5 minutes       | Unavailable storage, failed writes, or growth outside forecast |
| Laboratory/radiology probes  | Per approved interval | Order, result, study, or viewer flow fails                     |

## Dokploy operational views

Use Dokploy service monitoring, deployment history, real-time logs, and per-service terminal access. Limit dashboard access to named operators and protect it with HTTPS and strong authentication.

## Daily operations

* Review overnight backup status and backup age.
* Review container state, restarts, application errors, failed jobs, and scheduler execution.
* Check database and disk capacity trends.
* Review OIE failed or quarantined messages and radiology reconciliation queues.
* Confirm production simulators remain stopped.
* Record active incidents and unresolved warnings at shift handover.

## Weekly operations

* Review slow growth in queues, databases, logs, and imaging storage.
* Confirm certificate renewal and DNS health.
* Review failed login, privilege, token, and administrative events.
* Test a synthetic critical workflow in the approved production-monitoring manner.
* Confirm documentation, channel exports, contacts, and downtime materials remain current.

## Monthly maintenance

<Steps>
  <Step title="Review capacity">
    Forecast CPU, memory, database, system disk, and imaging storage for at least the next 12 months.
  </Step>

  <Step title="Patch staging">
    Apply operating-system, Dokploy, container, application, and dependency updates in staging. Re-run acceptance tests and document compatibility.
  </Step>

  <Step title="Schedule production patching">
    Use the release runbook, a current recovery point, and a maintenance window. Do not use floating image upgrades without staging evidence.
  </Step>

  <Step title="Review access and secrets">
    Remove stale accounts and tokens, confirm owners, and rotate any credential that reached its policy date or was exposed.
  </Step>
</Steps>

## Useful application checks

Run in the `app` service terminal:

```bash theme={null}
php artisan about --only=environment
php artisan migrate:status
php artisan queue:failed
php artisan schedule:list
```

Review logs in Dokploy rather than downloading broad production logs. Sanitize patient data and credentials before escalation.

## Alert ownership

Every alert needs a severity, response time, primary owner, backup owner, runbook link, escalation contact, and closure evidence. Repeated alerts must produce a corrective action rather than permanent acknowledgement.