> For the complete documentation index, see [llms.txt](https://spice.docs.medtroniclabs.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://spice.docs.medtroniclabs.org/deploy/deployment-guide/sizing-and-operations.md).

# Sizing & Operations

This page gives general guidance for running SPICE in production: how to think about capacity, how to scale, how to keep it running, how to back it up and how to upgrade it. It is deliberately qualitative. Capacity depends on the number of users and records, how data is distributed over areas, and how often devices sync, so measure your own deployment rather than copy someone else's numbers.

## Sizing approach

Start small, load realistic data, measure, and then size from the measurements. Watch CPU, memory, database connections and disk I/O on each component during a realistic migration or sync load.

| Component                                                                                      | What drives load                                                                        | Typically limited by                                                                                                 |
| ---------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| **FHIR database (PostgreSQL)**                                                                 | Every clinical write, every search from the sync service, and the reporting ETL's reads | CPU and disk I/O, then memory for cache. It is the single largest data store, and it grows with each record.         |
| **FHIR server**                                                                                | Concurrent searches and bundle writes, indexing of search parameters                    | CPU and memory (JVM heap), database connections                                                                      |
| **FHIR mapper**                                                                                | Parallel conversion of bundles, outbound calls to the FHIR server                       | Memory (large bundles in parallel), and the FHIR server's throughput                                                 |
| **Offline sync service and message queue**                                                     | Many devices syncing at once, first-time syncs and large uploads                        | Queue consumer concurrency, memory, and downstream services. A first sync of a large area is the heaviest operation. |
| **Platform database (PostgreSQL)**                                                             | Users, hierarchy, facilities, configuration and sync bookkeeping                        | Connections more than raw size                                                                                       |
| **Other backend services** (authentication, user, admin, clinical workflow, notification, CQL) | Request rate from devices and the web apps                                              | Memory and connection pools. Mostly light.                                                                           |
| **Reporting ETL**                                                                              | Run once per schedule. Reads from both databases and writes the reporting database.     | Database I/O during the run. Short, bursty.                                                                          |
| **Redis**                                                                                      | Cache and shared state                                                                  | Memory. Small.                                                                                                       |
| **Reporting database**                                                                         | Report queries and the ETL's writes                                                     | I/O. Size it separately from the transactional databases.                                                            |

General principles:

* **Give the FHIR database the most attention.** Separate it from the platform database, put it on fast storage, and monitor its size and slow queries.
* **Heavy first-time operations** (initial device sync for a large area, bulk migration) are the peak. Size and test for them, not for the average day.
* **Throughput is usually limited by the database and the FHIR server, not by the scripts or services in front of them.** Adding workers to a migration or sync beyond what they can absorb slows everything down.
* **Plan for growth in data, not just users.** The FHIR database grows with every visit, observation and audit record. Plan storage and backup windows for the size in two to three years.

### Defaults shipped in the repository

These are the values in the services' own configuration. They are defaults that let a service start, not recommendations for any particular load.

| Setting                                                                          | Default                                                  | Where                                                        |
| -------------------------------------------------------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------ |
| FHIR server database pool (`FHIR_DB_MAX_POOL_SIZE`)                              | 100                                                      | FHIR server                                                  |
| FHIR server web threads (`TOMCAT_THREADS_MAX`)                                   | 200                                                      | FHIR server                                                  |
| FHIR server search thread pool (`HAPI_SEARCH_COORD_CORE` / `_MAX`)               | 20 / 60                                                  | FHIR server                                                  |
| Database pool (`DB_MAX_POOL_SIZE`)                                               | 30 (20 for the CQL service)                              | FHIR mapper, admin, clinical workflow, offline sync services |
| Concurrent calls from mapper to FHIR server                                      | 80                                                       | FHIR mapper                                                  |
| Queue consumers (`RABBIT_MQ_CONSUMER_CONCURRENCY` / `RABBIT_MQ_MAX_CONCURRENCY`) | 5 / 10                                                   | Offline sync service                                         |
| JVM heap                                                                         | 75% of the container memory limit, exit on out-of-memory | FHIR mapper image                                            |
| Reference compose database                                                       | PostgreSQL 16                                            | FHIR server `docker-compose.yml`                             |

## Scaling

* **Backend services scale out.** The services are REST services whose durable state lives in PostgreSQL, Redis and the message queue, so you can run several replicas behind a load balancer or service.
* **Scheduled jobs are safe to replicate.** Jobs in the notification and offline sync services use database-backed locks (ShedLock), so only one instance runs a given job at a time.
* **The offline sync queue scales by consumers.** More concurrent consumers drain the queue faster, but each consumer calls the FHIR mapper and FHIR server, so raise it only while the FHIR tier has spare capacity.
* **Scale the FHIR server vertically first, then horizontally.** Several FHIR server replicas share one database. Make sure the combined pool size fits the database (see below).
* **The database scales vertically.** Use read replicas for reporting reads if you need them. For very large programmes, administrative-area partitioning in the FHIR server (see [FHIR Server & Mapper](https://spice.docs.medtroniclabs.org/deploy/deployment-guide/fhir-server-and-mapper.md#administrative-area-partitioning)) prepares the data to be split across database servers by area.
* **The ETL is a batch job.** Run one instance at a time.

### Database connections

Each service holds a pool of connections, and PostgreSQL has a hard limit (`max_connections`). With many replicas, the sum of all pools can exceed it, and services then fail with connection timeouts.

* Add up `replicas x pool size` for every service on a database and keep the total comfortably below `max_connections`, leaving room for administrators and backups.
* **Use a connection pooler** such as [PgBouncer](https://www.pgbouncer.org/) in front of PostgreSQL if you scale out, so many application connections share fewer database connections. Choose a pooling mode that your services' use of transactions supports, and test it under load before production.
* Lower pool sizes for services that do not need them. Connection leaks and long outbound calls inside transactions are the usual cause of pool exhaustion, so look at slow requests before raising limits.

## Backups and restore

Back up everything that cannot be rebuilt.

| What                          | Notes                                                                                                                                                        |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **FHIR database**             | The clinical record. Highest priority.                                                                                                                       |
| **Platform database**         | Users, hierarchy, facilities, configuration and sync bookkeeping                                                                                             |
| **Reporting database**        | Can be rebuilt by re-running the ETL, but a restore is much faster than a full rebuild. Back it up if the rebuild time matters.                              |
| **Object storage**            | Files such as signatures, consent forms, exports and sync files                                                                                              |
| **Secrets and configuration** | Environment files, credentials, signing keys and certificates, stored in a secrets manager or an encrypted backup, never in plain text with the data backups |
| **Message queue**             | Treat as transient. Let the queue drain before a planned shutdown.                                                                                           |
| **Redis**                     | Treat as a rebuildable cache.                                                                                                                                |

For PostgreSQL, take regular full backups and, for the FHIR and platform databases, consider continuous archiving (point-in-time recovery) so you can restore to a moment just before an incident. A logical dump is the simplest starting point:

```bash
pg_dump -F c -h <db-host> -U <db-user> <database> > <database>_$(date +%F).dump
pg_restore -h <db-host> -U <db-user> -d <empty-database> <database>_<date>.dump
```

The repository's example backup script takes a daily custom-format dump of the platform and FHIR databases, uploads them to object storage and keeps the most recent ten copies. Treat that retention as an example only. Set yours from your recovery needs and data-protection obligations.

Good practice:

* Encrypt backups, store them away from the primary site, and restrict access to them. They contain personal health information.
* Back up the platform and FHIR databases at about the same time, because they reference each other.
* **Test a restore regularly**, into an empty environment, and time it. A backup you have never restored is not a backup.
* If you use per-area resource-id sequences in the FHIR database, the restore must include them (they are part of the database).

## Monitoring and logging

* **Health endpoints.** Every backend service exposes `/actuator/health` (and `/actuator/info`) on its own port, and the FHIR server also enables liveness and readiness probes at `/actuator/health/liveness` and `/actuator/health/readiness`. Use them for container health checks and load balancer checks. Keep them reachable from your monitoring tools only.
* **Logs.** Services log to the console, and the FHIR mapper also writes rotating log files (by default 30 days of history, capped at 3 GB). Ship console logs to a central log store so they survive restarts. Look for application errors, `401` spikes and connection timeouts.
* **Metrics.** Watch CPU, memory, restarts, database connections and slow queries, database disk space and growth, and message queue depth. A queue that keeps growing means the consumers or the FHIR tier cannot keep up. The FHIR server also has a Prometheus metrics endpoint enabled in its configuration.
* **Application performance monitoring (optional).** The FHIR server image bundles an OpenTelemetry Java agent, and the ETL can report to an Elastic APM server when configured. Neither is required.
* **Reporting ETL.** Check its exit code on every run: `0` means success, `1` means it completed but some transformers failed (the affected watermarks stay where they were, so the next run retries), and `2` means a fatal error such as bad configuration or no database connection. Alert on non-zero.
* **Alerting.** At minimum, alert on: a failed health check, repeated restarts, database disk above a threshold, connection pool exhaustion, a growing queue, a failed backup, a failed ETL run and certificates nearing expiry.

## Upgrades

Upgrade in a maintenance window, after a verified backup, and rehearse on a non-production copy first.

| Order | Step                                                                                     | Notes                                                                                                                                                                                                                                                                                                          |
| ----- | ---------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1     | Read the release notes.                                                                  | Look for configuration changes, new required settings and manual steps.                                                                                                                                                                                                                                        |
| 2     | Back up the databases and configuration.                                                 | See above.                                                                                                                                                                                                                                                                                                     |
| 3     | **Shared common library.** If the common library changed, rebuild every backend service. | The library is compiled into each service, so a service built against an older version does not get the change.                                                                                                                                                                                                |
| 4     | **Schema migrations.**                                                                   | The user service, the facility registry service and the FHIR server run their own Flyway migrations at start-up. The reporting ETL applies its own migrations at the start of each real run. Start them first, and wait for migrations to finish before the rest. Never edit a migration that has already run. |
| 5     | **Backend services.**                                                                    | Roll out one service at a time. Start with the authentication, user and admin services, then the FHIR server, then the FHIR mapper, then the offline sync and clinical workflow services. Keep old and new versions of dependent services compatible during the roll-out.                                      |
| 6     | **Web applications.**                                                                    | Admin portal and reporting web app.                                                                                                                                                                                                                                                                            |
| 7     | **Mobile app.**                                                                          | Release through your normal channel. Devices may run older versions for a while, so keep the server compatible with the previous app version where possible.                                                                                                                                                   |
| 8     | Verify.                                                                                  | Check health endpoints, sign in, run a sync from a test device and confirm the queue drains.                                                                                                                                                                                                                   |

Use a unique, immutable image tag for every build, and roll back by redeploying the previous tag. Database migrations are one-way, so a rollback after a migration means restoring the backup. A major FHIR server (HAPI) upgrade is described in [FHIR Server & Mapper](https://spice.docs.medtroniclabs.org/deploy/deployment-guide/fhir-server-and-mapper.md#upgrading-hapi-fhir).

## Data retention and archival

* Clinical data is a medical record. Keep it for as long as your legal and programme requirements say, and set that period before go-live.
* Logs, backups and exports contain personal data too. Apply retention limits to them and delete on schedule.
* Deactivate users and facilities rather than deleting them, so history keeps its context.
* The FHIR server discards cached search results after a short time (five minutes by default). This is separate from clinical data retention.
*

## Disaster recovery

1. **Define targets.** Decide how much data you can afford to lose (recovery point) and how long you can be down (recovery time). Continuous archiving gives a short recovery point. Daily dumps give up to a day.
2. **Keep a copy off-site.** Store backups and the configuration needed to rebuild in a second location.
3. **Document the rebuild.** Write down the order: databases, secrets, queue and cache, backend services, web apps, then reconnect devices. Keep the document where you can reach it when the main site is down.
4. **Rehearse.** Restore into a clean environment at least once a year, and after major changes.
5. **After a restore, expect devices to re-sync.** Check the queue, then run the reporting ETL so the reporting database catches up.

## Production readiness checklist

**Security**

* [ ] TLS everywhere users and devices connect, with certificates that renew automatically
* [ ] Authentication enabled on the FHIR server (`FHIR_SERVER_AUTH_ENABLED` not set to `false`)
* [ ] No default or example credentials anywhere. Every password, key and salt is generated for your deployment.
* [ ] Secrets in a secrets manager or equivalent, not in images, source control or plain environment files on shared machines
* [ ] CORS origins restricted to your web applications (not `*`)
* [ ] API documentation (Swagger) disabled in production (`SWAGGER_ENABLED=false`, the default in every service)
* [ ] The migration password and any migration service accounts removed or rotated after data migration

**Network**

* [ ] Only the reverse proxy or API gateway is reachable from the internet
* [ ] Databases, Redis, the message queue and the FHIR server on a private network reachable only by the services that need them
* [ ] Management ports (queue console, database administration) not exposed publicly

**Resources and reliability**

* [ ] Container CPU and memory limits set for every service, with restart policies or an orchestrator that restarts failures
* [ ] Database connection totals below `max_connections`, with a pooler if you scale out
* [ ] Database storage on durable, monitored disks with room to grow
* [ ] The partitioning mode of the FHIR server chosen before loading data (see [FHIR Server & Mapper](https://spice.docs.medtroniclabs.org/deploy/deployment-guide/fhir-server-and-mapper.md#administrative-area-partitioning))
* [ ] Time synchronised across all hosts (NTP)

**Operations**

* [ ] Health checks wired to your orchestrator or load balancer
* [ ] Central logs with retention set
* [ ] Alerts configured (see Monitoring) and sent to a person who will act on them
* [ ] Backups scheduled, encrypted and **restore tested**
* [ ] The reporting ETL scheduled and monitored
* [ ] A written upgrade and rollback procedure, and a disaster recovery plan
* [ ] Data retention and access policies agreed and documented
