> For the complete documentation index, see [llms.txt](https://spice.docs.medtroniclabs.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://spice.docs.medtroniclabs.org/deploy/deployment-guide/fhir-server-and-mapper.md).

# FHIR Server & Mapper

SPICE stores clinical data as FHIR R4 resources. Two services are involved:

| Service         | Role                                                                                                                                                                                                                                                                                                         | Default port                         |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------ |
| **FHIR server** | A [HAPI FHIR JPA server](https://github.com/hapifhir/hapi-fhir-jpaserver-starter) (HAPI 8.x starter project) backed by PostgreSQL, with SPICE additions: token validation, extra search parameters, database migrations and optional administrative-area partitioning. The REST API is served under `/fhir`. | `8080` (set with `FHIR_SERVER_PORT`) |
| **FHIR mapper** | Translates the platform's internal data into FHIR bundles and posts them to the FHIR server. It also registers SPICE's custom search parameters on start-up.                                                                                                                                                 | `8092`                               |

Deploy the FHIR server first, then the FHIR mapper. The other backend services (authentication, user, admin, offline sync, clinical workflow) are described in [Backend Services](https://spice.docs.medtroniclabs.org/deploy/deployment-guide/server.md).

## Prerequisites

* A PostgreSQL database for the FHIR server, managed or self-hosted. The reference compose file uses PostgreSQL 16. Keep it separate from the platform database used by the other services.
* The authentication service reachable from the FHIR server (it validates tokens).
* For the FHIR mapper: the platform PostgreSQL database, Redis, and the authentication, user, admin, clinical workflow and CQL services (see [Connect the mapper](#connect-the-mapper-to-the-fhir-server-and-platform-services)).
* Java 25 and Maven to build the FHIR server image from source (the Dockerfile uses a Java 25 runtime image). The mapper image is built from its own Dockerfile.

## FHIR server

### Build and run

The container image packages `target/ROOT.war`, so build the WAR first, then the image:

```bash
cd fhir-server
mvn clean package spring-boot:repackage -Pboot -DskipTests
docker build -t fhir-server .
```

For a quick local trial the repository includes a `docker-compose.yml` that starts the server and a PostgreSQL container (the credentials in that file are examples only):

```bash
docker compose up -d --build
```

On first start the server applies its Flyway migrations (`src/main/resources/db/migration`). They create the schema, load SPICE's custom search parameters and set up partition support. Hibernate never changes the schema itself: `HIBERNATE_HBM2DDL_AUTO` defaults to `validate`, so the server refuses to start if the schema has drifted from what it expects. Leave this setting alone.

### Required settings

Set these as environment variables. Use your own secret store for the password.

| Variable                                                                                        | Purpose                                                                                                                                                                | Default                           |
| ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
| `FHIR_DATASOURCE_URL`                                                                           | JDBC URL of the FHIR PostgreSQL database, for example `jdbc:postgresql://<db-host>:5432/<fhir-db>`                                                                     | none (required)                   |
| `FHIR_DATASOURCE_USERNAME` / `FHIR_DATASOURCE_PASSWORD`                                         | Database credentials. The user must be able to run migrations (create and alter tables).                                                                               | none (required)                   |
| `FHIR_SERVER_PORT`                                                                              | HTTP port                                                                                                                                                              | none (set it, for example `8080`) |
| `APP_VALIDATE_TOKEN_API`                                                                        | URL of the authentication service endpoint that validates bearer tokens                                                                                                | none (required)                   |
| `FHIR_SERVER_AUTH_ENABLED`                                                                      | Set to `false` to switch authentication off. For local development only.                                                                                               | `true`                            |
| `INTERNAL_ASSERTION_PUBLIC_KEY`                                                                 | Public key (base64 X.509 DER, ES256) used to verify the internal assertion header that the authentication service issues and the mapper forwards. Leave blank to skip. | blank                             |
| `HAPI_FHIR_CORS_ALLOWED_ORIGIN`                                                                 | Allowed CORS origin. Restrict it in production.                                                                                                                        | `*`                               |
| `HAPI_FHIR_CORS_ALLOW_CREDENTIALS`                                                              | CORS credentials                                                                                                                                                       | `false`                           |
| `FHIR_PARTITION_ENABLED`, `FHIR_PARTITION_REQUEST_TENANT`, `FHIR_PARTITION_PER_DISTRICT_RES_ID` | Partitioning mode. See [Administrative-area partitioning](#administrative-area-partitioning).                                                                          | `true`, `true`, `true`            |

Authentication works as follows. Every request to `/fhir` must carry either a valid internal assertion, or a bearer token plus a `client` header that the authentication service accepts. Anything else gets `401`. Do not expose the FHIR server directly to the internet. Only the SPICE services and, optionally, trusted integrations should reach it.

Tuning settings you may need later (all optional):

| Variable                                                                     | Purpose                                                                                                                                                                         | Default              |
| ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- |
| `FHIR_DB_MAX_POOL_SIZE`, `FHIR_DB_MIN_IDLE`                                  | Database connection pool size                                                                                                                                                   | `100`, `10`          |
| `TOMCAT_THREADS_MAX`, `TOMCAT_ACCEPT_COUNT`, `TOMCAT_MAX_CONNECTIONS`        | Web server threads and queues                                                                                                                                                   | `200`, `100`, `2000` |
| `HAPI_SEARCH_COORD_CORE`, `HAPI_SEARCH_COORD_MAX`, `HAPI_SEARCH_COORD_QUEUE` | Search coordinator thread pool                                                                                                                                                  | `20`, `60`, `500`    |
| `HAPI_RETAIN_CACHED_SEARCHES_MINS`                                           | How long HAPI keeps cached search results                                                                                                                                       | `5`                  |
| `HAPI_REUSE_CACHED_SEARCH_MILLIS`                                            | Cached search reuse window. Keep at `0`: HAPI does not invalidate a cached search when a resource is written, so a non-zero value can return stale results right after a write. | `0`                  |
| `HAPI_FHIR_ADVANCED_LUCENE_INDEXING`                                         | Advanced full-text indexing (needs a search backend)                                                                                                                            | `false`              |

The defaults above come from `application.yaml`. They are starting points, not sizing advice. See [Sizing & Operations](https://spice.docs.medtroniclabs.org/deploy/deployment-guide/sizing-and-operations.md).

### Search parameters and implementation guides

SPICE relies on a few custom FHIR search parameters, for example a national-ID search on Patient and RelatedPerson, an enrolment-location search and an active-diagnosis search on Condition. They are loaded in two ways, and you do not need to load them by hand:

* The FHIR server's migrations insert the ones needed for SPICE's own searches.
* On start-up, the FHIR mapper registers the search parameters bundled with it (`fhir/searchparams/*.json`) through the FHIR API. If it creates any new ones, it asks the server to mark all resources for reindexing. Re-running is safe: parameters that already exist are skipped.

Runtime implementation guide upload is switched off (`ig_runtime_upload_enabled: false`), and no implementation guides are loaded by default.

### Administrative-area partitioning

The FHIR server can split clinical data into **partitions, one per administrative area** (for example one per district). Resources shared by everyone (practitioners, organisations, code systems and similar) live in a shared partition `0`. Clinical resources (households, members, encounters, observations and so on) live in the partition of their area, and requests are routed by a URL path segment such as `/fhir/<area-code>/Patient`. The partition identifier is the numeric code of the area in the administrative hierarchy. Each area also gets its own block of resource ids, which prepares the ground for moving areas to separate database servers later without renumbering.

Pick one of two modes **before the first start against an empty database, and before any data is loaded**:

|                                      | Multi-area mode (default)                                                                                                            | Single-partition mode                                                   |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------- |
| Use when                             | The programme covers many areas, and you want per-area partitions, partition pruning on large tables, and a path to sharding by area | One national or pilot deployment where you do not need per-area routing |
| `FHIR_PARTITION_ENABLED`             | `true`                                                                                                                               | `true`                                                                  |
| `FHIR_PARTITION_REQUEST_TENANT`      | `true`                                                                                                                               | `false`                                                                 |
| `FHIR_PARTITION_PER_DISTRICT_RES_ID` | `true`                                                                                                                               | `false`                                                                 |
| Mapper `FHIR_PARTITION_MODE_ENABLED` | `true`                                                                                                                               | `false`                                                                 |
| Extra work                           | Onboard each area before it writes (below)                                                                                           | None. Every operation is pinned to the default partition `0`.           |

Why the decision comes first:

* The schema is always created in its partitioned form. `partition_id` is a mandatory part of the primary keys, so **do not set `FHIR_PARTITION_ENABLED=false`**: every write fails. Single-partition mode is selected with `FHIR_PARTITION_REQUEST_TENANT=false`, not by disabling partitioning.
* `FHIR_PARTITION_REQUEST_TENANT` is also read by the database migrations, which decide whether to keep or remove the example area partition seeded by earlier migrations. It must have its final value the first time the server starts.
* Data written to partitions cannot be moved to a different mode simply by flipping a variable. Changing mode on a database that already holds data is not a supported operation.
* The mapper's `FHIR_PARTITION_MODE_ENABLED` must match: with it on, the mapper posts each bundle to the partition for the facility's area; with it off, it posts to the plain `/fhir` base URL. The same switch exists in the CQL service. The reporting ETL has a partition id setting (`REPORT_ETL_FHIR_PARTITION_ID`) that a single-partition install sets to `0`.

**Onboarding an area (multi-area mode).** Before facilities in a new area start writing data, an administrator creates that area's partition and its block of resource ids. The repository provides `partitioning/onboard_district.sql`, which registers the partition, creates the next resource-id sequence and records the mapping. Edit the area code and name at the top of the script, then run it:

```bash
psql "$FHIR_DATASOURCE_URL" -v ON_ERROR_STOP=1 -f partitioning/onboard_district.sql
```

Run it once per area, and before that area's first write. Rows for an area without a partition land in a default catch-all partition that gets no pruning, and moving them out later needs a full table scan and an exclusive lock. Treat rows in the catch-all partition as an alert. The script header also describes registering the partition through the REST operation when the server is already running.

Reader services (clinical workflow, offline sync, CQL) and the reporting ETL also need to know about partitioning. See each service's configuration for its partition setting.

### Health checks

The FHIR server exposes Spring Boot actuator endpoints on its HTTP port, outside `/fhir` and without authentication:

```bash
curl -s http://<fhir-server-host>:8080/actuator/health
```

Probe endpoints are enabled, so `/actuator/health/liveness` and `/actuator/health/readiness` are available for container orchestration. To check the FHIR API itself, request the capability statement with credentials:

```bash
curl -s -H "Authorization: Bearer <token>" -H "client: <client-name>" \
  http://<fhir-server-host>:8080/fhir/metadata
```

A `401` here with a healthy actuator endpoint means the server is up and authentication is working. A Prometheus metrics endpoint is also enabled in the configuration.

### Upgrading HAPI FHIR

The FHIR server tracks the upstream starter project. `CUSTOMIZATIONS.md` in the repository lists every file SPICE adds or changes. In summary:

1. Add the upstream project as a git remote, fetch the target release tag and merge it.
2. Expect merge conflicts only in the files `CUSTOMIZATIONS.md` lists as modified (the application configuration, `pom.xml`, the Dockerfile, the Helm chart and the partition configuration class). Keep the Flyway dependencies, the authentication interceptor registration and SPICE's partition code.
3. Compare the upstream Dockerfile for version bumps of the build image and the OpenTelemetry agent.
4. Build and run the tests. Smoke-test that an unauthenticated request returns `401`, and boot once in each partitioning mode.
5. **Back up the database first.** Never edit a migration that has already run anywhere. Add a new one. A major HAPI upgrade can change the database schema.

### Backups

Back up the FHIR database with your normal PostgreSQL tooling (see [Sizing & Operations](https://spice.docs.medtroniclabs.org/deploy/deployment-guide/sizing-and-operations.md#backups-and-restore)). It is the only copy of the clinical record, so treat it as the most important database in the platform. A logical dump looks like this:

```bash
pg_dump -F c -h <db-host> -U <db-user> <fhir-db> > fhir_db_$(date +%F).dump
```

Test restores regularly. Restore into an empty database and start the FHIR server against it. If you use native partitioning or per-area resource-id sequences, a restore must include the sequences and the resource-id block mapping table.

## FHIR mapper

### Required settings

| Variable                                                                                                            | Purpose                                                                                                                                                                                                                                                                                                           |
| ------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DATABASE_URL`, `DATABASE_USERNAME`, `DATABASE_PASSWORD`                                                            | Platform PostgreSQL database (the same one the other backend services use)                                                                                                                                                                                                                                        |
| `REDIS_HOST`, `REDIS_PORT`                                                                                          | Redis                                                                                                                                                                                                                                                                                                             |
| `HAPI_SERVER_URL`                                                                                                   | Base URL of the FHIR API **with a trailing slash**, for example `http://<fhir-server-host>:8080/fhir/`                                                                                                                                                                                                            |
| `HAPI_IDENTIFIER_URL`                                                                                               | Namespace URL used to build identifier systems (national ID, village ID and so on) and the custom search parameter URLs. Include the trailing slash. **Choose it once, before loading any data.** Changing it later makes existing identifiers unreachable by lookup. The mapper refuses to start if it is blank. |
| `AUTH_SERVICE`, `ADMIN_SERVICE`, `USER_SERVICE`, `SPICE_SERVICE`, `CQL_SERVICE`                                     | Base URLs of the other platform services                                                                                                                                                                                                                                                                          |
| `JOB_USER_NAME`, `JOB_USER_PASSWORD`                                                                                | A service account the mapper signs in with to register search parameters. It must exist in the platform before the mapper starts.                                                                                                                                                                                 |
| `FHIR_PARTITION_MODE_ENABLED`                                                                                       | Must match the FHIR server's mode (default `false`). See above.                                                                                                                                                                                                                                                   |
| `INTERNAL_ASSERTION_PUBLIC_KEY`                                                                                     | Public key to verify the internal assertion header on inbound calls. Leave blank to skip.                                                                                                                                                                                                                         |
| `SMART_ANC_ENABLED`, `LAB_TEST_INTENSIFICATION_LIST`, `PRESCRIPTION_PREDICTION_DAYS`, `TB_CONFIRMATION_LAB_TEST_ID` | Clinical behaviour settings the configuration requires.                                                                                                                                                                                                                                                           |

Optional settings: `DB_MAX_POOL_SIZE` (default `30`), `SERVER_MAX_HTTP_HEADER_SIZE` (default 64 KB; the internal assertion header can be large for users scoped to many facilities), `SWAGGER_ENABLED` (default `false`) and the outbound client limits `FHIR_CLIENT_MAX_CONCURRENT_REQUESTS` (default `80`) and `FHIR_CLIENT_RETRY_MAX_ATTEMPTS` (default `6`).

The container image runs the JVM with `-XX:MaxRAMPercentage=75` and exits on out-of-memory, so set a container memory limit and let your orchestrator restart it. A heap dump is written to `/tmp`.

### Connect the mapper to the FHIR server and platform services

1. Start the authentication, user and admin services and the FHIR server first.
2. Create the job service account in the platform (a user the mapper can sign in as) and set `JOB_USER_NAME` and `JOB_USER_PASSWORD`.
3. Start the mapper. On start-up it signs in, then checks and registers the custom search parameters. If the FHIR server or authentication service is not ready yet, it retries with exponential backoff (up to 10 attempts by default, roughly three minutes). If all attempts fail, it logs an error and carries on, so check the logs and restart the mapper once the dependencies are healthy.
4. Confirm the mapper can reach the FHIR server by checking the logs for `Found <n> SearchParameter files` followed by either a reindex message or `No new SearchParameters created`.

### Health check

```bash
curl -s http://<mapper-host>:8092/actuator/health
```

## Common problems

| Symptom                                                                                        | Likely cause and fix                                                                                                                                                                                                                                                 |
| ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| FHIR server fails on start with a schema validation error                                      | The database schema does not match the application. Do not change `HIBERNATE_HBM2DDL_AUTO`. Check that migrations ran against the right database and that the database user can alter tables.                                                                        |
| FHIR server fails on start with a Flyway history error after moving from an older HAPI release | The configuration already ignores history rows whose script is missing (`ignore-migration-patterns: "*:missing"`). Check that you did not override it.                                                                                                               |
| Every write fails with a not-null violation on `partition_id`                                  | `FHIR_PARTITION_ENABLED` was set to `false`. Set it back to `true` and use `FHIR_PARTITION_REQUEST_TENANT=false` for single-partition mode.                                                                                                                          |
| Server fails on start in single-partition mode with an error about resource-id blocks          | `FHIR_PARTITION_PER_DISTRICT_RES_ID` is still `true`. Set it to `false`.                                                                                                                                                                                             |
| All FHIR calls return `401`                                                                    | The token validation URL is wrong or unreachable, the `client` header is missing, or the token has expired. Check `APP_VALIDATE_TOKEN_API`.                                                                                                                          |
| A resource that was just created does not appear in a search                                   | A cached search is being reused. Check `HAPI_REUSE_CACHED_SEARCH_MILLIS` is `0`.                                                                                                                                                                                     |
| Mapper refuses to start with an identifier base error                                          | `HAPI_IDENTIFIER_URL` is blank.                                                                                                                                                                                                                                      |
| Custom searches (such as national ID) return nothing                                           | Search parameters were not registered because the mapper could not reach the FHIR server or sign in. Check the mapper logs, fix connectivity or the job user, and restart the mapper. Reindexing may take time on a large database.                                  |
| Mapper logs `Failed to connect` during a FHIR server restart                                   | The mapper retries connection failures for about 15 seconds. A longer restart window needs a higher `FHIR_CLIENT_RETRY_MAX_ATTEMPTS`, or start the services in order.                                                                                                |
| Mapper returns `Request header is too large`                                                   | An internal assertion header exceeded the limit. Raise `SERVER_MAX_HTTP_HEADER_SIZE` on the mapper.                                                                                                                                                                  |
| Mapper restarts repeatedly                                                                     | Out-of-memory under concurrent bundle processing. Increase the container memory limit or lower `FHIR_CLIENT_MAX_CONCURRENT_REQUESTS`.                                                                                                                                |
| FHIR server runs out of database connections                                                   | The combined pool sizes of all services exceed PostgreSQL `max_connections`. Lower the pool sizes or add a connection pooler. See [Sizing & Operations](https://spice.docs.medtroniclabs.org/deploy/deployment-guide/sizing-and-operations.md#database-connections). |
