1
0
mirror of https://github.com/pcvolkmer/mv64e-etl-processor synced 2025-09-13 09:02:50 +00:00

154 Commits

Author SHA1 Message Date
b7aa187293 fix: do not set unexpected config values (#141) 2025-08-26 09:16:07 +02:00
8402462c3b chore: use apache image including SSL config (#140)
The main purpose is to abandon bitnami kafka image.

The examples now include localhost certs and keys for development
purposes only.
More advanced support for SSL connections to kafka will be
available in later versions.
2025-08-25 12:43:32 +02:00
d3e6aa5821 fix: mime type representation in kafka header (#139) 2025-08-25 12:13:44 +02:00
eed0972018 docs: update README.md and add current changes (#137) 2025-08-21 15:51:36 +02:00
jlidke
3b66f42eb2 feat: configuration of genomDe test submission via 'app.genomDeTestSubmission' = 'true', is implemented, now. (#136)
Co-authored-by: Paul-Christian Volkmer <code@pcvolkmer.de>
2025-08-20 10:47:38 +02:00
c40fd7f816 feat: do not default to test submissions (#135) 2025-08-18 13:25:34 +02:00
1759729931 fix: add /mtb path alias for /mtbfile (#134) 2025-08-18 12:51:22 +02:00
jlidke
7f80224eac 132 fix consent check (#133) 2025-08-18 12:30:19 +02:00
3eb1c79cec feat: check consent for DNPM 2.1 requests (#126)
Co-authored-by: Jakub Lidke <jakub.lidke@uni-marburg.de>
2025-08-15 12:37:42 +02:00
jlidke
be513f305a 108 anonym id mtb v2 (#131) 2025-08-14 10:33:55 +02:00
2e88157893 refactor: remove obsolete bwHC data model V1.0 (#129) 2025-08-12 23:11:50 +02:00
bf898e5c25 docs: cleanup README file (#127) 2025-07-23 23:18:19 +02:00
e5693736d8 refactor: simple code cleanups (#125) 2025-07-23 22:45:04 +02:00
jlidke
dfc9de78ce 119 add transaction (#124) 2025-07-23 22:11:47 +02:00
jlidke
199511e567 63 check consent status (#120)
Co-authored-by: Paul-Christian Volkmer <code@pcvolkmer.de>
2025-07-22 20:02:15 +02:00
1319be8b3f chore: update dependencies (#122) 2025-07-20 12:04:38 +02:00
1a5737189c chore: update mtb data model example file (#123) 2025-07-20 11:57:53 +02:00
7543785116 chore: update to Spring Boot 3.5.3 (#116) 2025-06-26 01:14:07 +02:00
858189aa59 chore: data model changes (#117)
See: 3234082af1
2025-06-26 01:08:30 +02:00
17f4dc3512 chore: update to Spring Boot 3.5.0 (#115) 2025-06-08 20:02:16 +02:00
1dd601e8db chore: update dnpm mtb dto library (#113) 2025-05-30 22:36:38 +02:00
b748603c06 chore: update Spring Boot (#112) 2025-05-24 09:46:20 +02:00
b939b2bf57 chore: update Spring Boot (#111) 2025-04-26 11:18:40 +02:00
c6b37fda69 feat: support multiple request content types (#109) 2025-04-06 22:17:46 +02:00
8e3de6a220 feat: add pseudonymization for patient IDs (#107) 2025-04-06 14:42:09 +02:00
c5c553f817 refactor: move CustomMediaType into types.kt (#105) 2025-04-06 13:43:58 +02:00
7d97365aea feat: add endpoint for DNPM-Datamodel V2 using content negotiation (#104)
This simply adds an REST endpoint without proper implementation. The goal is to accept DNPM V2 JSON data.
2025-04-06 13:36:30 +02:00
48b1e62e22 feat: remove obsolete config params (#101) 2025-04-04 17:31:50 +02:00
66cc818755 feat: remove SSL-CA-Location config (#99) 2025-04-04 17:06:09 +02:00
9d4786fae3 refactor: update use of deprecated methods (#96) 2025-04-04 16:39:47 +02:00
b78dc3519b refactor: replace deprecated MockBean annotations (#95) 2025-04-04 16:13:07 +02:00
46015c5b66 chore: update to Spring Boot 3.4 2025-04-04 15:33:49 +02:00
a4d0b73d2b docs: update some information in README.md 2025-04-04 14:55:14 +02:00
9307fc0dad docs: change etl image and highlight important information 2025-04-04 14:49:37 +02:00
586d388e57 docs: add info about DNPM:DIP support 2025-04-04 14:36:30 +02:00
7ae34719fd feat: add new MTB endpoint path (#93) 2025-04-04 14:34:31 +02:00
033750eb10 feat: show issue path if available in response body (#92) 2025-04-04 13:59:51 +02:00
befeef3153 feat: use issue severity to create status (#90) 2025-04-03 17:06:03 +02:00
98b971d7db feat: do not retry on validation issues (#89)
This will prevent retry if response is HTTP 400 or HTTP 422.
2025-03-23 13:35:24 +01:00
56a63b276e Code cleanup (#87)
* refactor: Replace usage of Void with Kotlins Unit

* refactor: make ConnectionCheckService a functional interface

* refactor: ignore unused exception

* refactor: use property access syntax

* refactor: use const value for login path
2025-03-23 12:09:34 +01:00
c0ea5fcd51 test: use Europe/Berlin as timezone in tests 2025-03-23 01:05:06 +01:00
d4fd54f51f Merge branch '0.9.x' 2025-03-22 23:58:47 +01:00
d49671f0d4 build: update image name 2025-03-22 23:40:13 +01:00
84868dc22c Merge branch '0.9.x' 2025-03-22 11:26:16 +01:00
4ad6c4bd0a feat: handle and save issue report for non HTTP 2xx responses 2025-03-22 11:04:32 +01:00
9bdd8ba375 chore: update Spring Boot 2025-03-22 10:20:52 +01:00
f027339425 chore: update gradle 2025-03-22 10:20:35 +01:00
3c5639708f chore: highlight selected config tab 2025-03-21 19:26:25 +01:00
639159c677 docs: add information about DNPM:DIP dev environment 2025-03-21 19:15:20 +01:00
38261d6d2c chore: update bwhc-dto-java
This enables use of WHOGrading version 2021.
2025-03-21 19:11:24 +01:00
47ebe46974 feat: add checks for DNPM:DIP backend
Since DNPM:DIP responds with HTTP 404 on API base path, the Kaplan-Meier Config
endpoint will be used to check availability of DNPM:DIP backend API.
2025-03-20 14:39:40 +01:00
f347653be8 refactor: use UriComponentsBuilder to build URL to be used
This prevents problems using trailing slash in remote API URL.
2025-03-20 14:19:25 +01:00
775a7df1ce chore: use API URL to DNPM:DIP 2025-03-20 14:13:21 +01:00
f66b737f11 docs: add example APP_REST_URI for use with dnpm:dip 2025-03-19 15:29:55 +01:00
3a19212a78 chore: update Spring Boot 2025-03-09 09:38:59 +01:00
280fbd445e chore: update Spring Boot 2025-03-09 09:24:51 +01:00
91e2cf5ef1 refactor: use different sender classes for bwHC and DIP 2025-03-08 11:42:14 +01:00
262c54f2e5 fix: use patient pseudonym value 2025-03-08 11:42:14 +01:00
Niklas Sombert
b25e580113 feat: Support POSTing data to dnpm:dip 2025-03-08 11:42:14 +01:00
ff27b7157d chore: update spring boot and dependency management plugin 2025-02-10 19:18:27 +01:00
1e652a7856 test: explicit request URI check and fix use of expect() 2025-02-09 11:19:35 +01:00
74ff9f08a4 chore: update to mockito-kotlin 5.4.0
With this change, `anyValueClass()` from mockito-kotlin
replaces own implementation.
2024-12-23 18:16:37 +01:00
23cc2f365a chore: update dependencies 2024-12-20 10:45:16 +01:00
53b4cf1a95 chore: update dependencies 2024-11-25 21:02:55 +01:00
5ce13e962b build: add description and group to task 2024-11-02 15:08:25 +01:00
3257493b6a build: update HAPI dependencies
This also overrides 'commons-io' due to CVE-2024-47554
2024-11-02 15:08:25 +01:00
2036077c06 build: do not hard code version numbers in dependencies 2024-11-02 15:08:25 +01:00
8ce3aed870 docs: modify changed urls 2024-11-01 15:38:47 +01:00
998989d319 Merge branch '0.9.x' 2024-11-01 15:32:40 +01:00
46ba565c29 chore: update to Spring Boot 3.3.5 (#79) 2024-11-01 14:05:20 +01:00
Niklas
6cdbd35e64 feat: Allow configuring basic auth for the rest uri (#75) 2024-11-01 13:56:54 +01:00
jlidke
d258d9081b chore: gPas health check, fetch metadata instead of send invalid gPas request (#73) 2024-11-01 13:54:40 +01:00
eb49ba611b chore: update to Spring Boot 3.3.4 (#78) 2024-11-01 13:51:06 +01:00
efa736f232 chore: update to Spring Boot 3.3.3 (#77) 2024-11-01 13:42:20 +01:00
4a7030e85b chore: update to Spring Boot 3.3.2 (#76) 2024-11-01 13:27:31 +01:00
464c8b8c1d refactor: use dedicated type for path param 2024-07-15 11:59:00 +02:00
3f1bb4f4e2 refactor: rename template attribute to reflect content 2024-07-15 11:51:33 +02:00
370ea87095 refactor: rename db column name to reflect content 2024-07-15 11:44:19 +02:00
c8f6e6efc8 refactor: add types for patient id and pseudonym 2024-07-15 10:31:52 +02:00
c949ec07e5 Merge branch 0.9.x 2024-07-15 08:11:11 +02:00
6e0ec6b95a style: use sans-serif font everywhere 2024-06-25 16:16:13 +02:00
0ff56416dd chore: update dev-compose.yml 2024-06-25 16:13:55 +02:00
3a2f6a2bb6 chore: update oidc client local dev password 2024-06-25 16:05:41 +02:00
3eb9e68786 fix: use versions param for commons-compress 2024-06-25 08:18:02 +02:00
59403d1dba chore: update dependency version definitions
This will use httpclient5 version as defined within spring boot.
2024-06-24 10:18:18 +02:00
9f5ac664af chore: update to spring boot 3.3.1 2024-06-24 10:02:12 +02:00
5867ed9dd3 Merge pull request #67 from CCC-MF/66-update-auf-spring-boot-33x
chore: update to spring boot 3.3.0
2024-06-20 11:37:34 +02:00
4d6d1879e6 chore: update to spring boot 3.3.0
This also adds dependency to org.flywaydb:flyway-database-postgresql
2024-06-20 09:14:17 +02:00
2a34c0efc9 fix: downgrade echarts due to dependency issues 2024-06-19 14:25:58 +02:00
0ee00de5aa chore: update webjars dependencies 2024-06-19 14:16:52 +02:00
baeebdb9b8 Merge branch 0.9.x 2024-05-31 13:31:40 +02:00
5320466b6c test: exclude test and integrationTest source set
This replaces filtering for classes that contain 'Test' or 'Tests' in simple name.
2024-05-30 14:29:31 +02:00
263cb02416 test: use Kotlin KArgumentCaptor 2024-05-27 15:22:54 +02:00
0b37fd7091 chore: update kotlin version 2024-05-27 12:33:40 +02:00
bdee969409 chore: update spring dependency-management plugin 2024-05-27 12:33:15 +02:00
4c39920afd build: use compilerOptions instead of kotlinOptions
Config kotlinOptions will be deprecated in Kotlin >= 2.0
2024-05-27 12:31:05 +02:00
5e836c48b0 docs: add information about other credentials for MTBfile endpoint 2024-05-27 12:23:50 +02:00
fb5a3c062c feat: allow access to MTBFile endpoint for non-token users 2024-05-27 12:19:24 +02:00
8fc0609aa4 feat: use RequestId type 2024-05-27 11:23:20 +02:00
a846a8765a chore: update spring boot dependencies 2024-05-27 09:08:08 +02:00
8645becd82 chore: update bwhc-dto-java dependency 2024-05-16 17:59:22 +02:00
011511d5ef feat: use details as alias for message in data quality report 2024-05-16 09:08:21 +02:00
e9839c2731 fix: add missing 'fatal' severity 2024-05-15 17:27:04 +02:00
86bee9e2cf feat: show info if no requests present 2024-05-08 13:08:15 +02:00
f419acb924 test: mock UserRoleService only in nested test class 2024-05-08 10:28:11 +02:00
52171e8ebe test: add test for config SSE endpoint 2024-05-08 09:18:29 +02:00
a2124ba83d test: add test for SSE endpoint 2024-05-07 18:45:07 +02:00
a046203339 test: test statistics json response 2024-05-07 16:57:51 +02:00
b40d41ce8c test: add initial tests for StatisticsRestController 2024-05-07 15:57:35 +02:00
57de96771c test: add test for StatisticsController 2024-05-07 15:48:39 +02:00
3bc148f7ea refactor: move classes into package 'security' 2024-05-07 08:58:00 +02:00
8e6b1ec799 refactor: use whenever() instead of when in tests 2024-05-06 19:11:54 +02:00
8e5f5c73ec test: add tests for UserRoleService 2024-05-06 18:54:28 +02:00
d4f984b138 test: add tests for HomeController 2024-05-06 18:01:08 +02:00
24ebbf3b50 test: add tests to and reorganize ConfigControllerTest 2024-05-06 17:14:22 +02:00
9c6bd64a7e test: ensure correct view is rendered 2024-05-06 13:33:36 +02:00
6567aa803c test: add htmlunit based test for LoginController 2024-05-06 13:26:29 +02:00
e874350712 test: add tests for user role requests 2024-05-06 11:53:54 +02:00
94d7b4c4f0 test: add tests for token requests 2024-05-06 11:42:11 +02:00
107429fda7 fix: handle null values in MtbFile
This should not occur but if, it should not result in NPE except for

* Patient
* Consent
* Episode
2024-05-06 10:43:03 +02:00
26b2f65e67 chore: remove println 2024-05-06 10:03:36 +02:00
e863269a42 build: add jacoco code coverage 2024-05-06 09:44:19 +02:00
4ab95ef11f build: update gradle version 2024-05-03 16:44:41 +02:00
2244ef1b86 build: register task to run all tests at once 2024-05-03 16:37:16 +02:00
c3ddb387e2 test: add some basic arch unit tests 2024-05-02 12:30:12 +02:00
ae5d8341cc refactor: use RequestService in ResponseProcessor 2024-05-02 10:12:40 +02:00
40b2558943 refactor: use RequestService in controllers 2024-05-02 09:58:20 +02:00
9a6a0c6138 refactor: use Fingerprint type instead of plain String 2024-04-30 11:08:27 +02:00
5985327219 refactor: use type alias 2024-04-29 18:40:16 +02:00
06f9e8ace9 test: add initial RequestRepository test 2024-04-29 17:34:27 +02:00
365a651918 chore: update dependency to mockito-kotlin 2024-04-29 16:45:25 +02:00
5fcc24f915 refactor: add additional constructors 2024-04-29 10:11:25 +02:00
3bd7239812 chore: use java records 2024-04-26 12:35:37 +02:00
1672ad8640 chore: update webjars-locator dependency 2024-04-25 09:33:35 +02:00
710aeb1f18 docs: fix CVE number in dependency comment 2024-04-25 09:10:33 +02:00
06d11790b6 chore: update integration test dependency
This mitigates CVE-204-26308 and CVE-2024-25710
2024-04-24 12:06:08 +02:00
959f6889d4 refactor: extract custom SSL gPAS rest template creation 2024-04-24 08:56:31 +02:00
0f5a68660d refactor: move method content and call this method 2024-04-24 08:41:47 +02:00
b809a2da02 refactor: move custom rest template init to config class 2024-04-24 08:22:13 +02:00
effffcfc1a test: add tests with simulated gPAS responses 2024-04-23 15:53:22 +02:00
7b3151d227 fix: custom rest template generation after ssl context creation 2024-04-23 15:52:40 +02:00
26b415f336 docs: add deprecation warnings for config options 2024-04-23 15:02:14 +02:00
bda3c30a74 refactor: use default RestTemplate bean if no custom one required 2024-04-23 14:48:24 +02:00
8779600330 chore: change private method name to fix typo 2024-04-23 14:39:37 +02:00
159fb46009 chore: update GitHub workflow actions 2024-04-22 19:30:46 +02:00
eabbbfbb68 refactor: rename method for custom rest template configuration 2024-04-22 19:13:47 +02:00
4db38ef2f0 refactor: use more convenient way to set basic auth header 2024-04-22 19:11:19 +02:00
ed6d21e920 Merge branch 'v0.9.x' 2024-04-19 09:50:19 +02:00
550bee5ad3 chore: use snapshot version 2024-04-19 09:49:10 +02:00
bd7dccbd87 Merge pull request #61 from CCC-MF/issue_55
Anzeige der letzten Verbindungsstatusänderung
2024-03-26 10:06:27 +01:00
8ae958b8c4 feat: show information if no output is defined 2024-03-26 09:56:31 +01:00
0f144568e3 style: change tab content background to white 2024-03-25 17:27:08 +01:00
08540e3bd7 feat: add timestamp of last connection check change 2024-03-25 17:24:33 +01:00
43af1aa103 feat: add timestamp to connection check display 2024-03-25 17:09:27 +01:00
103 changed files with 13292 additions and 2032 deletions

2
.gitignore vendored
View File

@@ -39,3 +39,5 @@ out/
.vscode/
/dev/gpas*
/deploy/.env
/dev/gICS*
/dev/gPAS*

344
README.md
View File

@@ -1,33 +1,81 @@
# ETL-Processor for bwHC data [![Run Tests](https://github.com/CCC-MF/etl-processor/actions/workflows/test.yml/badge.svg)](https://github.com/CCC-MF/etl-processor/actions/workflows/test.yml)
# ETL-Processor für das MV gem. §64e und DNPM:DIP
[![Run Tests](https://github.com/pcvolkmer/etl-processor/actions/workflows/test.yml/badge.svg)](https://github.com/pcvolkmer/etl-processor/actions/workflows/test.yml)
Diese Anwendung versendet ein bwHC-MTB-File an das bwHC-Backend und pseudonymisiert die Patienten-ID.
Diese Anwendung pseudonymisiert/anonymisiert Daten im DNPM-Datenmodell 2.1 für das Modellvorhaben
Genomsequenzierung nach §64e unter Beachtung des Consents und sendet sie an DNPM:DIP.
## Einordnung innerhalb einer DNPM-ETL-Strecke
Diese Anwendung erlaubt das Entgegennehmen von HTTP/REST-Anfragen aus dem Onkostar-Plugin **[onkostar-plugin-dnpmexport](https://github.com/CCC-MF/onkostar-plugin-dnpmexport)**.
Diese Anwendung erlaubt das Entgegennehmen von HTTP/REST-Anfragen aus dem Onkostar-Plugin
**[mv64e-onkostar-plugin-export](https://github.com/pcvolkmer/mv64e-onkostar-plugin-export)**.
Der Inhalt einer Anfrage, wenn ein bwHC-MTBFile, wird pseudonymisiert und auf Duplikate geprüft.
Der Inhalt einer Anfrage, wenn ein MTB-File, wird pseudonymisiert und auf Duplikate geprüft.
Duplikate werden verworfen, Änderungen werden weitergeleitet.
Löschanfragen werden immer als Löschanfrage an das bwHC-backend weitergeleitet.
Löschanfragen werden immer als Löschanfrage an DNPM:DIP weitergeleitet.
Zudem ist eine minimalistische Weboberfläche integriert, die einen Einblick in den aktuellen Zustand der Anwendung gewährt.
![Modell DNPM-ETL-Strecke](docs/etl.png)
### 🔥 Wichtige Änderungen in Version 0.11
Ab Version 0.11 wird ausschließlich [DNPM:DIP](https://github.com/dnpm-dip) unterstützt.
Zudem wurde der Name des Pakets in **mv64e-etl-processor** geändert.
## Funktionsweise
### Duplikaterkennung
Die Erkennung von Duplikaten ist normalerweise immer aktiv, kann jedoch über den Konfigurationsparameter
Die Erkennung von Duplikaten ist normalerweise immer aktiv, kann jedoch über den
Konfigurationsparameter
`APP_DUPLICATION_DETECTION=false` deaktiviert werden.
### Modelvorhaben genomDE §64e
#### Vorgangsummern
Zusätzlich zur Patienten Identifier Pseudonymisierung müssen Vorgangsummern generiert werden, die
jede Übertragung eindeutig identifizieren aber gleichzeitig dem Patienten zugeordnet werden können.
Dies lässt sich durch weitere Pseudonyme abbilden, allerdings werden pro Originalwert mehrere
Pseudonyme benötigt.
Zu diesem Zweck muss in gPas eine **Multi-Pseudonym-Domäne** konfiguriert werden (siehe auch
*APP_PSEUDONYMIZE_GPAS_CCDN*).
**WICHTIG:** Deaktivierte Pseudonymisierung ist nur für Tests nutzbar. Vorgangsummern sind zufällig
und werden anschließend verworfen.
#### Test Betriebsbereitschaft
Um die voll Betriebsbereitschaft herzustellen, muss eine erfolgreiche Übertragung mit dem
Submission-Typ *Test* erfolgt sein. Über die Umgebungsvariable wird dieser Übertragungsmodus
aktiviert. Alle Datensätze mit erteilter Teilnahme am Modelvorhaben werden mit der Test-Submission-Kennung
übertragen, unabhängig vom ursprünglichen Wert.
`APP_GENOM_DE_TEST_SUBMISSION` -> `true` | `false` (falls fehlt, wird `false` angenommen)
### Datenübermittlung über HTTP/REST
Anfragen werden, wenn nicht als Duplikat behandelt, nach der Pseudonymisierung direkt an das bwHC-Backend gesendet.
Anfragen werden, wenn nicht als Duplikat behandelt, nach der Pseudonymisierung direkt an DNPM:DIP
gesendet.
Ein HTTP-Request kann, angenommen die Installation erfolgte auf dem Host `dnpm.example.com` an
nachfolgende URLs gesendet werden:
| HTTP-Request | URL | Consent-Status im Datensatz | Bemerkung |
|--------------|-----------------------------------------|-----------------------------|---------------------------------------------------------------------------------|
| `POST` | `https://dnpm.example.com/mtb` | `ACTIVE` | Die Anwendung verarbeitet den eingehenden Datensatz |
| `POST` | `https://dnpm.example.com/mtb` | `REJECT` | Die Anwendung sendet einen Lösch-Request für die im Datensatz angegebene Pat-ID |
| `DELETE` | `https://dnpm.example.com/mtb/12345678` | - | Die Anwendung sendet einen Lösch-Request für Pat-ID `12345678` |
Anstelle des Pfads `/mtb` kann auch, wie in Version 0.9 und älter üblich, `/mtbfile` verwendet
werden.
### Datenübermittlung mit Apache Kafka
Anfragen werden, wenn nicht als Duplikat behandelt, nach der Pseudonymisierung an Apache Kafka übergeben.
Eine Antwort wird dabei ebenfalls mithilfe von Apache Kafka übermittelt und nach der Entgegennahme verarbeitet.
Anfragen werden, wenn nicht als Duplikat behandelt, nach der Pseudonymisierung an Apache Kafka
übergeben.
Eine Antwort wird dabei ebenfalls mithilfe von Apache Kafka übermittelt und nach der Entgegennahme
verarbeitet.
Siehe hierzu auch: https://github.com/CCC-MF/kafka-to-bwhc
@@ -41,37 +89,93 @@ Ist diese nicht gesetzt. wird intern eine Anonymisierung der Patienten-ID vorgen
* `APP_PSEUDONYMIZE_PREFIX`: Standortbezogenes Präfix - `UNKNOWN`, wenn nicht gesetzt
* `APP_PSEUDONYMIZE_GENERATOR`: `BUILDIN` oder `GPAS` - `BUILDIN`, wenn nicht gesetzt
**Hinweise**:
**Hinweis**
* Der alte Konfigurationsparameter `APP_PSEUDONYMIZER` mit den Werten `GPAS` oder `BUILDIN` sollte nicht mehr verwendet
werden.
* Die Pseudonymisierung erfolgt im ETL-Prozessor nur für die Patienten-ID.
Andere IDs werden mithilfe des standortbezogenen Präfixes (erneut) anonymisiert, um für den aktuellen Kontext nicht
vergleichbare IDs bereitzustellen.
Die Pseudonymisierung erfolgt im ETL-Prozessor nur für die Patienten-ID.
Andere IDs werden mithilfe des standortbezogenen Präfixes (erneut) anonymisiert, um für den
aktuellen Kontext nicht
vergleichbare IDs bereitzustellen.
#### Eingebaute Anonymisierung
Wurde keine oder die Verwendung der eingebauten Anonymisierung konfiguriert, so wird für die Patienten-ID der
entsprechende SHA-256-Hash gebildet und Base64-codiert - hier ohne endende "=" - zuzüglich des konfigurierten Präfixes
als Patienten-Pseudonym verwendet.
Wurde keine oder die Verwendung der eingebauten Anonymisierung konfiguriert, so wird für die
Patienten-ID der entsprechende SHA-256-Hash gebildet und Base64-codiert - hier ohne endende
"=" - zuzüglich des konfigurierten Präfixes als Patienten-Pseudonym verwendet.
#### Pseudonymisierung mit gPAS
Wurde die Verwendung von gPAS konfiguriert, so sind weitere Angaben zu konfigurieren.
Wurde die Verwendung von gPAS konfiguriert, so sind weitere Angaben zu konfigurieren.
* `APP_PSEUDONYMIZE_GPAS_URI`: URI der gPAS-Instanz inklusive Endpoint (z.B. `http://localhost:8080/ttp-fhir/fhir/gpas/$$pseudonymizeAllowCreate`)
* `APP_PSEUDONYMIZE_GPAS_TARGET`: gPas Domänenname
Ab Version 2025.1 (Multi-Pseudonym Support)
* `APP_PSEUDONYMIZE_GPAS_URI`: URI der gPAS-Instanz REST API (e.g. http://127.0.0.1:9990/ttp-fhir/fhir/gpas)
* `APP_PSEUDONYMIZE_GPAS_USERNAME`: gPas Basic-Auth Benutzername
* `APP_PSEUDONYMIZE_GPAS_PASSWORD`: gPas Basic-Auth Passwort
* ~~`APP_PSEUDONYMIZE_GPAS_SSLCALOCATION`~~: **Veraltet** - Root Zertifikat für gPas, falls es dediziert hinzugefügt werden muss.
* `APP_PSEUDONYMIZE_GPAS_PID_DOMAIN`: gPas Domänenname für Patienten ID
* `APP_PSEUDONYMIZE_GPAS_GENOM_DE_TAN_DOMAIN`: gPas Multi-Pseudonym-Domäne für genomDE Vorgangsnummern (
Clinical data node)
Der Konfigurationsparameter `APP_PSEUDONYMIZE_GPAS_SSLCALOCATION` sollte nicht mehr verwendet werden und wird in einer kommenden Version entfernt.
Stattdessen sollte das Root Zertifikat wie unter [_Integration eines eigenen Root CA Zertifikats_](#integration-eines-eigenen-root-ca-zertifikats) beschrieben eingebunden werden.
### (Externe) Consent-Services
Consent-Services können konfiguriert werden.
* `APP_CONSENT_SERVICE`: Zu verwendender (externer) Consent-Service:
* `NONE`: Verwende Consent-Angaben im MTB-File v1 und ändere diese nicht. Für MTB-File v2 wird
die Prüfung übersprungen.
* `GICS`: Verwende gICS der Greiswalder Tools (siehe unten).
#### Einwilligung gICS
Ab gIcs Version 2.13.0 kann im ETL-Processor
per [REST-Schnittstelle](https://simplifier.net/guide/ttp-fhir-gateway-ig/ImplementationGuide-markdown-Einwilligungsmanagement-Operations-isConsented?version=current)
der Einwilligungsstatus abgefragt werden.
Vor der MTB-Übertragung kann der zum Sendezeitpunkt verfügbarer Einwilligungsstatus über Endpunkt
*isConsented* (MTB-File v1) und *currentPolicyStatesForPerson* (MTB-File v2) abgefragt werden.
Falls Anbindung an gICS aktiviert wurde, wird der Einwilligungsstatus der MTB Datei ignoriert.
Stattdessen werden vorhandene Einwilligungen abgefragt und in die MTB Datei eingebettet.
Es werden zwei Einwilligungsdomänen unterstützt, eine für Broad Consent und als zweites GenomDE
Modelvorhaben §64e.
##### Hinweise
1. Die aktuelle Impl. nimmt an, dass die hinterlegten Domänen der Einwilligungen ausschließlich für
die genannten Art von Einwilligungen genutzt werden. Es finde keine weitere Filterung statt. Wir
fragen pro Domäne die Schnittstelle `CurrentPolicyStatesForPerson` - siehe
auch [IG TTP-FHIR Gateway
](https://www.ths-greifswald.de/wp-content/uploads/tools/fhirgw/ig/2024-3-0/ImplementationGuide-markdown-Einwilligungsmanagement-Operations-currentPolicyStatesForPerson.html)
ab.
2. Die Einwilligung wird für den Patienten-Identifier der MTB abgerufen und anschließend durch das
DNPM Pseudonym ersetzt.
3. Abfragen von Einwilligungen über gesonderte Pseudonyme anstatt des MTB-Identifiers fehlt in der
ersten Implementierung.
4. Bei Verarbeitung von MTB Version 1.x Inhalten ist eine positive Einwilligung für die
Weiterverarbeitung notwendig. Das Fehlen einer Einwilligung löst die Löschung des Patienten im
Brückenkopf aus.
##### Konfiguration
* `APP_CONSENT_SERVICE`: Muss Wert `GICS` gesetzt sein um die Abfragen zu aktivieren. Der Wert
`NONE` deaktiviert die Abfrage in gICS.
* `APP_CONSENT_GICS_URI`: URI der gICS-Instanz (z.B. `http://localhost:8090/ttp-fhir/fhir/gics`)
* `APP_CONSENT_GICS_USERNAME`: gIcs Basic-Auth Benutzername
* `APP_CONSENT_GICS_PASSWORD`: gIcs Basic-Auth Passwort
* `APP_CONSENT_GICS_PERSONIDENTIFIERSYSTEM`: Derzeit wird nur die PID unterstützt. wenn leer wird
`https://ths-greifswald.de/fhir/gics/identifiers/Patienten-ID` angenommen
* `APP_CONSENT_GICS_BROADCONSENTDOMAINNAME`: Domäne in der gIcs Broad Consent Einwilligungen
verwaltet. Falls Wert leer, wird `MII` angenommen.
* `APP_CONSENT_GICS_GNOMDECONSENTDOMAINNAME`: Domäne in der gIcs GenomDE Modelvorhaben §64e
Einwilligungen verwaltet. Falls Wert leer, wird `GenomDE_MV` angenommen.
* `APP_CONSENT_GICS_POLICYCODE`: Die entscheidende Objekt-ID der zu prüfenden Einwilligung-Regel.
Falls leer wird `2.16.840.1.113883.3.1937.777.24.5.3.6` angenommen.
* `APP_CONSENT_GICS_POLICYSYSTEM`: Das System der Einwilligung-Regel der Objekt-IDs. Falls leer wird
`urn:oid:2.16.840.1.113883.3.1937.777.24.5.3` angenommen.
### Anmeldung mit einem Passwort
Ein initialer Administrator-Account kann optional konfiguriert werden und sorgt dafür, dass bestimmte Bereiche nur nach
einem erfolgreichen Login erreichbar sind.
Ein initialer Administrator-Account kann optional konfiguriert werden und sorgt dafür, dass
bestimmte Bereiche nur nach einem erfolgreichen Login erreichbar sind.
* `APP_SECURITY_ADMIN_USER`: Muss angegeben werden zur Aktivierung der Zugriffsbeschränkung.
* `APP_SECURITY_ADMIN_PASSWORD`: Das Passwort für den Administrator (Empfohlen).
@@ -84,27 +188,34 @@ Hier Beispiele für das Beispielpasswort `very-secret`:
* `{bcrypt}$2y$05$CCkfsMr/wbTleMyjVIK8g.Aa3RCvrvoLXVAsL.f6KeouS88vXD9b6`
* `{sha256}9a34717f0646b5e9cfcba70055de62edb026ff4f68671ba3db96aa29297d2df5f1a037d58c745657`
Wird kein Administrator-Passwort angegeben, wird ein zufälliger Wert generiert und beim Start der Anwendung in den Logs
Wird kein Administrator-Passwort angegeben, wird ein zufälliger Wert generiert und beim Start der
Anwendung in den Logs
angezeigt.
#### Weitere (nicht administrative) Nutzer mit OpenID Connect
Die folgenden Konfigurationsparameter werden benötigt, um die Authentifizierung weiterer Benutzer an einen OIDC-Provider
Die folgenden Konfigurationsparameter werden benötigt, um die Authentifizierung weiterer Benutzer an
einen OIDC-Provider
zu delegieren.
Ein Admin-Benutzer muss dabei konfiguriert sein.
* `APP_SECURITY_ENABLE_OIDC`: Aktiviert die Nutzung von OpenID Connect. Damit sind weitere Parameter erforderlich
* `SPRING_SECURITY_OAUTH2_CLIENT_REGISTRATION_CUSTOM_CLIENT_NAME`: Name. Wird beim zusätzlichen Loginbutton angezeigt.
* `APP_SECURITY_ENABLE_OIDC`: Aktiviert die Nutzung von OpenID Connect. Damit sind weitere Parameter
erforderlich
* `SPRING_SECURITY_OAUTH2_CLIENT_REGISTRATION_CUSTOM_CLIENT_NAME`: Name. Wird beim zusätzlichen
Loginbutton angezeigt.
* `SPRING_SECURITY_OAUTH2_CLIENT_REGISTRATION_CUSTOM_CLIENT_ID`: Client-ID
* `SPRING_SECURITY_OAUTH2_CLIENT_REGISTRATION_CUSTOM_CLIENT_SECRET`: Client-Secret
* `SPRING_SECURITY_OAUTH2_CLIENT_REGISTRATION_CUSTOM_CLIENT_SCOPE[0]`: Hier sollte immer `openid` angegeben werden.
* `SPRING_SECURITY_OAUTH2_CLIENT_REGISTRATION_CUSTOM_CLIENT_SCOPE[0]`: Hier sollte immer `openid`
angegeben werden.
* `SPRING_SECURITY_OAUTH2_CLIENT_PROVIDER_CUSTOM_ISSUER_URI`: Die URI des Providers,
z.B. `https://auth.example.com/realm/example`
* `SPRING_SECURITY_OAUTH2_CLIENT_PROVIDER_CUSTOM_USER_NAME_ATTRIBUTE`: Name des Attributes, welches den Benutzernamen
* `SPRING_SECURITY_OAUTH2_CLIENT_PROVIDER_CUSTOM_USER_NAME_ATTRIBUTE`: Name des Attributes, welches
den Benutzernamen
enthält.
Oft verwendet: `preferred_username`
Ist die Nutzung von OpenID Connect konfiguriert, erscheint ein zusätzlicher Login-Button zur Nutzung mit OpenID Connect
Ist die Nutzung von OpenID Connect konfiguriert, erscheint ein zusätzlicher Login-Button zur Nutzung
mit OpenID Connect
und dem konfigurierten `CLIENT_NAME`.
![Login mit OpenID Connect](docs/login.png)
@@ -115,60 +226,75 @@ zu finden.
#### Rollenbasierte Berechtigungen
Wird OpenID Connect verwendet, gibt es eine rollenbasierte Berechtigungszuweisung.
Wird OpenID Connect verwendet, gibt es eine rollenbasierte Berechtigungszuweisung.
Die Standardrolle für neue OIDC-Benutzer kann mit der Option `APP_SECURITY_DEFAULT_USER_ROLE` festgelegt werden.
Die Standardrolle für neue OIDC-Benutzer kann mit der Option `APP_SECURITY_DEFAULT_USER_ROLE`
festgelegt werden.
Mögliche Werte sind `user` oder `guest`. Standardwert ist `user`.
Benutzer mit der Rolle "Gast" sehen nur die Inhalte, die auch nicht angemeldete Benutzer sehen.
Hierdurch ist es möglich, einzelne Benutzer einzuschränken oder durch Änderung der Standardrolle auf `guest` nur
Hierdurch ist es möglich, einzelne Benutzer einzuschränken oder durch Änderung der Standardrolle auf
`guest` nur
einzelne Benutzer als vollwertige Nutzer zuzulassen.
![Rollenverwaltung](docs/userroles.png)
Benutzer werden nach dem Entfernen oder der Änderung der vergebenen Rolle automatisch abgemeldet und müssen sich neu anmelden.
Benutzer werden nach dem Entfernen oder der Änderung der vergebenen Rolle automatisch abgemeldet und
müssen sich neu anmelden.
Sie bekommen dabei wieder die Standardrolle zugewiesen.
#### Auswirkungen auf den dargestellten Inhalt
Nur Administratoren haben Zugriff auf den Konfigurationsbereich, nur angemeldete Benutzer können die anonymisierte oder
pseudonymisierte Patienten-ID sowie den Qualitätsbericht des bwHC-Backends einsehen.
Nur Administratoren haben Zugriff auf den Konfigurationsbereich, nur angemeldete Benutzer können die
anonymisierte oder
pseudonymisierte Patienten-ID sowie den Qualitätsbericht von DNPM:DIP einsehen.
Wurde kein Administrator-Account konfiguriert, sind diese Inhalte generell nicht verfügbar.
### Tokenbasierte Authentifizierung für MTBFile-Endpunkt
Die Anwendung unterstützt das Erstellen und Nutzen einer tokenbasierten Authentifizierung für den MTB-File-Endpunkt.
Die Anwendung unterstützt das Erstellen und Nutzen einer tokenbasierten Authentifizierung für den
MTB-File-Endpunkt.
Dies kann mit der Umgebungsvariable `APP_SECURITY_ENABLE_TOKENS` aktiviert (`true` oder `false`) werden
Dies kann mit der Umgebungsvariable `APP_SECURITY_ENABLE_TOKENS` aktiviert (`true` oder `false`)
werden
und ist als Standardeinstellung nicht aktiv.
Ist diese Einstellung aktiviert worden, ist es Administratoren möglich, Zugriffstokens für Onkostar zu erstellen, die
Ist diese Einstellung aktiviert worden, ist es Administratoren möglich, Zugriffstokens für Onkostar
zu erstellen, die
zur Nutzung des MTB-File-Endpunkts eine HTTP-Basic-Authentifizierung voraussetzen.
![Tokenverwaltung](docs/tokens.png)
In diesem Fall können den Endpunkt für das Onkostar-Plugin **[onkostar-plugin-dnpmexport](https://github.com/CCC-MF/onkostar-plugin-dnpmexport)** wie folgt konfigurieren:
In diesem Fall kann der Endpunkt für das Onkostar-Plugin *
*[onkostar-plugin-dnpmexport](https://github.com/CCC-MF/onkostar-plugin-dnpmexport)** wie folgt
konfiguriert werden:
```
https://testonkostar:MTg1NTL...NGU4@etl.example.com/mtbfile
```
Ist die Verwendung von Tokens aktiv, werden Anfragen ohne die Angabe der Token-Information abgelehnt.
Ist die Verwendung von Tokens aktiv, werden Anfragen ohne die Angabe der Token-Information
abgelehnt.
Alternativ kann eine Authentifizierung über Benutzername/Passwort oder OIDC erfolgen.
### Transformation von Werten
In Onkostar kann es vorkommen, dass ein Wert eines Merkmalskatalogs an einem Standort angepasst wurde und dadurch nicht dem Wert entspricht,
der vom bwHC-Backend akzeptiert wird.
In Onkostar kann es vorkommen, dass ein Wert eines Merkmalskatalogs an einem Standort angepasst
wurde und dadurch nicht dem Wert entspricht,
der von DNPM:DIP akzeptiert wird.
Diese Anwendung bietet daher die Möglichkeit, eine Transformation vorzunehmen. Hierzu muss der "Pfad" innerhalb des JSON-MTB-Files angegeben werden und
welcher Wert wie ersetzt werden soll.
Diese Anwendung bietet daher die Möglichkeit, eine Transformation vorzunehmen. Hierzu muss der "Pfad"
innerhalb des JSON-MTB-Files angegeben werden und welcher Wert wie ersetzt werden soll.
Hier ein Beispiel für die erste (Index 0 - weitere dann mit 1,2, ...) Transformationsregel:
* `APP_TRANSFORMATIONS_0_PATH`: Pfad zum Wert in der JSON-MTB-Datei. Beispiel: `diagnoses[*].icd10.version` für **alle** Diagnosen
* `APP_TRANSFORMATIONS_0_FROM`: Angabe des Werts, der ersetzt werden soll. Andere Werte bleiben dabei unverändert.
* `APP_TRANSFORMATIONS_0_PATH`: Pfad zum Wert in der JSON-MTB-Datei. Beispiel:
`diagnoses[*].icd10.version` für **alle** Diagnosen
* `APP_TRANSFORMATIONS_0_FROM`: Angabe des Werts, der ersetzt werden soll. Andere Werte bleiben
dabei unverändert.
* `APP_TRANSFORMATIONS_0_TO`: Angabe des neuen Werts.
### Mögliche Endpunkte zur Datenübermittlung
@@ -180,51 +306,61 @@ Werden sowohl REST als auch Kafka-Endpunkt konfiguriert, wird nur der REST-Endpu
#### REST
Folgende Umgebungsvariablen müssen gesetzt sein, damit ein bwHC-MTB-File an das bwHC-Backend gesendet wird:
Folgende Umgebungsvariablen müssen gesetzt sein, damit ein MTB-File an DNPM:DIP gesendet wird:
* `APP_REST_URI`: URI der zu benutzenden API der bwHC-Backend-Instanz. z.B.: `http://localhost:9000/bwhc/etl/api`
* `APP_REST_URI`: URI der zu benutzenden API der Backend-Instanz. Zum Beispiel `http://localhost:9000/api`
* `APP_REST_USERNAME`: Basic-Auth-Benutzername für den REST-Endpunkt
* `APP_REST_PASSWORD`: Basic-Auth-Passwort für den REST-Endpunkt
#### Kafka-Topics
Folgende Umgebungsvariablen müssen gesetzt sein, damit ein bwHC-MTB-File an ein Kafka-Topic übermittelt wird:
Folgende Umgebungsvariablen müssen gesetzt sein, damit ein MTB-File an ein Kafka-Topic
übermittelt wird:
* `APP_KAFKA_OUTPUT_TOPIC`: Zu verwendendes Topic zum Versenden von Anfragen.
Ersetzt in einer kommenden Version `APP_KAFKA_TOPIC`.
* `APP_KAFKA_OUTPUT_RESPONSE_TOPIC`: Topic mit Antworten über den Erfolg des Versendens. Standardwert: `APP_KAFKA_TOPIC` mit Anhang "_response".
Ersetzt in einer kommenden Version `APP_KAFKA_RESPONSE_TOPIC`.
* `APP_KAFKA_GROUP_ID`: Kafka GroupID des Consumers. Standardwert: `APP_KAFKA_TOPIC` mit Anhang "_group".
* `APP_KAFKA_OUTPUT_TOPIC`: Zu verwendendes Topic zum Versenden von Anfragen.
* `APP_KAFKA_OUTPUT_RESPONSE_TOPIC`: Topic mit Antworten über den Erfolg des Versendens.
Standardwert: `APP_KAFKA_TOPIC` mit Anhang "_response".
* `APP_KAFKA_GROUP_ID`: Kafka GroupID des Consumers. Standardwert: `APP_KAFKA_TOPIC` mit Anhang "_
group".
* `APP_KAFKA_SERVERS`: Zu verwendende Kafka-Bootstrap-Server als kommagetrennte Liste
Wird keine Rückantwort über Apache Kafka empfangen und es gibt keine weitere Möglichkeit den Status festzustellen, verbleibt der Status auf `UNKNOWN`.
Wird keine Rückantwort über Apache Kafka empfangen und es gibt keine weitere Möglichkeit den Status
festzustellen, verbleibt der Status auf `UNKNOWN`.
Weitere Einstellungen können über die Parameter von Spring Kafka konfiguriert werden.
Lässt sich keine Verbindung zu dem bwHC-Backend aufbauen, wird eine Rückantwort mit Status-Code `900` erwartet, welchen es
Lässt sich keine Verbindung zu dem Backend aufbauen, wird eine Rückantwort mit Status-Code `900`
erwartet, welchen es
für HTTP nicht gibt.
Wird die Umgebungsvariable `APP_KAFKA_INPUT_TOPIC` gesetzt, kann eine Nachricht auch über dieses Kafka-Topic an den ETL-Prozessor übermittelt werden.
Wird die Umgebungsvariable `APP_KAFKA_INPUT_TOPIC` gesetzt, kann eine Nachricht auch über dieses
Kafka-Topic an den ETL-Prozessor übermittelt werden.
##### Retention Time
Generell werden in Apache Kafka alle Records entsprechend der Konfiguration vorgehalten.
So wird ohne spezielle Konfiguration ein Record für 7 Tage in Apache Kafka gespeichert.
Es sind innerhalb dieses Zeitraums auch alte Informationen weiterhin enthalten, wenn der Consent später abgelehnt wurde.
Es sind innerhalb dieses Zeitraums auch alte Informationen weiterhin enthalten, wenn der Consent
später abgelehnt wurde.
Durch eine entsprechende Konfiguration des Topics kann dies verhindert werden.
Beispiel - auszuführen innerhalb des Kafka-Containers: Löschen alter Records nach einem Tag
```
kafka-configs.sh --bootstrap-server localhost:9092 --alter --topic test --add-config retention.ms=86400000
```
##### Key based Retention
Möchten Sie hingegen immer nur die letzte Meldung für einen Patienten und eine Erkrankung in Apache Kafka vorhalten,
Möchten Sie hingegen immer nur die letzte Meldung für einen Patienten und eine Erkrankung in Apache
Kafka vorhalten,
so ist die nachfolgend genannte Konfiguration der Kafka-Topics hilfreich.
* `retention.ms`: Möglichst kurze Zeit in der alte Records noch erhalten bleiben, z.B. 10 Sekunden 10000
* `cleanup.policy`: Löschen alter Records und Beibehalten des letzten Records zu einem Key [delete,compact]
* `retention.ms`: Möglichst kurze Zeit in der alte Records noch erhalten bleiben, z.B. 10 Sekunden
10000
* `cleanup.policy`: Löschen alter Records und Beibehalten des letzten Records zu einem
Key [delete,compact]
Beispiele für ein Topic `test`, hier bitte an die verwendeten Topics anpassen.
@@ -233,23 +369,41 @@ kafka-configs.sh --bootstrap-server localhost:9092 --alter --topic test --add-co
kafka-configs.sh --bootstrap-server localhost:9092 --alter --topic test --add-config cleanup.policy=[delete,compact]
```
Da als Key eines Records die (pseudonymisierte) Patienten-ID verwendet wird, stehen mit obiger Konfiguration
der Kafka-Topics nach 10 Sekunden nur noch der jeweils letzte Eintrag für den entsprechenden Key zur Verfügung.
Da als Key eines Records die (pseudonymisierte) Patienten-ID verwendet wird, stehen mit obiger
Konfiguration
der Kafka-Topics nach 10 Sekunden nur noch der jeweils letzte Eintrag für den entsprechenden Key zur
Verfügung.
Da der Key sowohl für die Records in Richtung bwHC-Backend für die Rückantwort identisch aufgebaut ist, lassen sich so
auch im Falle eines Consent-Widerspruchs die enthaltenen Daten als auch die Offenlegung durch Verifikationsdaten in der
Da der Key sowohl für die Records in Richtung DNPM:DIP, als auch für die Rückantwort identisch
aufgebaut ist, lassen sich so
auch im Falle eines Consent-Widerspruchs die enthaltenen Daten als auch die Offenlegung durch
Verifikationsdaten in der
Antwort effektiv verhindern, da diese nach 10 Sekunden gelöscht werden.
Es steht dann nur noch die jeweils letzten Information zur Verfügung, dass für einen Patienten/eine Erkrankung
Es steht dann nur noch die jeweils letzten Information zur Verfügung, dass für einen Patienten/eine
Erkrankung
ein Consent-Widerspruch erfolgte.
Dieses Vorgehen empfiehlt sich, wenn Sie gespeicherte Records nachgelagert für andere Auswertungen verwenden möchten.
Dieses Vorgehen empfiehlt sich, wenn Sie gespeicherte Records nachgelagert für andere Auswertungen
verwenden möchten.
### Antworten und Statusauswertung
Seit Version 0.10 wird die Issue-Liste der Antwort verwendet und die darion enthaltene höchste
Severity-Stufe als Ergebnis verwendet.
| Höchste Severity | Status |
|------------------|-----------|
| `info` | `SUCCESS` |
| `warning` | `WARNING` |
| `error`, `fatal` | `ERROR` |
## Docker-Images
Diese Anwendung ist auch als Docker-Image verfügbar: https://github.com/CCC-MF/etl-processor/pkgs/container/etl-processor
Diese Anwendung ist auch als Docker-Image
verfügbar: https://github.com/pcvolkmer/etl-processor/pkgs/container/etl-processor
### Images lokal bauen
### Images lokal bauen
```bash
./gradlew bootBuildImage
@@ -257,20 +411,25 @@ Diese Anwendung ist auch als Docker-Image verfügbar: https://github.com/CCC-MF/
### Integration eines eigenen Root CA Zertifikats
Wird eine eigene Root CA verwendet, die nicht offiziell signiert ist, wird es zu Problemen beim SSL-Handshake kommen, wenn z.B. gPAS zur Generierung von Pseudonymen verwendet wird.
Wird eine eigene Root CA verwendet, die nicht offiziell signiert ist, wird es zu Problemen beim
SSL-Handshake kommen, wenn z.B. gPAS zur Generierung von Pseudonymen verwendet wird.
Hier bietet es sich an, das Root CA Zertifikat in das Image zu integrieren.
#### Integration beim Bauen des Images
Hier muss die Zeile `"BP_EMBED_CERTS" to "true"` in der Datei `build.gradle.kts` verwendet werden und darf nicht als Kommentar verwendet werden.
Hier muss die Zeile `"BP_EMBED_CERTS" to "true"` in der Datei `build.gradle.kts` verwendet werden
und darf nicht als Kommentar verwendet werden.
Die PEM-Datei mit dem/den Root CA Zertifikat(en) muss dabei im vorbereiteten Verzeichnis [`bindings/ca-certificates`](bindings/ca-certificates) enthalten sein.
Die PEM-Datei mit dem/den Root CA Zertifikat(en) muss dabei im vorbereiteten Verzeichnis [
`bindings/ca-certificates`](bindings/ca-certificates) enthalten sein.
#### Integration zur Laufzeit
Hier muss die Umgebungsvariable `SERVICE_BINDING_ROOT` z.B. auf den Wert `/bindings` gesetzt sein.
Zudem muss ein Verzeichnis `bindings/ca-certificates` - analog zum Verzeichnis [`bindings/ca-certificates`](bindings/ca-certificates) mit einer PEM-Datei als Docker-Volume eingebunden werden.
Zudem muss ein Verzeichnis `bindings/ca-certificates` - analog zum Verzeichnis [
`bindings/ca-certificates`](bindings/ca-certificates) mit einer PEM-Datei als Docker-Volume
eingebunden werden.
Beispiel für Docker-Compose:
@@ -285,12 +444,14 @@ Beispiel für Docker-Compose:
```
## Deployment
*Ausführen als Docker Container:*
```bash
cd ./deploy
cp env-sample.env .env
```
Wenn gewünscht, Änderungen in der `.env` vornehmen.
```bash
@@ -299,15 +460,19 @@ docker compose up -d
### Einfaches Beispiel für ein eigenes Docker-Compose-File
Die Datei [`docs/docker-compose.yml`](docs/docker-compose.yml) zeigt eine einfache Konfiguration für REST-Requests basierend
Die Datei [`docs/docker-compose.yml`](docs/docker-compose.yml) zeigt eine einfache Konfiguration für
REST-Requests basierend
auf Docker-Compose mit der gestartet werden kann.
### Betrieb hinter einem Reverse-Proxy
Die Anwendung verarbeitet `X-Forwarded`-HTTP-Header und kann daher auch hinter einem Reverse-Proxy betrieben werden.
Die Anwendung verarbeitet `X-Forwarded`-HTTP-Header und kann daher auch hinter einem Reverse-Proxy
betrieben werden.
Dabei werden, je nachdem welche Header durch den Reverse-Proxy gesendet werden auch Protokoll, Host oder auch Path-Präfix
automatisch erkannt und verwendet werden. Dadurch ist z.B. eine abweichende Angabe des Pfads problemlos möglich.
Dabei werden, je nachdem welche Header durch den Reverse-Proxy gesendet werden auch Protokoll, Host
oder auch Path-Präfix
automatisch erkannt und verwendet werden. Dadurch ist z.B. eine abweichende Angabe des Pfads
problemlos möglich.
#### Beispiel *Traefik* (mit Docker-Labels):
@@ -343,13 +508,17 @@ Das folgende Beispiel zeigt die Konfiguration einer _location_ in einer nginx-Ko
## Entwicklungssetup
Zum Starten einer lokalen Entwicklungs- und Testumgebung kann die beiliegende Datei `dev-compose.yml` verwendet werden.
Zum Starten einer lokalen Entwicklungs- und Testumgebung kann die beiliegende Datei
`dev-compose.yml` verwendet werden.
Diese kann zur Nutzung der Datenbanken **MariaDB** als auch **PostgreSQL** angepasst werden.
Zur Nutzung von Apache Kafka muss dazu ein Eintrag im hosts-File vorgenommen werden und der Hostname `kafka` auf die lokale
IP-Adresse verweisen. Ohne diese Einstellung ist eine Nutzung von Apache Kafka außerhalb der Docker-Umgebung nicht möglich.
Zur Nutzung von Apache Kafka muss dazu ein Eintrag im hosts-File vorgenommen werden und der Hostname
`kafka` auf die lokale
IP-Adresse verweisen. Ohne diese Einstellung ist eine Nutzung von Apache Kafka außerhalb der
Docker-Umgebung nicht möglich.
Beim Start der Anwendung mit dem Profil `dev` wird die in `dev-compose.yml` definierte Umgebung beim Start der
Beim Start der Anwendung mit dem Profil `dev` wird die in `dev-compose.yml` definierte Umgebung beim
Start der
Anwendung mit gestartet:
```
@@ -360,3 +529,6 @@ Die Datei `application-dev.yml` enthält hierzu die Konfiguration für das Profi
Beim Ausführen der Integrationstests wird eine Testdatenbank in einem Docker-Container gestartet.
Siehe hier auch die Klasse `AbstractTestcontainerTest` unter `src/integrationTest`.
Ein einfaches Entwickler-Setup inklusive DNPM:DIP ist mit Hilfe
von https://github.com/pcvolkmer/dnpmdip-devenv realisierbar.

View File

@@ -1,30 +1,35 @@
import org.gradle.api.tasks.testing.logging.TestLogEvent
import org.jetbrains.kotlin.gradle.dsl.JvmTarget
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
import org.springframework.boot.gradle.tasks.bundling.BootBuildImage
plugins {
war
id("org.springframework.boot") version "3.2.11"
id("io.spring.dependency-management") version "1.1.5"
kotlin("jvm") version "1.9.24"
kotlin("plugin.spring") version "1.9.24"
id("org.springframework.boot") version "3.5.3"
id("io.spring.dependency-management") version "1.1.7"
kotlin("jvm") version "1.9.25"
kotlin("plugin.spring") version "1.9.25"
jacoco
}
group = "dev.dnpm"
version = "0.9-SNAPSHOT"
version = "0.11.0-SNAPSHOT"
var versions = mapOf(
"bwhc-dto-java" to "0.3.0",
"hapi-fhir" to "6.10.5",
"httpclient5" to "5.2.3",
"mockito-kotlin" to "5.3.1",
"mtb-dto" to "0.1.0-SNAPSHOT",
"hapi-fhir" to "7.6.1",
"mockito-kotlin" to "5.4.0",
"archunit" to "1.3.0",
// Webjars
"webjars-locator" to "0.52",
"echarts" to "5.4.3",
"htmx.org" to "1.9.12"
)
java {
sourceCompatibility = JavaVersion.VERSION_21
toolchain {
languageVersion = JavaLanguageVersion.of(21)
}
}
sourceSets {
@@ -43,9 +48,18 @@ configurations {
compileOnly {
extendsFrom(configurations.annotationProcessor.get())
}
all {
resolutionStrategy {
cacheChangingModulesFor(5, "minutes")
}
}
}
repositories {
maven {
url = uri("https://git.dnpm.dev/api/packages/public-snapshots/maven")
}
maven {
url = uri("https://git.dnpm.dev/api/packages/public/maven")
}
@@ -62,37 +76,47 @@ dependencies {
implementation("org.thymeleaf.extras:thymeleaf-extras-springsecurity6")
implementation("com.fasterxml.jackson.module:jackson-module-kotlin")
implementation("org.springframework.kafka:spring-kafka")
implementation("org.flywaydb:flyway-database-postgresql")
implementation("org.flywaydb:flyway-mysql")
implementation("commons-codec:commons-codec")
implementation("io.projectreactor.kotlin:reactor-kotlin-extensions")
implementation("de.ukw.ccc:bwhc-dto-java:${versions["bwhc-dto-java"]}")
implementation("dev.pcvolkmer.mv64e:mtb-dto:${versions["mtb-dto"]}") { isChanging = true }
implementation("ca.uhn.hapi.fhir:hapi-fhir-base:${versions["hapi-fhir"]}")
implementation("ca.uhn.hapi.fhir:hapi-fhir-structures-r4:${versions["hapi-fhir"]}")
implementation("org.apache.httpcomponents.client5:httpclient5:${versions["httpclient5"]}")
implementation("org.apache.httpcomponents.client5:httpclient5")
implementation("com.jayway.jsonpath:json-path")
implementation("org.webjars:webjars-locator:0.52")
implementation("org.webjars:webjars-locator:${versions["webjars-locator"]}")
implementation("org.webjars.npm:echarts:${versions["echarts"]}")
implementation("org.webjars.npm:htmx.org:${versions["htmx.org"]}")
// Fix for CVE-2025-48924
implementation("org.apache.commons:commons-lang3:3.18.0")
runtimeOnly("org.mariadb.jdbc:mariadb-java-client")
runtimeOnly("org.postgresql:postgresql")
developmentOnly("org.springframework.boot:spring-boot-devtools")
developmentOnly("org.springframework.boot:spring-boot-docker-compose")
annotationProcessor("org.springframework.boot:spring-boot-configuration-processor")
providedRuntime("org.springframework.boot:spring-boot-starter-tomcat")
testImplementation("org.springframework.boot:spring-boot-starter-test")
testImplementation("org.springframework.security:spring-security-test")
testImplementation("io.projectreactor:reactor-test")
testImplementation("org.mockito.kotlin:mockito-kotlin:${versions["mockito-kotlin"]}")
integrationTestImplementation("org.testcontainers:junit-jupiter")
integrationTestImplementation("org.testcontainers:postgresql")
// Override dependency version from org.testcontainers:junit-jupiter - CVE-2024-26308, CVE-2024-25710
integrationTestImplementation("org.apache.commons:commons-compress:1.26.1")
integrationTestImplementation("com.tngtech.archunit:archunit:${versions["archunit"]}")
integrationTestImplementation("org.htmlunit:htmlunit")
integrationTestImplementation("org.springframework:spring-webflux")
}
tasks.withType<KotlinCompile> {
kotlinOptions {
freeCompilerArgs += "-Xjsr305=strict"
jvmTarget = "21"
compilerOptions {
freeCompilerArgs.add("-Xjsr305=strict")
jvmTarget.set(JvmTarget.JVM_21)
}
}
@@ -103,8 +127,9 @@ tasks.withType<Test> {
}
}
task<Test>("integrationTest") {
tasks.register<Test>("integrationTest") {
description = "Runs integration tests"
group = "verification"
testClassesDirs = sourceSets["integrationTest"].output.classesDirs
classpath = sourceSets["integrationTest"].runtimeClasspath
@@ -112,8 +137,24 @@ task<Test>("integrationTest") {
shouldRunAfter("test")
}
tasks.register("allTests") {
description = "Run all tests"
group = JavaBasePlugin.VERIFICATION_GROUP
dependsOn(tasks.withType<Test>())
}
tasks.jacocoTestReport {
dependsOn("allTests")
executionData(fileTree(project.rootDir.absolutePath).include("**/build/jacoco/*.exec"))
reports {
xml.required = true
}
}
tasks.named<BootBuildImage>("bootBuildImage") {
imageName.set("ghcr.io/ccc-mf/etl-processor")
imageName.set("ghcr.io/pcvolkmer/mv64e-etl-processor")
// Binding for CA Certs
bindings.set(listOf(
@@ -123,8 +164,8 @@ tasks.named<BootBuildImage>("bootBuildImage") {
environment.set(environment.get() + mapOf(
// Enable this line to embed CA Certs into image on build time
//"BP_EMBED_CERTS" to "true",
"BP_OCI_SOURCE" to "https://github.com/pcvolkmer/etl-processor",
"BP_OCI_SOURCE" to "https://github.com/pcvolkmer/mv64e-etl-processor",
"BP_OCI_LICENSES" to "AGPLv3",
"BP_OCI_DESCRIPTION" to "ETL Processor for bwHC MTB files"
"BP_OCI_DESCRIPTION" to "ETL Processor for MV § 64e and DNPM:DIP"
))
}

View File

@@ -18,6 +18,9 @@ services:
APP_KAFKA_GROUP_ID: ${DNPM_KAFKA_GROUP_ID}
APP_KAFKA_RESPONSE_TOPIC: ${DNPM_KAFKA_RESPONSE_TOPIC}
APP_REST_URI: ${DNPM_BWHC_REST_URI}
APP_REST_USERNAME: ${DNPM_BWHC_REST_USERNAME}
APP_REST_PASSWORD: ${DNPM_BWHC_REST_PASSWORD}
APP_REST_IS_BWHC: ${DNPM_BWHC_REST_IS_BWHC}
APP_SECURITY_ADMIN_USER: ${DNPM_ADMIN_USER}
APP_SECURITY_ADMIN_PASSWORD: ${DNPM_ADMIN_PASSWORD}
SPRING_DATASOURCE_URL: ${DNPM_DATASOURCE_URL}

View File

@@ -28,6 +28,9 @@ DNPM_DATASOURCE_URL=jdbc:mariadb://dnpm-monitor-db:3306/$DNPM_MARIADB_DB
## TARGET SYSTEMS CONFIG
# in case of direct access to bwhc enter endpoint url here
DNPM_BWHC_REST_URI=
DNPM_BWHC_REST_USERNAME=
DNPM_BWHC_REST_PASSWORD=
DNPM_BWHC_REST_IS_BWHC=false
# produce mtb files to this topic - values 'false' disabling kafka processing
DNPM_KAFKA_TOPIC=false

View File

@@ -1,24 +1,125 @@
services:
kafka:
image: bitnami/kafka
image: apache/kafka
hostname: kafka
ports:
- "9092:9092"
- "9094:9094"
environment:
ALLOW_PLAINTEXT_LISTENER: "yes"
KAFKA_CFG_NODE_ID: "0"
KAFKA_CFG_PROCESS_ROLES: "controller,broker"
KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:9094
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
KAFKA_CFG_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: true
KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 0@kafka:9093
KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_NODE_ID: "0"
KAFKA_KRAFT_CLUSTER_ID: "mv64e-etl-processor-dev"
KAFKA_CONTROLLER_QUORUM_VOTERS: 0@kafka:9093
KAFKA_PROCESS_ROLES: "controller,broker"
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: "1"
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: "1"
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: "1"
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: "0"
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
# Without SSL
KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:9094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
# Using SSL
# KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
# KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:9094
# KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,EXTERNAL:SSL,PLAINTEXT:PLAINTEXT
# KAFKA_SSL_KEYSTORE_TYPE: "PEM"
# KAFKA_SSL_KEYSTORE_CERTIFICATE_CHAIN: -----BEGIN CERTIFICATE-----
# MIIDCzCCAfOgAwIBAgIUaXNh4PahaKeLUaab2rUPSVESx28wDQYJKoZIhvcNAQEL
# BQAwFTETMBEGA1UEAwwKRXhhbXBsZSBDQTAeFw0yNTA4MjExODEyMTFaFw0zNTA4
# MTkxODEyMTFaMBUxEzARBgNVBAMMCkV4YW1wbGUgQ0EwggEiMA0GCSqGSIb3DQEB
# AQUAA4IBDwAwggEKAoIBAQCsqalqVOLFglVbX9oSHU91ebyL1kPyb/2N90UGQIcD
# UAjzKxxysId1Vdvtbbwgli6UgfPwlzFP2Wlw51h496yL4QU/9tNV956UJ1RoS/fG
# qBAEHctqavfMI27UQmIzw4pGMkGzEQxRMc6a9pdabBhbMMTJsjtmOv2YMYHj1HHK
# Dr7wTBTt2l0eRyCR0kZ8XGIMWhYowPa4EMpC7+4e5Nf/7LSJZWLLy9jXPpazsjkJ
# jEmDNlFfx2tZiq0Wz2Xj1pZSDLbcuIX4IHcLfMvagibfrCMX/h6+WuW42sWPRuBW
# wB6cHGlXs+K/gBBWxtD7sOTacO5hbHFsfaJOhSEIGoIpAgMBAAGjUzBRMB0GA1Ud
# DgQWBBT2S/C2++ECY+CSuN5KKql0umfbTDAfBgNVHSMEGDAWgBT2S/C2++ECY+CS
# uN5KKql0umfbTDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBj
# H4DdwqrOHg7sVsqiwDsZfTharpUDCYeG5XhrJQlnA9eKwyofTb929W/fjOwBdDtg
# 9THT/omR0lA8/UyHtezMT6nMsCn4HG2mXvx6ghgvA3jrFTEY7R80dHkboLMTV3u4
# RYgC9S3BJPcbJYpM0cXzkp2T0F4FxWZlfqefuedHuX3zcCxpgVD56qQb2a131TX7
# O3UDJfVg8a65IFtehndqILgLVrf7w6+pbmDAlCg5RKrt2USEYyZXYdyTryJbdtn4
# BCLp0avYtSYVUGwgH0oUCpkjQRwMg1003TTz8SNnmE7mAXHYljyYejnjL8vBHfch
# 8tTDVXQn08BT9H3jZTnF
# -----END CERTIFICATE-----
# -----BEGIN CERTIFICATE-----
# MIIC+TCCAeGgAwIBAgIUUoCwz8GS6xQ3mmI7RUUYSNPIOi4wDQYJKoZIhvcNAQEL
# BQAwFTETMBEGA1UEAwwKRXhhbXBsZSBDQTAeFw0yNTA4MjExODE0NDhaFw0zNTA4
# MTkxODE0NDhaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
# BQADggEPADCCAQoCggEBAL9PW99MhhBwdEmTHyZgfnhfTrxZPrNU6z1UdV8b82Lk
# 3p75o8eCKa9iOd7DDQlo75hQBhhX0+Xc3mucrstx5p8TYFMbypif8ojWh3LM++P8
# tz3ezQZlq86ycyKpm8dqlA03b227tFDfiYTev2eD2HN40BU7yDAYhhqd/QW8+MV2
# jkcRGv5cE21GZxWmPUpkVN+bNoBC8H90WmkST90LfeYF+wZnlsAJZH6AQzR1GnGD
# ICE5evMhC78hvRnpgeA310SyxssZEigkePL5lTZOBPY2IuzBqL05agyVTiVq4Xd6
# y3xOqXoxxOhZu06yd3nymorqeTgbF1fW8wQF0u3KsFECAwEAAaNCMEAwHQYDVR0O
# BBYEFHk9jMWRAAt2YsBSxUcOQVoWayoHMB8GA1UdIwQYMBaAFPZL8Lb74QJj4JK4
# 3koqqXS6Z9tMMA0GCSqGSIb3DQEBCwUAA4IBAQBqabAA9INONDaLHqs9i9YQHm/g
# AnB7xRl/RFbERKKCTSMZUYM8oEaaH0W2ENoPMc/7xOB/R8a7Rm62PTr6syxwhZrY
# 5NtGKJOD+rh90/5l83tulf93KqOJtGkiv6NBDvCNrITcA+UKRk/z4GcFi2YjWAl4
# wvY44lzTasMKSpjUQ5N0VNANcW3nVuEgPQ8rrr0NOK/5j4guPjsXDsixa47gqblA
# 5xGfBKeVmEXdPbzawZfP4hPIw7DpX2m8Y0erswF1ZxkIV73V3TDsFSLcqSKSzZr6
# mtj8COlV9Us7zqaJbV5eOl7GN1T9orZJwZmX1Z46gCkkSLYDP/dqtl2j9JgN
# -----END CERTIFICATE-----
# KAFKA_SSL_KEYSTORE_KEY: -----BEGIN PRIVATE KEY-----
# MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC/T1vfTIYQcHRJ
# kx8mYH54X068WT6zVOs9VHVfG/Ni5N6e+aPHgimvYjneww0JaO+YUAYYV9Pl3N5r
# nK7LceafE2BTG8qYn/KI1odyzPvj/Lc93s0GZavOsnMiqZvHapQNN29tu7RQ34mE
# 3r9ng9hzeNAVO8gwGIYanf0FvPjFdo5HERr+XBNtRmcVpj1KZFTfmzaAQvB/dFpp
# Ek/dC33mBfsGZ5bACWR+gEM0dRpxgyAhOXrzIQu/Ib0Z6YHgN9dEssbLGRIoJHjy
# +ZU2TgT2NiLswai9OWoMlU4lauF3est8Tql6McToWbtOsnd58pqK6nk4GxdX1vME
# BdLtyrBRAgMBAAECggEAC1wXfPlqxoQe65WAVoOJTvV90+JKvlRPCZu/wm+C8r7b
# Vz5Ekt6wQflHrWoQlpv0CivKSNzCONZ2IJazrGHti0mXwSeXzptEyApRDaiNVnrV
# mKdnrjcQThw7iPXgSaWS9/vwMmhgayLy5ABkBi4GhsjINlKP7wctw1vZP+N6NCNd
# Ql3taStvDKmG0SfJHF6/2o/XBpof3IJEL7ghbzyTTbWWaO34J1mJ8A+AmjGhj9GE
# Dp3XuOrO9W7MVd1nfZDtGBS8qf80AwROyodZZRma9vZuWJZ5aQFi2CnUEtU1T+Uv
# tW+F6tg2FOMr8M0Fb79wGIDwSF8u/QcTvwhEzZAfiQKBgQDioOofnE1oB1DOMnqZ
# SOFjs+vsirvS6G3lo27+HkE3TgvCHR4sk1305AiXtjmPu8iaUCo9qn18MtduY2RS
# CcKMOG/FxhmDyP5I29DhJRhvERIpJd0kcSDQOgtaoVPC1XzIlyTqte6nGX9kAnA/
# x/OOXrZ0hjhMNDcZzf2NasPYJQKBgQDYGqTobkVBk+eekNWklnTh41/649rUIgTu
# JStArtY2hgaEInYcGa2e7cEj7nIHA0iGy3EJ2yvwoUIyxtoXVcGohu2IrzlhS33T
# R4jA7nE2/yHZrEMEJovuSU0eMw7rgvEtL79Q0RToYnTY1EU6X/BBoFfiiEeNMHKz
# zjDOOQ6ZvQKBgGCWChIc0FSkwYiPtPZ9PCn89XCjk/cIPkYfiF9fT5Ydeh9pv4Fp
# 8SI8yXi3HgMnGhDCV65eagqztGMEky3voO2X4/MbQaaL0+wDWxuJbsdvNBk7XOt6
# F20HP+2JUiR4Ti1DVWV+0k5/LG7YJzTXp/KmZQZ2aan4mv8xbn2F4h/NAoGAI4ou
# OLN53FEQtHkpSYoc6tFUBZTXdi+qE+g09sxKGmlsROrN9c0bSpnbO6eJRTH7CYAH
# tRFAZrB+jI87ar8FvEuEYQhALYoWxVpsWR5drCfFT2EPHG2icavIbQEEoSLFuyKx
# Gf9oqtcWVFqEkBcbEg/mpDC5Y7TmCEAOsrubdRkCgYEAl7B+EzIdG0rabGoti09q
# QXfyiTjR7nQYkhpLxMCeNlCpQ8Y15XSa8bm1UIGYqj/ZBpeBNhrj64IHoub5Vd43
# tzbb8yNgoLUd16TU1VvyccCMGQVPIF8RkDsAtEawV2eoXbHAstN99xbC8jsLNZRQ
# fcfgTiQaXRJmlVx6jfbfZd4=
# -----END PRIVATE KEY-----
# #KAFKA_SSL_KEYSTORE_CREDENTIALS: example
# KAFKA_SSL_KEY_CREDENTIALS: example
# KAFKA_SSL_TRUSTSTORE_TYPE: "PEM"
# KAFKA_SSL_TRUSTSTORE_CERTIFICATES: -----BEGIN CERTIFICATE-----
# MIIDCzCCAfOgAwIBAgIUaXNh4PahaKeLUaab2rUPSVESx28wDQYJKoZIhvcNAQEL
# BQAwFTETMBEGA1UEAwwKRXhhbXBsZSBDQTAeFw0yNTA4MjExODEyMTFaFw0zNTA4
# MTkxODEyMTFaMBUxEzARBgNVBAMMCkV4YW1wbGUgQ0EwggEiMA0GCSqGSIb3DQEB
# AQUAA4IBDwAwggEKAoIBAQCsqalqVOLFglVbX9oSHU91ebyL1kPyb/2N90UGQIcD
# UAjzKxxysId1Vdvtbbwgli6UgfPwlzFP2Wlw51h496yL4QU/9tNV956UJ1RoS/fG
# qBAEHctqavfMI27UQmIzw4pGMkGzEQxRMc6a9pdabBhbMMTJsjtmOv2YMYHj1HHK
# Dr7wTBTt2l0eRyCR0kZ8XGIMWhYowPa4EMpC7+4e5Nf/7LSJZWLLy9jXPpazsjkJ
# jEmDNlFfx2tZiq0Wz2Xj1pZSDLbcuIX4IHcLfMvagibfrCMX/h6+WuW42sWPRuBW
# wB6cHGlXs+K/gBBWxtD7sOTacO5hbHFsfaJOhSEIGoIpAgMBAAGjUzBRMB0GA1Ud
# DgQWBBT2S/C2++ECY+CSuN5KKql0umfbTDAfBgNVHSMEGDAWgBT2S/C2++ECY+CS
# uN5KKql0umfbTDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBj
# H4DdwqrOHg7sVsqiwDsZfTharpUDCYeG5XhrJQlnA9eKwyofTb929W/fjOwBdDtg
# 9THT/omR0lA8/UyHtezMT6nMsCn4HG2mXvx6ghgvA3jrFTEY7R80dHkboLMTV3u4
# RYgC9S3BJPcbJYpM0cXzkp2T0F4FxWZlfqefuedHuX3zcCxpgVD56qQb2a131TX7
# O3UDJfVg8a65IFtehndqILgLVrf7w6+pbmDAlCg5RKrt2USEYyZXYdyTryJbdtn4
# BCLp0avYtSYVUGwgH0oUCpkjQRwMg1003TTz8SNnmE7mAXHYljyYejnjL8vBHfch
# 8tTDVXQn08BT9H3jZTnF
# -----END CERTIFICATE-----
# KAFKA_SSL_CLIENT_AUTH: none
###
## Use AKHQ as Kafka web frontend
akhq:
image: tchiotludo/akhq:0.21.0
image: tchiotludo/akhq:0.25.0
environment:
AKHQ_CONFIGURATION: |
akhq:
@@ -32,6 +133,8 @@ services:
ports:
- "8084:8080"
## For use with MariaDB
mariadb:
image: mariadb:10
ports:
@@ -42,6 +145,7 @@ services:
MARIADB_PASSWORD: dev
MARIADB_ROOT_PASSWORD: dev
## For use with Postgres
# postgres:
# image: postgres:alpine
# ports:
@@ -49,4 +153,4 @@ services:
# environment:
# POSTGRES_DB: dev
# POSTGRES_USER: dev
# POSTGRES_PASSWORD: dev
# POSTGRES_PASSWORD: dev

View File

@@ -2,31 +2,55 @@ version: '3.7'
services:
zoo1:
image: zookeeper:3.8.0
hostname: zoo1
zoo:
image: zookeeper:3.9.2
restart: unless-stopped
ports:
- "2181:2181"
environment:
ZOO_MY_ID: 1
ZOO_PORT: 2181
ZOO_SERVERS: server.1=zoo1:2888:3888;2181
ZOO_SERVERS: server.1=zoo:2888:3888;2181
kafka1:
image: confluentinc/cp-kafka:7.2.1
hostname: kafka1
kafka:
image: confluentinc/cp-kafka:7.6.1
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:19092,LISTENER_DOCKER_EXTERNAL://172.17.0.1:9093,LISTENER_EXTERNAL://127.0.0.1:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT,LISTENER_EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
KAFKA_ZOOKEEPER_CONNECT: zoo:2181
KAFKA_BROKER_ID: 1
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
KAFKA_LOG4J_LOGGERS: kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_MESSAGE_MAX_BYTES: 5242880
KAFKA_REPLICA_FETCH_MAX_BYTES: 5242880
KAFKA_COMPRESSION_TYPE: gzip
depends_on:
- zoo1
- zoo
healthcheck:
test: kafka-topics --bootstrap-server kafka:9092 --list
interval: 30s
timeout: 10s
retries: 3
akhq:
image: tchiotludo/akhq:0.25.0
environment:
AKHQ_CONFIGURATION: |
akhq:
ui-options:
topic.show-all-consumer-groups: true
topic-data.sort: NEWEST
connections:
docker-kafka-server:
properties:
bootstrap.servers: "kafka:19092"
ports:
- "9000:8080"
depends_on:
- kafka
kafka-rest-proxy:
image: confluentinc/cp-kafka-rest:7.2.1
@@ -40,8 +64,8 @@ services:
KAFKA_REST_HOST_NAME: kafka-rest-proxy
KAFKA_REST_BOOTSTRAP_SERVERS: PLAINTEXT://kafka1:19092
depends_on:
- zoo1
- kafka1
- zoo
- kafka
kafka-connect:
image: confluentinc/cp-kafka-connect:7.2.1
@@ -67,24 +91,6 @@ services:
#volumes:
# - ./connectors:/etc/kafka-connect/jars/
depends_on:
- zoo1
- kafka1
- zoo
- kafka
- kafka-rest-proxy
akhq:
image: tchiotludo/akhq:0.21.0
environment:
AKHQ_CONFIGURATION: |
akhq:
connections:
docker-kafka-server:
properties:
bootstrap.servers: "kafka1:19092"
connect:
- name: "kafka-connect"
url: "http://kafka-connect:8083"
ports:
- "8084:8080"
depends_on:
- kafka1
- kafka-connect

Binary file not shown.

Before

Width:  |  Height:  |  Size: 120 KiB

After

Width:  |  Height:  |  Size: 115 KiB

View File

@@ -1,6 +1,6 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-8.6-bin.zip
distributionUrl=https\://services.gradle.org/distributions/gradle-8.13-bin.zip
networkTimeout=10000
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists

View File

@@ -1 +1 @@
rootProject.name = "etl-processor"
rootProject.name = "mv64e-etl-processor"

View File

@@ -20,10 +20,11 @@
package dev.dnpm.etl.processor
import com.fasterxml.jackson.databind.ObjectMapper
import de.ukw.ccc.bwhc.dto.*
import dev.dnpm.etl.processor.monitoring.RequestRepository
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.output.DnpmV2MtbFileRequest
import dev.dnpm.etl.processor.output.MtbFileSender
import dev.pcvolkmer.mv64e.mtb.*
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Nested
@@ -33,10 +34,10 @@ import org.mockito.kotlin.*
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc
import org.springframework.boot.test.context.SpringBootTest
import org.springframework.boot.test.mock.mockito.MockBean
import org.springframework.context.ApplicationContext
import org.springframework.http.MediaType
import org.springframework.test.context.TestPropertySource
import org.springframework.test.context.bean.override.mockito.MockitoBean
import org.springframework.test.context.junit.jupiter.SpringExtension
import org.springframework.test.web.servlet.MockMvc
import org.springframework.test.web.servlet.post
@@ -45,11 +46,12 @@ import org.testcontainers.junit.jupiter.Testcontainers
@Testcontainers
@ExtendWith(SpringExtension::class)
@SpringBootTest
@MockBean(MtbFileSender::class)
@MockitoBean(types = [MtbFileSender::class])
@TestPropertySource(
properties = [
"app.rest.uri=http://example.com",
"app.pseudonymize.generator=buildin"
"app.pseudonymize.generator=buildin",
"app.consent.service=none"
]
)
class EtlProcessorApplicationTests : AbstractTestcontainerTest() {
@@ -66,14 +68,15 @@ class EtlProcessorApplicationTests : AbstractTestcontainerTest() {
@TestPropertySource(
properties = [
"app.pseudonymize.generator=buildin",
"app.transformations[0].path=diagnoses[*].icd10.version",
"app.consent.service=none",
"app.transformations[0].path=diagnoses[*].code.version",
"app.transformations[0].from=2013",
"app.transformations[0].to=2014",
]
)
inner class TransformationTest {
@MockBean
@MockitoBean
private lateinit var mtbFileSender: MtbFileSender
@Autowired
@@ -91,36 +94,33 @@ class EtlProcessorApplicationTests : AbstractTestcontainerTest() {
fun mtbFileIsTransformed() {
doAnswer {
MtbFileSender.Response(RequestStatus.SUCCESS)
}.whenever(mtbFileSender).send(any<MtbFileSender.MtbFileRequest>())
}.whenever(mtbFileSender).send(any<DnpmV2MtbFileRequest>())
val mtbFile = MtbFile.builder()
.withPatient(
val mtbFile = Mtb.builder()
.patient(
Patient.builder()
.withId("TEST_12345678")
.withBirthDate("2000-08-08")
.withGender(Patient.Gender.MALE)
.id("TEST_12345678")
.build()
)
.withConsent(
Consent.builder()
.withId("1")
.withStatus(Consent.Status.ACTIVE)
.withPatient("TEST_12345678")
.metadata(
MvhMetadata
.builder()
.modelProjectConsent(
ModelProjectConsent
.builder()
.provisions(
listOf(Provision.builder().type(ConsentProvision.PERMIT).purpose(ModelProjectConsentPurpose.SEQUENCING).build())
).build()
)
.build()
)
.withEpisode(
Episode.builder()
.withId("1")
.withPatient("TEST_12345678")
.withPeriod(PeriodStart("2023-08-08"))
.build()
)
.withDiagnoses(
.diagnoses(
listOf(
Diagnosis.builder()
.withId("1234")
.withIcd10(Icd10.builder().withCode("F79.9").withVersion("2013").build())
.build()
MtbDiagnosis.builder()
.id("1234")
.patient(Reference.builder().id("TEST_12345678").build())
.code(Coding.builder().code("F79.9").version("2013").build())
.build(),
)
)
.build()
@@ -134,10 +134,10 @@ class EtlProcessorApplicationTests : AbstractTestcontainerTest() {
}
}
val captor = argumentCaptor<MtbFileSender.MtbFileRequest>()
val captor = argumentCaptor<DnpmV2MtbFileRequest>()
verify(mtbFileSender).send(captor.capture())
assertThat(captor.firstValue.mtbFile.diagnoses).hasSize(1).allMatch { diagnosis ->
diagnosis.icd10.version == "2014"
assertThat(captor.firstValue.content.diagnoses).hasSize(1).allMatch { diagnosis ->
diagnosis.code.version == "2014"
}
}
}

View File

@@ -0,0 +1,73 @@
package dev.dnpm.etl.processor
import com.tngtech.archunit.core.domain.JavaClasses
import com.tngtech.archunit.core.importer.ClassFileImporter
import com.tngtech.archunit.lang.syntax.ArchRuleDefinition.classes
import com.tngtech.archunit.lang.syntax.ArchRuleDefinition.noClasses
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.springframework.data.repository.Repository
class EtlProcessorArchTest {
private lateinit var noTestClasses: JavaClasses
@BeforeEach
fun setUp() {
this.noTestClasses = ClassFileImporter()
.withImportOption { !(it.contains("/test/") || it.contains("/integrationTest/")) }
.importPackages("dev.dnpm.etl.processor")
}
@Test
fun noClassesInInputPackageShouldDependOnMonitoringPackage() {
val rule = noClasses()
.that()
.resideInAPackage("..input")
.should().dependOnClassesThat()
.resideInAnyPackage("..monitoring")
rule.check(noTestClasses)
}
@Test
fun noClassesInInputPackageShouldDependOnRepositories() {
val rule = noClasses()
.that()
.resideInAPackage("..input")
.should().dependOnClassesThat().haveSimpleNameEndingWith("Repository")
rule.check(noTestClasses)
}
@Test
fun noClassesInOutputPackageShouldDependOnRepositories() {
val rule = noClasses()
.that()
.resideInAPackage("..output")
.should().dependOnClassesThat().haveSimpleNameEndingWith("Repository")
rule.check(noTestClasses)
}
@Test
fun noClassesInWebPackageShouldDependOnRepositories() {
val rule = noClasses()
.that()
.resideInAPackage("..web")
.should().dependOnClassesThat().haveSimpleNameEndingWith("Repository")
rule.check(noTestClasses)
}
@Test
fun repositoryClassNamesShouldEndWithRepository() {
val rule = classes()
.that()
.areInterfaces().and().areAssignableTo(Repository::class.java)
.should().haveSimpleNameEndingWith("Repository")
rule.check(noTestClasses)
}
}

View File

@@ -20,15 +20,18 @@
package dev.dnpm.etl.processor.config
import com.fasterxml.jackson.databind.ObjectMapper
import dev.dnpm.etl.processor.consent.ConsentEvaluator
import dev.dnpm.etl.processor.consent.GicsConsentService
import dev.dnpm.etl.processor.consent.MtbFileConsentService
import dev.dnpm.etl.processor.input.KafkaInputListener
import dev.dnpm.etl.processor.monitoring.RequestRepository
import dev.dnpm.etl.processor.output.KafkaMtbFileSender
import dev.dnpm.etl.processor.output.RestMtbFileSender
import dev.dnpm.etl.processor.pseudonym.AnonymizingGenerator
import dev.dnpm.etl.processor.pseudonym.GpasPseudonymGenerator
import dev.dnpm.etl.processor.security.TokenRepository
import dev.dnpm.etl.processor.security.TokenService
import dev.dnpm.etl.processor.services.RequestProcessor
import dev.dnpm.etl.processor.services.TokenRepository
import dev.dnpm.etl.processor.services.TokenService
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.Nested
import org.junit.jupiter.api.Test
@@ -36,24 +39,26 @@ import org.junit.jupiter.api.assertThrows
import org.springframework.beans.factory.NoSuchBeanDefinitionException
import org.springframework.boot.autoconfigure.kafka.KafkaAutoConfiguration
import org.springframework.boot.test.context.SpringBootTest
import org.springframework.boot.test.mock.mockito.MockBean
import org.springframework.boot.test.mock.mockito.MockBeans
import org.springframework.context.ApplicationContext
import org.springframework.retry.support.RetryTemplate
import org.springframework.security.crypto.password.PasswordEncoder
import org.springframework.security.provisioning.InMemoryUserDetailsManager
import org.springframework.test.context.ContextConfiguration
import org.springframework.test.context.TestPropertySource
import org.springframework.test.context.bean.override.mockito.MockitoBean
@SpringBootTest
@ContextConfiguration(classes = [
AppConfiguration::class,
AppSecurityConfiguration::class,
KafkaAutoConfiguration::class,
AppKafkaConfiguration::class,
AppRestConfiguration::class
])
@MockBean(ObjectMapper::class)
@ContextConfiguration(
classes = [
AppConfiguration::class,
AppSecurityConfiguration::class,
KafkaAutoConfiguration::class,
AppKafkaConfiguration::class,
AppRestConfiguration::class,
ConsentEvaluator::class
]
)
@MockitoBean(types = [ObjectMapper::class])
@TestPropertySource(
properties = [
"app.pseudonymize.generator=BUILDIN",
@@ -86,7 +91,7 @@ class AppConfigurationTest {
"app.kafka.group-id=test"
]
)
@MockBean(RequestRepository::class)
@MockitoBean(types = [RequestRepository::class])
inner class AppConfigurationKafkaTest(private val context: ApplicationContext) {
@Test
@@ -145,7 +150,7 @@ class AppConfigurationTest {
"app.kafka.group-id=test"
]
)
@MockBean(RequestProcessor::class)
@MockitoBean(types = [RequestProcessor::class])
inner class AppConfigurationUsingKafkaInputTest(private val context: ApplicationContext) {
@Test
@@ -181,40 +186,7 @@ class AppConfigurationTest {
@Nested
@TestPropertySource(
properties = [
"app.pseudonymize.generator=",
"app.pseudonymizer=buildin",
]
)
inner class AppConfigurationPseudonymizerBuildinTest(private val context: ApplicationContext) {
@Test
fun shouldUseConfiguredGenerator() {
assertThat(context.getBean(AnonymizingGenerator::class.java)).isNotNull
}
}
@Nested
@TestPropertySource(
properties = [
"app.pseudonymize.generator=",
"app.pseudonymizer=gpas",
]
)
inner class AppConfigurationPseudonymizerGpasTest(private val context: ApplicationContext) {
@Test
fun shouldUseConfiguredGenerator() {
assertThat(context.getBean(GpasPseudonymGenerator::class.java)).isNotNull
}
}
@Nested
@TestPropertySource(
properties = [
"app.pseudonymize.generator=buildin",
"app.pseudonymizer=",
"app.pseudonymize.generator=buildin"
]
)
inner class AppConfigurationPseudonymizeGeneratorBuildinTest(private val context: ApplicationContext) {
@@ -229,8 +201,7 @@ class AppConfigurationTest {
@Nested
@TestPropertySource(
properties = [
"app.pseudonymize.generator=gpas",
"app.pseudonymizer=",
"app.pseudonymize.generator=gpas"
]
)
inner class AppConfigurationPseudonymizeGeneratorGpasTest(private val context: ApplicationContext) {
@@ -248,11 +219,13 @@ class AppConfigurationTest {
"app.security.enable-tokens=true"
]
)
@MockBeans(value = [
MockBean(InMemoryUserDetailsManager::class),
MockBean(PasswordEncoder::class),
MockBean(TokenRepository::class)
])
@MockitoBean(
types = [
InMemoryUserDetailsManager::class,
PasswordEncoder::class,
TokenRepository::class
]
)
inner class AppConfigurationTokenEnabledTest(private val context: ApplicationContext) {
@Test
@@ -263,11 +236,13 @@ class AppConfigurationTest {
}
@Nested
@MockBeans(value = [
MockBean(InMemoryUserDetailsManager::class),
MockBean(PasswordEncoder::class),
MockBean(TokenRepository::class)
])
@MockitoBean(
types = [
InMemoryUserDetailsManager::class,
PasswordEncoder::class,
TokenRepository::class
]
)
inner class AppConfigurationTokenDisabledTest(private val context: ApplicationContext) {
@Test
@@ -305,4 +280,30 @@ class AppConfigurationTest {
}
}
@Nested
@TestPropertySource(
properties = [
"app.consent.service=GICS",
"app.consent.gics.uri=http://localhost:9000",
]
)
inner class AppConfigurationConsentGicsTest(private val context: ApplicationContext) {
@Test
fun shouldUseConfiguredGenerator() {
assertThat(context.getBean(GicsConsentService::class.java)).isNotNull
}
}
@Nested
inner class AppConfigurationConsentBuildinTest(private val context: ApplicationContext) {
@Test
fun shouldUseConfiguredGenerator() {
assertThat(context.getBean(MtbFileConsentService::class.java)).isNotNull
}
}
}

View File

@@ -0,0 +1,30 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor
import org.mockito.ArgumentMatchers
@Suppress("UNCHECKED_CAST")
inline fun <reified T> anyValueClass(): T {
val unboxedClass = T::class.java.declaredFields.first().type
return ArgumentMatchers.any(unboxedClass as Class<T>)
?: T::class.java.getDeclaredMethod("box-impl", unboxedClass)
.invoke(null, null) as T
}

View File

@@ -20,41 +20,47 @@
package dev.dnpm.etl.processor.input
import com.fasterxml.jackson.databind.ObjectMapper
import de.ukw.ccc.bwhc.dto.*
import dev.dnpm.etl.processor.config.AppSecurityConfiguration
import dev.dnpm.etl.processor.consent.ConsentEvaluation
import dev.dnpm.etl.processor.consent.ConsentEvaluator
import dev.dnpm.etl.processor.consent.MtbFileConsentService
import dev.dnpm.etl.processor.consent.TtpConsentStatus
import dev.dnpm.etl.processor.security.TokenRepository
import dev.dnpm.etl.processor.security.UserRoleRepository
import dev.dnpm.etl.processor.services.RequestProcessor
import dev.dnpm.etl.processor.services.TokenRepository
import dev.pcvolkmer.mv64e.mtb.*
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Nested
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.ArgumentMatchers.anyString
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.any
import org.mockito.kotlin.never
import org.mockito.kotlin.times
import org.mockito.kotlin.verify
import org.mockito.kotlin.*
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest
import org.springframework.boot.test.mock.mockito.MockBean
import org.springframework.http.MediaType
import org.springframework.security.oauth2.client.registration.ClientRegistrationRepository
import org.springframework.security.test.web.servlet.request.SecurityMockMvcRequestPostProcessors.anonymous
import org.springframework.security.test.web.servlet.request.SecurityMockMvcRequestPostProcessors.user
import org.springframework.test.context.ContextConfiguration
import org.springframework.test.context.TestPropertySource
import org.springframework.test.context.bean.override.mockito.MockitoBean
import org.springframework.test.context.junit.jupiter.SpringExtension
import org.springframework.test.web.servlet.MockMvc
import org.springframework.test.web.servlet.delete
import org.springframework.test.web.servlet.post
import java.time.Instant
import java.util.*
@WebMvcTest(controllers = [MtbFileRestController::class])
@ExtendWith(value = [MockitoExtension::class, SpringExtension::class])
@ContextConfiguration(
classes = [
MtbFileRestController::class,
AppSecurityConfiguration::class
AppSecurityConfiguration::class,
MtbFileConsentService::class
]
)
@MockBean(TokenRepository::class, RequestProcessor::class)
@MockitoBean(types = [TokenRepository::class, RequestProcessor::class, ConsentEvaluator::class])
@TestPropertySource(
properties = [
"app.pseudonymize.generator=BUILDIN",
@@ -65,17 +71,23 @@ import org.springframework.test.web.servlet.post
)
class MtbFileRestControllerTest {
private lateinit var mockMvc: MockMvc
private lateinit var requestProcessor: RequestProcessor
lateinit var mockMvc: MockMvc
lateinit var requestProcessor: RequestProcessor
lateinit var consentEvaluator: ConsentEvaluator
@BeforeEach
fun setup(
@Autowired mockMvc: MockMvc,
@Autowired requestProcessor: RequestProcessor
@Autowired requestProcessor: RequestProcessor,
@Autowired consentEvaluator: ConsentEvaluator
) {
this.mockMvc = mockMvc
this.requestProcessor = requestProcessor
this.consentEvaluator = consentEvaluator
doAnswer {
ConsentEvaluation(TtpConsentStatus.BROAD_CONSENT_GIVEN, true)
}.whenever(consentEvaluator).check(any())
}
@Test
@@ -88,7 +100,20 @@ class MtbFileRestControllerTest {
status { isAccepted() }
}
verify(requestProcessor, times(1)).processMtbFile(any())
verify(requestProcessor, times(1)).processMtbFile(any<Mtb>())
}
@Test
fun testShouldGrantPermissionToSendMtbFileToAdminUser() {
mockMvc.post("/mtbfile") {
with(user("onkostarserver").roles("ADMIN"))
contentType = MediaType.APPLICATION_JSON
content = ObjectMapper().writeValueAsString(mtbFile)
}.andExpect {
status { isAccepted() }
}
verify(requestProcessor, times(1)).processMtbFile(any<Mtb>())
}
@Test
@@ -101,7 +126,20 @@ class MtbFileRestControllerTest {
status { isUnauthorized() }
}
verify(requestProcessor, never()).processMtbFile(any())
verify(requestProcessor, never()).processMtbFile(any<Mtb>())
}
@Test
fun testShouldDenyPermissionToSendMtbFileForUser() {
mockMvc.post("/mtbfile") {
with(user("fakeuser").roles("USER"))
contentType = MediaType.APPLICATION_JSON
content = ObjectMapper().writeValueAsString(mtbFile)
}.andExpect {
status { isForbidden() }
}
verify(requestProcessor, never()).processMtbFile(any<Mtb>())
}
@Test
@@ -112,7 +150,7 @@ class MtbFileRestControllerTest {
status { isAccepted() }
}
verify(requestProcessor, times(1)).processDeletion(anyString())
verify(requestProcessor, times(1)).processDeletion(anyValueClass(), eq(TtpConsentStatus.UNKNOWN_CHECK_FILE))
}
@Test
@@ -123,32 +161,64 @@ class MtbFileRestControllerTest {
status { isUnauthorized() }
}
verify(requestProcessor, never()).processDeletion(anyString())
verify(requestProcessor, never()).processDeletion(anyValueClass(), any())
}
@Nested
@MockitoBean(types = [UserRoleRepository::class, ClientRegistrationRepository::class])
@TestPropertySource(
properties = [
"app.pseudonymize.generator=BUILDIN",
"app.security.admin-user=admin",
"app.security.admin-password={noop}very-secret",
"app.security.enable-tokens=true",
"app.security.enable-oidc=true"
]
)
inner class WithOidcEnabled {
@Test
fun testShouldGrantPermissionToSendMtbFileToAdminUser() {
mockMvc.post("/mtbfile") {
with(user("onkostarserver").roles("ADMIN"))
contentType = MediaType.APPLICATION_JSON
content = ObjectMapper().writeValueAsString(mtbFile)
}.andExpect {
status { isAccepted() }
}
verify(requestProcessor, times(1)).processMtbFile(any<Mtb>())
}
@Test
fun testShouldGrantPermissionToSendMtbFileToUser() {
mockMvc.post("/mtbfile") {
with(user("onkostarserver").roles("USER"))
contentType = MediaType.APPLICATION_JSON
content = ObjectMapper().writeValueAsString(mtbFile)
}.andExpect {
status { isAccepted() }
}
verify(requestProcessor, times(1)).processMtbFile(any<Mtb>())
}
}
companion object {
val mtbFile: MtbFile = MtbFile.builder()
.withPatient(
val mtbFile = Mtb.builder()
.patient(
Patient.builder()
.withId("PID")
.withBirthDate("2000-08-08")
.withGender(Patient.Gender.MALE)
.id("PID")
.build()
)
.withConsent(
Consent.builder()
.withId("1")
.withStatus(Consent.Status.ACTIVE)
.withPatient("PID")
.build()
)
.withEpisode(
Episode.builder()
.withId("1")
.withPatient("PID")
.withPeriod(PeriodStart("2023-08-08"))
.build()
.episodesOfCare(
listOf(
MtbEpisodeOfCare.builder()
.id("1")
.patient(Reference.builder().id("PID").build())
.period(PeriodDate.builder().start(Date.from(Instant.parse("2023-08-08T02:00:00.00Z"))).build())
.build()
)
)
.build()

View File

@@ -0,0 +1,75 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.monitoring
import dev.dnpm.etl.processor.*
import dev.dnpm.etl.processor.output.MtbFileSender
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.autoconfigure.data.jdbc.DataJdbcTest
import org.springframework.boot.test.autoconfigure.jdbc.AutoConfigureTestDatabase
import org.springframework.test.context.TestPropertySource
import org.springframework.test.context.bean.override.mockito.MockitoBean
import org.springframework.test.context.junit.jupiter.SpringExtension
import org.springframework.transaction.annotation.Transactional
import org.testcontainers.junit.jupiter.Testcontainers
import java.time.Instant
@Testcontainers
@ExtendWith(SpringExtension::class)
@DataJdbcTest
@AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE)
@Transactional
@MockitoBean(types = [MtbFileSender::class])
@TestPropertySource(
properties = [
"app.pseudonymize.generator=buildin",
"app.rest.uri=http://example.com"
]
)
class RequestRepositoryTest : AbstractTestcontainerTest() {
private lateinit var requestRepository: RequestRepository
@BeforeEach
fun setUp(
@Autowired requestRepository: RequestRepository
) {
this.requestRepository = requestRepository
}
@Test
fun shouldSaveRequest() {
val request = Request(
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.MTB_FILE,
RequestStatus.WARNING,
Instant.parse("2023-07-07T00:00:00Z")
)
requestRepository.save(request)
}
}

View File

@@ -0,0 +1,147 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.pseudonym
import dev.dnpm.etl.processor.config.AppFhirConfig
import dev.dnpm.etl.processor.config.GPasConfigProperties
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.assertThrows
import org.springframework.http.HttpHeaders
import org.springframework.http.HttpMethod
import org.springframework.http.HttpStatus
import org.springframework.retry.policy.SimpleRetryPolicy
import org.springframework.retry.support.RetryTemplateBuilder
import org.springframework.test.web.client.MockRestServiceServer
import org.springframework.test.web.client.match.MockRestRequestMatchers.method
import org.springframework.test.web.client.match.MockRestRequestMatchers.requestTo
import org.springframework.test.web.client.response.MockRestResponseCreators.withException
import org.springframework.test.web.client.response.MockRestResponseCreators.withStatus
import org.springframework.web.client.RestTemplate
import java.io.IOException
class GpasPseudonymGeneratorTest {
private lateinit var mockRestServiceServer: MockRestServiceServer
private lateinit var generator: GpasPseudonymGenerator
private lateinit var restTemplate: RestTemplate
private var appFhirConfig: AppFhirConfig = AppFhirConfig()
@BeforeEach
fun setup() {
val retryTemplate = RetryTemplateBuilder().customPolicy(SimpleRetryPolicy(1)).build()
val gPasConfigProperties = GPasConfigProperties(
"https://localhost:9990/ttp-fhir/fhir/gpas",
"test", "test2",
null,
null
)
this.restTemplate = RestTemplate()
this.mockRestServiceServer = MockRestServiceServer.createServer(restTemplate)
this.generator =
GpasPseudonymGenerator(gPasConfigProperties, retryTemplate, restTemplate, appFhirConfig)
}
@Test
fun shouldReturnExpectedPseudonym() {
this.mockRestServiceServer.expect {
method(HttpMethod.POST)
requestTo("https://localhost/ttp-fhir/fhir/gpas/\$pseudonymizeAllowCreate")
}.andRespond {
withStatus(HttpStatus.OK).body(
getDummyResponseBody(
"1234",
"test",
"test1234ABCDEF567890"
)
)
.createResponse(it)
}
assertThat(this.generator.generate("ID1234")).isEqualTo("test1234ABCDEF567890")
}
@Test
fun shouldThrowExceptionIfGpasNotAvailable() {
this.mockRestServiceServer.expect {
method(HttpMethod.POST)
requestTo("https://localhost/ttp-fhir/fhir/gpas/\$pseudonymizeAllowCreate")
}.andRespond {
withException(IOException("Simulated IO error")).createResponse(it)
}
assertThrows<PseudonymRequestFailed> { this.generator.generate("ID1234") }
}
@Test
fun shouldThrowExceptionIfGpasDoesNotReturn2xxResponse() {
this.mockRestServiceServer.expect {
method(HttpMethod.POST)
requestTo("https://localhost/ttp-fhir/fhir/gpas/\$pseudonymizeAllowCreate")
}.andRespond {
withStatus(HttpStatus.FOUND)
.header(
HttpHeaders.LOCATION,
"https://localhost/ttp-fhir/fhir/gpas/\$pseudonymizeAllowCreate"
)
.createResponse(it)
}
assertThrows<PseudonymRequestFailed> { this.generator.generate("ID1234") }
}
companion object {
fun getDummyResponseBody(original: String, target: String, pseudonym: String) = """{
"resourceType": "Parameters",
"parameter": [
{
"name": "pseudonym",
"part": [
{
"name": "original",
"valueIdentifier": {
"system": "https://ths-greifswald.de/gpas",
"value": "$original"
}
},
{
"name": "target",
"valueIdentifier": {
"system": "https://ths-greifswald.de/gpas",
"value": "$target"
}
},
{
"name": "pseudonym",
"valueIdentifier": {
"system": "https://ths-greifswald.de/gpas",
"value": "$pseudonym"
}
}
]
}
]
}""".trimIndent()
}
}

View File

@@ -19,7 +19,7 @@
package dev.dnpm.etl.processor.services
import dev.dnpm.etl.processor.AbstractTestcontainerTest
import dev.dnpm.etl.processor.*
import dev.dnpm.etl.processor.monitoring.Request
import dev.dnpm.etl.processor.monitoring.RequestRepository
import dev.dnpm.etl.processor.monitoring.RequestStatus
@@ -31,19 +31,18 @@ import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.context.SpringBootTest
import org.springframework.boot.test.mock.mockito.MockBean
import org.springframework.test.context.TestPropertySource
import org.springframework.test.context.bean.override.mockito.MockitoBean
import org.springframework.test.context.junit.jupiter.SpringExtension
import org.springframework.transaction.annotation.Transactional
import org.testcontainers.junit.jupiter.Testcontainers
import java.time.Instant
import java.util.*
@Testcontainers
@ExtendWith(SpringExtension::class)
@SpringBootTest
@Transactional
@MockBean(MtbFileSender::class)
@MockitoBean(types = [MtbFileSender::class])
@TestPropertySource(
properties = [
"app.pseudonymize.generator=buildin",
@@ -66,7 +65,7 @@ class RequestServiceIntegrationTest : AbstractTestcontainerTest() {
@Test
fun shouldResultInEmptyRequestList() {
val actual = requestService.allRequestsByPatientPseudonym("TEST_12345678901")
val actual = requestService.allRequestsByPatientPseudonym(TEST_PATIENT_PSEUDONYM)
assertThat(actual).isEmpty()
}
@@ -76,33 +75,33 @@ class RequestServiceIntegrationTest : AbstractTestcontainerTest() {
this.requestRepository.saveAll(
listOf(
Request(
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "0123456789abcdef1",
type = RequestType.MTB_FILE,
status = RequestStatus.SUCCESS,
processedAt = Instant.parse("2023-07-07T02:00:00Z")
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS,
Instant.parse("2023-07-07T02:00:00Z")
),
// Should be ignored - wrong patient ID -->
Request(
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678902",
pid = "P2",
fingerprint = "0123456789abcdef2",
type = RequestType.MTB_FILE,
status = RequestStatus.WARNING,
processedAt = Instant.parse("2023-08-08T00:00:00Z")
randomRequestId(),
PatientPseudonym("TEST_12345678902"),
PatientId("P2"),
Fingerprint("0123456789abcdef2"),
RequestType.MTB_FILE,
RequestStatus.WARNING,
Instant.parse("2023-08-08T00:00:00Z")
),
// <--
Request(
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P2",
fingerprint = "0123456789abcdee1",
type = RequestType.DELETE,
status = RequestStatus.SUCCESS,
processedAt = Instant.parse("2023-08-08T02:00:00Z")
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P2"),
Fingerprint("0123456789abcdee1"),
RequestType.DELETE,
RequestStatus.SUCCESS,
Instant.parse("2023-08-08T02:00:00Z")
)
)
)
@@ -112,18 +111,18 @@ class RequestServiceIntegrationTest : AbstractTestcontainerTest() {
fun shouldResultInSortedRequestList() {
setupTestData()
val actual = requestService.allRequestsByPatientPseudonym("TEST_12345678901")
val actual = requestService.allRequestsByPatientPseudonym(TEST_PATIENT_PSEUDONYM)
assertThat(actual).hasSize(2)
assertThat(actual[0].fingerprint).isEqualTo("0123456789abcdee1")
assertThat(actual[1].fingerprint).isEqualTo("0123456789abcdef1")
assertThat(actual[0].fingerprint).isEqualTo(Fingerprint("0123456789abcdee1"))
assertThat(actual[1].fingerprint).isEqualTo(Fingerprint("0123456789abcdef1"))
}
@Test
fun shouldReturnDeleteRequestAsLastRequest() {
setupTestData()
val actual = requestService.isLastRequestWithKnownStatusDeletion("TEST_12345678901")
val actual = requestService.isLastRequestWithKnownStatusDeletion(TEST_PATIENT_PSEUDONYM)
assertThat(actual).isTrue()
}
@@ -132,10 +131,14 @@ class RequestServiceIntegrationTest : AbstractTestcontainerTest() {
fun shouldReturnLastMtbFileRequest() {
setupTestData()
val actual = requestService.lastMtbFileRequestForPatientPseudonym("TEST_12345678901")
val actual = requestService.lastMtbFileRequestForPatientPseudonym(TEST_PATIENT_PSEUDONYM)
assertThat(actual).isNotNull
assertThat(actual?.fingerprint).isEqualTo("0123456789abcdef1")
assertThat(actual?.fingerprint).isEqualTo(Fingerprint("0123456789abcdef1"))
}
companion object {
val TEST_PATIENT_PSEUDONYM = PatientPseudonym("TEST_12345678901")
}
}

View File

@@ -21,30 +21,50 @@ package dev.dnpm.etl.processor.web
import dev.dnpm.etl.processor.config.AppConfiguration
import dev.dnpm.etl.processor.config.AppSecurityConfiguration
import dev.dnpm.etl.processor.monitoring.ConnectionCheckService
import dev.dnpm.etl.processor.monitoring.ConnectionCheckResult
import dev.dnpm.etl.processor.monitoring.GIcsConnectionCheckService
import dev.dnpm.etl.processor.monitoring.GPasConnectionCheckService
import dev.dnpm.etl.processor.monitoring.RestConnectionCheckService
import dev.dnpm.etl.processor.output.MtbFileSender
import dev.dnpm.etl.processor.pseudonym.Generator
import dev.dnpm.etl.processor.security.Role
import dev.dnpm.etl.processor.security.TokenService
import dev.dnpm.etl.processor.security.UserRoleService
import dev.dnpm.etl.processor.services.RequestProcessor
import dev.dnpm.etl.processor.services.TokenRepository
import dev.dnpm.etl.processor.services.TransformationService
import org.assertj.core.api.Assertions.assertThat
import org.htmlunit.WebClient
import org.htmlunit.html.HtmlPage
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Nested
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.ArgumentMatchers.anyString
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.argumentCaptor
import org.mockito.kotlin.times
import org.mockito.kotlin.verify
import org.mockito.kotlin.whenever
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest
import org.springframework.boot.test.mock.mockito.MockBean
import org.springframework.http.HttpHeaders
import org.springframework.http.MediaType
import org.springframework.http.MediaType.TEXT_EVENT_STREAM
import org.springframework.security.test.context.support.WithMockUser
import org.springframework.security.test.web.servlet.request.SecurityMockMvcRequestPostProcessors.anonymous
import org.springframework.security.test.web.servlet.request.SecurityMockMvcRequestPostProcessors.user
import org.springframework.test.context.ContextConfiguration
import org.springframework.test.context.TestPropertySource
import org.springframework.test.context.bean.override.mockito.MockitoBean
import org.springframework.test.context.junit.jupiter.SpringExtension
import org.springframework.test.web.servlet.MockMvc
import org.springframework.test.web.servlet.get
import org.springframework.test.web.reactive.server.WebTestClient
import org.springframework.test.web.servlet.*
import org.springframework.test.web.servlet.client.MockMvcWebTestClient
import org.springframework.test.web.servlet.htmlunit.MockMvcWebClientBuilder
import org.springframework.web.context.WebApplicationContext
import reactor.core.publisher.Sinks
import reactor.test.StepVerifier
import java.time.Instant
abstract class MockSink : Sinks.Many<Boolean>
@@ -59,44 +79,51 @@ abstract class MockSink : Sinks.Many<Boolean>
)
@TestPropertySource(
properties = [
"app.pseudonymize.generator=BUILDIN",
"app.security.admin-user=admin",
"app.security.admin-password={noop}very-secret",
"app.security.enable-tokens=true"
"app.pseudonymize.generator=BUILDIN"
]
)
@MockBean(name = "configsUpdateProducer", classes = [MockSink::class])
@MockBean(
Generator::class,
MtbFileSender::class,
ConnectionCheckService::class,
RequestProcessor::class,
TransformationService::class,
TokenRepository::class,
RestConnectionCheckService::class
@MockitoBean(name = "configsUpdateProducer", types = [MockSink::class])
@MockitoBean(
types = [
Generator::class,
MtbFileSender::class,
RequestProcessor::class,
TransformationService::class,
GPasConnectionCheckService::class,
RestConnectionCheckService::class,
GIcsConnectionCheckService::class
]
)
class ConfigControllerTest {
private lateinit var mockMvc: MockMvc
private lateinit var webClient: WebClient
private lateinit var requestProcessor: RequestProcessor
private lateinit var connectionCheckUpdateProducer: Sinks.Many<ConnectionCheckResult>
@BeforeEach
fun setup(
@Autowired mockMvc: MockMvc,
@Autowired requestProcessor: RequestProcessor
@Autowired requestProcessor: RequestProcessor,
@Autowired connectionCheckUpdateProducer: Sinks.Many<ConnectionCheckResult>
) {
this.mockMvc = mockMvc
this.webClient = MockMvcWebClientBuilder.mockMvcSetup(mockMvc).build()
this.requestProcessor = requestProcessor
this.connectionCheckUpdateProducer = connectionCheckUpdateProducer
webClient.options.isThrowExceptionOnScriptError = false
}
@Test
fun testShouldShowConfigPageIfLoggedIn() {
fun testShouldRequestConfigPageIfLoggedIn() {
mockMvc.get("/configs") {
with(user("admin").roles("ADMIN"))
accept(MediaType.TEXT_HTML)
}.andExpect {
status { isOk() }
view { name("configs") }
}
}
@@ -113,4 +140,244 @@ class ConfigControllerTest {
}
}
@Nested
@TestPropertySource(
properties = [
"app.security.enable-tokens=true",
"app.security.admin-user=admin"
]
)
@MockitoBean(
types = [
TokenService::class
]
)
inner class WithTokensEnabled {
private lateinit var tokenService: TokenService
@BeforeEach
fun setup(
@Autowired tokenService: TokenService
) {
webClient.options.isThrowExceptionOnScriptError = false
this.tokenService = tokenService
}
@Test
fun testShouldSaveNewToken() {
mockMvc.post("/configs/tokens") {
with(user("admin").roles("ADMIN"))
accept(MediaType.TEXT_HTML)
contentType = MediaType.APPLICATION_FORM_URLENCODED
content = "name=Testtoken"
}.andExpect {
status { is2xxSuccessful() }
view { name("configs/tokens") }
}
val captor = argumentCaptor<String>()
verify(tokenService, times(1)).addToken(captor.capture())
assertThat(captor.firstValue).isEqualTo("Testtoken")
}
@Test
fun testShouldNotSaveTokenWithExstingName() {
whenever(tokenService.addToken(anyString())).thenReturn(
Result.failure(
RuntimeException(
"Testfailure"
)
)
)
mockMvc.post("/configs/tokens") {
with(user("admin").roles("ADMIN"))
accept(MediaType.TEXT_HTML)
contentType = MediaType.APPLICATION_FORM_URLENCODED
content = "name=Testtoken"
}.andExpect {
status { is2xxSuccessful() }
view { name("configs/tokens") }
}
val captor = argumentCaptor<String>()
verify(tokenService, times(1)).addToken(captor.capture())
assertThat(captor.firstValue).isEqualTo("Testtoken")
}
@Test
fun testShouldDeleteToken() {
mockMvc.delete("/configs/tokens/42") {
with(user("admin").roles("ADMIN"))
accept(MediaType.TEXT_HTML)
}.andExpect {
status { is2xxSuccessful() }
view { name("configs/tokens") }
}
val captor = argumentCaptor<Long>()
verify(tokenService, times(1)).deleteToken(captor.capture())
assertThat(captor.firstValue).isEqualTo(42)
}
@Test
@WithMockUser(username = "admin", roles = ["ADMIN"])
fun testShouldRenderConfigPageWithTokens() {
val page = webClient.getPage<HtmlPage>("http://localhost/configs")
assertThat(
page.getElementById("tokens")
).isNotNull
}
}
@Nested
@TestPropertySource(
properties = [
"app.security.enable-tokens=false"
]
)
inner class WithTokensDisabled {
@BeforeEach
fun setup() {
webClient.options.isThrowExceptionOnScriptError = false
}
@Test
@WithMockUser(username = "admin", roles = ["ADMIN"])
fun testShouldRenderConfigPageWithoutTokens() {
val page = webClient.getPage<HtmlPage>("http://localhost/configs")
assertThat(
page.getElementById("tokens")
).isNull()
}
}
@Nested
@TestPropertySource(
properties = [
"app.security.enable-tokens=false",
"app.security.admin-user=admin",
"app.security.admin-password={noop}very-secret"
]
)
@MockitoBean(
types = [
UserRoleService::class
]
)
inner class WithUserRolesEnabled {
private lateinit var userRoleService: UserRoleService
@BeforeEach
fun setup(
@Autowired userRoleService: UserRoleService
) {
webClient.options.isThrowExceptionOnScriptError = false
this.userRoleService = userRoleService
}
@Test
fun testShouldDeleteUserRole() {
mockMvc.delete("/configs/userroles/42") {
with(user("admin").roles("ADMIN"))
accept(MediaType.TEXT_HTML)
}.andExpect {
status { is2xxSuccessful() }
view { name("configs/userroles") }
}
val captor = argumentCaptor<Long>()
verify(userRoleService, times(1)).deleteUserRole(captor.capture())
assertThat(captor.firstValue).isEqualTo(42)
}
@Test
fun testShouldUpdateUserRole() {
mockMvc.put("/configs/userroles/42") {
with(user("admin").roles("ADMIN"))
accept(MediaType.TEXT_HTML)
contentType = MediaType.APPLICATION_FORM_URLENCODED
content = "role=ADMIN"
}.andExpect {
status { is2xxSuccessful() }
view { name("configs/userroles") }
}
val idCaptor = argumentCaptor<Long>()
val roleCaptor = argumentCaptor<Role>()
verify(userRoleService, times(1)).updateUserRole(
idCaptor.capture(),
roleCaptor.capture()
)
assertThat(idCaptor.firstValue).isEqualTo(42)
assertThat(roleCaptor.firstValue).isEqualTo(Role.ADMIN)
}
@Test
@WithMockUser(username = "admin", roles = ["ADMIN"])
fun testShouldRenderConfigPageWithUserRoles() {
val page = webClient.getPage<HtmlPage>("http://localhost/configs")
assertThat(
page.getElementById("userroles")
).isNotNull
}
}
@Nested
inner class WithUserRolesDisabled {
@BeforeEach
fun setup() {
webClient.options.isThrowExceptionOnScriptError = false
}
@Test
fun testShouldRenderConfigPageWithoutUserRoles() {
val page = webClient.getPage<HtmlPage>("http://localhost/configs")
assertThat(
page.getElementById("userroles")
).isNull()
}
}
@Nested
inner class SseTest {
private lateinit var webClient: WebTestClient
@BeforeEach
fun setup(
applicationContext: WebApplicationContext
) {
this.webClient = MockMvcWebTestClient
.bindToApplicationContext(applicationContext).build()
}
@Test
fun testShouldRequestGPasSSE() {
val expectedEvent =
ConnectionCheckResult.GPasConnectionCheckResult(true, Instant.now(), Instant.now())
connectionCheckUpdateProducer.tryEmitNext(expectedEvent)
connectionCheckUpdateProducer.emitComplete { _, _ -> true }
val result =
webClient.get().uri("http://localhost/configs/events").accept(TEXT_EVENT_STREAM)
.exchange()
.expectStatus().isOk()
.expectHeader().contentType(TEXT_EVENT_STREAM)
.returnResult(ConnectionCheckResult.GPasConnectionCheckResult::class.java)
StepVerifier.create(result.responseBody)
.expectNext(expectedEvent)
.expectComplete()
.verify()
}
}
}

View File

@@ -0,0 +1,287 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.web
import dev.dnpm.etl.processor.*
import dev.dnpm.etl.processor.config.AppConfiguration
import dev.dnpm.etl.processor.config.AppSecurityConfiguration
import dev.dnpm.etl.processor.monitoring.Report
import dev.dnpm.etl.processor.monitoring.Request
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.monitoring.RequestType
import dev.dnpm.etl.processor.services.RequestService
import org.assertj.core.api.Assertions.assertThat
import org.htmlunit.WebClient
import org.htmlunit.html.HtmlPage
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Nested
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.assertThrows
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.any
import org.mockito.kotlin.whenever
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest
import org.springframework.data.domain.Page
import org.springframework.data.domain.PageImpl
import org.springframework.data.domain.Pageable
import org.springframework.security.test.context.support.WithMockUser
import org.springframework.test.context.ContextConfiguration
import org.springframework.test.context.TestPropertySource
import org.springframework.test.context.bean.override.mockito.MockitoBean
import org.springframework.test.context.junit.jupiter.SpringExtension
import org.springframework.test.web.servlet.MockMvc
import org.springframework.test.web.servlet.get
import org.springframework.test.web.servlet.htmlunit.MockMvcWebClientBuilder
import java.io.IOException
import java.time.Instant
import java.util.*
@WebMvcTest(controllers = [HomeController::class])
@ExtendWith(value = [MockitoExtension::class, SpringExtension::class])
@ContextConfiguration(
classes = [
HomeController::class,
AppConfiguration::class,
AppSecurityConfiguration::class
]
)
@TestPropertySource(
properties = [
"app.pseudonymize.generator=BUILDIN",
"app.security.admin-user=admin",
"app.security.admin-password={noop}very-secret"
]
)
@MockitoBean(
types = [RequestService::class]
)
class HomeControllerTest {
private lateinit var mockMvc: MockMvc
private lateinit var webClient: WebClient
@BeforeEach
fun setup(
@Autowired mockMvc: MockMvc,
@Autowired requestService: RequestService
) {
this.mockMvc = mockMvc
this.webClient = MockMvcWebClientBuilder.mockMvcSetup(mockMvc).build()
whenever(requestService.findAll(any<Pageable>())).thenReturn(Page.empty())
}
@Test
fun testShouldRequestHomePage() {
mockMvc.get("/").andExpect {
status { isOk() }
view { name("index") }
}
}
@Nested
inner class WithRequests {
private lateinit var requestService: RequestService
@BeforeEach
fun setup(
@Autowired requestService: RequestService
) {
this.requestService = requestService
}
@Test
fun testShouldShowHomePage() {
whenever(requestService.findAll(any<Pageable>())).thenReturn(
PageImpl(
listOf(
Request(
2L,
randomRequestId(),
PatientPseudonym("PSEUDO1"),
PatientId("PATIENT1"),
Fingerprint("ashdkasdh"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS
),
Request(
1L,
randomRequestId(),
PatientPseudonym("PSEUDO1"),
PatientId("PATIENT1"),
Fingerprint("asdasdasd"),
RequestType.MTB_FILE,
RequestStatus.ERROR
)
)
)
)
val page = webClient.getPage<HtmlPage>("http://localhost/")
assertThat(page.querySelectorAll("tbody tr")).hasSize(2)
assertThat(page.querySelectorAll("div.notification.info")).isEmpty()
}
@Test
@WithMockUser(username = "admin", roles = ["ADMIN"])
fun testShouldShowRequestDetails() {
val requestId = randomRequestId()
whenever(requestService.findByUuid(anyValueClass())).thenReturn(
Optional.of(
Request(
2L,
requestId,
PatientPseudonym("PSEUDO1"),
PatientId("PATIENT1"),
Fingerprint("ashdkasdh"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS,
Instant.now(),
Report("Test")
)
)
)
val page = webClient.getPage<HtmlPage>("http://localhost/report/${requestId.value}")
assertThat(page.querySelectorAll("tbody tr")).hasSize(1)
assertThat(page.querySelectorAll("div.notification.info")).isEmpty()
}
@Test
@WithMockUser(username = "admin", roles = ["ADMIN"])
fun testShouldShowPatientDetails() {
whenever(requestService.findRequestByPatientId(anyValueClass(), any<Pageable>())).thenReturn(
PageImpl(
listOf(
Request(
2L,
randomRequestId(),
PatientPseudonym("PSEUDO1"),
PatientId("PATIENT1"),
Fingerprint("ashdkasdh"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS
),
Request(
1L,
randomRequestId(),
PatientPseudonym("PSEUDO1"),
PatientId("PATIENT1"),
Fingerprint("asdasdasd"),
RequestType.MTB_FILE,
RequestStatus.ERROR
)
)
)
)
val page = webClient.getPage<HtmlPage>("http://localhost/patient/PSEUDO1")
assertThat(page.querySelectorAll("tbody tr")).hasSize(2)
assertThat(page.querySelectorAll("div.notification.info")).isEmpty()
}
@Test
@WithMockUser(username = "admin", roles = ["ADMIN"])
fun testShouldShowPatientPseudonym() {
whenever(requestService.findRequestByPatientId(anyValueClass(), any<Pageable>())).thenReturn(
PageImpl(
listOf(
Request(
2L,
randomRequestId(),
PatientPseudonym("PSEUDO1"),
PatientId("PATIENT1"),
Fingerprint("ashdkasdh"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS
),
Request(
1L,
randomRequestId(),
PatientPseudonym("PSEUDO1"),
PatientId("PATIENT1"),
Fingerprint("asdasdasd"),
RequestType.MTB_FILE,
RequestStatus.ERROR
)
)
)
)
val page = webClient.getPage<HtmlPage>("http://localhost/patient/PSEUDO1")
assertThat(page.querySelectorAll("h2 > span")).hasSize(1)
assertThat(page.querySelectorAll("h2 > span").first().textContent).isEqualTo("PSEUDO1")
}
}
@Nested
inner class WithoutRequests {
private lateinit var requestService: RequestService
@BeforeEach
fun setup(
@Autowired requestService: RequestService
) {
this.requestService = requestService
whenever(requestService.findAll(any<Pageable>())).thenReturn(Page.empty())
}
@Test
fun testShouldShowHomePage() {
val page = webClient.getPage<HtmlPage>("http://localhost/")
assertThat(page.querySelectorAll("tbody tr")).isEmpty()
assertThat(page.querySelectorAll("div.notification.info")).hasSize(1)
}
@Test
@WithMockUser(username = "admin", roles = ["ADMIN"])
fun testShouldThrowNotFoundExceptionForUnknownReport() {
val requestId = randomRequestId()
whenever(requestService.findByUuid(anyValueClass())).thenReturn(
Optional.empty()
)
assertThrows<IOException> {
webClient.getPage<HtmlPage>("http://localhost/report/${requestId.value}")
}.also {
assertThat(it).hasRootCauseInstanceOf(NotFoundException::class.java)
}
}
@Test
@WithMockUser(username = "admin", roles = ["ADMIN"])
fun testShouldShowEmptyPatientDetails() {
whenever(requestService.findRequestByPatientId(anyValueClass(), any<Pageable>())).thenReturn(Page.empty())
val page = webClient.getPage<HtmlPage>("http://localhost/patient/PSEUDO1")
assertThat(page.querySelectorAll("tbody tr")).isEmpty()
assertThat(page.querySelectorAll("div.notification.info")).hasSize(1)
}
}
}

View File

@@ -0,0 +1,88 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.web
import dev.dnpm.etl.processor.config.AppConfiguration
import dev.dnpm.etl.processor.config.AppSecurityConfiguration
import dev.dnpm.etl.processor.security.TokenService
import org.assertj.core.api.Assertions.assertThat
import org.htmlunit.WebClient
import org.htmlunit.html.HtmlPage
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.junit.jupiter.MockitoExtension
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest
import org.springframework.test.context.ContextConfiguration
import org.springframework.test.context.TestPropertySource
import org.springframework.test.context.bean.override.mockito.MockitoBean
import org.springframework.test.context.junit.jupiter.SpringExtension
import org.springframework.test.web.servlet.MockMvc
import org.springframework.test.web.servlet.get
import org.springframework.test.web.servlet.htmlunit.MockMvcWebClientBuilder
@WebMvcTest(controllers = [LoginController::class])
@ExtendWith(value = [MockitoExtension::class, SpringExtension::class])
@ContextConfiguration(
classes = [
LoginController::class,
AppConfiguration::class,
AppSecurityConfiguration::class
]
)
@TestPropertySource(
properties = [
"app.pseudonymize.generator=BUILDIN",
"app.security.admin-user=admin",
"app.security.admin-password={noop}very-secret",
"app.security.enable-tokens=true"
]
)
@MockitoBean(
types = [TokenService::class]
)
class LoginControllerTest {
private lateinit var mockMvc: MockMvc
private lateinit var webClient: WebClient
@BeforeEach
fun setup(@Autowired mockMvc: MockMvc) {
this.mockMvc = mockMvc
this.webClient = MockMvcWebClientBuilder.mockMvcSetup(mockMvc).build()
}
@Test
fun testShouldRequestLoginPage() {
mockMvc.get("/login").andExpect {
status { isOk() }
view { name("login") }
}
}
@Test
fun testShouldShowLoginForm() {
val page = webClient.getPage<HtmlPage>("http://localhost/login")
assertThat(
page.getElementsByTagName("main").first().firstElementChild.getAttribute("class")
).isEqualTo("login-form")
}
}

View File

@@ -0,0 +1,73 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.web
import dev.dnpm.etl.processor.config.AppConfiguration
import dev.dnpm.etl.processor.config.AppSecurityConfiguration
import org.htmlunit.WebClient
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.junit.jupiter.MockitoExtension
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest
import org.springframework.test.context.ContextConfiguration
import org.springframework.test.context.TestPropertySource
import org.springframework.test.context.junit.jupiter.SpringExtension
import org.springframework.test.web.servlet.MockMvc
import org.springframework.test.web.servlet.get
import org.springframework.test.web.servlet.htmlunit.MockMvcWebClientBuilder
@WebMvcTest(controllers = [StatisticsController::class])
@ExtendWith(value = [MockitoExtension::class, SpringExtension::class])
@ContextConfiguration(
classes = [
StatisticsController::class,
AppConfiguration::class,
AppSecurityConfiguration::class
]
)
@TestPropertySource(
properties = [
"app.pseudonymize.generator=BUILDIN",
"app.security.admin-user=admin",
"app.security.admin-password={noop}very-secret"
]
)
class StatisticsControllerTest {
private lateinit var mockMvc: MockMvc
private lateinit var webClient: WebClient
@BeforeEach
fun setup(@Autowired mockMvc: MockMvc) {
this.mockMvc = mockMvc
this.webClient = MockMvcWebClientBuilder.mockMvcSetup(mockMvc).build()
}
@Test
fun testShouldRequestLoginPage() {
mockMvc.get("/statistics").andExpect {
status { isOk() }
view { name("statistics") }
}
}
}

View File

@@ -0,0 +1,314 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.web
import dev.dnpm.etl.processor.Fingerprint
import dev.dnpm.etl.processor.PatientId
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.config.AppConfiguration
import dev.dnpm.etl.processor.config.AppSecurityConfiguration
import dev.dnpm.etl.processor.monitoring.CountedState
import dev.dnpm.etl.processor.monitoring.Request
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.monitoring.RequestType
import dev.dnpm.etl.processor.randomRequestId
import dev.dnpm.etl.processor.services.RequestService
import org.hamcrest.Matchers.equalTo
import org.hamcrest.Matchers.hasSize
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Nested
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.doAnswer
import org.mockito.kotlin.whenever
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest
import org.springframework.http.MediaType.TEXT_EVENT_STREAM
import org.springframework.test.context.ContextConfiguration
import org.springframework.test.context.TestPropertySource
import org.springframework.test.context.bean.override.mockito.MockitoBean
import org.springframework.test.context.junit.jupiter.SpringExtension
import org.springframework.test.web.reactive.server.WebTestClient
import org.springframework.test.web.servlet.MockMvc
import org.springframework.test.web.servlet.client.MockMvcWebTestClient
import org.springframework.test.web.servlet.get
import org.springframework.web.context.WebApplicationContext
import reactor.core.publisher.Sinks
import reactor.test.StepVerifier
import java.time.Instant
import java.time.ZoneId
import java.time.temporal.ChronoUnit
@WebMvcTest(controllers = [StatisticsRestController::class])
@ExtendWith(value = [MockitoExtension::class, SpringExtension::class])
@ContextConfiguration(
classes = [
StatisticsRestController::class,
AppConfiguration::class,
AppSecurityConfiguration::class
]
)
@TestPropertySource(
properties = [
"app.pseudonymize.generator=BUILDIN",
"app.security.admin-user=admin",
"app.security.admin-password={noop}very-secret"
]
)
@MockitoBean(
types = [RequestService::class]
)
class StatisticsRestControllerTest {
private lateinit var mockMvc: MockMvc
private lateinit var statisticsUpdateProducer: Sinks.Many<Any>
private lateinit var requestService: RequestService
@BeforeEach
fun setup(
@Autowired mockMvc: MockMvc,
@Autowired statisticsUpdateProducer: Sinks.Many<Any>,
@Autowired requestService: RequestService
) {
this.mockMvc = mockMvc
this.statisticsUpdateProducer = statisticsUpdateProducer
this.requestService = requestService
}
@Nested
inner class RequestStatesTest {
@Test
fun testShouldRequestStatesForMtbFiles() {
doAnswer { _ ->
listOf(
CountedState(42, RequestStatus.WARNING),
CountedState(1, RequestStatus.UNKNOWN)
)
}.whenever(requestService).countStates()
mockMvc.get("/statistics/requeststates").andExpect {
status { isOk() }.also {
jsonPath("$", hasSize<Int>(2))
jsonPath("$[0].name", equalTo(RequestStatus.WARNING.name))
jsonPath("$[0].value", equalTo(42))
jsonPath("$[1].name", equalTo(RequestStatus.UNKNOWN.name))
jsonPath("$[1].value", equalTo(1))
}
}
}
@Test
fun testShouldRequestStatesForDeletes() {
doAnswer { _ ->
listOf(
CountedState(42, RequestStatus.SUCCESS),
CountedState(1, RequestStatus.ERROR)
)
}.whenever(requestService).countDeleteStates()
mockMvc.get("/statistics/requeststates?delete=true").andExpect {
status { isOk() }.also {
jsonPath("$", hasSize<Int>(2))
jsonPath("$[0].name", equalTo(RequestStatus.SUCCESS.name))
jsonPath("$[0].value", equalTo(42))
jsonPath("$[1].name", equalTo(RequestStatus.ERROR.name))
jsonPath("$[1].value", equalTo(1))
}
}
}
}
@Nested
inner class PatientRequestStatesTest {
@Test
fun testShouldRequestPatientStatesForMtbFiles() {
doAnswer { _ ->
listOf(
CountedState(42, RequestStatus.WARNING),
CountedState(1, RequestStatus.UNKNOWN)
)
}.whenever(requestService).findPatientUniqueStates()
mockMvc.get("/statistics/requestpatientstates").andExpect {
status { isOk() }.also {
jsonPath("$", hasSize<Int>(2))
jsonPath("$[0].name", equalTo(RequestStatus.WARNING.name))
jsonPath("$[0].value", equalTo(42))
jsonPath("$[1].name", equalTo(RequestStatus.UNKNOWN.name))
jsonPath("$[1].value", equalTo(1))
}
}
}
@Test
fun testShouldRequestPatientStatesForDeletes() {
doAnswer { _ ->
listOf(
CountedState(42, RequestStatus.SUCCESS),
CountedState(1, RequestStatus.ERROR)
)
}.whenever(requestService).findPatientUniqueDeleteStates()
mockMvc.get("/statistics/requestpatientstates?delete=true").andExpect {
status { isOk() }.also {
jsonPath("$", hasSize<Int>(2))
jsonPath("$[0].name", equalTo(RequestStatus.SUCCESS.name))
jsonPath("$[0].value", equalTo(42))
jsonPath("$[1].name", equalTo(RequestStatus.ERROR.name))
jsonPath("$[1].value", equalTo(1))
}
}
}
}
@Nested
inner class LastMonthStatesTest {
@BeforeEach
fun setup() {
val zoneId = ZoneId.of("Europe/Berlin")
doAnswer { _ ->
listOf(
Request(
1,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS,
Instant.now().atZone(zoneId).truncatedTo(ChronoUnit.DAYS).minus(2, ChronoUnit.DAYS).toInstant()
),
Request(
2,
randomRequestId(),
PatientPseudonym("TEST_12345678902"),
PatientId("P2"),
Fingerprint("0123456789abcdef2"),
RequestType.MTB_FILE,
RequestStatus.WARNING,
Instant.now().atZone(zoneId).truncatedTo(ChronoUnit.DAYS).minus(2, ChronoUnit.DAYS).toInstant()
),
Request(
3,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P2"),
Fingerprint("0123456789abcdee1"),
RequestType.DELETE,
RequestStatus.ERROR,
Instant.now().atZone(zoneId).truncatedTo(ChronoUnit.DAYS).minus(1, ChronoUnit.DAYS).toInstant()
),
Request(
4,
randomRequestId(),
PatientPseudonym("TEST_12345678902"),
PatientId("P2"),
Fingerprint("0123456789abcdef2"),
RequestType.MTB_FILE,
RequestStatus.DUPLICATION,
Instant.now().atZone(zoneId).truncatedTo(ChronoUnit.DAYS).minus(1, ChronoUnit.DAYS).toInstant()
),
Request(
5,
randomRequestId(),
PatientPseudonym("TEST_12345678902"),
PatientId("P2"),
Fingerprint("0123456789abcdef2"),
RequestType.DELETE,
RequestStatus.UNKNOWN,
Instant.now().atZone(zoneId).truncatedTo(ChronoUnit.DAYS).toInstant()
),
)
}.whenever(requestService).findAll()
}
@Test
fun testShouldRequestLastMonthForMtbFiles() {
mockMvc.get("/statistics/requestslastmonth").andExpect {
status { isOk() }.also {
jsonPath("$", hasSize<Int>(31))
}.also {
jsonPath("$[28].nameValues.error", equalTo(0))
jsonPath("$[28].nameValues.warning", equalTo(1))
jsonPath("$[28].nameValues.success", equalTo(1))
jsonPath("$[28].nameValues.duplication", equalTo(0))
jsonPath("$[28].nameValues.unknown", equalTo(0))
jsonPath("$[29].nameValues.error", equalTo(0))
jsonPath("$[29].nameValues.warning", equalTo(0))
jsonPath("$[29].nameValues.success", equalTo(0))
jsonPath("$[29].nameValues.duplication", equalTo(1))
jsonPath("$[29].nameValues.unknown", equalTo(0))
}
}
}
@Test
fun testShouldRequestLastMonthForDeletes() {
mockMvc.get("/statistics/requestslastmonth?delete=true").andExpect {
status { isOk() }.also {
jsonPath("$", hasSize<Int>(31))
}.also {
jsonPath("$[29].nameValues.error", equalTo(1))
jsonPath("$[29].nameValues.warning", equalTo(0))
jsonPath("$[29].nameValues.success", equalTo(0))
jsonPath("$[29].nameValues.duplication", equalTo(0))
jsonPath("$[29].nameValues.unknown", equalTo(0))
jsonPath("$[30].nameValues.error", equalTo(0))
jsonPath("$[30].nameValues.warning", equalTo(0))
jsonPath("$[30].nameValues.success", equalTo(0))
jsonPath("$[30].nameValues.duplication", equalTo(0))
jsonPath("$[30].nameValues.unknown", equalTo(1))
}
}
}
}
@Nested
inner class SseTest {
private lateinit var webClient: WebTestClient
@BeforeEach
fun setup(
applicationContext: WebApplicationContext,
) {
this.webClient = MockMvcWebTestClient
.bindToApplicationContext(applicationContext).build()
}
@Test
fun testShouldRequestSSE() {
statisticsUpdateProducer.emitComplete { _, _ -> true }
val result = webClient.get().uri("http://localhost/statistics/events").accept(TEXT_EVENT_STREAM).exchange()
.expectStatus().isOk()
.expectHeader().contentType(TEXT_EVENT_STREAM)
.returnResult(String::class.java)
StepVerifier.create(result.responseBody)
.expectComplete()
.verify()
}
}
}

View File

@@ -0,0 +1,13 @@
package dev.dnpm.etl.processor.consent;
public enum ConsentDomain {
/**
* MII Broad consent
*/
BROAD_CONSENT,
/**
* GenomDe Modellvorhaben §64e
*/
MODELLVORHABEN_64E
}

View File

@@ -0,0 +1,307 @@
package dev.dnpm.etl.processor.consent;
import ca.uhn.fhir.context.FhirContext;
import ca.uhn.fhir.parser.DataFormatException;
import dev.dnpm.etl.processor.config.AppFhirConfig;
import dev.dnpm.etl.processor.config.GIcsConfigProperties;
import org.apache.commons.lang3.StringUtils;
import org.hl7.fhir.r4.model.*;
import org.hl7.fhir.r4.model.Parameters.ParametersParameterComponent;
import org.jetbrains.annotations.NotNull;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.MediaType;
import org.springframework.retry.TerminatedRetryException;
import org.springframework.retry.support.RetryTemplate;
import org.springframework.web.client.RestClientException;
import org.springframework.web.client.RestTemplate;
import org.springframework.web.util.UriComponentsBuilder;
import java.net.URI;
import java.util.Date;
/**
* Service to request Consent from remote gICS installation
*
* @since 0.11
*/
public class GicsConsentService implements IConsentService {
private final Logger log = LoggerFactory.getLogger(GicsConsentService.class);
public static final String IS_CONSENTED_ENDPOINT = "/$isConsented";
public static final String IS_POLICY_STATES_FOR_PERSON_ENDPOINT = "/$currentPolicyStatesForPerson";
private final RetryTemplate retryTemplate;
private final RestTemplate restTemplate;
private final FhirContext fhirContext;
private final GIcsConfigProperties gIcsConfigProperties;
public GicsConsentService(
GIcsConfigProperties gIcsConfigProperties,
RetryTemplate retryTemplate,
RestTemplate restTemplate,
AppFhirConfig appFhirConfig
) {
this.retryTemplate = retryTemplate;
this.restTemplate = restTemplate;
this.fhirContext = appFhirConfig.fhirContext();
this.gIcsConfigProperties = gIcsConfigProperties;
log.info("GicsConsentService initialized...");
}
protected Parameters getFhirRequestParameters(
String personIdentifierValue
) {
var result = new Parameters();
result.addParameter(
new ParametersParameterComponent()
.setName("personIdentifier")
.setValue(
new Identifier()
.setValue(personIdentifierValue)
.setSystem(this.gIcsConfigProperties.getPersonIdentifierSystem())
)
);
result.addParameter(
new ParametersParameterComponent()
.setName("domain")
.setValue(
new StringType()
.setValue(this.gIcsConfigProperties.getBroadConsentDomainName())
)
);
result.addParameter(
new ParametersParameterComponent()
.setName("policy")
.setValue(
new Coding()
.setCode(this.gIcsConfigProperties.getBroadConsentPolicyCode())
.setSystem(this.gIcsConfigProperties.getBroadConsentPolicySystem())
)
);
/*
* is mandatory parameter, but we ignore it via additional configuration parameter
* 'ignoreVersionNumber'.
*/
result.addParameter(
new ParametersParameterComponent()
.setName("version")
.setValue(new StringType().setValue("1.1")
)
);
/* add config parameter with:
* ignoreVersionNumber -> true ->> Reason is we cannot know which policy version each patient
* has possibly signed or not, therefore we are happy with any version found.
* unknownStateIsConsideredAsDecline -> true
*/
var config = new ParametersParameterComponent()
.setName("config")
.addPart(
new ParametersParameterComponent()
.setName("ignoreVersionNumber")
.setValue(new BooleanType().setValue(true))
)
.addPart(
new ParametersParameterComponent()
.setName("unknownStateIsConsideredAsDecline")
.setValue(new BooleanType().setValue(false))
);
result.addParameter(config);
return result;
}
private URI endpointUri(String endpoint) {
assert this.gIcsConfigProperties.getUri() != null;
return UriComponentsBuilder.fromUriString(this.gIcsConfigProperties.getUri()).path(endpoint).build().toUri();
}
private HttpHeaders headersWithHttpBasicAuth() {
assert this.gIcsConfigProperties.getUri() != null;
var headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_XML);
if (
StringUtils.isBlank(this.gIcsConfigProperties.getUsername())
|| StringUtils.isBlank(this.gIcsConfigProperties.getPassword())
) {
return headers;
}
headers.setBasicAuth(this.gIcsConfigProperties.getUsername(), this.gIcsConfigProperties.getPassword());
return headers;
}
protected String callGicsApi(Parameters parameter, String endpoint) {
var parameterAsXml = fhirContext.newXmlParser().encodeResourceToString(parameter);
HttpEntity<String> requestEntity = new HttpEntity<>(parameterAsXml, this.headersWithHttpBasicAuth());
try {
var responseEntity = retryTemplate.execute(
ctx -> restTemplate.exchange(endpointUri(endpoint), HttpMethod.POST, requestEntity, String.class)
);
if (responseEntity.getStatusCode().is2xxSuccessful()) {
return responseEntity.getBody();
} else {
var msg = String.format(
"Trusted party system reached but request failed! code: '%s' response: '%s'",
responseEntity.getStatusCode(), responseEntity.getBody());
log.error(msg);
return null;
}
} catch (RestClientException e) {
var msg = String.format("Get consents status request failed reason: '%s",
e.getMessage());
log.error(msg);
return null;
} catch (TerminatedRetryException terminatedRetryException) {
var msg = String.format(
"Get consents status process has been terminated. termination reason: '%s",
terminatedRetryException.getMessage());
log.error(msg);
return null;
}
}
@Override
public TtpConsentStatus getTtpBroadConsentStatus(String personIdentifierValue) {
var consentStatusResponse = callGicsApi(
getFhirRequestParameters(personIdentifierValue),
GicsConsentService.IS_CONSENTED_ENDPOINT
);
return evaluateConsentResponse(consentStatusResponse);
}
protected Bundle currentConsentForPersonAndTemplate(
String personIdentifierValue,
ConsentDomain consentDomain,
Date requestDate
) {
var requestParameter = buildRequestParameterCurrentPolicyStatesForPerson(
personIdentifierValue,
requestDate,
consentDomain
);
var consentDataSerialized = callGicsApi(requestParameter,
GicsConsentService.IS_POLICY_STATES_FOR_PERSON_ENDPOINT);
if (consentDataSerialized == null) {
// error occurred - should not process further!
throw new IllegalStateException(
"consent data request failed - stopping processing! - try again or fix other problems first.");
}
var iBaseResource = fhirContext.newJsonParser()
.parseResource(consentDataSerialized);
if (iBaseResource instanceof OperationOutcome) {
// log error - very likely a configuration error
String errorMessage =
"Consent request failed! Check outcome:\n " + consentDataSerialized;
log.error(errorMessage);
throw new IllegalStateException(errorMessage);
} else if (iBaseResource instanceof Bundle bundle) {
return bundle;
} else {
String errorMessage = "Consent request failed! Unexpected response received! -> "
+ consentDataSerialized;
log.error(errorMessage);
throw new IllegalStateException(errorMessage);
}
}
@NotNull
private String getConsentDomainName(ConsentDomain targetConsentDomain) {
return switch (targetConsentDomain) {
case BROAD_CONSENT -> gIcsConfigProperties.getBroadConsentDomainName();
case MODELLVORHABEN_64E -> gIcsConfigProperties.getGenomDeConsentDomainName();
};
}
protected Parameters buildRequestParameterCurrentPolicyStatesForPerson(
String personIdentifierValue,
Date requestDate,
ConsentDomain consentDomain
) {
var requestParameter = new Parameters();
requestParameter.addParameter(
new ParametersParameterComponent()
.setName("personIdentifier")
.setValue(
new Identifier()
.setValue(personIdentifierValue)
.setSystem(this.gIcsConfigProperties.getPersonIdentifierSystem())
)
);
requestParameter.addParameter(
new ParametersParameterComponent()
.setName("domain")
.setValue(new StringType().setValue(getConsentDomainName(consentDomain)))
);
Parameters nestedConfigParameters = new Parameters();
nestedConfigParameters
.addParameter(
new ParametersParameterComponent()
.setName("idMatchingType")
.setValue(new Coding()
.setSystem("https://ths-greifswald.de/fhir/CodeSystem/gics/IdMatchingType")
.setCode("AT_LEAST_ONE")
)
)
.addParameter("ignoreVersionNumber", false)
.addParameter("unknownStateIsConsideredAsDecline", false)
.addParameter("requestDate", new DateType().setValue(requestDate));
requestParameter.addParameter(
new ParametersParameterComponent().setName("config").addPart().setResource(nestedConfigParameters)
);
return requestParameter;
}
private TtpConsentStatus evaluateConsentResponse(String consentStatusResponse) {
if (consentStatusResponse == null) {
return TtpConsentStatus.FAILED_TO_ASK;
}
try {
var response = fhirContext.newJsonParser().parseResource(consentStatusResponse);
if (response instanceof Parameters responseParameters) {
var responseValue = responseParameters.getParameter("consented").getValue();
var isConsented = responseValue.castToBoolean(responseValue);
if (!isConsented.hasValue()) {
return TtpConsentStatus.FAILED_TO_ASK;
}
if (isConsented.booleanValue()) {
return TtpConsentStatus.BROAD_CONSENT_GIVEN;
} else {
return TtpConsentStatus.BROAD_CONSENT_MISSING_OR_REJECTED;
}
} else if (response instanceof OperationOutcome outcome) {
log.error("failed to get consent status from ttp. probably configuration error. "
+ "outcome: '{}'", fhirContext.newJsonParser().encodeToString(outcome));
}
} catch (DataFormatException dfe) {
log.error("failed to parse response to FHIR R4 resource.", dfe);
}
return TtpConsentStatus.FAILED_TO_ASK;
}
@Override
public Bundle getConsent(String patientId, Date requestDate, ConsentDomain consentDomain) {
return currentConsentForPersonAndTemplate(patientId, consentDomain, requestDate);
}
}

View File

@@ -0,0 +1,27 @@
package dev.dnpm.etl.processor.consent;
import java.util.Date;
import org.hl7.fhir.r4.model.Bundle;
public interface IConsentService {
/**
* Get broad consent status for a patient identifier
*
* @param personIdentifierValue patient identifier used for consent data
* @return status of broad consent
* @apiNote cannot not differ between not asked and rejected
*
*/
TtpConsentStatus getTtpBroadConsentStatus(String personIdentifierValue);
/**
* Get broad consent policies with respect to a request date
*
* @param personIdentifierValue patient identifier used for consent data
* @param requestDate target date until consent data should be considered
* @return consent policies as bundle; <p>if empty patient has not been asked, yet.</p>
*/
Bundle getConsent(String personIdentifierValue, Date requestDate, ConsentDomain consentDomain);
}

View File

@@ -0,0 +1,31 @@
package dev.dnpm.etl.processor.consent;
import java.util.Date;
import org.hl7.fhir.r4.model.Bundle;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class MtbFileConsentService implements IConsentService {
private static final Logger log = LoggerFactory.getLogger(MtbFileConsentService.class);
public MtbFileConsentService() {
log.info("ConsentCheckFileBased initialized...");
}
@Override
public TtpConsentStatus getTtpBroadConsentStatus(String personIdentifierValue) {
return TtpConsentStatus.UNKNOWN_CHECK_FILE;
}
/**
* EMPTY METHOD: NOT IMPLEMENTED
*
* @return empty bundle
*/
@Override
public Bundle getConsent(String personIdentifierValue, Date requestDate,
ConsentDomain consentDomain) {
return new Bundle();
}
}

View File

@@ -0,0 +1,38 @@
package dev.dnpm.etl.processor.consent;
public enum TtpConsentStatus {
/**
* Valid consent found
*/
BROAD_CONSENT_GIVEN,
/**
* Missing or rejected...actually unknown
*/
BROAD_CONSENT_MISSING_OR_REJECTED,
/**
* No Broad consent policy found
*/
BROAD_CONSENT_MISSING,
/**
* Research policy has been rejected
*/
BROAD_CONSENT_REJECTED,
GENOM_DE_CONSENT_SEQUENCING_PERMIT,
/**
* No GenomDE consent policy found
*/
GENOM_DE_CONSENT_MISSING,
/**
* GenomDE consent policy found, but has been rejected
*/
GENOM_DE_SEQUENCING_REJECTED,
/**
* Consent status is validate via file property 'consent.status'
*/
UNKNOWN_CHECK_FILE,
/**
* Due technical problems consent status is unknown
*/
FAILED_TO_ASK
}

View File

@@ -23,4 +23,6 @@ public interface Generator {
String generate(String id);
String generateGenomDeTan(String id);
}

View File

@@ -21,91 +21,92 @@ package dev.dnpm.etl.processor.pseudonym;
import ca.uhn.fhir.context.FhirContext;
import ca.uhn.fhir.parser.IParser;
import dev.dnpm.etl.processor.config.AppFhirConfig;
import dev.dnpm.etl.processor.config.GPasConfigProperties;
import java.net.URI;
import java.net.URISyntaxException;
import org.apache.commons.lang3.NotImplementedException;
import org.apache.commons.lang3.StringUtils;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.client5.http.impl.io.BasicHttpClientConnectionManager;
import org.apache.hc.client5.http.socket.ConnectionSocketFactory;
import org.apache.hc.client5.http.socket.PlainConnectionSocketFactory;
import org.apache.hc.client5.http.ssl.SSLConnectionSocketFactory;
import org.apache.hc.core5.http.config.Registry;
import org.apache.hc.core5.http.config.RegistryBuilder;
import org.apache.hc.core5.net.URIBuilder;
import org.hl7.fhir.r4.model.Identifier;
import org.hl7.fhir.r4.model.Parameters;
import org.hl7.fhir.r4.model.Parameters.ParametersParameterComponent;
import org.hl7.fhir.r4.model.StringType;
import org.jetbrains.annotations.NotNull;
import org.jetbrains.annotations.Nullable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.*;
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory;
import org.springframework.retry.support.RetryTemplate;
import org.springframework.web.client.HttpClientErrorException.BadRequest;
import org.springframework.web.client.HttpClientErrorException.Unauthorized;
import org.springframework.web.client.RestClientException;
import org.springframework.web.client.RestTemplate;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManagerFactory;
import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.IOException;
import java.security.KeyManagementException;
import java.security.KeyStore;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.cert.CertificateException;
import java.security.cert.CertificateFactory;
import java.security.cert.X509Certificate;
import java.util.Base64;
public class GpasPseudonymGenerator implements Generator {
private final static FhirContext r4Context = FhirContext.forR4();
private final FhirContext r4Context;
private final String gPasUrl;
private final String psnTargetDomain;
private final HttpHeaders httpHeader;
private final RetryTemplate retryTemplate;
private final Logger log = LoggerFactory.getLogger(GpasPseudonymGenerator.class);
private final RestTemplate restTemplate;
private final @NotNull String genomDeTanDomain;
private final @NotNull String pidPsnDomain;
protected final static String createOrGetPsn = "$pseudonymizeAllowCreate";
protected final static String createMultiDomainPsn = "$pseudonymize-secondary";
private final static String SINGLE_PSN_PART_NAME = "pseudonym";
private final static String MULTI_PSN_PART_NAME = "value";
private SSLContext customSslContext;
public GpasPseudonymGenerator(GPasConfigProperties gpasCfg, RetryTemplate retryTemplate) {
public GpasPseudonymGenerator(GPasConfigProperties gpasCfg, RetryTemplate retryTemplate,
RestTemplate restTemplate, AppFhirConfig appFhirConfig) {
this.retryTemplate = retryTemplate;
this.restTemplate = getRestTemplete();
this.restTemplate = restTemplate;
this.gPasUrl = gpasCfg.getUri();
this.psnTargetDomain = gpasCfg.getTarget();
this.pidPsnDomain = gpasCfg.getPatientDomain();
this.genomDeTanDomain = gpasCfg.getGenomDeTanDomain();
this.r4Context = appFhirConfig.fhirContext();
httpHeader = getHttpHeaders(gpasCfg.getUsername(), gpasCfg.getPassword());
try {
if (StringUtils.isNotBlank(gpasCfg.getSslCaLocation())) {
customSslContext = getSslContext(gpasCfg.getSslCaLocation());
log.warn(String.format("%s has been initialized with SSL certificate %s. This is deprecated in favor of including Root CA.",
this.getClass().getName(), gpasCfg.getSslCaLocation()));
}
} catch (IOException | KeyManagementException | KeyStoreException | CertificateException |
NoSuchAlgorithmException e) {
throw new RuntimeException(e);
}
log.debug(String.format("%s has been initialized", this.getClass().getName()));
log.debug("{} has been initialized", this.getClass().getName());
}
@Override
public String generate(String id) {
var gPasRequestBody = getGpasRequestBody(id);
var responseEntity = getGpasPseudonym(gPasRequestBody);
var gPasPseudonymResult = (Parameters) r4Context.newJsonParser()
.parseResource(responseEntity.getBody());
return generate(id, PsnDomainType.SINGLE_PSN_DOMAIN);
}
return unwrapPseudonym(gPasPseudonymResult);
@Override
public String generateGenomDeTan(String id) {
return generate(id, PsnDomainType.MULTI_PSN_DOMAIN);
}
protected String generate(String id, PsnDomainType domainType) {
switch (domainType) {
case SINGLE_PSN_DOMAIN -> {
final var requestBody = createSinglePsnRequestBody(id, pidPsnDomain);
final var responseEntity = getGpasPseudonym(requestBody, createOrGetPsn);
final var gPasPseudonymResult = (Parameters) r4Context.newJsonParser()
.parseResource(responseEntity.getBody());
return unwrapPseudonym(gPasPseudonymResult, SINGLE_PSN_PART_NAME);
}
case MULTI_PSN_DOMAIN -> {
final var requestBody = createMultiPsnRequestBody(id, genomDeTanDomain);
final var responseEntity = getGpasPseudonym(requestBody, createMultiDomainPsn);
final var gPasPseudonymResult = (Parameters) r4Context.newJsonParser()
.parseResource(responseEntity.getBody());
return unwrapPseudonym(gPasPseudonymResult, MULTI_PSN_PART_NAME);
}
}
throw new NotImplementedException(
"give domain type '%s' is unexpected and is currently not supported!".formatted(
domainType));
}
@NotNull
public static String unwrapPseudonym(Parameters gPasPseudonymResult) {
public static String unwrapPseudonym(Parameters gPasPseudonymResult, String targetPartName) {
final var parameters = gPasPseudonymResult.getParameter().stream().findFirst();
if (parameters.isEmpty()) {
@@ -113,7 +114,7 @@ public class GpasPseudonymGenerator implements Generator {
}
final var identifier = (Identifier) parameters.get().getPart().stream()
.filter(a -> a.getName().equals("pseudonym"))
.filter(a -> a.getName().equals(targetPartName))
.findFirst()
.orElseGet(ParametersParameterComponent::new).getValue();
@@ -135,42 +136,80 @@ public class GpasPseudonymGenerator implements Generator {
return psnValue.replaceAll(forbiddenCharsRegex, "_");
}
@NotNull
protected ResponseEntity<String> getGpasPseudonym(String gPasRequestBody) {
protected ResponseEntity<String> getGpasPseudonym(String gPasRequestBody, String apiEndpoint) {
HttpEntity<String> requestEntity = new HttpEntity<>(gPasRequestBody, this.httpHeader);
ResponseEntity<String> responseEntity;
try {
responseEntity = retryTemplate.execute(
ctx -> restTemplate.exchange(gPasUrl, HttpMethod.POST, requestEntity,
var targetUrl = buildRequestUrl(apiEndpoint);
ResponseEntity<String> responseEntity = retryTemplate.execute(
ctx -> restTemplate.exchange(targetUrl, HttpMethod.POST, requestEntity,
String.class));
if (responseEntity.getStatusCode().is2xxSuccessful()) {
log.debug("API request succeeded. Response: {}", responseEntity.getStatusCode());
} else {
log.warn("API request unsuccessful. Response: {}", requestEntity.getBody());
throw new PseudonymRequestFailed("API request unsuccessful gPas unsuccessful.");
return responseEntity;
}
} catch (RestClientException rce) {
if (rce instanceof BadRequest) {
String msg = "gPas or request configuration is incorrect. Please check both."
+ rce.getMessage();
log.debug(
msg);
throw new PseudonymRequestFailed(msg, rce);
}
if (rce instanceof Unauthorized) {
var msg = "gPas access credentials are invalid check your configuration. msg: '%s".formatted(
rce.getMessage());
log.error(msg);
throw new PseudonymRequestFailed(msg, rce);
}
return responseEntity;
} catch (Exception unexpected) {
throw new PseudonymRequestFailed(
"API request due unexpected error unsuccessful gPas unsuccessful.", unexpected);
}
throw new PseudonymRequestFailed(
"API request due unexpected error unsuccessful gPas unsuccessful.");
}
protected String getGpasRequestBody(String id) {
var requestParameters = new Parameters();
protected URI buildRequestUrl(String apiEndpoint) throws URISyntaxException {
var gPasUrl1 = gPasUrl;
if (gPasUrl.lastIndexOf("/") == gPasUrl.length() - 1) {
gPasUrl1 = gPasUrl.substring(0, gPasUrl.length() - 1);
}
var urlBuilder = new URIBuilder(new URI(gPasUrl1)).appendPath(apiEndpoint);
return urlBuilder.build();
}
protected String createSinglePsnRequestBody(String id, String targetDomain) {
final var requestParameters = new Parameters();
requestParameters.addParameter().setName("target")
.setValue(new StringType().setValue(psnTargetDomain));
.setValue(new StringType().setValue(targetDomain));
requestParameters.addParameter().setName("original")
.setValue(new StringType().setValue(id));
final IParser iParser = r4Context.newJsonParser();
return iParser.encodeResourceToString(requestParameters);
}
protected String createMultiPsnRequestBody(String id, String targetDomain) {
final var param = new Parameters();
ParametersParameterComponent targetParam = param.addParameter().setName("original");
targetParam.addPart(
new ParametersParameterComponent().setName("target")
.setValue(new StringType(targetDomain)));
targetParam.addPart(
new ParametersParameterComponent().setName("value").setValue(new StringType(id)));
targetParam
.addPart(new ParametersParameterComponent().setName("count").setValue(
new StringType("1")));
final IParser iParser = r4Context.newJsonParser();
return iParser.encodeResourceToString(param);
}
@NotNull
protected HttpHeaders getHttpHeaders(String gPasUserName, String gPasPassword) {
var headers = new HttpHeaders();
@@ -180,67 +219,7 @@ public class GpasPseudonymGenerator implements Generator {
return headers;
}
String authHeader = gPasUserName + ":" + gPasPassword;
byte[] authHeaderBytes = authHeader.getBytes();
byte[] encodedAuthHeaderBytes = Base64.getEncoder().encode(authHeaderBytes);
String encodedAuthHeader = new String(encodedAuthHeaderBytes);
if (StringUtils.isNotBlank(gPasUserName) && StringUtils.isNotBlank(gPasPassword)) {
headers.set("Authorization", "Basic " + encodedAuthHeader);
}
headers.setBasicAuth(gPasUserName, gPasPassword);
return headers;
}
/**
* Read SSL root certificate and return SSLContext
*
* @param certificateLocation file location to root certificate (PEM)
* @return initialized SSLContext
* @throws IOException file cannot be read
* @throws CertificateException in case we have an invalid certificate of type X.509
* @throws KeyStoreException keystore cannot be initialized
* @throws NoSuchAlgorithmException missing trust manager algorithmus
* @throws KeyManagementException key management failed at init SSLContext
*/
@Nullable
protected SSLContext getSslContext(String certificateLocation)
throws IOException, CertificateException, KeyStoreException, KeyManagementException, NoSuchAlgorithmException {
KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());
FileInputStream fis = new FileInputStream(certificateLocation);
X509Certificate ca = (X509Certificate) CertificateFactory.getInstance("X.509")
.generateCertificate(new BufferedInputStream(fis));
ks.load(null, null);
ks.setCertificateEntry(Integer.toString(1), ca);
TrustManagerFactory tmf = TrustManagerFactory.getInstance(
TrustManagerFactory.getDefaultAlgorithm());
tmf.init(ks);
SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(null, tmf.getTrustManagers(), null);
return sslContext;
}
protected RestTemplate getRestTemplete() {
if (customSslContext == null) {
return new RestTemplate();
}
final var sslsf = new SSLConnectionSocketFactory(customSslContext);
final Registry<ConnectionSocketFactory> socketFactoryRegistry = RegistryBuilder.<ConnectionSocketFactory>create()
.register("https", sslsf).register("http", new PlainConnectionSocketFactory()).build();
final BasicHttpClientConnectionManager connectionManager = new BasicHttpClientConnectionManager(
socketFactoryRegistry);
final CloseableHttpClient httpClient = HttpClients.custom()
.setConnectionManager(connectionManager).build();
final HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(
httpClient);
return new RestTemplate(requestFactory);
}
}

View File

@@ -0,0 +1,12 @@
package dev.dnpm.etl.processor.pseudonym;
public enum PsnDomainType {
/**
* one pseudonym per original value
*/
SINGLE_PSN_DOMAIN,
/**
* multiple pseudonymes for one original value
*/
MULTI_PSN_DOMAIN
}

View File

@@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@@ -21,19 +21,13 @@ package dev.dnpm.etl.processor.config
import dev.dnpm.etl.processor.security.Role
import org.springframework.boot.context.properties.ConfigurationProperties
import org.springframework.boot.context.properties.DeprecatedConfigurationProperty
@ConfigurationProperties(AppConfigProperties.NAME)
data class AppConfigProperties(
var bwhcUri: String?,
@get:DeprecatedConfigurationProperty(
reason = "Deprecated in favor of 'app.pseudonymize.generator'",
replacement = "app.pseudonymize.generator"
)
var pseudonymizer: PseudonymGenerator = PseudonymGenerator.BUILDIN,
var transformations: List<TransformationProperties> = listOf(),
var maxRetryAttempts: Int = 3,
var duplicationDetection: Boolean = true
var duplicationDetection: Boolean = true,
var genomDeTestSubmission: Boolean = false
) {
companion object {
const val NAME = "app"
@@ -53,22 +47,87 @@ data class PseudonymizeConfigProperties(
@ConfigurationProperties(GPasConfigProperties.NAME)
data class GPasConfigProperties(
val uri: String?,
val target: String = "etl-processor",
val patientDomain: String = "etl-processor",
val genomDeTanDomain: String = "ccdn",
val username: String?,
val password: String?,
@get:DeprecatedConfigurationProperty(
reason = "Deprecated in favor of including Root CA"
)
val sslCaLocation: String?
) {
companion object {
const val NAME = "app.pseudonymize.gpas"
}
}
@ConfigurationProperties(ConsentConfigProperties.NAME)
data class ConsentConfigProperties(
var service: ConsentService = ConsentService.NONE
) {
companion object {
const val NAME = "app.consent"
}
}
@ConfigurationProperties(GIcsConfigProperties.NAME)
data class GIcsConfigProperties(
/**
* Base URL to gICS System
*
*/
val uri: String?,
val username: String? = null,
val password: String? = null,
/**
* gICS specific system
* **/
val personIdentifierSystem: String =
"https://ths-greifswald.de/fhir/gics/identifiers/Patienten-ID",
/**
* Domain of broad consent resources
**/
val broadConsentDomainName: String = "MII",
/**
* Domain of Modelvorhaben 64e consent resources
**/
val genomDeConsentDomainName: String = "GenomDE_MV",
/**
* Value to expect in case of positiv consent
*/
val broadConsentPolicyCode: String = "2.16.840.1.113883.3.1937.777.24.5.3.6",
/**
* Consent Policy which should be used for consent check
*/
val broadConsentPolicySystem: String = "urn:oid:2.16.840.1.113883.3.1937.777.24.5.3",
/**
* Value to expect in case of positiv consent
*/
val genomeDePolicyCode: String = "sequencing",
/**
* Consent Policy which should be used for consent check
*/
val genomeDePolicySystem: String = "https://ths-greifswald.de/fhir/CodeSystem/gics/Policy/GenomDE_MV",
/**
* Consent version (fixed version)
*
*/
val genomeDeConsentVersion: String = "2.0"
) {
companion object {
const val NAME = "app.consent.gics"
}
}
@ConfigurationProperties(RestTargetProperties.NAME)
data class RestTargetProperties(
val uri: String?,
val username: String?,
val password: String?
) {
companion object {
const val NAME = "app.rest"
@@ -79,18 +138,8 @@ data class RestTargetProperties(
data class KafkaProperties(
val inputTopic: String?,
val outputTopic: String = "etl-processor",
@get:DeprecatedConfigurationProperty(
reason = "Deprecated",
replacement = "outputTopic"
)
val topic: String = outputTopic,
val outputResponseTopic: String = "${outputTopic}_response",
@get:DeprecatedConfigurationProperty(
reason = "Deprecated",
replacement = "outputResponseTopic"
)
val responseTopic: String = outputResponseTopic,
val groupId: String = "${topic}_group",
val groupId: String = "${outputTopic}_group",
val servers: String = ""
) {
companion object {
@@ -116,8 +165,13 @@ enum class PseudonymGenerator {
GPAS
}
enum class ConsentService {
NONE,
GICS
}
data class TransformationProperties(
val path: String,
val from: String,
val to: String
)
)

View File

@@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@@ -20,21 +20,29 @@
package dev.dnpm.etl.processor.config
import com.fasterxml.jackson.databind.ObjectMapper
import dev.dnpm.etl.processor.consent.MtbFileConsentService
import dev.dnpm.etl.processor.consent.GicsConsentService
import dev.dnpm.etl.processor.consent.IConsentService
import dev.dnpm.etl.processor.monitoring.*
import dev.dnpm.etl.processor.pseudonym.AnonymizingGenerator
import dev.dnpm.etl.processor.pseudonym.Generator
import dev.dnpm.etl.processor.pseudonym.GpasPseudonymGenerator
import dev.dnpm.etl.processor.pseudonym.PseudonymizeService
import dev.dnpm.etl.processor.services.TokenRepository
import dev.dnpm.etl.processor.services.TokenService
import dev.dnpm.etl.processor.security.TokenRepository
import dev.dnpm.etl.processor.security.TokenService
import dev.dnpm.etl.processor.services.ConsentProcessor
import dev.dnpm.etl.processor.services.Transformation
import dev.dnpm.etl.processor.services.TransformationService
import org.slf4j.LoggerFactory
import org.springframework.boot.autoconfigure.condition.AnyNestedCondition
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty
import org.springframework.boot.context.properties.EnableConfigurationProperties
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Conditional
import org.springframework.context.annotation.Configuration
import org.springframework.context.annotation.ConfigurationCondition
import org.springframework.data.jdbc.repository.config.AbstractJdbcConfiguration
import org.springframework.retry.RetryCallback
import org.springframework.retry.RetryContext
import org.springframework.retry.RetryListener
@@ -44,6 +52,7 @@ import org.springframework.retry.support.RetryTemplateBuilder
import org.springframework.scheduling.annotation.EnableScheduling
import org.springframework.security.crypto.password.PasswordEncoder
import org.springframework.security.provisioning.InMemoryUserDetailsManager
import org.springframework.web.client.HttpClientErrorException
import org.springframework.web.client.RestTemplate
import reactor.core.publisher.Sinks
import kotlin.time.Duration.Companion.seconds
@@ -55,7 +64,9 @@ import kotlin.time.toJavaDuration
value = [
AppConfigProperties::class,
PseudonymizeConfigProperties::class,
GPasConfigProperties::class
GPasConfigProperties::class,
ConsentConfigProperties::class,
GIcsConfigProperties::class
]
)
@EnableScheduling
@@ -68,29 +79,31 @@ class AppConfiguration {
return RestTemplate()
}
@Bean
fun appFhirConfig(): AppFhirConfig {
return AppFhirConfig()
}
@ConditionalOnProperty(value = ["app.pseudonymize.generator"], havingValue = "GPAS")
@Bean
fun gpasPseudonymGenerator(configProperties: GPasConfigProperties, retryTemplate: RetryTemplate): Generator {
return GpasPseudonymGenerator(configProperties, retryTemplate)
fun gpasPseudonymGenerator(
configProperties: GPasConfigProperties,
retryTemplate: RetryTemplate,
restTemplate: RestTemplate,
appFhirConfig: AppFhirConfig
): Generator {
logger.info("Selected 'GpasPseudonym Generator'")
return GpasPseudonymGenerator(configProperties, retryTemplate, restTemplate, appFhirConfig)
}
@ConditionalOnProperty(value = ["app.pseudonymize.generator"], havingValue = "BUILDIN", matchIfMissing = true)
@ConditionalOnProperty(
value = ["app.pseudonymize.generator"],
havingValue = "BUILDIN",
matchIfMissing = true
)
@Bean
fun buildinPseudonymGenerator(): Generator {
return AnonymizingGenerator()
}
@ConditionalOnProperty(value = ["app.pseudonymizer"], havingValue = "GPAS")
@ConditionalOnMissingBean
@Bean
fun gpasPseudonymGeneratorOnDeprecatedProperty(configProperties: GPasConfigProperties, retryTemplate: RetryTemplate): Generator {
return GpasPseudonymGenerator(configProperties, retryTemplate)
}
@ConditionalOnProperty(value = ["app.pseudonymizer"], havingValue = "BUILDIN")
@ConditionalOnMissingBean
@Bean
fun buildinPseudonymGeneratorOnDeprecatedProperty(): Generator {
logger.info("Selected 'BUILDIN Pseudonym Generator'")
return AnonymizingGenerator()
}
@@ -103,17 +116,21 @@ class AppConfiguration {
}
@Bean
fun reportService(objectMapper: ObjectMapper): ReportService {
return ReportService(objectMapper)
fun reportService(): ReportService {
return ReportService(getObjectMapper())
}
@Bean
fun getObjectMapper(): ObjectMapper {
return JacksonConfig().objectMapper()
}
@Bean
fun transformationService(
objectMapper: ObjectMapper,
configProperties: AppConfigProperties
): TransformationService {
logger.info("Apply ${configProperties.transformations.size} transformation rules")
return TransformationService(objectMapper, configProperties.transformations.map {
return TransformationService(getObjectMapper(), configProperties.transformations.map {
Transformation.of(it.path) from it.from to it.to
})
}
@@ -122,6 +139,8 @@ class AppConfiguration {
fun retryTemplate(configProperties: AppConfigProperties): RetryTemplate {
return RetryTemplateBuilder()
.notRetryOn(IllegalArgumentException::class.java)
.notRetryOn(HttpClientErrorException.BadRequest::class.java)
.notRetryOn(HttpClientErrorException.UnprocessableEntity::class.java)
.exponentialBackoff(2.seconds.toJavaDuration(), 1.25, 5.seconds.toJavaDuration())
.customPolicy(SimpleRetryPolicy(configProperties.maxRetryAttempts))
.withListener(object : RetryListener {
@@ -130,7 +149,11 @@ class AppConfiguration {
callback: RetryCallback<T, E>,
throwable: Throwable
) {
logger.warn("Error occured: {}. Retrying {}", throwable.message, context.retryCount)
logger.warn(
"Error occured: {}. Retrying {}",
throwable.message,
context.retryCount
)
}
})
.build()
@@ -138,7 +161,11 @@ class AppConfiguration {
@ConditionalOnProperty(value = ["app.security.enable-tokens"], havingValue = "true")
@Bean
fun tokenService(userDetailsManager: InMemoryUserDetailsManager, passwordEncoder: PasswordEncoder, tokenRepository: TokenRepository): TokenService {
fun tokenService(
userDetailsManager: InMemoryUserDetailsManager,
passwordEncoder: PasswordEncoder,
tokenRepository: TokenRepository
): TokenService {
return TokenService(userDetailsManager, passwordEncoder, tokenRepository)
}
@@ -159,7 +186,11 @@ class AppConfiguration {
gPasConfigProperties: GPasConfigProperties,
connectionCheckUpdateProducer: Sinks.Many<ConnectionCheckResult>
): ConnectionCheckService {
return GPasConnectionCheckService(restTemplate, gPasConfigProperties, connectionCheckUpdateProducer)
return GPasConnectionCheckService(
restTemplate,
gPasConfigProperties,
connectionCheckUpdateProducer
)
}
@ConditionalOnProperty(value = ["app.pseudonymizer"], havingValue = "GPAS")
@@ -170,8 +201,81 @@ class AppConfiguration {
gPasConfigProperties: GPasConfigProperties,
connectionCheckUpdateProducer: Sinks.Many<ConnectionCheckResult>
): ConnectionCheckService {
return GPasConnectionCheckService(restTemplate, gPasConfigProperties, connectionCheckUpdateProducer)
return GPasConnectionCheckService(
restTemplate,
gPasConfigProperties,
connectionCheckUpdateProducer
)
}
@Bean
fun jdbcConfiguration(): AbstractJdbcConfiguration {
return AppJdbcConfiguration()
}
@Conditional(GicsEnabledCondition::class)
@Bean
fun gicsConsentService(
gIcsConfigProperties: GIcsConfigProperties,
retryTemplate: RetryTemplate,
restTemplate: RestTemplate,
appFhirConfig: AppFhirConfig
): IConsentService {
return GicsConsentService(
gIcsConfigProperties,
retryTemplate,
restTemplate,
appFhirConfig
)
}
@Conditional(GicsEnabledCondition::class)
@Bean
fun consentProcessor(
configProperties: AppConfigProperties,
gIcsConfigProperties: GIcsConfigProperties,
getObjectMapper: ObjectMapper,
appFhirConfig: AppFhirConfig,
gicsConsentService: IConsentService
): ConsentProcessor {
return ConsentProcessor(
configProperties,
gIcsConfigProperties,
getObjectMapper,
appFhirConfig.fhirContext(),
gicsConsentService
)
}
@Conditional(GicsEnabledCondition::class)
@Bean
fun gIcsConnectionCheckService(
restTemplate: RestTemplate,
gIcsConfigProperties: GIcsConfigProperties,
connectionCheckUpdateProducer: Sinks.Many<ConnectionCheckResult>
): ConnectionCheckService {
return GIcsConnectionCheckService(
restTemplate,
gIcsConfigProperties,
connectionCheckUpdateProducer
)
}
@Bean
@ConditionalOnMissingBean
fun iGetConsentService(): IConsentService {
return MtbFileConsentService()
}
}
class GicsEnabledCondition :
AnyNestedCondition(ConfigurationCondition.ConfigurationPhase.REGISTER_BEAN) {
@ConditionalOnProperty(name = ["app.consent.service"], havingValue = "gics")
@ConditionalOnProperty(name = ["app.consent.gics.uri"])
class OnGicsServiceSelected {
// Just for Condition
}
}

View File

@@ -0,0 +1,16 @@
package dev.dnpm.etl.processor.config
import ca.uhn.fhir.context.FhirContext
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
@Configuration
class AppFhirConfig {
private val fhirCtx: FhirContext = FhirContext.forR4()
@Bean
fun fhirContext(): FhirContext {
return fhirCtx
}
}

View File

@@ -0,0 +1,25 @@
package dev.dnpm.etl.processor.config
import dev.dnpm.etl.processor.Fingerprint
import org.springframework.context.annotation.Configuration
import org.springframework.core.convert.converter.Converter
import org.springframework.data.jdbc.repository.config.AbstractJdbcConfiguration
@Configuration
class AppJdbcConfiguration : AbstractJdbcConfiguration() {
override fun userConverters(): MutableList<*> {
return mutableListOf(StringToFingerprintConverter(), FingerprintToStringConverter())
}
}
class StringToFingerprintConverter : Converter<String, Fingerprint> {
override fun convert(source: String): Fingerprint {
return Fingerprint(source)
}
}
class FingerprintToStringConverter : Converter<Fingerprint, String> {
override fun convert(source: Fingerprint): String {
return source.value
}
}

View File

@@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@@ -20,6 +20,7 @@
package dev.dnpm.etl.processor.config
import com.fasterxml.jackson.databind.ObjectMapper
import dev.dnpm.etl.processor.consent.ConsentEvaluator
import dev.dnpm.etl.processor.input.KafkaInputListener
import dev.dnpm.etl.processor.monitoring.ConnectionCheckResult
import dev.dnpm.etl.processor.monitoring.ConnectionCheckService
@@ -71,7 +72,7 @@ class AppKafkaConfiguration {
kafkaProperties: KafkaProperties,
kafkaResponseProcessor: KafkaResponseProcessor
): KafkaMessageListenerContainer<String, String> {
val containerProperties = ContainerProperties(kafkaProperties.responseTopic)
val containerProperties = ContainerProperties(kafkaProperties.outputResponseTopic)
containerProperties.messageListener = kafkaResponseProcessor
return KafkaMessageListenerContainer(consumerFactory, containerProperties)
}
@@ -100,9 +101,10 @@ class AppKafkaConfiguration {
@ConditionalOnProperty(value = ["app.kafka.input-topic"])
fun kafkaInputListener(
requestProcessor: RequestProcessor,
objectMapper: ObjectMapper
objectMapper: ObjectMapper,
consentEvaluator: ConsentEvaluator
): KafkaInputListener {
return KafkaInputListener(requestProcessor, objectMapper)
return KafkaInputListener(requestProcessor, consentEvaluator, objectMapper)
}
@Bean
@@ -113,4 +115,4 @@ class AppKafkaConfiguration {
return KafkaConnectionCheckService(consumerFactory.createConsumer(), connectionCheckUpdateProducer)
}
}
}

View File

@@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@@ -21,9 +21,10 @@ package dev.dnpm.etl.processor.config
import dev.dnpm.etl.processor.monitoring.ConnectionCheckResult
import dev.dnpm.etl.processor.monitoring.ConnectionCheckService
import dev.dnpm.etl.processor.monitoring.ReportService
import dev.dnpm.etl.processor.monitoring.RestConnectionCheckService
import dev.dnpm.etl.processor.output.MtbFileSender
import dev.dnpm.etl.processor.output.RestMtbFileSender
import dev.dnpm.etl.processor.output.RestDipMtbFileSender
import org.slf4j.LoggerFactory
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty
@@ -52,10 +53,11 @@ class AppRestConfiguration {
fun restMtbFileSender(
restTemplate: RestTemplate,
restTargetProperties: RestTargetProperties,
retryTemplate: RetryTemplate
retryTemplate: RetryTemplate,
reportService: ReportService,
): MtbFileSender {
logger.info("Selected 'RestMtbFileSender'")
return RestMtbFileSender(restTemplate, restTargetProperties, retryTemplate)
logger.info("Selected 'RestDipMtbFileSender'")
return RestDipMtbFileSender(restTemplate, restTargetProperties, retryTemplate, reportService)
}
@Bean

View File

@@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2023 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@@ -21,7 +21,7 @@ package dev.dnpm.etl.processor.config
import dev.dnpm.etl.processor.security.UserRole
import dev.dnpm.etl.processor.security.UserRoleRepository
import dev.dnpm.etl.processor.services.UserRoleService
import dev.dnpm.etl.processor.security.UserRoleService
import org.slf4j.LoggerFactory
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty
import org.springframework.boot.context.properties.EnableConfigurationProperties
@@ -44,6 +44,8 @@ import org.springframework.security.web.SecurityFilterChain
import java.util.*
private const val LOGIN_PATH = "/login"
@Configuration
@EnableConfigurationProperties(
value = [
@@ -85,11 +87,17 @@ class AppSecurityConfiguration(
@Bean
@ConditionalOnProperty(value = ["app.security.enable-oidc"], havingValue = "true")
fun filterChainOidc(http: HttpSecurity, passwordEncoder: PasswordEncoder, userRoleRepository: UserRoleRepository, sessionRegistry: SessionRegistry): SecurityFilterChain {
fun filterChainOidc(
http: HttpSecurity,
passwordEncoder: PasswordEncoder,
userRoleRepository: UserRoleRepository,
sessionRegistry: SessionRegistry
): SecurityFilterChain {
http {
authorizeRequests {
authorizeHttpRequests {
authorize("/configs/**", hasRole("ADMIN"))
authorize("/mtbfile/**", hasAnyRole("MTBFILE"))
authorize("/mtbfile/**", hasAnyRole("MTBFILE", "ADMIN", "USER"))
authorize("/mtb/**", hasAnyRole("MTBFILE", "ADMIN", "USER"))
authorize("/report/**", hasAnyRole("ADMIN", "USER"))
authorize("*.css", permitAll)
authorize("*.ico", permitAll)
@@ -104,15 +112,15 @@ class AppSecurityConfiguration(
realmName = "ETL-Processor"
}
formLogin {
loginPage = "/login"
loginPage = LOGIN_PATH
}
oauth2Login {
loginPage = "/login"
loginPage = LOGIN_PATH
}
sessionManagement {
sessionConcurrency {
maximumSessions = 1
expiredUrl = "/login?expired"
expiredUrl = "$LOGIN_PATH?expired"
}
sessionFixation {
newSession()
@@ -125,13 +133,22 @@ class AppSecurityConfiguration(
@Bean
@ConditionalOnProperty(value = ["app.security.enable-oidc"], havingValue = "true")
fun grantedAuthoritiesMapper(userRoleRepository: UserRoleRepository, appSecurityConfigProperties: SecurityConfigProperties): GrantedAuthoritiesMapper {
fun grantedAuthoritiesMapper(
userRoleRepository: UserRoleRepository,
appSecurityConfigProperties: SecurityConfigProperties
): GrantedAuthoritiesMapper {
return GrantedAuthoritiesMapper { grantedAuthority ->
grantedAuthority.filterIsInstance<OidcUserAuthority>()
.onEach {
val userRole = userRoleRepository.findByUsername(it.userInfo.preferredUsername)
if (userRole.isEmpty) {
userRoleRepository.save(UserRole(null, it.userInfo.preferredUsername, appSecurityConfigProperties.defaultNewUserRole))
userRoleRepository.save(
UserRole(
null,
it.userInfo.preferredUsername,
appSecurityConfigProperties.defaultNewUserRole
)
)
}
}
.map {
@@ -145,9 +162,10 @@ class AppSecurityConfiguration(
@ConditionalOnProperty(value = ["app.security.enable-oidc"], havingValue = "false", matchIfMissing = true)
fun filterChain(http: HttpSecurity, passwordEncoder: PasswordEncoder): SecurityFilterChain {
http {
authorizeRequests {
authorizeHttpRequests {
authorize("/configs/**", hasRole("ADMIN"))
authorize("/mtbfile/**", hasAnyRole("MTBFILE"))
authorize("/mtbfile/**", hasAnyRole("MTBFILE", "ADMIN"))
authorize("/mtb/**", hasAnyRole("MTBFILE", "ADMIN"))
authorize("/report/**", hasRole("ADMIN"))
authorize(anyRequest, permitAll)
}
@@ -155,7 +173,7 @@ class AppSecurityConfiguration(
realmName = "ETL-Processor"
}
formLogin {
loginPage = "/login"
loginPage = LOGIN_PATH
}
csrf { disable() }
}

View File

@@ -0,0 +1,18 @@
package dev.dnpm.etl.processor.config
import com.fasterxml.jackson.core.JsonParser
import com.fasterxml.jackson.databind.DeserializationContext
import com.fasterxml.jackson.databind.JsonDeserializer
import com.fasterxml.jackson.databind.JsonNode
import org.hl7.fhir.r4.model.Consent
class ConsentResourceDeserializer : JsonDeserializer<Consent>() {
override fun deserialize(p: JsonParser?, ctxt: DeserializationContext?): Consent {
val jsonNode = p?.readValueAsTree<JsonNode>()
val json = jsonNode?.toString()
return JacksonConfig.fhirContext().newJsonParser().parseResource(json) as Consent
}
}

View File

@@ -0,0 +1,15 @@
package dev.dnpm.etl.processor.config
import com.fasterxml.jackson.core.JsonGenerator
import com.fasterxml.jackson.databind.JsonSerializer
import com.fasterxml.jackson.databind.SerializerProvider
import org.hl7.fhir.r4.model.Consent
class ConsentResourceSerializer : JsonSerializer<Consent>() {
override fun serialize(
value: Consent, gen: JsonGenerator, serializers: SerializerProvider
) {
val json = JacksonConfig.fhirContext().newJsonParser().encodeResourceToString(value)
gen.writeRawValue(json)
}
}

View File

@@ -0,0 +1,12 @@
package dev.dnpm.etl.processor.config
import com.fasterxml.jackson.databind.module.SimpleModule
import org.hl7.fhir.r4.model.Consent
class FhirResourceModule : SimpleModule() {
init {
addSerializer(Consent::class.java, ConsentResourceSerializer())
addDeserializer(Consent::class.java, ConsentResourceDeserializer())
}
}

View File

@@ -0,0 +1,27 @@
package dev.dnpm.etl.processor.config
import ca.uhn.fhir.context.FhirContext
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.databind.SerializationFeature
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule
@Configuration
class JacksonConfig {
companion object {
var fhirContext: FhirContext = FhirContext.forR4()
@JvmStatic
fun fhirContext(): FhirContext {
return fhirContext
}
}
@Bean
fun objectMapper(): ObjectMapper = ObjectMapper().registerModule(FhirResourceModule())
.disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS).registerModule(
JavaTimeModule()
)
}

View File

@@ -0,0 +1,66 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.consent
import dev.pcvolkmer.mv64e.mtb.ConsentProvision
import dev.pcvolkmer.mv64e.mtb.ModelProjectConsentPurpose
import dev.pcvolkmer.mv64e.mtb.Mtb
import org.springframework.stereotype.Service
/**
* Evaluates consent using provided consent service and file based consent information
*/
@Service
class ConsentEvaluator(
private val consentService: IConsentService
) {
fun check(mtbFile: Mtb): ConsentEvaluation {
val ttpConsentStatus = consentService.getTtpBroadConsentStatus(mtbFile.patient.id)
val consentGiven = ttpConsentStatus == TtpConsentStatus.BROAD_CONSENT_GIVEN
|| ttpConsentStatus == TtpConsentStatus.GENOM_DE_CONSENT_SEQUENCING_PERMIT
// Aktuell nur Modellvorhaben Consent im File
|| ttpConsentStatus == TtpConsentStatus.UNKNOWN_CHECK_FILE && mtbFile.metadata?.modelProjectConsent?.provisions?.any {
it.purpose == ModelProjectConsentPurpose.SEQUENCING
&& it.type == ConsentProvision.PERMIT
} == true
return ConsentEvaluation(ttpConsentStatus, consentGiven)
}
}
data class ConsentEvaluation(private val ttpConsentStatus: TtpConsentStatus, private val consentGiven: Boolean) {
/**
* Checks if any required consent is present
*/
fun hasConsent(): Boolean {
return consentGiven
}
/**
* Returns the consent status
*/
fun getStatus(): TtpConsentStatus {
if (ttpConsentStatus == TtpConsentStatus.UNKNOWN_CHECK_FILE) {
// in case ttp check is disabled - we propagate rejected status anyway
return TtpConsentStatus.BROAD_CONSENT_MISSING_OR_REJECTED
}
return ttpConsentStatus
}
}

View File

@@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@@ -20,29 +20,56 @@
package dev.dnpm.etl.processor.input
import com.fasterxml.jackson.databind.ObjectMapper
import de.ukw.ccc.bwhc.dto.Consent
import de.ukw.ccc.bwhc.dto.MtbFile
import dev.dnpm.etl.processor.CustomMediaType
import dev.dnpm.etl.processor.PatientId
import dev.dnpm.etl.processor.RequestId
import dev.dnpm.etl.processor.consent.ConsentEvaluator
import dev.dnpm.etl.processor.consent.TtpConsentStatus
import dev.dnpm.etl.processor.services.RequestProcessor
import dev.pcvolkmer.mv64e.mtb.Mtb
import org.apache.kafka.clients.consumer.ConsumerRecord
import org.slf4j.LoggerFactory
import org.springframework.http.MediaType
import org.springframework.kafka.listener.MessageListener
import java.nio.charset.Charset
class KafkaInputListener(
private val requestProcessor: RequestProcessor,
private val consentEvaluator: ConsentEvaluator,
private val objectMapper: ObjectMapper
) : MessageListener<String, String> {
private val logger = LoggerFactory.getLogger(KafkaInputListener::class.java)
override fun onMessage(data: ConsumerRecord<String, String>) {
val mtbFile = objectMapper.readValue(data.value(), MtbFile::class.java)
val firstRequestIdHeader = data.headers().headers("requestId")?.firstOrNull()
val requestId = if (null != firstRequestIdHeader) {
String(firstRequestIdHeader.value())
} else {
""
override fun onMessage(record: ConsumerRecord<String, String>) {
when (guessMimeType(record)) {
MediaType.APPLICATION_JSON_VALUE -> handleDnpmV2Message(record)
CustomMediaType.APPLICATION_VND_DNPM_V2_MTB_JSON_VALUE -> handleDnpmV2Message(record)
else -> {
/* ignore other messages */
}
}
}
private fun guessMimeType(record: ConsumerRecord<String, String>): String? {
if (record.headers().headers("contentType").toList().isEmpty()) {
// Fallback if no contentType set (old behavior)
return MediaType.APPLICATION_JSON_VALUE
}
if (mtbFile.consent.status == Consent.Status.ACTIVE) {
return record.headers().headers("contentType")?.firstOrNull()?.value()?.toString(Charset.forName("UTF-8"))
}
private fun handleDnpmV2Message(record: ConsumerRecord<String, String>) {
val mtbFile = objectMapper.readValue(record.value(), Mtb::class.java)
val patientId = PatientId(mtbFile.patient.id)
val firstRequestIdHeader = record.headers().headers("requestId")?.firstOrNull()
val requestId = if (null != firstRequestIdHeader) {
RequestId(String(firstRequestIdHeader.value()))
} else {
RequestId("")
}
if (consentEvaluator.check(mtbFile).hasConsent()) {
logger.debug("Accepted MTB File for processing")
if (requestId.isBlank()) {
requestProcessor.processMtbFile(mtbFile)
@@ -52,10 +79,15 @@ class KafkaInputListener(
} else {
logger.debug("Accepted MTB File and process deletion")
if (requestId.isBlank()) {
requestProcessor.processDeletion(mtbFile.patient.id)
requestProcessor.processDeletion(patientId, TtpConsentStatus.UNKNOWN_CHECK_FILE)
} else {
requestProcessor.processDeletion(mtbFile.patient.id, requestId)
requestProcessor.processDeletion(
patientId,
requestId,
TtpConsentStatus.UNKNOWN_CHECK_FILE
)
}
}
}
}
}

View File

@@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@@ -19,19 +19,23 @@
package dev.dnpm.etl.processor.input
import de.ukw.ccc.bwhc.dto.Consent
import de.ukw.ccc.bwhc.dto.MtbFile
import dev.dnpm.etl.processor.CustomMediaType
import dev.dnpm.etl.processor.PatientId
import dev.dnpm.etl.processor.consent.ConsentEvaluator
import dev.dnpm.etl.processor.consent.TtpConsentStatus
import dev.dnpm.etl.processor.services.RequestProcessor
import dev.pcvolkmer.mv64e.mtb.Mtb
import org.slf4j.LoggerFactory
import org.springframework.http.MediaType
import org.springframework.http.ResponseEntity
import org.springframework.web.bind.annotation.*
@RestController
@RequestMapping(path = ["mtbfile"])
@RequestMapping(path = ["mtbfile", "mtb"])
class MtbFileRestController(
private val requestProcessor: RequestProcessor,
private val consentEvaluator: ConsentEvaluator
) {
private val logger = LoggerFactory.getLogger(MtbFileRestController::class.java)
@GetMapping
@@ -39,23 +43,25 @@ class MtbFileRestController(
return ResponseEntity.ok("Test")
}
@PostMapping
fun mtbFile(@RequestBody mtbFile: MtbFile): ResponseEntity<Void> {
if (mtbFile.consent.status == Consent.Status.ACTIVE) {
logger.debug("Accepted MTB File for processing")
@PostMapping(consumes = [MediaType.APPLICATION_JSON_VALUE, CustomMediaType.APPLICATION_VND_DNPM_V2_MTB_JSON_VALUE])
fun mtbFile(@RequestBody mtbFile: Mtb): ResponseEntity<Unit> {
val consentEvaluation = consentEvaluator.check(mtbFile)
if (consentEvaluation.hasConsent()) {
logger.debug("Accepted MTB File (DNPM V2) for processing")
requestProcessor.processMtbFile(mtbFile)
} else {
logger.debug("Accepted MTB File and process deletion")
requestProcessor.processDeletion(mtbFile.patient.id)
logger.debug("Accepted MTB File (DNPM V2) and process deletion")
val patientId = PatientId(mtbFile.patient.id)
requestProcessor.processDeletion(patientId, consentEvaluation.getStatus())
}
return ResponseEntity.accepted().build()
}
@DeleteMapping(path = ["{patientId}"])
fun deleteData(@PathVariable patientId: String): ResponseEntity<Void> {
fun deleteData(@PathVariable patientId: String): ResponseEntity<Unit> {
logger.debug("Accepted patient ID to process deletion")
requestProcessor.processDeletion(patientId)
requestProcessor.processDeletion(PatientId(patientId), TtpConsentStatus.UNKNOWN_CHECK_FILE)
return ResponseEntity.accepted().build()
}
}
}

View File

@@ -20,28 +20,25 @@
package dev.dnpm.etl.processor.monitoring
import dev.dnpm.etl.processor.config.GIcsConfigProperties
import dev.dnpm.etl.processor.config.GPasConfigProperties
import dev.dnpm.etl.processor.config.RestTargetProperties
import jakarta.annotation.PostConstruct
import org.apache.kafka.clients.consumer.Consumer
import org.apache.kafka.common.errors.TimeoutException
import org.springframework.beans.factory.annotation.Qualifier
import org.springframework.http.HttpEntity
import org.springframework.http.HttpHeaders
import org.springframework.http.HttpMethod
import org.springframework.http.HttpStatus
import org.springframework.http.MediaType
import org.springframework.http.RequestEntity
import org.springframework.http.*
import org.springframework.scheduling.annotation.Scheduled
import org.springframework.web.client.RestTemplate
import org.springframework.web.util.UriComponentsBuilder
import reactor.core.publisher.Sinks
import java.time.Instant
import kotlin.time.Duration.Companion.seconds
import kotlin.time.toJavaDuration
interface ConnectionCheckService {
fun interface ConnectionCheckService {
fun connectionAvailable(): Boolean
fun connectionAvailable(): ConnectionCheckResult
}
@@ -51,9 +48,33 @@ sealed class ConnectionCheckResult {
abstract val available: Boolean
data class KafkaConnectionCheckResult(override val available: Boolean) : ConnectionCheckResult()
data class RestConnectionCheckResult(override val available: Boolean) : ConnectionCheckResult()
data class GPasConnectionCheckResult(override val available: Boolean) : ConnectionCheckResult()
abstract val timestamp: Instant
abstract val lastChange: Instant
data class KafkaConnectionCheckResult(
override val available: Boolean,
override val timestamp: Instant,
override val lastChange: Instant
) : ConnectionCheckResult()
data class RestConnectionCheckResult(
override val available: Boolean,
override val timestamp: Instant,
override val lastChange: Instant
) : ConnectionCheckResult()
data class GPasConnectionCheckResult(
override val available: Boolean,
override val timestamp: Instant,
override val lastChange: Instant
) : ConnectionCheckResult()
data class GIcsConnectionCheckResult(
override val available: Boolean,
override val timestamp: Instant,
override val lastChange: Instant
) : ConnectionCheckResult()
}
class KafkaConnectionCheckService(
@@ -62,25 +83,33 @@ class KafkaConnectionCheckService(
private val connectionCheckUpdateProducer: Sinks.Many<ConnectionCheckResult>
) : OutputConnectionCheckService {
private var connectionAvailable: Boolean = false
private var result = ConnectionCheckResult.KafkaConnectionCheckResult(false, Instant.now(), Instant.now())
@PostConstruct
@Scheduled(cron = "0 * * * * *")
fun check() {
connectionAvailable = try {
null != consumer.listTopics(5.seconds.toJavaDuration())
} catch (e: TimeoutException) {
false
result = try {
val available = null != consumer.listTopics(5.seconds.toJavaDuration())
ConnectionCheckResult.KafkaConnectionCheckResult(
available,
Instant.now(),
if (result.available == available) { result.lastChange } else { Instant.now() }
)
} catch (_: TimeoutException) {
ConnectionCheckResult.KafkaConnectionCheckResult(
false,
Instant.now(),
if (!result.available) { result.lastChange } else { Instant.now() }
)
}
connectionCheckUpdateProducer.emitNext(
ConnectionCheckResult.KafkaConnectionCheckResult(connectionAvailable),
result,
Sinks.EmitFailureHandler.FAIL_FAST
)
}
override fun connectionAvailable(): Boolean {
return this.connectionAvailable
override fun connectionAvailable(): ConnectionCheckResult.KafkaConnectionCheckResult {
return this.result
}
}
@@ -92,27 +121,41 @@ class RestConnectionCheckService(
private val connectionCheckUpdateProducer: Sinks.Many<ConnectionCheckResult>
) : OutputConnectionCheckService {
private var connectionAvailable: Boolean = false
private var result = ConnectionCheckResult.RestConnectionCheckResult(false, Instant.now(), Instant.now())
@PostConstruct
@Scheduled(cron = "0 * * * * *")
fun check() {
connectionAvailable = try {
restTemplate.getForEntity(
restTargetProperties.uri?.replace("/etl/api", "").toString(),
result = try {
val available = restTemplate.getForEntity(
UriComponentsBuilder.fromUriString(restTargetProperties.uri.toString())
.pathSegment("mtb")
.pathSegment("kaplan-meier")
.pathSegment("config")
.toUriString(),
String::class.java
).statusCode == HttpStatus.OK
} catch (e: Exception) {
false
ConnectionCheckResult.RestConnectionCheckResult(
available,
Instant.now(),
if (result.available == available) { result.lastChange } else { Instant.now() }
)
} catch (_: Exception) {
ConnectionCheckResult.RestConnectionCheckResult(
false,
Instant.now(),
if (!result.available) { result.lastChange } else { Instant.now() }
)
}
connectionCheckUpdateProducer.emitNext(
ConnectionCheckResult.RestConnectionCheckResult(connectionAvailable),
result,
Sinks.EmitFailureHandler.FAIL_FAST
)
}
override fun connectionAvailable(): Boolean {
return this.connectionAvailable
override fun connectionAvailable(): ConnectionCheckResult.RestConnectionCheckResult {
return this.result
}
}
@@ -123,40 +166,101 @@ class GPasConnectionCheckService(
private val connectionCheckUpdateProducer: Sinks.Many<ConnectionCheckResult>
) : ConnectionCheckService {
private var connectionAvailable: Boolean = false
private var result = ConnectionCheckResult.GPasConnectionCheckResult(false, Instant.now(), Instant.now())
@PostConstruct
@Scheduled(cron = "0 * * * * *")
fun check() {
connectionAvailable = try {
result = try {
val uri = UriComponentsBuilder.fromUriString(
gPasConfigProperties.uri?.replace("/\$pseudonymizeAllowCreate", "/\$pseudonymize").toString()
)
.queryParam("target", gPasConfigProperties.target)
.queryParam("original", "???")
.build().toUri()
gPasConfigProperties.uri?.replace("/\$pseudonymizeAllowCreate", "/metadata").toString()
).build().toUri()
val headers = HttpHeaders()
headers.contentType = MediaType.APPLICATION_JSON
if (!gPasConfigProperties.username.isNullOrBlank() && !gPasConfigProperties.password.isNullOrBlank()) {
headers.setBasicAuth(gPasConfigProperties.username, gPasConfigProperties.password)
}
restTemplate.exchange(
val available = restTemplate.exchange(
uri,
HttpMethod.GET,
HttpEntity<Void>(headers),
Void::class.java
).statusCode == HttpStatus.OK
} catch (e: Exception) {
false
ConnectionCheckResult.GPasConnectionCheckResult(
available,
Instant.now(),
if (result.available == available) { result.lastChange } else { Instant.now() }
)
} catch (_: Exception) {
ConnectionCheckResult.GPasConnectionCheckResult(
false,
Instant.now(),
if (!result.available) { result.lastChange } else { Instant.now() }
)
}
connectionCheckUpdateProducer.emitNext(
ConnectionCheckResult.GPasConnectionCheckResult(connectionAvailable),
result,
Sinks.EmitFailureHandler.FAIL_FAST
)
}
override fun connectionAvailable(): Boolean {
return this.connectionAvailable
override fun connectionAvailable(): ConnectionCheckResult.GPasConnectionCheckResult {
return this.result
}
}
}
class GIcsConnectionCheckService(
private val restTemplate: RestTemplate,
private val gIcsConfigProperties: GIcsConfigProperties,
@Qualifier("connectionCheckUpdateProducer")
private val connectionCheckUpdateProducer: Sinks.Many<ConnectionCheckResult>
) : ConnectionCheckService {
private var result = ConnectionCheckResult.GIcsConnectionCheckResult(false, Instant.now(), Instant.now())
@PostConstruct
@Scheduled(cron = "0 * * * * *")
fun check() {
result = try {
val uri = UriComponentsBuilder.fromUriString(
gIcsConfigProperties.uri.toString()).path("/metadata").build().toUri()
val headers = HttpHeaders()
headers.contentType = MediaType.APPLICATION_JSON
if (!gIcsConfigProperties.username.isNullOrBlank() && !gIcsConfigProperties.password.isNullOrBlank()) {
headers.setBasicAuth(gIcsConfigProperties.username, gIcsConfigProperties.password)
}
val available = restTemplate.exchange(
uri,
HttpMethod.GET,
HttpEntity<Void>(headers),
Void::class.java
).statusCode == HttpStatus.OK
ConnectionCheckResult.GIcsConnectionCheckResult(
available,
Instant.now(),
if (result.available == available) { result.lastChange } else { Instant.now() }
)
} catch (_: Exception) {
ConnectionCheckResult.GIcsConnectionCheckResult(
false,
Instant.now(),
if (!result.available) { result.lastChange } else { Instant.now() }
)
}
connectionCheckUpdateProducer.emitNext(
result,
Sinks.EmitFailureHandler.FAIL_FAST
)
}
override fun connectionAvailable(): ConnectionCheckResult.GIcsConnectionCheckResult {
return this.result
}
}

View File

@@ -19,11 +19,15 @@
package dev.dnpm.etl.processor.monitoring
import com.fasterxml.jackson.annotation.JsonAlias
import com.fasterxml.jackson.annotation.JsonIgnoreProperties
import com.fasterxml.jackson.annotation.JsonValue
import com.fasterxml.jackson.core.JsonParseException
import com.fasterxml.jackson.databind.JsonMappingException
import com.fasterxml.jackson.databind.ObjectMapper
import dev.dnpm.etl.processor.monitoring.ReportService.Issue
import dev.dnpm.etl.processor.monitoring.ReportService.Severity
import java.util.Optional
class ReportService(
private val objectMapper: ObjectMapper
@@ -54,7 +58,11 @@ class ReportService(
private data class DataQualityReport(val issues: List<Issue>)
@JsonIgnoreProperties(ignoreUnknown = true)
data class Issue(val severity: Severity, val message: String)
data class Issue(
val severity: Severity,
@JsonAlias("details") val message: String,
val path: Optional<String> = Optional.empty()
)
enum class Severity(@JsonValue val value: String) {
FATAL("fatal"),
@@ -62,4 +70,13 @@ class ReportService(
WARNING("warning"),
INFO("info")
}
}
fun List<Issue>.asRequestStatus(): RequestStatus {
val severity = this.minOfOrNull { it.severity }
return when (severity) {
Severity.FATAL, Severity.ERROR -> RequestStatus.ERROR
Severity.WARNING -> RequestStatus.WARNING
else -> RequestStatus.SUCCESS
}
}

View File

@@ -19,10 +19,12 @@
package dev.dnpm.etl.processor.monitoring
import dev.dnpm.etl.processor.*
import org.springframework.data.annotation.Id
import org.springframework.data.domain.Page
import org.springframework.data.domain.Pageable
import org.springframework.data.jdbc.repository.query.Query
import org.springframework.data.relational.core.mapping.Column
import org.springframework.data.relational.core.mapping.Embedded
import org.springframework.data.relational.core.mapping.Table
import org.springframework.data.repository.CrudRepository
@@ -30,26 +32,48 @@ import org.springframework.data.repository.PagingAndSortingRepository
import java.time.Instant
import java.util.*
typealias RequestId = UUID
@Table("request")
data class Request(
@Id val id: Long? = null,
val uuid: String = RequestId.randomUUID().toString(),
val patientId: String,
val pid: String,
val fingerprint: String,
val uuid: RequestId = randomRequestId(),
val patientPseudonym: PatientPseudonym,
val pid: PatientId,
@Column("fingerprint")
val fingerprint: Fingerprint,
val type: RequestType,
var status: RequestStatus,
var processedAt: Instant = Instant.now(),
@Embedded.Nullable var report: Report? = null
)
) {
constructor(
uuid: RequestId,
patientPseudonym: PatientPseudonym,
pid: PatientId,
fingerprint: Fingerprint,
type: RequestType,
status: RequestStatus
) :
this(null, uuid, patientPseudonym, pid, fingerprint, type, status, Instant.now())
constructor(
uuid: RequestId,
patientPseudonym: PatientPseudonym,
pid: PatientId,
fingerprint: Fingerprint,
type: RequestType,
status: RequestStatus,
processedAt: Instant
) :
this(null, uuid, patientPseudonym, pid, fingerprint, type, status, processedAt)
}
@JvmRecord
data class Report(
val description: String,
val dataQualityReport: String = ""
)
@JvmRecord
data class CountedState(
val count: Int,
val status: RequestStatus,
@@ -57,17 +81,17 @@ data class CountedState(
interface RequestRepository : CrudRepository<Request, Long>, PagingAndSortingRepository<Request, Long> {
fun findAllByPatientIdOrderByProcessedAtDesc(patientId: String): List<Request>
fun findAllByPatientPseudonymOrderByProcessedAtDesc(patientId: PatientPseudonym): List<Request>
fun findByUuidEquals(uuid: String): Optional<Request>
fun findByUuidEquals(uuid: RequestId): Optional<Request>
fun findRequestByPatientId(patientId: String, pageable: Pageable): Page<Request>
fun findRequestByPatientPseudonym(patientPseudonym: PatientPseudonym, pageable: Pageable): Page<Request>
@Query("SELECT count(*) AS count, status FROM request WHERE type = 'MTB_FILE' GROUP BY status ORDER BY status, count DESC;")
fun countStates(): List<CountedState>
@Query("SELECT count(*) AS count, status FROM (" +
"SELECT status, rank() OVER (PARTITION BY patient_id ORDER BY processed_at DESC) AS rank FROM request " +
"SELECT status, rank() OVER (PARTITION BY patient_pseudonym ORDER BY processed_at DESC) AS rank FROM request " +
"WHERE type = 'MTB_FILE' AND status NOT IN ('DUPLICATION') " +
") rank WHERE rank = 1 GROUP BY status ORDER BY status, count DESC;")
fun findPatientUniqueStates(): List<CountedState>
@@ -76,7 +100,7 @@ interface RequestRepository : CrudRepository<Request, Long>, PagingAndSortingRep
fun countDeleteStates(): List<CountedState>
@Query("SELECT count(*) AS count, status FROM (" +
"SELECT status, rank() OVER (PARTITION BY patient_id ORDER BY processed_at DESC) AS rank FROM request " +
"SELECT status, rank() OVER (PARTITION BY patient_pseudonym ORDER BY processed_at DESC) AS rank FROM request " +
"WHERE type = 'DELETE'" +
") rank WHERE rank = 1 GROUP BY status ORDER BY status, count DESC;")
fun findPatientUniqueDeleteStates(): List<CountedState>

View File

@@ -24,5 +24,6 @@ enum class RequestStatus(val value: String) {
WARNING("warning"),
ERROR("error"),
UNKNOWN("unknown"),
DUPLICATION("duplication")
DUPLICATION("duplication"),
NO_CONSENT("no-consent")
}

View File

@@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@@ -20,11 +20,14 @@
package dev.dnpm.etl.processor.output
import com.fasterxml.jackson.databind.ObjectMapper
import de.ukw.ccc.bwhc.dto.Consent
import de.ukw.ccc.bwhc.dto.MtbFile
import dev.dnpm.etl.processor.CustomMediaType
import dev.dnpm.etl.processor.config.KafkaProperties
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.pcvolkmer.mv64e.mtb.Mtb
import dev.pcvolkmer.mv64e.mtb.MvhMetadata
import org.apache.kafka.clients.producer.ProducerRecord
import org.slf4j.LoggerFactory
import org.springframework.http.MediaType
import org.springframework.kafka.core.KafkaTemplate
import org.springframework.retry.support.RetryTemplate
@@ -37,14 +40,24 @@ class KafkaMtbFileSender(
private val logger = LoggerFactory.getLogger(KafkaMtbFileSender::class.java)
override fun send(request: MtbFileSender.MtbFileRequest): MtbFileSender.Response {
override fun <T> send(request: MtbFileRequest<T>): MtbFileSender.Response {
return try {
return retryTemplate.execute<MtbFileSender.Response, Exception> {
val result = kafkaTemplate.send(
kafkaProperties.topic,
key(request),
objectMapper.writeValueAsString(Data(request.requestId, request.mtbFile))
)
val record =
ProducerRecord(
kafkaProperties.outputTopic,
key(request),
objectMapper.writeValueAsString(request)
)
when (request) {
is DnpmV2MtbFileRequest -> record.headers()
.add(
"contentType",
CustomMediaType.APPLICATION_VND_DNPM_V2_MTB_JSON_VALUE.toByteArray()
)
}
val result = kafkaTemplate.send(record)
if (result.get() != null) {
logger.debug("Sent file via KafkaMtbFileSender")
MtbFileSender.Response(RequestStatus.UNKNOWN)
@@ -58,24 +71,27 @@ class KafkaMtbFileSender(
}
}
override fun send(request: MtbFileSender.DeleteRequest): MtbFileSender.Response {
val dummyMtbFile = MtbFile.builder()
.withConsent(
Consent.builder()
.withPatient(request.patientId)
.withStatus(Consent.Status.REJECTED)
.build()
)
override fun send(request: DeleteRequest): MtbFileSender.Response {
val dummyMtbFile = Mtb.builder()
.metadata(MvhMetadata())
.build()
return try {
return retryTemplate.execute<MtbFileSender.Response, Exception> {
val result = kafkaTemplate.send(
kafkaProperties.topic,
key(request),
objectMapper.writeValueAsString(Data(request.requestId, dummyMtbFile))
)
val record =
ProducerRecord(
kafkaProperties.outputTopic,
key(request),
// Always use old BwhcV1FileRequest with Consent REJECT
objectMapper.writeValueAsString(
DnpmV2MtbFileRequest(
request.requestId,
dummyMtbFile
)
)
)
val result = kafkaTemplate.send(record)
if (result.get() != null) {
logger.debug("Sent deletion request via KafkaMtbFileSender")
MtbFileSender.Response(RequestStatus.UNKNOWN)
@@ -90,16 +106,14 @@ class KafkaMtbFileSender(
}
override fun endpoint(): String {
return "${this.kafkaProperties.servers} (${this.kafkaProperties.topic}/${this.kafkaProperties.responseTopic})"
return "${this.kafkaProperties.servers} (${this.kafkaProperties.outputTopic}/${this.kafkaProperties.outputResponseTopic})"
}
private fun key(request: MtbFileSender.MtbFileRequest): String {
return "{\"pid\": \"${request.mtbFile.patient.id}\"}"
private fun key(request: MtbRequest): String {
return when (request) {
is DnpmV2MtbFileRequest -> "{\"pid\": \"${request.content.patient.id}\"}"
is DeleteRequest -> "{\"pid\": \"${request.patientId.value}\"}"
else -> throw IllegalArgumentException("Unsupported request type: ${request::class.simpleName}")
}
}
private fun key(request: MtbFileSender.DeleteRequest): String {
return "{\"pid\": \"${request.patientId}\"}"
}
data class Data(val requestId: String, val content: MtbFile)
}
}

View File

@@ -19,23 +19,17 @@
package dev.dnpm.etl.processor.output
import de.ukw.ccc.bwhc.dto.MtbFile
import dev.dnpm.etl.processor.monitoring.RequestStatus
import org.springframework.http.HttpStatusCode
interface MtbFileSender {
fun send(request: MtbFileRequest): Response
fun <T> send(request: MtbFileRequest<T>): Response
fun send(request: DeleteRequest): Response
fun endpoint(): String
data class Response(val status: RequestStatus, val body: String = "")
data class MtbFileRequest(val requestId: String, val mtbFile: MtbFile)
data class DeleteRequest(val requestId: String, val patientId: String)
}
fun Int.asRequestStatus(): RequestStatus {
@@ -49,4 +43,4 @@ fun Int.asRequestStatus(): RequestStatus {
fun HttpStatusCode.asRequestStatus(): RequestStatus {
return this.value().asRequestStatus()
}
}

View File

@@ -0,0 +1,49 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.output
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.RequestId
import dev.pcvolkmer.mv64e.mtb.Mtb
interface MtbRequest {
val requestId: RequestId
}
sealed interface MtbFileRequest<out T> : MtbRequest {
override val requestId: RequestId
val content: T
fun patientPseudonym(): PatientPseudonym
}
data class DnpmV2MtbFileRequest(
override val requestId: RequestId,
override val content: Mtb
) : MtbFileRequest<Mtb> {
override fun patientPseudonym(): PatientPseudonym {
return PatientPseudonym(content.patient.id)
}
}
data class DeleteRequest(
override val requestId: RequestId,
val patientId: PatientPseudonym
) : MtbRequest

View File

@@ -0,0 +1,55 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.output
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.config.RestTargetProperties
import dev.dnpm.etl.processor.monitoring.ReportService
import org.springframework.retry.support.RetryTemplate
import org.springframework.web.client.RestTemplate
import org.springframework.web.util.UriComponentsBuilder
class RestDipMtbFileSender(
restTemplate: RestTemplate,
private val restTargetProperties: RestTargetProperties,
retryTemplate: RetryTemplate,
reportService: ReportService
) : RestMtbFileSender(restTemplate, restTargetProperties, retryTemplate, reportService) {
override fun sendUrl(): String {
return UriComponentsBuilder
.fromUriString(restTargetProperties.uri.toString())
.pathSegment("mtb")
.pathSegment("etl")
.pathSegment("patient-record")
.toUriString()
}
override fun deleteUrl(patientId: PatientPseudonym): String {
return UriComponentsBuilder
.fromUriString(restTargetProperties.uri.toString())
.pathSegment("mtb")
.pathSegment("etl")
.pathSegment("patient")
.pathSegment(patientId.value)
.toUriString()
}
}

View File

@@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@@ -19,62 +19,71 @@
package dev.dnpm.etl.processor.output
import dev.dnpm.etl.processor.CustomMediaType
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.config.RestTargetProperties
import dev.dnpm.etl.processor.monitoring.ReportService
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.monitoring.asRequestStatus
import org.slf4j.LoggerFactory
import org.springframework.http.HttpEntity
import org.springframework.http.HttpHeaders
import org.springframework.http.MediaType
import org.springframework.retry.support.RetryTemplate
import org.springframework.web.client.RestClientException
import org.springframework.web.client.RestClientResponseException
import org.springframework.web.client.RestTemplate
class RestMtbFileSender(
abstract class RestMtbFileSender(
private val restTemplate: RestTemplate,
private val restTargetProperties: RestTargetProperties,
private val retryTemplate: RetryTemplate
private val retryTemplate: RetryTemplate,
private val reportService: ReportService
) : MtbFileSender {
private val logger = LoggerFactory.getLogger(RestMtbFileSender::class.java)
override fun send(request: MtbFileSender.MtbFileRequest): MtbFileSender.Response {
abstract fun sendUrl(): String
abstract fun deleteUrl(patientId: PatientPseudonym): String
override fun <T> send(request: MtbFileRequest<T>): MtbFileSender.Response {
try {
return retryTemplate.execute<MtbFileSender.Response, Exception> {
val headers = HttpHeaders()
headers.contentType = MediaType.APPLICATION_JSON
val entityReq = HttpEntity(request.mtbFile, headers)
val headers = getHttpHeaders(request)
val entityReq = HttpEntity(request.content, headers)
val response = restTemplate.postForEntity(
"${restTargetProperties.uri}/MTBFile",
sendUrl(),
entityReq,
String::class.java
)
if (!response.statusCode.is2xxSuccessful) {
logger.warn("Error sending to remote system: {}", response.body)
return@execute MtbFileSender.Response(
response.statusCode.asRequestStatus(),
reportService.deserialize(response.body).asRequestStatus(),
"Status-Code: ${response.statusCode.value()}"
)
}
logger.debug("Sent file via RestMtbFileSender")
return@execute MtbFileSender.Response(response.statusCode.asRequestStatus(), response.body.orEmpty())
return@execute MtbFileSender.Response(reportService.deserialize(response.body).asRequestStatus(), response.body.orEmpty())
}
} catch (e: IllegalArgumentException) {
logger.error("Not a valid URI to export to: '{}'", restTargetProperties.uri!!)
} catch (e: RestClientException) {
} catch (e: RestClientResponseException) {
logger.info(restTargetProperties.uri!!.toString())
logger.error("Cannot send data to remote system", e)
logger.error("Request data not accepted by remote system", e)
return MtbFileSender.Response(reportService.deserialize(e.responseBodyAsString).asRequestStatus(), e.responseBodyAsString)
}
return MtbFileSender.Response(RequestStatus.ERROR, "Sonstiger Fehler bei der Übertragung")
}
override fun send(request: MtbFileSender.DeleteRequest): MtbFileSender.Response {
override fun send(request: DeleteRequest): MtbFileSender.Response {
try {
return retryTemplate.execute<MtbFileSender.Response, Exception> {
val headers = HttpHeaders()
headers.contentType = MediaType.APPLICATION_JSON
val headers = getHttpHeaders(request)
val entityReq = HttpEntity(null, headers)
restTemplate.delete(
"${restTargetProperties.uri}/Patient/${request.patientId}",
deleteUrl(request.patientId),
entityReq,
String::class.java
)
@@ -94,4 +103,21 @@ class RestMtbFileSender(
return this.restTargetProperties.uri.orEmpty()
}
}
private fun getHttpHeaders(request: MtbRequest): HttpHeaders {
val username = restTargetProperties.username
val password = restTargetProperties.password
val headers = HttpHeaders()
headers.contentType = when (request) {
is DnpmV2MtbFileRequest -> CustomMediaType.APPLICATION_VND_DNPM_V2_MTB_JSON
else -> MediaType.APPLICATION_JSON
}
if (username.isNullOrBlank() || password.isNullOrBlank()) {
return headers
}
headers.setBasicAuth(username, password)
return headers
}
}

View File

@@ -21,9 +21,12 @@ package dev.dnpm.etl.processor.pseudonym
import org.apache.commons.codec.binary.Base32
import org.apache.commons.codec.digest.DigestUtils
import java.security.SecureRandom
class AnonymizingGenerator : Generator {
companion object fun getSecureRandom() : SecureRandom {
return SecureRandom()
}
override fun generate(id: String): String {
return Base32().encodeAsString(DigestUtils.sha256(id))
@@ -31,4 +34,14 @@ class AnonymizingGenerator : Generator {
.lowercase()
}
@OptIn(ExperimentalStdlibApi::class)
override fun generateGenomDeTan(id: String?): String {
val bytes = ByteArray(64 / 2)
getSecureRandom().nextBytes(bytes)
return bytes.joinToString("") { "%02x".format(it) }
}
}

View File

@@ -19,6 +19,8 @@
package dev.dnpm.etl.processor.pseudonym
import dev.dnpm.etl.processor.PatientId
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.config.PseudonymizeConfigProperties
class PseudonymizeService(
@@ -26,13 +28,17 @@ class PseudonymizeService(
private val configProperties: PseudonymizeConfigProperties
) {
fun patientPseudonym(patientId: String): String {
fun patientPseudonym(patientId: PatientId): PatientPseudonym {
return when (generator) {
is GpasPseudonymGenerator -> generator.generate(patientId)
else -> "${configProperties.prefix}_${generator.generate(patientId)}"
is GpasPseudonymGenerator -> PatientPseudonym(generator.generate(patientId.value))
else -> PatientPseudonym("${configProperties.prefix}_${generator.generate(patientId.value)}")
}
}
fun genomDeTan(patientId: PatientId): String {
return generator.generateGenomDeTan(patientId.value)
}
fun prefix(): String {
return configProperties.prefix
}

View File

@@ -19,53 +19,96 @@
package dev.dnpm.etl.processor.pseudonym
import de.ukw.ccc.bwhc.dto.MtbFile
import dev.dnpm.etl.processor.PatientId
import dev.pcvolkmer.mv64e.mtb.ModelProjectConsent
import dev.pcvolkmer.mv64e.mtb.Mtb
import dev.pcvolkmer.mv64e.mtb.MvhMetadata
import org.apache.commons.codec.digest.DigestUtils
/** Replaces patient ID with generated patient pseudonym
*
* @param pseudonymizeService The pseudonymizeService to be used
* @since 0.11.0
*
* @param pseudonymizeService The pseudonymizeService to be used
* @return The MTB file containing patient pseudonymes
*/
infix fun MtbFile.pseudonymizeWith(pseudonymizeService: PseudonymizeService) {
val patientPseudonym = pseudonymizeService.patientPseudonym(this.patient.id)
infix fun Mtb.pseudonymizeWith(pseudonymizeService: PseudonymizeService) {
val patientPseudonym = pseudonymizeService.patientPseudonym(PatientId(this.patient.id)).value
this.episode?.patient = patientPseudonym
this.carePlans?.forEach { it.patient = patientPseudonym }
this.patient.id = patientPseudonym
this.claims?.forEach { it.patient = patientPseudonym }
this.consent?.patient = patientPseudonym
this.claimResponses?.forEach { it.patient = patientPseudonym }
this.diagnoses?.forEach { it.patient = patientPseudonym }
this.ecogStatus?.forEach { it.patient = patientPseudonym }
this.familyMemberDiagnoses?.forEach { it.patient = patientPseudonym }
this.geneticCounsellingRequests?.forEach { it.patient = patientPseudonym }
this.histologyReevaluationRequests?.forEach { it.patient = patientPseudonym }
this.histologyReports?.forEach {
it.patient = patientPseudonym
it.tumorMorphology?.patient = patientPseudonym
this.episodesOfCare?.forEach { it.patient.id = patientPseudonym }
this.carePlans?.forEach {
it.patient.id = patientPseudonym
it.rebiopsyRequests?.forEach { it.patient.id = patientPseudonym }
it.histologyReevaluationRequests?.forEach { it.patient.id = patientPseudonym }
it.medicationRecommendations.forEach { it.patient.id = patientPseudonym }
it.studyEnrollmentRecommendations?.forEach { it.patient.id = patientPseudonym }
it.procedureRecommendations?.forEach { it.patient.id = patientPseudonym }
it.geneticCounselingRecommendation.patient.id = patientPseudonym
}
this.diagnoses?.forEach { it.patient.id = patientPseudonym }
this.guidelineTherapies?.forEach { it.patient.id = patientPseudonym }
this.guidelineProcedures?.forEach { it.patient.id = patientPseudonym }
this.patient.id = patientPseudonym
this.claims?.forEach { it.patient.id = patientPseudonym }
this.claimResponses?.forEach { it.patient.id = patientPseudonym }
this.diagnoses?.forEach { it.patient.id = patientPseudonym }
this.familyMemberHistories?.forEach { it.patient.id = patientPseudonym }
this.histologyReports?.forEach {
it.patient.id = patientPseudonym
it.results.tumorMorphology?.patient?.id = patientPseudonym
it.results.tumorCellContent?.patient?.id = patientPseudonym
}
this.ngsReports?.forEach {
it.patient.id = patientPseudonym
it.results.simpleVariants?.forEach { it.patient.id = patientPseudonym }
it.results.copyNumberVariants?.forEach { it.patient.id = patientPseudonym }
it.results.dnaFusions?.forEach { it.patient.id = patientPseudonym }
it.results.rnaFusions?.forEach { it.patient.id = patientPseudonym }
it.results.tumorCellContent?.patient?.id = patientPseudonym
it.results.brcaness?.patient?.id = patientPseudonym
it.results.tmb?.patient?.id = patientPseudonym
it.results.hrdScore?.patient?.id = patientPseudonym
}
this.ihcReports?.forEach {
it.patient.id = patientPseudonym
it.results.msiMmr?.forEach { it.patient.id = patientPseudonym }
it.results.proteinExpression?.forEach { it.patient.id = patientPseudonym }
}
this.responses?.forEach { it.patient.id = patientPseudonym }
this.specimens?.forEach { it.patient.id = patientPseudonym }
this.priorDiagnosticReports?.forEach { it.patient.id = patientPseudonym }
this.performanceStatus?.forEach { it.patient.id = patientPseudonym }
this.systemicTherapies?.forEach {
it.history?.forEach {
it.patient.id = patientPseudonym
}
}
this.followUps?.forEach {
it.patient.id = patientPseudonym
}
this.msiFindings?.forEach { it -> it.patient.id = patientPseudonym }
this.metadata?.researchConsents?.forEach { it ->
val entry = it ?: return@forEach
if (entry.contains("patient")) {
// here we expect only a patient reference any other data like display
// need to be removed, since may contain unsecure data
entry.remove("patient")
entry["patient"] = mapOf("reference" to "Patient/$patientPseudonym")
}
}
this.lastGuidelineTherapies?.forEach { it.patient = patientPseudonym }
this.molecularPathologyFindings?.forEach { it.patient = patientPseudonym }
this.molecularTherapies?.forEach { molecularTherapy -> molecularTherapy.history.forEach { it.patient = patientPseudonym } }
this.ngsReports?.forEach { it.patient = patientPseudonym }
this.previousGuidelineTherapies?.forEach { it.patient = patientPseudonym }
this.rebiopsyRequests?.forEach { it.patient = patientPseudonym }
this.recommendations?.forEach { it.patient = patientPseudonym }
this.responses?.forEach { it.patient = patientPseudonym }
this.studyInclusionRequests?.forEach { it.patient = patientPseudonym }
this.specimens?.forEach { it.patient = patientPseudonym }
}
/**
* Creates new hash of content IDs with given prefix except for patient IDs
*
* @param pseudonymizeService The pseudonymizeService to be used
* @since 0.11.0
*
* @param pseudonymizeService The pseudonymizeService to be used
* @return The MTB file containing rehashed content IDs
*/
infix fun MtbFile.anonymizeContentWith(pseudonymizeService: PseudonymizeService) {
infix fun Mtb.anonymizeContentWith(pseudonymizeService: PseudonymizeService) {
val prefix = pseudonymizeService.prefix()
fun anonymize(id: String): String {
@@ -73,153 +116,199 @@ infix fun MtbFile.anonymizeContentWith(pseudonymizeService: PseudonymizeService)
return "$prefix$hash"
}
this.episode?.apply {
id = id?.let {
anonymize(it)
this.episodesOfCare?.forEach {
it?.apply { id = id?.let(::anonymize) }
it.diagnoses?.forEach { it ->
it?.id = it.id?.let(::anonymize)
}
}
this.carePlans?.onEach { carePlan ->
carePlan?.apply {
id = id?.let { anonymize(it) }
diagnosis = diagnosis?.let { anonymize(it) }
geneticCounsellingRequest = geneticCounsellingRequest?.let { anonymize(it) }
rebiopsyRequests = rebiopsyRequests.map { it?.let { anonymize(it) } }
recommendations = recommendations.map { it?.let { anonymize(it) } }
studyInclusionRequests = studyInclusionRequests.map { it?.let { anonymize(it) } }
}
}
this.claims?.onEach { claim ->
claim?.apply {
id = id?.let { anonymize(it) }
therapy = therapy?.let { anonymize(it) }
}
}
this.claimResponses?.onEach { claimResponse ->
claimResponse?.apply {
id = id?.let { anonymize(it) }
claim = claim?.let { anonymize(it) }
}
}
this.consent?.apply {
id = id?.let { anonymize(it) }
}
this.diagnoses?.onEach { diagnosis ->
diagnosis?.apply {
id = id?.let { anonymize(it) }
histologyResults = histologyResults?.map { it?.let { anonymize(it) } }
}
}
this.ecogStatus?.onEach { ecogStatus ->
ecogStatus?.apply {
id = id?.let { anonymize(it) }
}
}
this.familyMemberDiagnoses?.onEach { familyMemberDiagnosis ->
familyMemberDiagnosis?.apply {
id = id?.let { anonymize(it) }
}
}
this.geneticCounsellingRequests?.onEach { geneticCounsellingRequest ->
geneticCounsellingRequest?.apply {
id = id?.let { anonymize(it) }
}
}
this.histologyReevaluationRequests?.onEach { histologyReevaluationRequest ->
histologyReevaluationRequest?.apply {
id = id?.let { anonymize(it) }
specimen = specimen?.let { anonymize(it) }
}
}
this.histologyReports?.onEach { histologyReport ->
histologyReport?.apply {
id = id?.let { anonymize(it) }
specimen = specimen?.let { anonymize(it) }
tumorMorphology?.apply {
id = id?.let { anonymize(it) }
specimen = specimen?.let { anonymize(it) }
diagnoses?.forEach { it -> it?.id = it.id?.let(::anonymize) }
geneticCounselingRecommendation?.apply {
id = geneticCounselingRecommendation.id?.let(::anonymize)
}
tumorCellContent?.apply {
id = id?.let { anonymize(it) }
specimen = specimen?.let { anonymize(it) }
rebiopsyRequests?.forEach { it ->
it.id = it.id?.let(::anonymize)
it.tumorEntity?.id = it.tumorEntity?.id?.let(::anonymize)
}
}
}
this.lastGuidelineTherapies?.onEach { lastGuidelineTherapy ->
lastGuidelineTherapy?.apply {
id = id?.let { anonymize(it) }
diagnosis = diagnosis?.let { anonymize(it) }
}
}
this.molecularPathologyFindings?.onEach { molecularPathologyFinding ->
molecularPathologyFinding?.apply {
id = id?.let { anonymize(it) }
specimen = specimen?.let { anonymize(it) }
}
}
this.molecularTherapies?.onEach { molecularTherapy ->
molecularTherapy?.apply {
history?.onEach { history ->
history?.apply {
id = id?.let { anonymize(it) }
basedOn = basedOn?.let { anonymize(it) }
histologyReevaluationRequests?.forEach { it ->
it.id = it?.id?.let(::anonymize)
it.specimen?.id = it.specimen?.id?.let(::anonymize)
}
medicationRecommendations?.forEach { it ->
it.id = it?.id?.let(::anonymize)
it.supportingVariants?.forEach { it ->
it.variant?.id = it.variant?.id?.let(::anonymize)
}
it.reason?.id = it.reason?.id?.let(::anonymize)
}
reason?.id = reason?.id?.let(::anonymize)
studyEnrollmentRecommendations?.forEach { it ->
it?.reason?.id = it.reason?.id?.let(::anonymize)
}
procedureRecommendations?.forEach { it ->
it.id = it?.id?.let(::anonymize)
it.supportingVariants?.forEach { it ->
it.variant?.id = it.variant?.id?.let(::anonymize)
}
it.reason?.id = it.reason?.id?.let(::anonymize)
studyEnrollmentRecommendations?.forEach { it ->
it.id = it?.id?.let(::anonymize)
it.supportingVariants.forEach { it ->
it.variant?.id = it?.variant?.id?.let(::anonymize)
}
responses?.forEach { it ->
it.id = it?.id?.let(::anonymize)
it.id = it?.id?.let(::anonymize)
}
}
}
}
}
this.ngsReports?.onEach { ngsReport ->
ngsReport?.apply {
id = id?.let { anonymize(it) }
specimen = specimen?.let { anonymize(it) }
tumorCellContent?.apply {
id = id?.let { anonymize(it) }
specimen = specimen?.let { anonymize(it) }
}
simpleVariants?.onEach { simpleVariant ->
simpleVariant?.apply {
id = id?.let { anonymize(it) }
}
}
}
this.responses?.forEach { it ->
it?.id = it.id?.let(::anonymize)
it?.therapy?.id = it.therapy?.id?.let(::anonymize)
}
this.previousGuidelineTherapies?.onEach { previousGuidelineTherapy ->
previousGuidelineTherapy?.apply {
id = id?.let { anonymize(it) }
diagnosis = diagnosis?.let { anonymize(it) }
medication.forEach { medication ->
medication?.apply {
id = id?.let { anonymize(it) }
}
}
}
this.diagnoses?.forEach { it ->
it.id = it?.id?.let(::anonymize)
it.histology?.forEach { it -> it.id = it?.id?.let(::anonymize) }
}
this.rebiopsyRequests?.onEach { rebiopsyRequest ->
rebiopsyRequest?.apply {
id = id?.let { anonymize(it) }
specimen = specimen?.let { anonymize(it) }
this.ngsReports?.forEach { it ->
it.id = it?.id?.let(::anonymize)
it.results?.tumorCellContent?.id = it.results.tumorCellContent?.id?.let(::anonymize)
it.results?.tumorCellContent?.specimen?.id =
it.results?.tumorCellContent?.specimen?.id?.let(::anonymize)
it.results?.rnaFusions?.forEach { it ->
it?.id = it.id?.let(::anonymize)
}
}
this.recommendations?.onEach { recommendation ->
recommendation?.apply {
id = id?.let { anonymize(it) }
diagnosis = diagnosis?.let { anonymize(it) }
ngsReport = ngsReport?.let { anonymize(it) }
it.results?.simpleVariants?.forEach { it ->
it?.id = it.id?.let(::anonymize)
it?.transcriptId?.value = it.transcriptId?.value?.let(::anonymize)
}
it.results?.tmb?.id = it.results?.tmb?.id?.let(::anonymize)
it.results?.tmb?.specimen?.id = it.results?.tmb?.specimen?.id?.let(::anonymize)
it.results?.brcaness?.id = it.results?.brcaness?.id?.let(::anonymize)
it.results?.brcaness?.specimen?.id = it.results?.brcaness?.specimen?.id?.let(::anonymize)
it.results?.copyNumberVariants?.forEach { it -> it?.id = it.id?.let(::anonymize) }
it.results?.hrdScore?.id = it.results?.hrdScore?.id?.let(::anonymize)
it.results?.hrdScore?.specimen?.id = it.results?.hrdScore?.specimen?.id?.let(::anonymize)
it.results?.rnaSeqs?.forEach { it -> it?.id = it.id?.let(::anonymize) }
it.results?.dnaFusions?.forEach { it -> it?.id = it.id?.let(::anonymize) }
it.specimen?.id = it?.specimen?.id?.let(::anonymize)
}
this.responses?.onEach { response ->
response?.apply {
id = id?.let { anonymize(it) }
therapy = therapy?.let { anonymize(it) }
this.histologyReports?.forEach { it ->
it.id = it?.id?.let(::anonymize)
it.results?.tumorCellContent?.id = it.results?.tumorCellContent?.id?.let(::anonymize)
it.results?.tumorCellContent?.specimen?.id =
it.results?.tumorCellContent?.specimen?.id?.let(::anonymize)
it.results?.tumorMorphology?.id = it.results?.tumorMorphology?.id?.let(::anonymize)
it.results?.tumorMorphology?.specimen?.id =
it.results?.tumorMorphology?.specimen?.id?.let(::anonymize)
it.specimen?.id = it.specimen?.id?.let(::anonymize)
}
this.claimResponses?.forEach { it ->
it.id = it?.id?.let(::anonymize)
it.claim?.id = it.claim?.id?.let(::anonymize)
}
this.claims?.forEach { it ->
it.id = it?.id?.let(::anonymize)
it.recommendation?.id = it.recommendation?.id?.let(::anonymize)
}
this.familyMemberHistories?.forEach { it -> it.id = it?.id?.let(::anonymize) }
this.guidelineProcedures?.forEach { it ->
it.id = it?.id?.let(::anonymize)
it.reason?.id = it.reason?.id?.let(::anonymize)
it.basedOn?.id = it.basedOn?.id?.let(::anonymize)
}
this.guidelineTherapies?.forEach { it ->
it.id = it?.id?.let(::anonymize)
it.reason?.id = it.reason?.id?.let(::anonymize)
it.basedOn?.id = it.basedOn?.id?.let(::anonymize)
}
this.ihcReports?.forEach { it ->
it.id = it?.id?.let(::anonymize)
it.specimen?.id = it.specimen?.id?.let(::anonymize)
it.results.proteinExpression.forEach { it -> it?.id = it.id.let(::anonymize) }
}
this.msiFindings?.forEach { it ->
it.id = it?.id?.let(::anonymize)
it.specimen?.id = it.specimen?.id?.let(::anonymize)
}
this.performanceStatus?.forEach { it -> it.id = it?.id?.let(::anonymize) }
this.priorDiagnosticReports?.forEach { it ->
it.id = it?.id?.let(::anonymize)
it.specimen?.id = it.specimen?.id?.let(::anonymize)
}
this.specimens?.forEach { it ->
it.id = it?.id?.let(::anonymize)
it.diagnosis?.id = it.diagnosis?.id?.let(::anonymize)
}
this.systemicTherapies?.forEach { it ->
it.history?.forEach { it ->
it.id = it?.id?.let(::anonymize)
it.reason?.id = it.reason?.id?.let(::anonymize)
it.basedOn?.id = it.basedOn?.id?.let(::anonymize)
}
}
this.studyInclusionRequests?.onEach { studyInclusionRequest ->
studyInclusionRequest?.apply {
id = id?.let { anonymize(it) }
reason = reason?.let { anonymize(it) }
}
}
fun Mtb.ensureMetaDataIsInitialized() {
// init metadata if necessary
if (this.metadata == null) {
val mvhMetadata = MvhMetadata.builder().build()
this.metadata = mvhMetadata
}
this.specimens?.onEach { specimen ->
specimen?.apply {
id = id?.let { anonymize(it) }
}
if (this.metadata.researchConsents == null) {
this.metadata.researchConsents = mutableListOf()
}
}
if (this.metadata.modelProjectConsent == null) {
this.metadata.modelProjectConsent = ModelProjectConsent()
this.metadata.modelProjectConsent.provisions = mutableListOf()
} else if (this.metadata.modelProjectConsent.provisions != null) {
// make sure list can be changed
this.metadata.modelProjectConsent.provisions =
this.metadata.modelProjectConsent.provisions.toMutableList()
}
}
infix fun Mtb.addGenomDeTan(pseudonymizeService: PseudonymizeService) {
this.metadata.transferTan = pseudonymizeService.genomDeTan(PatientId(this.patient.id))
}

View File

@@ -17,7 +17,7 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.services
package dev.dnpm.etl.processor.security
import jakarta.annotation.PostConstruct
import org.springframework.data.annotation.Id

View File

@@ -17,11 +17,8 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.services
package dev.dnpm.etl.processor.security
import dev.dnpm.etl.processor.security.Role
import dev.dnpm.etl.processor.security.UserRole
import dev.dnpm.etl.processor.security.UserRoleRepository
import org.springframework.data.repository.findByIdOrNull
import org.springframework.security.core.session.SessionRegistry
import org.springframework.security.oauth2.core.oidc.user.OidcUser

View File

@@ -0,0 +1,273 @@
package dev.dnpm.etl.processor.services
import ca.uhn.fhir.context.FhirContext
import com.fasterxml.jackson.core.JsonProcessingException
import com.fasterxml.jackson.core.type.TypeReference
import com.fasterxml.jackson.databind.ObjectMapper
import dev.dnpm.etl.processor.config.AppConfigProperties
import dev.dnpm.etl.processor.config.GIcsConfigProperties
import dev.dnpm.etl.processor.consent.ConsentDomain
import dev.dnpm.etl.processor.consent.IConsentService
import dev.dnpm.etl.processor.consent.MtbFileConsentService
import dev.dnpm.etl.processor.pseudonym.ensureMetaDataIsInitialized
import dev.pcvolkmer.mv64e.mtb.*
import org.apache.commons.lang3.NotImplementedException
import org.hl7.fhir.r4.model.*
import org.hl7.fhir.r4.model.Bundle.BundleEntryComponent
import org.hl7.fhir.r4.model.Coding
import org.hl7.fhir.r4.model.Consent.ConsentState
import org.hl7.fhir.r4.model.Consent.ProvisionComponent
import org.slf4j.Logger
import org.slf4j.LoggerFactory
import org.springframework.stereotype.Service
import java.io.IOException
import java.time.Clock
import java.time.Instant
import java.util.*
@Service
class ConsentProcessor(
private val appConfigProperties: AppConfigProperties,
private val gIcsConfigProperties: GIcsConfigProperties,
private val objectMapper: ObjectMapper,
private val fhirContext: FhirContext,
private val consentService: IConsentService
) {
private var logger: Logger = LoggerFactory.getLogger("ConsentProcessor")
/**
* In case an instance of {@link ICheckConsent} is active, consent will be embedded and checked.
*
* Logic:
* * <c>true</c> IF consent check is disabled.
* * <c>true</c> IF broad consent (BC) has been given.
* * <c>true</c> BC has been asked AND declined but genomDe consent has been consented.
* * ELSE <c>false</c> is returned.
*
* @param mtbFile File v2 (will be enriched with consent data)
* @return true if consent is given
*
*/
fun consentGatedCheckAndTryEmbedding(mtbFile: Mtb): Boolean {
if (consentService is MtbFileConsentService) {
// consent check is disabled
return true
}
mtbFile.ensureMetaDataIsInitialized()
val personIdentifierValue = mtbFile.patient.id
val requestDate = Date.from(Instant.now(Clock.systemUTC()))
// 1. Broad consent Entry exists?
// 1.1. -> yes and research consent is given -> send mtb file
// 1.2. -> no -> return status error - consent has not been asked
// 2. -> Broad consent found but rejected -> is GenomDe consent provision 'sequencing' given?
// 2.1 -> yes -> send mtb file
// 2.2 -> no -> warn/info no consent given
/*
* broad consent
*/
val broadConsent = consentService.getConsent(
personIdentifierValue, requestDate, ConsentDomain.BROAD_CONSENT
)
val broadConsentHasBeenAsked = broadConsent.entry.isNotEmpty()
// fast exit - if patient has not been asked, we can skip and exit
if (!broadConsentHasBeenAsked) return false
val genomeDeConsent = consentService.getConsent(
personIdentifierValue, requestDate, ConsentDomain.MODELLVORHABEN_64E
)
addGenomeDbProvisions(mtbFile, genomeDeConsent)
if (genomeDeConsent.entry.isNotEmpty()) setGenomDeSubmissionType(mtbFile)
embedBroadConsentResources(mtbFile, broadConsent)
val broadConsentStatus = getProvisionTypeByPolicyCode(
broadConsent, requestDate, ConsentDomain.BROAD_CONSENT
)
val genomDeSequencingStatus = getProvisionTypeByPolicyCode(
genomeDeConsent, requestDate, ConsentDomain.MODELLVORHABEN_64E
)
if (Consent.ConsentProvisionType.NULL == broadConsentStatus) {
// bc not asked
return false
}
if (Consent.ConsentProvisionType.PERMIT == broadConsentStatus || Consent.ConsentProvisionType.PERMIT == genomDeSequencingStatus) return true
return false
}
fun embedBroadConsentResources(mtbFile: Mtb, broadConsent: Bundle) {
for (entry in broadConsent.entry) {
val resource = entry.resource
if (resource is Consent) {
// since jackson convertValue does not work here,
// we need another step to back to string, before we convert to object map
val asJsonString = fhirContext.newJsonParser().encodeResourceToString(resource)
try {
val mapOfJson: HashMap<String?, Any?>? =
objectMapper.readValue<HashMap<String?, Any?>?>(
asJsonString, object : TypeReference<HashMap<String?, Any?>?>() {})
mtbFile.metadata.researchConsents.add(mapOfJson)
} catch (e: JsonProcessingException) {
throw RuntimeException(e)
}
}
}
}
fun addGenomeDbProvisions(mtbFile: Mtb, consentGnomeDe: Bundle) {
for (entry in consentGnomeDe.entry) {
val resource = entry.resource
if (resource !is Consent) {
continue
}
// We expect only one provision in collection, therefore get first or none
val provisions = resource.provision.provision
if (provisions.isEmpty()) {
continue
}
val provisionComponent: ProvisionComponent = provisions.first()
var provisionCode: String? = null
if (provisionComponent.code != null && provisionComponent.code.isNotEmpty()) {
val codableConcept: CodeableConcept = provisionComponent.code.first()
if (codableConcept.coding != null && codableConcept.coding.isNotEmpty()) {
provisionCode = codableConcept.coding.first().code
}
}
if (provisionCode != null) {
try {
val modelProjectConsentPurpose =
ModelProjectConsentPurpose.forValue(provisionCode)
if (ModelProjectConsentPurpose.SEQUENCING == modelProjectConsentPurpose) {
// CONVENTION: wrapping date is date of SEQUENCING consent
mtbFile.metadata.modelProjectConsent.date = resource.dateTime
}
val provision = Provision.builder()
.type(ConsentProvision.valueOf(provisionComponent.type.name))
.date(provisionComponent.period.start)
.purpose(modelProjectConsentPurpose).build()
mtbFile.metadata.modelProjectConsent.provisions.add(provision)
} catch (ioe: IOException) {
logger.error(
"Provision code '$provisionCode' is unknown and cannot be mapped.",
ioe.toString()
)
}
}
if (mtbFile.metadata.modelProjectConsent.provisions.isNotEmpty()) {
mtbFile.metadata.modelProjectConsent.version =
gIcsConfigProperties.genomeDeConsentVersion
}
}
}
private fun setGenomDeSubmissionType(mtbFile: Mtb) {
if (appConfigProperties.genomDeTestSubmission) {
mtbFile.metadata.type = MvhSubmissionType.TEST
logger.info("genomeDe submission mit TEST")
} else {
mtbFile.metadata.type = when (mtbFile.metadata.type) {
null -> MvhSubmissionType.INITIAL
else -> mtbFile.metadata.type
}
}
}
/**
* @param consentBundle consent resource
* @param requestDate date which must be within validation period of provision
* @return type of provision, will be [org.hl7.fhir.r4.model.Consent.ConsentProvisionType.NULL] if none is found.
*/
fun getProvisionTypeByPolicyCode(
consentBundle: Bundle, requestDate: Date?, consentDomain: ConsentDomain
): Consent.ConsentProvisionType {
val code: String?
val system: String?
if (ConsentDomain.BROAD_CONSENT == consentDomain) {
code = gIcsConfigProperties.broadConsentPolicyCode
system = gIcsConfigProperties.broadConsentPolicySystem
} else if (ConsentDomain.MODELLVORHABEN_64E == consentDomain) {
code = gIcsConfigProperties.genomeDePolicyCode
system = gIcsConfigProperties.genomeDePolicySystem
} else {
throw NotImplementedException("unknown consent domain " + consentDomain.name)
}
val provisionTypeByPolicyCode = getProvisionTypeByPolicyCode(
consentBundle, code, system, requestDate
)
return provisionTypeByPolicyCode
}
/**
* @param consentBundle consent resource
* @param targetCode policyRule and provision code value
* @param targetSystem policyRule and provision system value
* @param requestDate date which must be within validation period of provision
* @return type of provision, will be [org.hl7.fhir.r4.model.Consent.ConsentProvisionType.NULL] if none is found.
*/
fun getProvisionTypeByPolicyCode(
consentBundle: Bundle, targetCode: String?, targetSystem: String?, requestDate: Date?
): Consent.ConsentProvisionType {
val entriesOfInterest = consentBundle.entry.filter { entry ->
val isConsentResource =
entry.resource.isResource && entry.resource.resourceType == ResourceType.Consent
val consentIsActive = (entry.resource as Consent).status == ConsentState.ACTIVE
isConsentResource && consentIsActive && checkCoding(
targetCode, targetSystem, (entry.resource as Consent).policyRule.coding
) && isRequestDateInRange(requestDate, (entry.resource as Consent).provision.period)
}.map { entry: BundleEntryComponent ->
val consent = (entry.getResource() as Consent)
consent.provision.provision.filter { subProvision ->
isRequestDateInRange(requestDate, subProvision.period)
// search coding entries of current provision for code and system
subProvision.code.map { c -> c.coding }.flatten().firstOrNull { code ->
targetCode.equals(code.code) && targetSystem.equals(code.system)
} != null
}.map { subProvision ->
subProvision
}
}.flatten()
if (entriesOfInterest.isNotEmpty()) {
return entriesOfInterest.first().type
}
return Consent.ConsentProvisionType.NULL
}
fun checkCoding(
researchAllowedPolicyOid: String?,
researchAllowedPolicySystem: String?,
policyRules: Collection<Coding>
): Boolean {
return policyRules.find { code ->
researchAllowedPolicySystem.equals(code.getSystem()) && (researchAllowedPolicyOid.equals(
code.getCode()
))
} != null
}
fun isRequestDateInRange(requestDate: Date?, provPeriod: Period): Boolean {
val isRequestDateAfterOrEqualStart = provPeriod.start.compareTo(requestDate)
val isRequestDateBeforeOrEqualEnd = provPeriod.end.compareTo(requestDate)
return isRequestDateAfterOrEqualStart <= 0 && isRequestDateBeforeOrEqualEnd >= 0
}
}

View File

@@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2023 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@@ -20,18 +20,28 @@
package dev.dnpm.etl.processor.services
import com.fasterxml.jackson.databind.ObjectMapper
import de.ukw.ccc.bwhc.dto.MtbFile
import dev.dnpm.etl.processor.*
import dev.dnpm.etl.processor.config.AppConfigProperties
import dev.dnpm.etl.processor.consent.TtpConsentStatus
import dev.dnpm.etl.processor.monitoring.Report
import dev.dnpm.etl.processor.monitoring.Request
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.monitoring.RequestType
import dev.dnpm.etl.processor.output.DeleteRequest
import dev.dnpm.etl.processor.output.DnpmV2MtbFileRequest
import dev.dnpm.etl.processor.output.MtbFileRequest
import dev.dnpm.etl.processor.output.MtbFileSender
import dev.dnpm.etl.processor.pseudonym.PseudonymizeService
import dev.dnpm.etl.processor.pseudonym.addGenomDeTan
import dev.dnpm.etl.processor.pseudonym.anonymizeContentWith
import dev.dnpm.etl.processor.pseudonym.pseudonymizeWith
import dev.pcvolkmer.mv64e.mtb.ConsentProvision
import dev.pcvolkmer.mv64e.mtb.ModelProjectConsentPurpose
import dev.pcvolkmer.mv64e.mtb.Mtb
import org.apache.commons.codec.binary.Base32
import org.apache.commons.codec.digest.DigestUtils
import org.slf4j.Logger
import org.slf4j.LoggerFactory
import org.springframework.context.ApplicationEventPublisher
import org.springframework.stereotype.Service
import java.time.Instant
@@ -45,38 +55,63 @@ class RequestProcessor(
private val requestService: RequestService,
private val objectMapper: ObjectMapper,
private val applicationEventPublisher: ApplicationEventPublisher,
private val appConfigProperties: AppConfigProperties
private val appConfigProperties: AppConfigProperties,
private val consentProcessor: ConsentProcessor?
) {
fun processMtbFile(mtbFile: MtbFile) {
processMtbFile(mtbFile, UUID.randomUUID().toString())
private var logger: Logger = LoggerFactory.getLogger("RequestProcessor")
fun processMtbFile(mtbFile: Mtb) {
processMtbFile(mtbFile, randomRequestId())
}
fun processMtbFile(mtbFile: MtbFile, requestId: String) {
val pid = mtbFile.patient.id
mtbFile pseudonymizeWith pseudonymizeService
mtbFile anonymizeContentWith pseudonymizeService
fun processMtbFile(mtbFile: Mtb, requestId: RequestId) {
val pid = PatientId(extractPatientIdentifier(mtbFile))
val request = MtbFileSender.MtbFileRequest(requestId, transformationService.transform(mtbFile))
val isConsentOk =
consentProcessor != null && consentProcessor.consentGatedCheckAndTryEmbedding(mtbFile) || consentProcessor == null
if (isConsentOk) {
if (isGenomDeConsented(mtbFile)) {
mtbFile addGenomDeTan pseudonymizeService
}
mtbFile pseudonymizeWith pseudonymizeService
mtbFile anonymizeContentWith pseudonymizeService
val request = DnpmV2MtbFileRequest(requestId, transformationService.transform(mtbFile))
saveAndSend(request, pid)
} else {
logger.warn("consent check failed file will not be processed further!")
applicationEventPublisher.publishEvent(
ResponseEvent(
requestId, Instant.now(), RequestStatus.NO_CONSENT
)
)
}
}
private fun isGenomDeConsented(mtbFile: Mtb): Boolean {
val isModelProjectConsented = mtbFile.metadata?.modelProjectConsent?.provisions?.any { p ->
p.purpose == ModelProjectConsentPurpose.SEQUENCING && p.type == ConsentProvision.PERMIT
} == true
return isModelProjectConsented
}
private fun <T> saveAndSend(request: MtbFileRequest<T>, pid: PatientId) {
requestService.save(
Request(
uuid = requestId,
patientId = request.mtbFile.patient.id,
pid = pid,
fingerprint = fingerprint(request.mtbFile),
status = RequestStatus.UNKNOWN,
type = RequestType.MTB_FILE
request.requestId,
request.patientPseudonym(),
pid,
fingerprint(request),
RequestType.MTB_FILE,
RequestStatus.UNKNOWN
)
)
if (appConfigProperties.duplicationDetection && isDuplication(mtbFile)) {
if (appConfigProperties.duplicationDetection && isDuplication(request)) {
applicationEventPublisher.publishEvent(
ResponseEvent(
requestId,
Instant.now(),
RequestStatus.DUPLICATION
request.requestId, Instant.now(), RequestStatus.DUPLICATION
)
)
return
@@ -86,67 +121,78 @@ class RequestProcessor(
applicationEventPublisher.publishEvent(
ResponseEvent(
requestId,
request.requestId,
Instant.now(),
responseStatus.status,
when (responseStatus.status) {
RequestStatus.WARNING -> Optional.of(responseStatus.body)
RequestStatus.ERROR, RequestStatus.WARNING -> Optional.of(responseStatus.body)
else -> Optional.empty()
}
)
)
}
private fun isDuplication(pseudonymizedMtbFile: MtbFile): Boolean {
private fun <T> isDuplication(pseudonymizedMtbFileRequest: MtbFileRequest<T>): Boolean {
val patientPseudonym = when (pseudonymizedMtbFileRequest) {
is DnpmV2MtbFileRequest -> PatientPseudonym(pseudonymizedMtbFileRequest.content.patient.id)
}
val lastMtbFileRequestForPatient =
requestService.lastMtbFileRequestForPatientPseudonym(pseudonymizedMtbFile.patient.id)
val isLastRequestDeletion = requestService.isLastRequestWithKnownStatusDeletion(pseudonymizedMtbFile.patient.id)
requestService.lastMtbFileRequestForPatientPseudonym(patientPseudonym)
val isLastRequestDeletion =
requestService.isLastRequestWithKnownStatusDeletion(patientPseudonym)
return null != lastMtbFileRequestForPatient
&& !isLastRequestDeletion
&& lastMtbFileRequestForPatient.fingerprint == fingerprint(pseudonymizedMtbFile)
return null != lastMtbFileRequestForPatient && !isLastRequestDeletion && lastMtbFileRequestForPatient.fingerprint == fingerprint(
pseudonymizedMtbFileRequest
)
}
fun processDeletion(patientId: String) {
processDeletion(patientId, UUID.randomUUID().toString())
fun processDeletion(patientId: PatientId, isConsented: TtpConsentStatus) {
processDeletion(patientId, randomRequestId(), isConsented)
}
fun processDeletion(patientId: String, requestId: String) {
fun processDeletion(patientId: PatientId, requestId: RequestId, isConsented: TtpConsentStatus) {
try {
val patientPseudonym = pseudonymizeService.patientPseudonym(patientId)
val requestStatus: RequestStatus = when (isConsented) {
TtpConsentStatus.BROAD_CONSENT_MISSING_OR_REJECTED, TtpConsentStatus.BROAD_CONSENT_MISSING, TtpConsentStatus.BROAD_CONSENT_REJECTED -> RequestStatus.NO_CONSENT
TtpConsentStatus.FAILED_TO_ASK -> RequestStatus.ERROR
TtpConsentStatus.BROAD_CONSENT_GIVEN, TtpConsentStatus.UNKNOWN_CHECK_FILE -> RequestStatus.UNKNOWN
TtpConsentStatus.GENOM_DE_CONSENT_SEQUENCING_PERMIT, TtpConsentStatus.GENOM_DE_CONSENT_MISSING, TtpConsentStatus.GENOM_DE_SEQUENCING_REJECTED -> {
throw RuntimeException("processDelete should never deal with '" + isConsented.name + "' consent status. This is a bug and need to be fixed!")
}
}
requestService.save(
Request(
uuid = requestId,
patientId = patientPseudonym,
pid = patientId,
fingerprint = fingerprint(patientPseudonym),
status = RequestStatus.UNKNOWN,
type = RequestType.DELETE
requestId,
patientPseudonym,
patientId,
fingerprint(patientPseudonym.value),
RequestType.DELETE,
requestStatus
)
)
val responseStatus = sender.send(MtbFileSender.DeleteRequest(requestId, patientPseudonym))
val responseStatus = sender.send(DeleteRequest(requestId, patientPseudonym))
applicationEventPublisher.publishEvent(
ResponseEvent(
requestId,
Instant.now(),
responseStatus.status,
when (responseStatus.status) {
requestId, Instant.now(), responseStatus.status, when (responseStatus.status) {
RequestStatus.WARNING, RequestStatus.ERROR -> Optional.of(responseStatus.body)
else -> Optional.empty()
}
)
)
} catch (e: Exception) {
} catch (_: Exception) {
requestService.save(
Request(
uuid = requestId,
patientId = "???",
patientPseudonym = emptyPatientPseudonym(),
pid = patientId,
fingerprint = "",
fingerprint = Fingerprint.empty(),
status = RequestStatus.ERROR,
type = RequestType.DELETE,
report = Report("Fehler bei der Pseudonymisierung")
@@ -155,14 +201,18 @@ class RequestProcessor(
}
}
private fun fingerprint(mtbFile: MtbFile): String {
return fingerprint(objectMapper.writeValueAsString(mtbFile))
private fun <T> fingerprint(request: MtbFileRequest<T>): Fingerprint {
return when (request) {
is DnpmV2MtbFileRequest -> fingerprint(objectMapper.writeValueAsString(request.content))
}
}
private fun fingerprint(s: String): String {
return Base32().encodeAsString(DigestUtils.sha256(s))
.replace("=", "")
.lowercase()
private fun fingerprint(s: String): Fingerprint {
return Fingerprint(
Base32().encodeAsString(DigestUtils.sha256(s)).replace("=", "").lowercase()
)
}
}
}
private fun extractPatientIdentifier(mtbFile: Mtb): String = mtbFile.patient.id

View File

@@ -19,11 +19,13 @@
package dev.dnpm.etl.processor.services
import dev.dnpm.etl.processor.monitoring.Request
import dev.dnpm.etl.processor.monitoring.RequestRepository
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.monitoring.RequestType
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.RequestId
import dev.dnpm.etl.processor.monitoring.*
import org.springframework.data.domain.Page
import org.springframework.data.domain.Pageable
import org.springframework.stereotype.Service
import java.util.*
@Service
class RequestService(
@@ -32,15 +34,32 @@ class RequestService(
fun save(request: Request) = requestRepository.save(request)
fun allRequestsByPatientPseudonym(patientPseudonym: String) = requestRepository
.findAllByPatientIdOrderByProcessedAtDesc(patientPseudonym)
fun findAll(): Iterable<Request> = requestRepository.findAll()
fun lastMtbFileRequestForPatientPseudonym(patientPseudonym: String) =
fun findAll(pageable: Pageable): Page<Request> = requestRepository.findAll(pageable)
fun findByUuid(uuid: RequestId): Optional<Request> =
requestRepository.findByUuidEquals(uuid)
fun findRequestByPatientId(patientPseudonym: PatientPseudonym, pageable: Pageable): Page<Request> = requestRepository.findRequestByPatientPseudonym(patientPseudonym, pageable)
fun allRequestsByPatientPseudonym(patientPseudonym: PatientPseudonym) = requestRepository
.findAllByPatientPseudonymOrderByProcessedAtDesc(patientPseudonym)
fun lastMtbFileRequestForPatientPseudonym(patientPseudonym: PatientPseudonym) =
Companion.lastMtbFileRequestForPatientPseudonym(allRequestsByPatientPseudonym(patientPseudonym))
fun isLastRequestWithKnownStatusDeletion(patientPseudonym: String) =
fun isLastRequestWithKnownStatusDeletion(patientPseudonym: PatientPseudonym) =
Companion.isLastRequestWithKnownStatusDeletion(allRequestsByPatientPseudonym(patientPseudonym))
fun countStates(): Iterable<CountedState> = requestRepository.countStates()
fun countDeleteStates(): Iterable<CountedState> = requestRepository.countDeleteStates()
fun findPatientUniqueStates(): List<CountedState> = requestRepository.findPatientUniqueStates()
fun findPatientUniqueDeleteStates(): List<CountedState> = requestRepository.findPatientUniqueDeleteStates()
companion object {
fun lastMtbFileRequestForPatientPseudonym(allRequests: List<Request>) = allRequests

View File

@@ -19,8 +19,8 @@
package dev.dnpm.etl.processor.services
import dev.dnpm.etl.processor.RequestId
import dev.dnpm.etl.processor.monitoring.Report
import dev.dnpm.etl.processor.monitoring.RequestRepository
import dev.dnpm.etl.processor.monitoring.RequestStatus
import org.slf4j.LoggerFactory
import org.springframework.context.event.EventListener
@@ -31,7 +31,7 @@ import java.util.*
@Service
class ResponseProcessor(
private val requestRepository: RequestRepository,
private val requestService: RequestService,
private val statisticsUpdateProducer: Sinks.Many<Any>
) {
@@ -39,7 +39,7 @@ class ResponseProcessor(
@EventListener(classes = [ResponseEvent::class])
fun handleResponseEvent(event: ResponseEvent) {
requestRepository.findByUuidEquals(event.requestUuid).ifPresentOrElse({
requestService.findByUuid(event.requestUuid).ifPresentOrElse({
it.processedAt = event.timestamp
it.status = event.status
@@ -70,13 +70,19 @@ class ResponseProcessor(
)
}
RequestStatus.NO_CONSENT -> {
it.report = Report(
"Einwilligung Status fehlt, widerrufen oder ungeklärt."
)
}
else -> {
logger.error("Cannot process response: Unknown response!")
return@ifPresentOrElse
}
}
requestRepository.save(it)
requestService.save(it)
statisticsUpdateProducer.emitNext("", Sinks.EmitFailureHandler.FAIL_FAST)
}, {
@@ -87,7 +93,7 @@ class ResponseProcessor(
}
data class ResponseEvent(
val requestUuid: String,
val requestUuid: RequestId,
val timestamp: Instant,
val status: RequestStatus,
val body: Optional<String> = Optional.empty()

View File

@@ -22,11 +22,17 @@ package dev.dnpm.etl.processor.services
import com.fasterxml.jackson.databind.ObjectMapper
import com.jayway.jsonpath.JsonPath
import com.jayway.jsonpath.PathNotFoundException
import de.ukw.ccc.bwhc.dto.MtbFile
import dev.pcvolkmer.mv64e.mtb.Mtb
class TransformationService(private val objectMapper: ObjectMapper, private val transformations: List<Transformation>) {
fun transform(mtbFile: MtbFile): MtbFile {
var json = objectMapper.writeValueAsString(mtbFile)
fun transform(mtbFile: Mtb): Mtb {
val json = transform(objectMapper.writeValueAsString(mtbFile))
return objectMapper.readValue(json, Mtb::class.java)
}
private fun transform(content: String): String {
var json = content
transformations.forEach { transformation ->
val jsonPath = JsonPath.parse(json)
@@ -48,7 +54,7 @@ class TransformationService(private val objectMapper: ObjectMapper, private val
json = jsonPath.jsonString()
}
return objectMapper.readValue(json, MtbFile::class.java)
return json
}
fun getTransformations(): List<Transformation> {

View File

@@ -22,6 +22,7 @@ package dev.dnpm.etl.processor.services.kafka
import com.fasterxml.jackson.annotation.JsonAlias
import com.fasterxml.jackson.annotation.JsonProperty
import com.fasterxml.jackson.databind.ObjectMapper
import dev.dnpm.etl.processor.RequestId
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.output.asRequestStatus
import dev.dnpm.etl.processor.services.ResponseEvent
@@ -47,7 +48,7 @@ class KafkaResponseProcessor(
Optional.empty()
}.ifPresentOrElse({ responseBody ->
val event = ResponseEvent(
responseBody.requestId,
RequestId(responseBody.requestId),
Instant.ofEpochMilli(data.timestamp()),
responseBody.statusCode.asRequestStatus(),
when (responseBody.statusCode.asRequestStatus()) {

View File

@@ -0,0 +1,63 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor
import org.springframework.http.MediaType
import java.util.*
class Fingerprint(val value: String) {
override fun hashCode() = value.hashCode()
override fun equals(other: Any?) = other is Fingerprint && other.value == value
companion object {
fun empty() = Fingerprint("")
}
}
@JvmInline
value class RequestId(val value: String) {
fun isBlank() = value.isBlank()
}
fun randomRequestId() = RequestId(UUID.randomUUID().toString())
@JvmInline
value class PatientId(val value: String)
@JvmInline
value class PatientPseudonym(val value: String)
fun emptyPatientPseudonym() = PatientPseudonym("")
/**
* Custom MediaTypes
*
* @since 0.11.0
*/
object CustomMediaType {
val APPLICATION_VND_DNPM_V2_MTB_JSON = MediaType("application", "vnd.dnpm.v2.mtb+json")
const val APPLICATION_VND_DNPM_V2_MTB_JSON_VALUE = "application/vnd.dnpm.v2.mtb+json"
val APPLICATION_VND_DNPM_V2_RD_JSON = MediaType("application", "vnd.dnpm.v2.rd+json")
const val APPLICATION_VND_DNPM_V2_RD_JSON_VALUE = "application/vnd.dnpm.v2.rd+json"
}

View File

@@ -19,18 +19,15 @@
package dev.dnpm.etl.processor.web
import dev.dnpm.etl.processor.monitoring.ConnectionCheckResult
import dev.dnpm.etl.processor.monitoring.ConnectionCheckService
import dev.dnpm.etl.processor.monitoring.GPasConnectionCheckService
import dev.dnpm.etl.processor.monitoring.OutputConnectionCheckService
import dev.dnpm.etl.processor.monitoring.*
import dev.dnpm.etl.processor.output.MtbFileSender
import dev.dnpm.etl.processor.pseudonym.Generator
import dev.dnpm.etl.processor.security.Role
import dev.dnpm.etl.processor.security.UserRole
import dev.dnpm.etl.processor.services.Token
import dev.dnpm.etl.processor.services.TokenService
import dev.dnpm.etl.processor.security.Token
import dev.dnpm.etl.processor.security.TokenService
import dev.dnpm.etl.processor.services.TransformationService
import dev.dnpm.etl.processor.services.UserRoleService
import dev.dnpm.etl.processor.security.UserRoleService
import org.springframework.beans.factory.annotation.Qualifier
import org.springframework.http.MediaType
import org.springframework.http.codec.ServerSentEvent
@@ -56,16 +53,20 @@ class ConfigController(
@GetMapping
fun index(model: Model): String {
val outputConnectionAvailable =
connectionCheckServices.filterIsInstance<OutputConnectionCheckService>().first().connectionAvailable()
connectionCheckServices.filterIsInstance<OutputConnectionCheckService>().firstOrNull()?.connectionAvailable()
val gPasConnectionAvailable =
connectionCheckServices.filterIsInstance<GPasConnectionCheckService>().firstOrNull()?.connectionAvailable()
val gIcsConnectionAvailable =
connectionCheckServices.filterIsInstance<GIcsConnectionCheckService>().firstOrNull()?.connectionAvailable()
model.addAttribute("pseudonymGenerator", pseudonymGenerator.javaClass.simpleName)
model.addAttribute("mtbFileSender", mtbFileSender.javaClass.simpleName)
model.addAttribute("mtbFileEndpoint", mtbFileSender.endpoint())
model.addAttribute("outputConnectionAvailable", outputConnectionAvailable)
model.addAttribute("gPasConnectionAvailable", gPasConnectionAvailable)
model.addAttribute("gIcsConnectionAvailable", gIcsConnectionAvailable)
model.addAttribute("tokensEnabled", tokenService != null)
if (tokenService != null) {
model.addAttribute("tokens", tokenService.findAll())
@@ -119,6 +120,24 @@ class ConfigController(
return "configs/gPasConnectionAvailable"
}
@GetMapping(params = ["gIcsConnectionAvailable"])
fun gIcsConnectionAvailable(model: Model): String {
val gIcsConnectionAvailable =
connectionCheckServices.filterIsInstance<GIcsConnectionCheckService>().firstOrNull()?.connectionAvailable()
model.addAttribute("mtbFileSender", mtbFileSender.javaClass.simpleName)
model.addAttribute("mtbFileEndpoint", mtbFileSender.endpoint())
model.addAttribute("gIcsConnectionAvailable", gIcsConnectionAvailable)
if (tokenService != null) {
model.addAttribute("tokensEnabled", true)
model.addAttribute("tokens", tokenService.findAll())
} else {
model.addAttribute("tokens", listOf<Token>())
}
return "configs/gIcsConnectionAvailable"
}
@PostMapping(path = ["tokens"])
fun addToken(@ModelAttribute("name") name: String, model: Model): String {
if (tokenService == null) {
@@ -127,10 +146,11 @@ class ConfigController(
} else {
model.addAttribute("tokensEnabled", true)
val result = tokenService.addToken(name)
if (result.isSuccess) {
model.addAttribute("newTokenValue", result.getOrDefault(""))
result.onSuccess {
model.addAttribute("newTokenValue", it)
model.addAttribute("success", true)
} else {
}
result.onFailure {
model.addAttribute("success", false)
}
model.addAttribute("tokens", tokenService.findAll())
@@ -182,12 +202,14 @@ class ConfigController(
}
@GetMapping(path = ["events"], produces = [MediaType.TEXT_EVENT_STREAM_VALUE])
@ResponseBody
fun events(): Flux<ServerSentEvent<Any>> {
return connectionCheckUpdateProducer.asFlux().map {
val event = when (it) {
is ConnectionCheckResult.KafkaConnectionCheckResult -> "output-connection-check"
is ConnectionCheckResult.RestConnectionCheckResult -> "output-connection-check"
is ConnectionCheckResult.GPasConnectionCheckResult -> "gpas-connection-check"
is ConnectionCheckResult.GIcsConnectionCheckResult -> "gics-connection-check"
}
ServerSentEvent.builder<Any>()

View File

@@ -20,9 +20,10 @@
package dev.dnpm.etl.processor.web
import dev.dnpm.etl.processor.NotFoundException
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.RequestId
import dev.dnpm.etl.processor.monitoring.ReportService
import dev.dnpm.etl.processor.monitoring.RequestId
import dev.dnpm.etl.processor.monitoring.RequestRepository
import dev.dnpm.etl.processor.services.RequestService
import org.springframework.data.domain.Pageable
import org.springframework.data.domain.Sort
import org.springframework.data.web.PageableDefault
@@ -35,7 +36,7 @@ import org.springframework.web.bind.annotation.RequestMapping
@Controller
@RequestMapping(path = ["/"])
class HomeController(
private val requestRepository: RequestRepository,
private val requestService: RequestService,
private val reportService: ReportService
) {
@@ -44,20 +45,20 @@ class HomeController(
@PageableDefault(page = 0, size = 20, sort = ["processedAt"], direction = Sort.Direction.DESC) pageable: Pageable,
model: Model
): String {
val requests = requestRepository.findAll(pageable)
val requests = requestService.findAll(pageable)
model.addAttribute("requests", requests)
return "index"
}
@GetMapping(path = ["patient/{patientId}"])
@GetMapping(path = ["patient/{patientPseudonym}"])
fun byPatient(
@PathVariable patientId: String,
@PathVariable patientPseudonym: PatientPseudonym,
@PageableDefault(page = 0, size = 20, sort = ["processedAt"], direction = Sort.Direction.DESC) pageable: Pageable,
model: Model
): String {
val requests = requestRepository.findRequestByPatientId(patientId, pageable)
model.addAttribute("patientId", patientId)
val requests = requestService.findRequestByPatientId(patientPseudonym, pageable)
model.addAttribute("patientPseudonym", patientPseudonym.value)
model.addAttribute("requests", requests)
return "index"
@@ -65,7 +66,7 @@ class HomeController(
@GetMapping(path = ["/report/{id}"])
fun report(@PathVariable id: RequestId, model: Model): String {
val request = requestRepository.findByUuidEquals(id.toString()).orElse(null) ?: throw NotFoundException()
val request = requestService.findByUuid(id).orElse(null) ?: throw NotFoundException()
model.addAttribute("request", request)
model.addAttribute("issues", reportService.deserialize(request.report?.dataQualityReport))

View File

@@ -19,9 +19,9 @@
package dev.dnpm.etl.processor.web
import dev.dnpm.etl.processor.monitoring.RequestRepository
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.monitoring.RequestType
import dev.dnpm.etl.processor.services.RequestService
import org.springframework.beans.factory.annotation.Qualifier
import org.springframework.http.MediaType
import org.springframework.http.codec.ServerSentEvent
@@ -41,15 +41,15 @@ import java.time.temporal.ChronoUnit
class StatisticsRestController(
@Qualifier("statisticsUpdateProducer")
private val statisticsUpdateProducer: Sinks.Many<Any>,
private val requestRepository: RequestRepository
private val requestService: RequestService
) {
@GetMapping(path = ["requeststates"])
fun requestStates(@RequestParam(required = false, defaultValue = "false") delete: Boolean): List<NameValue> {
val states = if (delete) {
requestRepository.countDeleteStates()
requestService.countDeleteStates()
} else {
requestRepository.countStates()
requestService.countStates()
}
return states
@@ -79,7 +79,7 @@ class StatisticsRestController(
}
val formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.of("Europe/Berlin"))
val data = requestRepository.findAll()
val data = requestService.findAll()
.filter { it.type == requestType }
.filter { it.processedAt.isAfter(Instant.now().minus(30, ChronoUnit.DAYS)) }
.groupBy { formatter.format(it.processedAt) }
@@ -115,9 +115,9 @@ class StatisticsRestController(
@GetMapping(path = ["requestpatientstates"])
fun requestPatientStates(@RequestParam(required = false, defaultValue = "false") delete: Boolean): List<NameValue> {
val states = if (delete) {
requestRepository.findPatientUniqueDeleteStates()
requestService.findPatientUniqueDeleteStates()
} else {
requestRepository.findPatientUniqueStates()
requestService.findPatientUniqueStates()
}
return states.map {

View File

@@ -3,17 +3,128 @@ spring:
compose:
file: ./dev-compose.yml
security:
oauth2:
client:
registration:
custom:
client-name: App-Dev
client-id: app-dev
client-secret: very-secret-ae3f7a-5a9f-1190
scope:
- openid
provider:
custom:
issuer-uri: https://dnpm.dev/auth/realms/intern
user-name-attribute: name
# kafka:
# security:
# protocol: "SSL"
# ssl:
# key-store-type: "PEM"
# key-store-certificate-chain: -----BEGIN CERTIFICATE-----
# MIIDCzCCAfOgAwIBAgIUaXNh4PahaKeLUaab2rUPSVESx28wDQYJKoZIhvcNAQEL
# BQAwFTETMBEGA1UEAwwKRXhhbXBsZSBDQTAeFw0yNTA4MjExODEyMTFaFw0zNTA4
# MTkxODEyMTFaMBUxEzARBgNVBAMMCkV4YW1wbGUgQ0EwggEiMA0GCSqGSIb3DQEB
# AQUAA4IBDwAwggEKAoIBAQCsqalqVOLFglVbX9oSHU91ebyL1kPyb/2N90UGQIcD
# UAjzKxxysId1Vdvtbbwgli6UgfPwlzFP2Wlw51h496yL4QU/9tNV956UJ1RoS/fG
# qBAEHctqavfMI27UQmIzw4pGMkGzEQxRMc6a9pdabBhbMMTJsjtmOv2YMYHj1HHK
# Dr7wTBTt2l0eRyCR0kZ8XGIMWhYowPa4EMpC7+4e5Nf/7LSJZWLLy9jXPpazsjkJ
# jEmDNlFfx2tZiq0Wz2Xj1pZSDLbcuIX4IHcLfMvagibfrCMX/h6+WuW42sWPRuBW
# wB6cHGlXs+K/gBBWxtD7sOTacO5hbHFsfaJOhSEIGoIpAgMBAAGjUzBRMB0GA1Ud
# DgQWBBT2S/C2++ECY+CSuN5KKql0umfbTDAfBgNVHSMEGDAWgBT2S/C2++ECY+CS
# uN5KKql0umfbTDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBj
# H4DdwqrOHg7sVsqiwDsZfTharpUDCYeG5XhrJQlnA9eKwyofTb929W/fjOwBdDtg
# 9THT/omR0lA8/UyHtezMT6nMsCn4HG2mXvx6ghgvA3jrFTEY7R80dHkboLMTV3u4
# RYgC9S3BJPcbJYpM0cXzkp2T0F4FxWZlfqefuedHuX3zcCxpgVD56qQb2a131TX7
# O3UDJfVg8a65IFtehndqILgLVrf7w6+pbmDAlCg5RKrt2USEYyZXYdyTryJbdtn4
# BCLp0avYtSYVUGwgH0oUCpkjQRwMg1003TTz8SNnmE7mAXHYljyYejnjL8vBHfch
# 8tTDVXQn08BT9H3jZTnF
# -----END CERTIFICATE-----
# -----BEGIN CERTIFICATE-----
# MIIC+TCCAeGgAwIBAgIUUoCwz8GS6xQ3mmI7RUUYSNPIOi4wDQYJKoZIhvcNAQEL
# BQAwFTETMBEGA1UEAwwKRXhhbXBsZSBDQTAeFw0yNTA4MjExODE0NDhaFw0zNTA4
# MTkxODE0NDhaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
# BQADggEPADCCAQoCggEBAL9PW99MhhBwdEmTHyZgfnhfTrxZPrNU6z1UdV8b82Lk
# 3p75o8eCKa9iOd7DDQlo75hQBhhX0+Xc3mucrstx5p8TYFMbypif8ojWh3LM++P8
# tz3ezQZlq86ycyKpm8dqlA03b227tFDfiYTev2eD2HN40BU7yDAYhhqd/QW8+MV2
# jkcRGv5cE21GZxWmPUpkVN+bNoBC8H90WmkST90LfeYF+wZnlsAJZH6AQzR1GnGD
# ICE5evMhC78hvRnpgeA310SyxssZEigkePL5lTZOBPY2IuzBqL05agyVTiVq4Xd6
# y3xOqXoxxOhZu06yd3nymorqeTgbF1fW8wQF0u3KsFECAwEAAaNCMEAwHQYDVR0O
# BBYEFHk9jMWRAAt2YsBSxUcOQVoWayoHMB8GA1UdIwQYMBaAFPZL8Lb74QJj4JK4
# 3koqqXS6Z9tMMA0GCSqGSIb3DQEBCwUAA4IBAQBqabAA9INONDaLHqs9i9YQHm/g
# AnB7xRl/RFbERKKCTSMZUYM8oEaaH0W2ENoPMc/7xOB/R8a7Rm62PTr6syxwhZrY
# 5NtGKJOD+rh90/5l83tulf93KqOJtGkiv6NBDvCNrITcA+UKRk/z4GcFi2YjWAl4
# wvY44lzTasMKSpjUQ5N0VNANcW3nVuEgPQ8rrr0NOK/5j4guPjsXDsixa47gqblA
# 5xGfBKeVmEXdPbzawZfP4hPIw7DpX2m8Y0erswF1ZxkIV73V3TDsFSLcqSKSzZr6
# mtj8COlV9Us7zqaJbV5eOl7GN1T9orZJwZmX1Z46gCkkSLYDP/dqtl2j9JgN
# -----END CERTIFICATE-----
# ### For dev/testing purposes only!
# key-store-key: -----BEGIN PRIVATE KEY-----
# MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC/T1vfTIYQcHRJ
# kx8mYH54X068WT6zVOs9VHVfG/Ni5N6e+aPHgimvYjneww0JaO+YUAYYV9Pl3N5r
# nK7LceafE2BTG8qYn/KI1odyzPvj/Lc93s0GZavOsnMiqZvHapQNN29tu7RQ34mE
# 3r9ng9hzeNAVO8gwGIYanf0FvPjFdo5HERr+XBNtRmcVpj1KZFTfmzaAQvB/dFpp
# Ek/dC33mBfsGZ5bACWR+gEM0dRpxgyAhOXrzIQu/Ib0Z6YHgN9dEssbLGRIoJHjy
# +ZU2TgT2NiLswai9OWoMlU4lauF3est8Tql6McToWbtOsnd58pqK6nk4GxdX1vME
# BdLtyrBRAgMBAAECggEAC1wXfPlqxoQe65WAVoOJTvV90+JKvlRPCZu/wm+C8r7b
# Vz5Ekt6wQflHrWoQlpv0CivKSNzCONZ2IJazrGHti0mXwSeXzptEyApRDaiNVnrV
# mKdnrjcQThw7iPXgSaWS9/vwMmhgayLy5ABkBi4GhsjINlKP7wctw1vZP+N6NCNd
# Ql3taStvDKmG0SfJHF6/2o/XBpof3IJEL7ghbzyTTbWWaO34J1mJ8A+AmjGhj9GE
# Dp3XuOrO9W7MVd1nfZDtGBS8qf80AwROyodZZRma9vZuWJZ5aQFi2CnUEtU1T+Uv
# tW+F6tg2FOMr8M0Fb79wGIDwSF8u/QcTvwhEzZAfiQKBgQDioOofnE1oB1DOMnqZ
# SOFjs+vsirvS6G3lo27+HkE3TgvCHR4sk1305AiXtjmPu8iaUCo9qn18MtduY2RS
# CcKMOG/FxhmDyP5I29DhJRhvERIpJd0kcSDQOgtaoVPC1XzIlyTqte6nGX9kAnA/
# x/OOXrZ0hjhMNDcZzf2NasPYJQKBgQDYGqTobkVBk+eekNWklnTh41/649rUIgTu
# JStArtY2hgaEInYcGa2e7cEj7nIHA0iGy3EJ2yvwoUIyxtoXVcGohu2IrzlhS33T
# R4jA7nE2/yHZrEMEJovuSU0eMw7rgvEtL79Q0RToYnTY1EU6X/BBoFfiiEeNMHKz
# zjDOOQ6ZvQKBgGCWChIc0FSkwYiPtPZ9PCn89XCjk/cIPkYfiF9fT5Ydeh9pv4Fp
# 8SI8yXi3HgMnGhDCV65eagqztGMEky3voO2X4/MbQaaL0+wDWxuJbsdvNBk7XOt6
# F20HP+2JUiR4Ti1DVWV+0k5/LG7YJzTXp/KmZQZ2aan4mv8xbn2F4h/NAoGAI4ou
# OLN53FEQtHkpSYoc6tFUBZTXdi+qE+g09sxKGmlsROrN9c0bSpnbO6eJRTH7CYAH
# tRFAZrB+jI87ar8FvEuEYQhALYoWxVpsWR5drCfFT2EPHG2icavIbQEEoSLFuyKx
# Gf9oqtcWVFqEkBcbEg/mpDC5Y7TmCEAOsrubdRkCgYEAl7B+EzIdG0rabGoti09q
# QXfyiTjR7nQYkhpLxMCeNlCpQ8Y15XSa8bm1UIGYqj/ZBpeBNhrj64IHoub5Vd43
# tzbb8yNgoLUd16TU1VvyccCMGQVPIF8RkDsAtEawV2eoXbHAstN99xbC8jsLNZRQ
# fcfgTiQaXRJmlVx6jfbfZd4=
# -----END PRIVATE KEY-----
# trust-store-type: "PEM"
# trust-store-certificates: -----BEGIN CERTIFICATE-----
# MIIDCzCCAfOgAwIBAgIUaXNh4PahaKeLUaab2rUPSVESx28wDQYJKoZIhvcNAQEL
# BQAwFTETMBEGA1UEAwwKRXhhbXBsZSBDQTAeFw0yNTA4MjExODEyMTFaFw0zNTA4
# MTkxODEyMTFaMBUxEzARBgNVBAMMCkV4YW1wbGUgQ0EwggEiMA0GCSqGSIb3DQEB
# AQUAA4IBDwAwggEKAoIBAQCsqalqVOLFglVbX9oSHU91ebyL1kPyb/2N90UGQIcD
# UAjzKxxysId1Vdvtbbwgli6UgfPwlzFP2Wlw51h496yL4QU/9tNV956UJ1RoS/fG
# qBAEHctqavfMI27UQmIzw4pGMkGzEQxRMc6a9pdabBhbMMTJsjtmOv2YMYHj1HHK
# Dr7wTBTt2l0eRyCR0kZ8XGIMWhYowPa4EMpC7+4e5Nf/7LSJZWLLy9jXPpazsjkJ
# jEmDNlFfx2tZiq0Wz2Xj1pZSDLbcuIX4IHcLfMvagibfrCMX/h6+WuW42sWPRuBW
# wB6cHGlXs+K/gBBWxtD7sOTacO5hbHFsfaJOhSEIGoIpAgMBAAGjUzBRMB0GA1Ud
# DgQWBBT2S/C2++ECY+CSuN5KKql0umfbTDAfBgNVHSMEGDAWgBT2S/C2++ECY+CS
# uN5KKql0umfbTDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBj
# H4DdwqrOHg7sVsqiwDsZfTharpUDCYeG5XhrJQlnA9eKwyofTb929W/fjOwBdDtg
# 9THT/omR0lA8/UyHtezMT6nMsCn4HG2mXvx6ghgvA3jrFTEY7R80dHkboLMTV3u4
# RYgC9S3BJPcbJYpM0cXzkp2T0F4FxWZlfqefuedHuX3zcCxpgVD56qQb2a131TX7
# O3UDJfVg8a65IFtehndqILgLVrf7w6+pbmDAlCg5RKrt2USEYyZXYdyTryJbdtn4
# BCLp0avYtSYVUGwgH0oUCpkjQRwMg1003TTz8SNnmE7mAXHYljyYejnjL8vBHfch
# 8tTDVXQn08BT9H3jZTnF
# -----END CERTIFICATE-----
app:
#rest:
# uri: http://localhost:9000/bwhc/etl/api
kafka:
topic: test
response-topic: test_response
servers: localhost:9094
#security:
# admin-user: admin
# admin-password: "{noop}very-secret"
rest:
uri: http://localhost/api
#kafka:
# servers: localhost:9094
# group-id: "test1234"
# input-topic: test_input
# output-topic: test_output
# output-response-topic: test_response
security:
admin-user: admin
admin-password: "{noop}very-secret"
enable-oidc: "true"
server:
port: 8000

View File

@@ -16,6 +16,5 @@ spring:
content:
enabled: true
paths: /**/*.js,/**/*.css,/**/*.svg,/**/*.jpeg
server:
forward-headers-strategy: framework
forward-headers-strategy: framework

View File

@@ -0,0 +1 @@
ALTER TABLE request RENAME COLUMN patient_id TO patient_pseudonym;

View File

@@ -0,0 +1 @@
ALTER TABLE request RENAME COLUMN patient_id TO patient_pseudonym;

View File

@@ -22,6 +22,10 @@
--bg-gray-op: rgba(112, 128, 144, .35);
}
* {
font-family: sans-serif;
}
html {
background: linear-gradient(-5deg, var(--bg-blue-op), transparent 10em);
min-height: 100vh;
@@ -30,7 +34,6 @@ html {
body {
margin: 0 0 5em 0;
font-family: sans-serif;
font-size: .8rem;
color: var(--text);
@@ -619,6 +622,10 @@ input.inline:focus-visible {
text-align: center;
}
.notification.info {
color: var(--bg-blue);
}
.notification.success {
color: var(--bg-green);
}
@@ -643,14 +650,16 @@ input.inline:focus-visible {
.tab:hover,
.tab.active {
background: var(--table-border);
background: var(--bg-gray);
color: white;
}
.tabcontent {
border: 1px solid var(--table-border);
border: 2px solid var(--bg-gray);
border-radius: 0 .5em .5em .5em;
display: none;
padding: 1em;
background: white;
}
.tabcontent.active {
@@ -688,4 +697,14 @@ a.reload {
.no-token {
padding: 1em;
background: var(--bg-red-op);
}
.issue-message {
font-family: monospace;
font-weight: bolder;
}
.issue-path {
font-family: monospace;
line-height: 1rem;
}

View File

@@ -49,6 +49,11 @@
</div>
</section>
<section hx-ext="sse" th:sse-connect="@{/configs/events}">
<div th:insert="~{configs/gIcsConnectionAvailable.html}" th:hx-get="@{/configs?gIcsConnectionAvailable}" hx-trigger="sse:gics-connection-check">
</div>
</section>
<section hx-ext="sse" th:sse-connect="@{/configs/events}">
<div th:insert="~{configs/outputConnectionAvailable.html}" th:hx-get="@{/configs?outputConnectionAvailable}" hx-trigger="sse:output-connection-check">
</div>

View File

@@ -0,0 +1,24 @@
<th:block th:if="${gIcsConnectionAvailable == null}">
<h2><span>🟦</span> gICS nicht konfiguriert - Einwilligung wird über Dateiinhalt geprüft</h2>
</th:block>
<th:block th:if="${gIcsConnectionAvailable != null}">
<h2><span th:if="${gIcsConnectionAvailable.available}"></span><span th:if="${not(gIcsConnectionAvailable.available)}"></span> Verbindung zu gICS</h2>
<div>
Stand: <time style="font-weight: bold" th:datetime="${#temporals.formatISO(gIcsConnectionAvailable.timestamp)}" th:text="${#temporals.formatISO(gIcsConnectionAvailable.timestamp)}"></time>
&nbsp;|&nbsp;
Letzte Änderung: <time style="font-weight: bold" th:datetime="${#temporals.formatISO(gIcsConnectionAvailable.lastChange)}" th:text="${#temporals.formatISO(gIcsConnectionAvailable.lastChange)}"></time>
</div>
<div>
<span>Die Verbindung ist aktuell</span>
<strong th:if="${gIcsConnectionAvailable.available}" style="color: green">verfügbar.</strong>
<strong th:if="${not(gIcsConnectionAvailable.available)}" style="color: red">nicht verfügbar.</strong>
</div>
<div class="connection-display border">
<img th:src="@{/server.png}" alt="ETL-Processor" />
<span class="connection" th:classappend="${gIcsConnectionAvailable.available ? 'available' : ''}"></span>
<img th:src="@{/server.png}" alt="gICS" />
<span>ETL-Processor</span>
<span></span>
<span>gICS</span>
</div>
</th:block>

View File

@@ -2,15 +2,20 @@
<h2><span>🟦</span> gPAS nicht konfiguriert - Patienten-IDs werden intern anonymisiert</h2>
</th:block>
<th:block th:if="${gPasConnectionAvailable != null}">
<h2><span th:if="${gPasConnectionAvailable}"></span><span th:if="${not(gPasConnectionAvailable)}"></span> Verbindung zu gPAS</h2>
<h2><span th:if="${gPasConnectionAvailable.available}"></span><span th:if="${not(gPasConnectionAvailable.available)}"></span> Verbindung zu gPAS</h2>
<div>
Die Verbindung ist aktuell
<strong th:if="${gPasConnectionAvailable}" style="color: green">verfügbar.</strong>
<strong th:if="${not(gPasConnectionAvailable)}" style="color: red">nicht verfügbar.</strong>
Stand: <time style="font-weight: bold" th:datetime="${#temporals.formatISO(gPasConnectionAvailable.timestamp)}" th:text="${#temporals.formatISO(gPasConnectionAvailable.timestamp)}"></time>
&nbsp;|&nbsp;
Letzte Änderung: <time style="font-weight: bold" th:datetime="${#temporals.formatISO(gPasConnectionAvailable.lastChange)}" th:text="${#temporals.formatISO(gPasConnectionAvailable.lastChange)}"></time>
</div>
<div>
<span>Die Verbindung ist aktuell</span>
<strong th:if="${gPasConnectionAvailable.available}" style="color: green">verfügbar.</strong>
<strong th:if="${not(gPasConnectionAvailable.available)}" style="color: red">nicht verfügbar.</strong>
</div>
<div class="connection-display border">
<img th:src="@{/server.png}" alt="ETL-Processor" />
<span class="connection" th:classappend="${gPasConnectionAvailable ? 'available' : ''}"></span>
<span class="connection" th:classappend="${gPasConnectionAvailable.available ? 'available' : ''}"></span>
<img th:src="@{/server.png}" alt="gPAS" />
<span>ETL-Processor</span>
<span></span>

View File

@@ -1,16 +1,27 @@
<h2><span th:if="${outputConnectionAvailable}"></span><span th:if="${not(outputConnectionAvailable)}"></span> MTB-File Verbindung</h2>
<div>
Verbindung über <code>[[ ${mtbFileSender} ]]</code>. Die Verbindung ist aktuell
<strong th:if="${outputConnectionAvailable}" style="color: green">verfügbar.</strong>
<strong th:if="${not(outputConnectionAvailable)}" style="color: red">nicht verfügbar.</strong>
</div>
<div class="connection-display border">
<img th:src="@{/server.png}" alt="ETL-Processor" />
<span class="connection" th:classappend="${outputConnectionAvailable ? 'available' : ''}"></span>
<img th:if="${mtbFileSender.startsWith('Rest')}" th:src="@{/server.png}" alt="bwHC-Backend" />
<img th:if="${mtbFileSender.startsWith('Kafka')}" th:src="@{/kafka.png}" alt="Kafka-Broker" />
<span>ETL-Processor</span>
<span></span>
<span th:if="${mtbFileSender.startsWith('Rest')}">bwHC-Backend</span>
<span th:if="${mtbFileSender.startsWith('Kafka')}">Kafka-Broker</span>
</div>
<th:block th:if="${outputConnectionAvailable == null}">
<h2><span>🟦</span> Keine Ausgabenkonfiguration</h2>
</th:block>
<th:block th:if="${outputConnectionAvailable != null}">
<h2><span th:if="${outputConnectionAvailable.available}"></span><span th:if="${not(outputConnectionAvailable.available)}"></span> MTB-File Verbindung</h2>
<div>
Stand: <time style="font-weight: bold" th:datetime="${#temporals.formatISO(outputConnectionAvailable.timestamp)}" th:text="${#temporals.formatISO(outputConnectionAvailable.timestamp)}"></time>
&nbsp;|&nbsp;
Letzte Änderung: <time style="font-weight: bold" th:datetime="${#temporals.formatISO(outputConnectionAvailable.lastChange)}" th:text="${#temporals.formatISO(outputConnectionAvailable.lastChange)}"></time>
</div>
<div>
Verbindung über <code>[[ ${mtbFileSender} ]]</code>. Die Verbindung ist aktuell
<strong th:if="${outputConnectionAvailable.available}" style="color: green">verfügbar.</strong>
<strong th:if="${not(outputConnectionAvailable.available)}" style="color: red">nicht verfügbar.</strong>
</div>
<div class="connection-display border">
<img th:src="@{/server.png}" alt="ETL-Processor" />
<span class="connection" th:classappend="${outputConnectionAvailable.available ? 'available' : ''}"></span>
<img th:if="${mtbFileSender.startsWith('Rest')}" th:src="@{/server.png}" alt="bwHC-Backend" />
<img th:if="${mtbFileSender.startsWith('Kafka')}" th:src="@{/kafka.png}" alt="Kafka-Broker" />
<span>ETL-Processor</span>
<span></span>
<span th:if="${mtbFileSender.startsWith('RestBwhc')}">bwHC-Backend</span>
<span th:if="${mtbFileSender.startsWith('RestDip')}">DNPM:DIP-Backend</span>
<span th:if="${mtbFileSender.startsWith('Kafka')}">Kafka-Broker</span>
</div>
</th:block>

View File

@@ -12,26 +12,30 @@
<h1>Alle Anfragen<a id="reload-notify" class="reload" title="Neue Anfragen" th:href="@{/}"></a></h1>
<div>
<h2 th:if="${patientId != null}">
Betreffend Patienten-Pseudonym <span class="monospace" th:text="${patientId}">***</span>
<a class="btn btn-blue" th:if="${patientId != null}" th:href="@{/}">Alle anzeigen</a>
<h2 th:if="${patientPseudonym != null}">
Betreffend Patienten-Pseudonym <span class="monospace" th:text="${patientPseudonym}">***</span>
<a class="btn btn-blue" th:if="${patientPseudonym != null}" th:href="@{/}">Alle anzeigen</a>
</h2>
</div>
<div class="border">
<div th:if="${patientId == null}" class="page-control">
<div class="border" th:if="${requests.totalElements == 0}">
<div class="notification info">Noch keine Anfragen eingegangen</div>
</div>
<div class="border" th:if="${requests.totalElements > 0}">
<div th:if="${patientPseudonym == null}" class="page-control">
<a id="first-page-link" th:href="@{/(page=${0})}" title="Zum Anfang: Taste W" th:if="${not requests.isFirst()}">&larrb;</a><a th:if="${requests.isFirst()}">&larrb;</a>
<a id="prev-page-link" th:href="@{/(page=${requests.getNumber() - 1})}" title="Seite zurück: Taste A" th:if="${not requests.isFirst()}">&larr;</a><a th:if="${requests.isFirst()}">&larr;</a>
<span>Seite [[ ${requests.getNumber() + 1} ]] von [[ ${requests.getTotalPages()} ]]</span>
<a id="next-page-link" th:href="@{/(page=${requests.getNumber() + 1})}" title="Seite vor: Taste D" th:if="${not requests.isLast()}">&rarr;</a><a th:if="${requests.isLast()}">&rarr;</a>
<a id="last-page-link" th:href="@{/(page=${requests.getTotalPages() - 1})}" title="Zum Ende: Taste S" th:if="${not requests.isLast()}">&rarrb;</a><a th:if="${requests.isLast()}">&rarrb;</a>
</div>
<div th:if="${patientId != null}" class="page-control">
<a id="first-page-link" th:href="@{/patient/{patientId}(patientId=${patientId},page=${0})}" title="Zum Anfang: Taste W" th:if="${not requests.isFirst()}">&larrb;</a><a th:if="${requests.isFirst()}">&larrb;</a>
<a id="prev-page-link" th:href="@{/patient/{patientId}(patientId=${patientId},page=${requests.getNumber() - 1})}" title="Seite zurück: Taste A" th:if="${not requests.isFirst()}">&larr;</a><a th:if="${requests.isFirst()}">&larr;</a>
<div th:if="${patientPseudonym != null}" class="page-control">
<a id="first-page-link" th:href="@{/patient/{patientPseudonym}(patientPseudonym=${patientPseudonym},page=${0})}" title="Zum Anfang: Taste W" th:if="${not requests.isFirst()}">&larrb;</a><a th:if="${requests.isFirst()}">&larrb;</a>
<a id="prev-page-link" th:href="@{/patient/{patientPseudonym}(patientPseudonym=${patientPseudonym},page=${requests.getNumber() - 1})}" title="Seite zurück: Taste A" th:if="${not requests.isFirst()}">&larr;</a><a th:if="${requests.isFirst()}">&larr;</a>
<span>Seite [[ ${requests.getNumber() + 1} ]] von [[ ${requests.getTotalPages()} ]]</span>
<a id="next-page-link" th:href="@{/patient/{patientId}(patientId=${patientId},page=${requests.getNumber() + 1})}" title="Seite vor: Taste D" th:if="${not requests.isLast()}">&rarr;</a><a th:if="${requests.isLast()}">&rarr;</a>
<a id="last-page-link" th:href="@{/patient/{patientId}(patientId=${patientId},page=${requests.getTotalPages() - 1})}" title="Zum Ende: Taste S" th:if="${not requests.isLast()}">&rarrb;</a><a th:if="${requests.isLast()}">&rarrb;</a>
<a id="next-page-link" th:href="@{/patient/{patientPseudonym}(patientPseudonym=${patientPseudonym},page=${requests.getNumber() + 1})}" title="Seite vor: Taste D" th:if="${not requests.isLast()}">&rarr;</a><a th:if="${requests.isLast()}">&rarr;</a>
<a id="last-page-link" th:href="@{/patient/{patientPseudonym}(patientPseudonym=${patientPseudonym},page=${requests.getTotalPages() - 1})}" title="Zum Ende: Taste S" th:if="${not requests.isLast()}">&rarrb;</a><a th:if="${requests.isLast()}">&rarrb;</a>
</div>
<table class="paged">
<thead>
@@ -57,11 +61,11 @@
<th:block sec:authorize="not (hasRole('USER') or hasRole('ADMIN'))">[[ ${request.uuid} ]]</th:block>
</td>
<td><time th:datetime="${request.processedAt}">[[ ${request.processedAt} ]]</time></td>
<td class="patient-id" th:if="${patientId != null}" sec:authorize="hasRole('USER') or hasRole('ADMIN')">
[[ ${request.patientId} ]]
<td class="patient-id" th:if="${patientPseudonym != null}" sec:authorize="hasRole('USER') or hasRole('ADMIN')">
[[ ${request.patientPseudonym} ]]
</td>
<td class="patient-id" th:if="${patientId == null}" sec:authorize="hasRole('USER') or hasRole('ADMIN')">
<a th:href="@{/patient/{pid}(pid=${request.patientId})}">[[ ${request.patientId} ]]</a>
<td class="patient-id" th:if="${patientPseudonym == null}" sec:authorize="hasRole('USER') or hasRole('ADMIN')">
<a th:href="@{/patient/{pid}(pid=${request.patientPseudonym})}">[[ ${request.patientPseudonym} ]]</a>
</td>
<td class="patient-id" sec:authorize="not (hasRole('USER') or hasRole('ADMIN'))">***</td>
</tr>

View File

@@ -31,7 +31,7 @@
<td th:style="${request.type.value == 'delete'} ? 'color: red;'"><small>[[ ${request.type} ]]</small></td>
<td>[[ ${request.uuid} ]]</td>
<td><time th:datetime="${request.processedAt}">[[ ${request.processedAt} ]]</time></td>
<td class="patient-id" sec:authorize="authenticated">[[ ${request.patientId} ]]</td>
<td class="patient-id" sec:authorize="authenticated">[[ ${request.patientPseudonym} ]]</td>
<td class="patient-id" sec:authorize="not authenticated">***</td>
</tr>
</tbody>
@@ -47,7 +47,7 @@
<thead>
<tr>
<th>Schweregrad</th>
<th>Beschreibung</th>
<th>Beschreibung und Pfad</th>
</tr>
</thead>
<tbody>
@@ -56,7 +56,11 @@
<td th:if="${issue.severity.value == 'warning'}" class="bg-yellow"><small>[[ ${issue.severity} ]]</small></td>
<td th:if="${issue.severity.value == 'error'}" class="bg-red"><small>[[ ${issue.severity} ]]</small></td>
<td th:if="${issue.severity.value == 'fatal'}" class="bg-red"><small>[[ ${issue.severity} ]]</small></td>
<td>[[ ${issue.message} ]]</td>
<td>
<div class="issue-message">[[ ${issue.message} ]]</div>
<div class="issue-path" th:if="${issue.path.isPresent()}">[[ ${issue.path.get()} ]]</div>
<div class="issue-path" th:if="${issue.path.isEmpty()}"><i>Keine Angabe</i></div>
</td>
</tr>
</tbody>
</table>

View File

@@ -0,0 +1,181 @@
package dev.dnpm.etl.processor.consent;
import com.fasterxml.jackson.databind.ObjectMapper;
import dev.dnpm.etl.processor.config.AppConfiguration;
import dev.dnpm.etl.processor.config.AppFhirConfig;
import dev.dnpm.etl.processor.config.GIcsConfigProperties;
import org.hl7.fhir.r4.model.*;
import org.hl7.fhir.r4.model.OperationOutcome.IssueSeverity;
import org.hl7.fhir.r4.model.OperationOutcome.IssueType;
import org.hl7.fhir.r4.model.OperationOutcome.OperationOutcomeIssueComponent;
import org.hl7.fhir.r4.model.Parameters.ParametersParameterComponent;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.client.RestClientTest;
import org.springframework.http.MediaType;
import org.springframework.retry.support.RetryTemplate;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.TestPropertySource;
import org.springframework.test.web.client.MockRestServiceServer;
import org.springframework.web.client.RestTemplate;
import java.time.Instant;
import java.util.Date;
import static org.assertj.core.api.Assertions.assertThat;
import static org.springframework.test.web.client.match.MockRestRequestMatchers.requestTo;
import static org.springframework.test.web.client.response.MockRestResponseCreators.withServerError;
import static org.springframework.test.web.client.response.MockRestResponseCreators.withSuccess;
@ContextConfiguration(classes = {AppConfiguration.class, ObjectMapper.class})
@TestPropertySource(properties = {
"app.consent.service=gics",
"app.consent.gics.uri=http://localhost:8090/ttp-fhir/fhir/gics"
})
@RestClientTest
class GicsConsentServiceTest {
static final String GICS_BASE_URI = "http://localhost:8090/ttp-fhir/fhir/gics";
MockRestServiceServer mockRestServiceServer;
AppFhirConfig appFhirConfig;
GIcsConfigProperties gIcsConfigProperties;
GicsConsentService gicsConsentService;
@BeforeEach
void setUp(
@Autowired AppFhirConfig appFhirConfig,
@Autowired GIcsConfigProperties gIcsConfigProperties
) {
this.appFhirConfig = appFhirConfig;
this.gIcsConfigProperties = gIcsConfigProperties;
var restTemplate = new RestTemplate();
this.mockRestServiceServer = MockRestServiceServer.createServer(restTemplate);
this.gicsConsentService = new GicsConsentService(
this.gIcsConfigProperties,
RetryTemplate.builder().maxAttempts(1).build(),
restTemplate,
this.appFhirConfig
);
}
@Test
void shouldReturnTtpBroadConsentStatus() {
final Parameters consentedResponse = new Parameters()
.addParameter(
new ParametersParameterComponent()
.setName("consented")
.setValue(new BooleanType().setValue(true))
);
mockRestServiceServer
.expect(
requestTo(
"http://localhost:8090/ttp-fhir/fhir/gics" + GicsConsentService.IS_CONSENTED_ENDPOINT)
)
.andRespond(
withSuccess(
appFhirConfig.fhirContext().newJsonParser().encodeResourceToString(consentedResponse),
MediaType.APPLICATION_JSON
)
);
var consentStatus = gicsConsentService.getTtpBroadConsentStatus("123456");
assertThat(consentStatus).isEqualTo(TtpConsentStatus.BROAD_CONSENT_GIVEN);
}
@Test
void shouldReturnRevokedConsent() {
final Parameters revokedResponse = new Parameters()
.addParameter(
new ParametersParameterComponent()
.setName("consented")
.setValue(new BooleanType().setValue(false))
);
mockRestServiceServer
.expect(
requestTo(
"http://localhost:8090/ttp-fhir/fhir/gics" + GicsConsentService.IS_CONSENTED_ENDPOINT)
)
.andRespond(
withSuccess(
appFhirConfig.fhirContext().newJsonParser().encodeResourceToString(revokedResponse),
MediaType.APPLICATION_JSON)
);
var consentStatus = gicsConsentService.getTtpBroadConsentStatus("123456");
assertThat(consentStatus).isEqualTo(TtpConsentStatus.BROAD_CONSENT_MISSING_OR_REJECTED);
}
@Test
void shouldReturnInvalidParameterResponse() {
final OperationOutcome responseWithErrorOutcome = new OperationOutcome()
.addIssue(
new OperationOutcomeIssueComponent()
.setSeverity(IssueSeverity.ERROR)
.setCode(IssueType.PROCESSING)
.setDiagnostics("Invalid policy parameter...")
);
mockRestServiceServer
.expect(
requestTo(GICS_BASE_URI + GicsConsentService.IS_CONSENTED_ENDPOINT)
)
.andRespond(
withSuccess(
appFhirConfig.fhirContext().newJsonParser().encodeResourceToString(responseWithErrorOutcome),
MediaType.APPLICATION_JSON
)
);
var consentStatus = gicsConsentService.getTtpBroadConsentStatus("123456");
assertThat(consentStatus).isEqualTo(TtpConsentStatus.FAILED_TO_ASK);
}
@Test
void shouldReturnRequestError() {
mockRestServiceServer
.expect(
requestTo(GICS_BASE_URI + GicsConsentService.IS_CONSENTED_ENDPOINT)
)
.andRespond(
withServerError()
);
var consentStatus = gicsConsentService.getTtpBroadConsentStatus("123456");
assertThat(consentStatus).isEqualTo(TtpConsentStatus.FAILED_TO_ASK);
}
@Test
void buildRequestParameterCurrentPolicyStatesForPersonTest() {
String pid = "12345678";
var result = gicsConsentService
.buildRequestParameterCurrentPolicyStatesForPerson(
pid,
Date.from(Instant.now()),
ConsentDomain.MODELLVORHABEN_64E
);
assertThat(result.getParameter())
.as("should contain 3 parameter resources")
.hasSize(3);
assertThat(((StringType) result.getParameter("domain").getValue()).getValue())
.isEqualTo(
gIcsConfigProperties.getGenomDeConsentDomainName()
);
assertThat(((Identifier) result.getParameter("personIdentifier").getValue()).getValue())
.isEqualTo(
pid
);
}
}

View File

@@ -0,0 +1,287 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.consent
import dev.dnpm.etl.processor.ArgProvider
import dev.pcvolkmer.mv64e.mtb.*
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Nested
import org.junit.jupiter.api.extension.ExtendWith
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.Arguments
import org.junit.jupiter.params.provider.ArgumentsSource
import org.mockito.ArgumentMatchers.anyString
import org.mockito.Mock
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.whenever
import java.time.Instant
import java.util.*
@ExtendWith(MockitoExtension::class)
class Dnpm21BasedConsentEvaluatorTest {
@Nested
inner class WithGicsConsentEnabled {
lateinit var consentService: GicsConsentService
lateinit var consentEvaluator: ConsentEvaluator
@BeforeEach
fun setUp(
@Mock consentService: GicsConsentService
) {
this.consentService = consentService
this.consentEvaluator = ConsentEvaluator(consentService)
}
@ParameterizedTest
@ArgumentsSource(WithGicsMtbFileProvider::class)
fun test(
mtbFile: Mtb,
ttpConsentStatus: TtpConsentStatus,
expectedConsentEvaluation: ConsentEvaluation
) {
whenever(consentService.getTtpBroadConsentStatus(anyString())).thenReturn(
ttpConsentStatus
)
assertThat(consentEvaluator.check(mtbFile)).isEqualTo(expectedConsentEvaluation)
}
}
@Nested
inner class WithFileConsentOnly {
lateinit var consentService: MtbFileConsentService
lateinit var consentEvaluator: ConsentEvaluator
@BeforeEach
fun setUp() {
this.consentService = MtbFileConsentService()
this.consentEvaluator = ConsentEvaluator(consentService)
}
@ParameterizedTest
@ArgumentsSource(MtbFileProvider::class)
fun test(mtbFile: Mtb, expectedConsentEvaluation: ConsentEvaluation) {
assertThat(consentEvaluator.check(mtbFile)).isEqualTo(expectedConsentEvaluation)
}
}
// Util classes
class WithGicsMtbFileProvider : ArgProvider(
// Has file ModelProjectConsent and broad consent => consent given
Arguments.of(
buildMtb(ConsentProvision.PERMIT),
TtpConsentStatus.BROAD_CONSENT_GIVEN,
ConsentEvaluation(TtpConsentStatus.BROAD_CONSENT_GIVEN, true)
),
// Has file ModelProjectConsent and broad consent missing => no consent given
Arguments.of(
buildMtb(ConsentProvision.PERMIT),
TtpConsentStatus.BROAD_CONSENT_MISSING,
ConsentEvaluation(TtpConsentStatus.BROAD_CONSENT_MISSING, false)
),
// Has file ModelProjectConsent and broad consent missing or rejected => no consent given
Arguments.of(
buildMtb(ConsentProvision.PERMIT),
TtpConsentStatus.BROAD_CONSENT_MISSING_OR_REJECTED,
ConsentEvaluation(TtpConsentStatus.BROAD_CONSENT_MISSING_OR_REJECTED, false)
),
// Has file ModelProjectConsent and MV consent => consent given
Arguments.of(
buildMtb(ConsentProvision.PERMIT),
TtpConsentStatus.GENOM_DE_CONSENT_SEQUENCING_PERMIT,
ConsentEvaluation(TtpConsentStatus.GENOM_DE_CONSENT_SEQUENCING_PERMIT, true)
),
// Has file ModelProjectConsent and MV consent rejected => no consent given
Arguments.of(
buildMtb(ConsentProvision.PERMIT),
TtpConsentStatus.GENOM_DE_SEQUENCING_REJECTED,
ConsentEvaluation(TtpConsentStatus.GENOM_DE_SEQUENCING_REJECTED, false)
),
// Has file ModelProjectConsent and MV consent missing => no consent given
Arguments.of(
buildMtb(ConsentProvision.PERMIT),
TtpConsentStatus.GENOM_DE_CONSENT_MISSING,
ConsentEvaluation(TtpConsentStatus.GENOM_DE_CONSENT_MISSING, false)
),
// Has file ModelProjectConsent and no broad consent result => consent given
Arguments.of(
buildMtb(ConsentProvision.PERMIT),
TtpConsentStatus.UNKNOWN_CHECK_FILE,
ConsentEvaluation(TtpConsentStatus.UNKNOWN_CHECK_FILE, true)
),
// Has file ModelProjectConsent and failed to ask => no consent given
Arguments.of(
buildMtb(ConsentProvision.PERMIT),
TtpConsentStatus.FAILED_TO_ASK,
ConsentEvaluation(TtpConsentStatus.FAILED_TO_ASK, false)
),
// File ModelProjectConsent rejected and broad consent => consent given
Arguments.of(
buildMtb(ConsentProvision.DENY),
TtpConsentStatus.BROAD_CONSENT_GIVEN,
ConsentEvaluation(TtpConsentStatus.BROAD_CONSENT_GIVEN, true)
),
// File ModelProjectConsent rejected and broad consent missing => no consent given
Arguments.of(
buildMtb(ConsentProvision.DENY),
TtpConsentStatus.BROAD_CONSENT_MISSING,
ConsentEvaluation(TtpConsentStatus.BROAD_CONSENT_MISSING, false)
),
// File ModelProjectConsent rejected and broad consent missing or rejected => no consent given
Arguments.of(
buildMtb(ConsentProvision.DENY),
TtpConsentStatus.BROAD_CONSENT_MISSING_OR_REJECTED,
ConsentEvaluation(TtpConsentStatus.BROAD_CONSENT_MISSING_OR_REJECTED, false)
),
// File ModelProjectConsent rejected and MV consent => consent given
Arguments.of(
buildMtb(ConsentProvision.DENY),
TtpConsentStatus.GENOM_DE_CONSENT_SEQUENCING_PERMIT,
ConsentEvaluation(TtpConsentStatus.GENOM_DE_CONSENT_SEQUENCING_PERMIT, true)
),
// File ModelProjectConsent rejected and MV consent rejected => no consent given
Arguments.of(
buildMtb(ConsentProvision.DENY),
TtpConsentStatus.GENOM_DE_SEQUENCING_REJECTED,
ConsentEvaluation(TtpConsentStatus.GENOM_DE_SEQUENCING_REJECTED, false)
),
// File ModelProjectConsent rejected and MV consent missing => no consent given
Arguments.of(
buildMtb(ConsentProvision.DENY),
TtpConsentStatus.GENOM_DE_CONSENT_MISSING,
ConsentEvaluation(TtpConsentStatus.GENOM_DE_CONSENT_MISSING, false)
),
// File ModelProjectConsent rejected and no broad consent result => no consent given
Arguments.of(
buildMtb(ConsentProvision.DENY),
TtpConsentStatus.UNKNOWN_CHECK_FILE,
ConsentEvaluation(TtpConsentStatus.UNKNOWN_CHECK_FILE, false)
),
// File ModelProjectConsent rejected and failed to ask => no consent given
Arguments.of(
buildMtb(ConsentProvision.DENY),
TtpConsentStatus.FAILED_TO_ASK,
ConsentEvaluation(TtpConsentStatus.FAILED_TO_ASK, false)
)
) {
companion object {
fun buildMtb(consentProvision: ConsentProvision): Mtb {
return Mtb.builder()
.patient(
Patient.builder().id("TEST_12345678")
.birthDate(Date.from(Instant.parse("2000-08-08T12:34:56Z"))).gender(
GenderCoding.builder().code(GenderCodingCode.MALE).build()
).build()
)
.metadata(
MvhMetadata.builder().modelProjectConsent(
ModelProjectConsent.builder().provisions(
listOf(
Provision.builder().date(Date()).type(consentProvision)
.purpose(ModelProjectConsentPurpose.SEQUENCING).build()
)
).build()
).build()
)
.episodesOfCare(
listOf(
MtbEpisodeOfCare.builder().id("1")
.patient(Reference.builder().id("TEST_12345678").build())
.build()
)
)
.build()
}
}
}
class MtbFileProvider : ArgProvider(
// Has file consent => consent given
Arguments.of(
buildMtb(ConsentProvision.PERMIT),
ConsentEvaluation(TtpConsentStatus.UNKNOWN_CHECK_FILE, true)
),
// File consent rejected => no consent given
Arguments.of(
buildMtb(ConsentProvision.DENY),
ConsentEvaluation(TtpConsentStatus.UNKNOWN_CHECK_FILE, false)
),
// policy REIDENTIFICATION has no effect on ConsentEvaluation
Arguments.of(
buildMtb(ModelProjectConsentPurpose.REIDENTIFICATION, ConsentProvision.DENY),
ConsentEvaluation(TtpConsentStatus.UNKNOWN_CHECK_FILE, false)
), Arguments.of(
buildMtb(ModelProjectConsentPurpose.REIDENTIFICATION, ConsentProvision.PERMIT),
ConsentEvaluation(TtpConsentStatus.UNKNOWN_CHECK_FILE, false)
),
// policy CASE_IDENTIFICATION has no effect on ConsentEvaluation
Arguments.of(
buildMtb(ModelProjectConsentPurpose.CASE_IDENTIFICATION, ConsentProvision.DENY),
ConsentEvaluation(TtpConsentStatus.UNKNOWN_CHECK_FILE, false)
), Arguments.of(
buildMtb(ModelProjectConsentPurpose.CASE_IDENTIFICATION, ConsentProvision.PERMIT),
ConsentEvaluation(TtpConsentStatus.UNKNOWN_CHECK_FILE, false)
)
) {
companion object {
fun buildMtb(consentProvision: ConsentProvision): Mtb {
return buildMtb(ModelProjectConsentPurpose.SEQUENCING, consentProvision)
}
fun buildMtb(
policy: ModelProjectConsentPurpose,
consentProvision: ConsentProvision
): Mtb {
return Mtb.builder()
.patient(
Patient.builder().id("TEST_12345678")
.birthDate(Date.from(Instant.parse("2000-08-08T12:34:56Z"))).gender(
GenderCoding.builder().code(GenderCodingCode.MALE).build()
).build()
)
.metadata(
MvhMetadata.builder().modelProjectConsent(
ModelProjectConsent.builder().provisions(
listOf(
Provision.builder().date(Date()).type(consentProvision)
.purpose(policy).build()
)
).build()
).build()
)
.episodesOfCare(
listOf(
MtbEpisodeOfCare.builder().id("1")
.patient(Reference.builder().id("TEST_12345678").build())
.build()
)
)
.build()
}
}
}
}

View File

@@ -0,0 +1,31 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor
import org.junit.jupiter.api.extension.ExtensionContext
import org.junit.jupiter.params.provider.Arguments
import org.junit.jupiter.params.provider.ArgumentsProvider
import java.util.stream.Stream
open class ArgProvider(vararg val data: Arguments) : ArgumentsProvider {
override fun provideArguments(
context: ExtensionContext?
): Stream<out Arguments> = Stream.of(*data)
}

View File

@@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@@ -20,10 +20,12 @@
package dev.dnpm.etl.processor.input
import com.fasterxml.jackson.databind.ObjectMapper
import de.ukw.ccc.bwhc.dto.Consent
import de.ukw.ccc.bwhc.dto.MtbFile
import de.ukw.ccc.bwhc.dto.Patient
import dev.dnpm.etl.processor.CustomMediaType
import dev.dnpm.etl.processor.consent.ConsentEvaluation
import dev.dnpm.etl.processor.consent.ConsentEvaluator
import dev.dnpm.etl.processor.consent.TtpConsentStatus
import dev.dnpm.etl.processor.services.RequestProcessor
import dev.pcvolkmer.mv64e.mtb.*
import org.apache.kafka.clients.consumer.ConsumerRecord
import org.apache.kafka.common.header.internals.RecordHeader
import org.apache.kafka.common.header.internals.RecordHeaders
@@ -31,82 +33,270 @@ import org.apache.kafka.common.record.TimestampType
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.ArgumentMatchers.anyString
import org.mockito.Mock
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.any
import org.mockito.kotlin.times
import org.mockito.kotlin.verify
import org.mockito.kotlin.*
import java.util.*
@ExtendWith(MockitoExtension::class)
class KafkaInputListenerTest {
private lateinit var requestProcessor: RequestProcessor
private lateinit var consentEvaluator: ConsentEvaluator
private lateinit var objectMapper: ObjectMapper
private lateinit var kafkaInputListener: KafkaInputListener
@BeforeEach
fun setup(
@Mock requestProcessor: RequestProcessor
@Mock requestProcessor: RequestProcessor,
@Mock consentEvaluator: ConsentEvaluator,
) {
this.requestProcessor = requestProcessor
this.consentEvaluator = consentEvaluator
this.objectMapper = ObjectMapper()
this.kafkaInputListener = KafkaInputListener(requestProcessor, objectMapper)
this.kafkaInputListener = KafkaInputListener(requestProcessor, consentEvaluator, objectMapper)
}
@Test
fun shouldProcessMtbFileRequest() {
val mtbFile = MtbFile.builder()
.withPatient(Patient.builder().withId("DUMMY_12345678").build())
.withConsent(Consent.builder().withStatus(Consent.Status.ACTIVE).build())
whenever(consentEvaluator.check(any())).thenReturn(
ConsentEvaluation(
TtpConsentStatus.BROAD_CONSENT_GIVEN,
true
)
)
val mtbFile = Mtb.builder()
.patient(Patient.builder().id("DUMMY_12345678").build())
.metadata(
MvhMetadata
.builder()
.modelProjectConsent(
ModelProjectConsent
.builder()
.provisions(
listOf(
Provision.builder().type(ConsentProvision.PERMIT)
.purpose(ModelProjectConsentPurpose.SEQUENCING).build()
)
).build()
)
.build()
)
.build()
kafkaInputListener.onMessage(ConsumerRecord("testtopic", 0, 0, "", this.objectMapper.writeValueAsString(mtbFile)))
kafkaInputListener.onMessage(
ConsumerRecord(
"testtopic",
0,
0,
"",
this.objectMapper.writeValueAsString(mtbFile)
)
)
verify(requestProcessor, times(1)).processMtbFile(any())
verify(requestProcessor, times(1)).processMtbFile(any<Mtb>())
}
@Test
fun shouldProcessDeleteRequest() {
val mtbFile = MtbFile.builder()
.withPatient(Patient.builder().withId("DUMMY_12345678").build())
.withConsent(Consent.builder().withStatus(Consent.Status.REJECTED).build())
whenever(consentEvaluator.check(any())).thenReturn(
ConsentEvaluation(
TtpConsentStatus.BROAD_CONSENT_GIVEN,
false
)
)
val mtbFile = Mtb.builder()
.patient(Patient.builder().id("DUMMY_12345678").build())
.metadata(
MvhMetadata
.builder()
.modelProjectConsent(
ModelProjectConsent
.builder()
.provisions(
listOf(
Provision.builder().type(ConsentProvision.DENY)
.purpose(ModelProjectConsentPurpose.SEQUENCING).build()
)
).build()
)
.build()
)
.build()
kafkaInputListener.onMessage(ConsumerRecord("testtopic", 0, 0, "", this.objectMapper.writeValueAsString(mtbFile)))
kafkaInputListener.onMessage(
ConsumerRecord(
"testtopic",
0,
0,
"",
this.objectMapper.writeValueAsString(mtbFile)
)
)
verify(requestProcessor, times(1)).processDeletion(anyString())
verify(requestProcessor, times(1)).processDeletion(
anyValueClass(),
eq(TtpConsentStatus.UNKNOWN_CHECK_FILE)
)
}
@Test
fun shouldProcessMtbFileRequestWithExistingRequestId() {
val mtbFile = MtbFile.builder()
.withPatient(Patient.builder().withId("DUMMY_12345678").build())
.withConsent(Consent.builder().withStatus(Consent.Status.ACTIVE).build())
whenever(consentEvaluator.check(any())).thenReturn(
ConsentEvaluation(
TtpConsentStatus.BROAD_CONSENT_GIVEN,
true
)
)
val mtbFile = Mtb.builder()
.patient(Patient.builder().id("DUMMY_12345678").build())
.metadata(
MvhMetadata
.builder()
.modelProjectConsent(
ModelProjectConsent
.builder()
.provisions(
listOf(
Provision.builder().type(ConsentProvision.PERMIT)
.purpose(ModelProjectConsentPurpose.SEQUENCING).build()
)
).build()
)
.build()
)
.build()
val headers = RecordHeaders(listOf(RecordHeader("requestId", UUID.randomUUID().toString().toByteArray())))
kafkaInputListener.onMessage(
ConsumerRecord("testtopic", 0, 0, -1L, TimestampType.NO_TIMESTAMP_TYPE, -1, -1, "", this.objectMapper.writeValueAsString(mtbFile), headers, Optional.empty())
ConsumerRecord(
"testtopic",
0,
0,
-1L,
TimestampType.NO_TIMESTAMP_TYPE,
-1,
-1,
"",
this.objectMapper.writeValueAsString(mtbFile),
headers,
Optional.empty()
)
)
verify(requestProcessor, times(1)).processMtbFile(any(), anyString())
verify(requestProcessor, times(1)).processMtbFile(any<Mtb>(), anyValueClass())
}
@Test
fun shouldProcessDeleteRequestWithExistingRequestId() {
val mtbFile = MtbFile.builder()
.withPatient(Patient.builder().withId("DUMMY_12345678").build())
.withConsent(Consent.builder().withStatus(Consent.Status.REJECTED).build())
whenever(consentEvaluator.check(any())).thenReturn(
ConsentEvaluation(
TtpConsentStatus.BROAD_CONSENT_GIVEN,
false
)
)
val mtbFile = Mtb.builder()
.patient(Patient.builder().id("DUMMY_12345678").build())
.metadata(
MvhMetadata
.builder()
.modelProjectConsent(
ModelProjectConsent
.builder()
.provisions(
listOf(
Provision.builder().type(ConsentProvision.DENY)
.purpose(ModelProjectConsentPurpose.SEQUENCING).build()
)
).build()
)
.build()
)
.build()
val headers = RecordHeaders(listOf(RecordHeader("requestId", UUID.randomUUID().toString().toByteArray())))
kafkaInputListener.onMessage(
ConsumerRecord("testtopic", 0, 0, -1L, TimestampType.NO_TIMESTAMP_TYPE, -1, -1, "", this.objectMapper.writeValueAsString(mtbFile), headers, Optional.empty())
ConsumerRecord(
"testtopic",
0,
0,
-1L,
TimestampType.NO_TIMESTAMP_TYPE,
-1,
-1,
"",
this.objectMapper.writeValueAsString(mtbFile),
headers,
Optional.empty()
)
)
verify(requestProcessor, times(1)).processDeletion(
anyValueClass(), anyValueClass(), eq(
TtpConsentStatus.UNKNOWN_CHECK_FILE
)
)
verify(requestProcessor, times(1)).processDeletion(anyString(), anyString())
}
}
@Test
fun shouldProcessDnpmV2Request() {
whenever(consentEvaluator.check(any())).thenReturn(
ConsentEvaluation(
TtpConsentStatus.BROAD_CONSENT_GIVEN,
false
)
)
val mtbFile = Mtb.builder()
.patient(Patient.builder().id("DUMMY_12345678").build())
.metadata(
MvhMetadata
.builder()
.modelProjectConsent(
ModelProjectConsent
.builder()
.provisions(
listOf(
Provision.builder().type(ConsentProvision.DENY)
.purpose(ModelProjectConsentPurpose.SEQUENCING).build()
)
).build()
)
.build()
)
.build()
val headers = RecordHeaders(
listOf(
RecordHeader("requestId", UUID.randomUUID().toString().toByteArray()),
RecordHeader("contentType", CustomMediaType.APPLICATION_VND_DNPM_V2_MTB_JSON_VALUE.toByteArray())
)
)
kafkaInputListener.onMessage(
ConsumerRecord(
"testtopic",
0,
0,
-1L,
TimestampType.NO_TIMESTAMP_TYPE,
-1,
-1,
"",
this.objectMapper.writeValueAsString(mtbFile),
headers,
Optional.empty()
)
)
verify(requestProcessor, times(1)).processDeletion(
anyValueClass(), anyValueClass(), eq(
TtpConsentStatus.UNKNOWN_CHECK_FILE
)
)
}
}

View File

@@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@@ -20,131 +20,207 @@
package dev.dnpm.etl.processor.input
import com.fasterxml.jackson.databind.ObjectMapper
import de.ukw.ccc.bwhc.dto.*
import dev.dnpm.etl.processor.ArgProvider
import dev.dnpm.etl.processor.CustomMediaType
import dev.dnpm.etl.processor.consent.ConsentEvaluation
import dev.dnpm.etl.processor.consent.ConsentEvaluator
import dev.dnpm.etl.processor.consent.TtpConsentStatus
import dev.dnpm.etl.processor.services.RequestProcessor
import org.assertj.core.api.Assertions.assertThat
import dev.pcvolkmer.mv64e.mtb.*
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Nested
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.Arguments
import org.junit.jupiter.params.provider.ArgumentsSource
import org.mockito.Mock
import org.mockito.Mockito.times
import org.mockito.Mockito.verify
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.any
import org.mockito.kotlin.argumentCaptor
import org.springframework.http.MediaType
import org.mockito.kotlin.anyValueClass
import org.mockito.kotlin.whenever
import org.springframework.core.io.ClassPathResource
import org.springframework.test.web.servlet.MockMvc
import org.springframework.test.web.servlet.delete
import org.springframework.test.web.servlet.post
import org.springframework.test.web.servlet.setup.MockMvcBuilders
import java.time.Instant
import java.util.*
@ExtendWith(MockitoExtension::class)
class MtbFileRestControllerTest {
private lateinit var mockMvc: MockMvc
private lateinit var requestProcessor: RequestProcessor
private val objectMapper = ObjectMapper()
@BeforeEach
fun setup(
@Mock requestProcessor: RequestProcessor
) {
this.requestProcessor = requestProcessor
val controller = MtbFileRestController(requestProcessor)
this.mockMvc = MockMvcBuilders.standaloneSetup(controller).build()
}
@Nested
inner class RequestsForDnpmDataModel21 {
@Test
fun shouldProcessMtbFilePostRequest() {
val mtbFile = MtbFile.builder()
.withPatient(
Patient.builder()
.withId("TEST_12345678")
.withBirthDate("2000-08-08")
.withGender(Patient.Gender.MALE)
.build()
)
.withConsent(
Consent.builder()
.withId("1")
.withStatus(Consent.Status.ACTIVE)
.withPatient("TEST_12345678")
.build()
)
.withEpisode(
Episode.builder()
.withId("1")
.withPatient("TEST_12345678")
.withPeriod(PeriodStart("2023-08-08"))
.build()
)
.build()
private lateinit var mockMvc: MockMvc
mockMvc.post("/mtbfile") {
content = objectMapper.writeValueAsString(mtbFile)
contentType = MediaType.APPLICATION_JSON
}.andExpect {
status {
isAccepted()
private lateinit var requestProcessor: RequestProcessor
private lateinit var consentEvaluator: ConsentEvaluator
@BeforeEach
fun setup(
@Mock requestProcessor: RequestProcessor,
@Mock consentEvaluator: ConsentEvaluator
) {
this.requestProcessor = requestProcessor
this.consentEvaluator = consentEvaluator
val controller = MtbFileRestController(
requestProcessor,
consentEvaluator
)
this.mockMvc = MockMvcBuilders.standaloneSetup(controller).build()
}
@Test
fun shouldRespondPostRequest() {
whenever(consentEvaluator.check(any())).thenReturn(
ConsentEvaluation(
TtpConsentStatus.BROAD_CONSENT_GIVEN,
true
)
)
val mtbFileContent =
ClassPathResource("mv64e-mtb-fake-patient.json").inputStream.readAllBytes().toString(Charsets.UTF_8)
mockMvc.post("/mtb") {
content = mtbFileContent
contentType = CustomMediaType.APPLICATION_VND_DNPM_V2_MTB_JSON
}.andExpect {
status {
isAccepted()
}
}
verify(requestProcessor, times(1)).processMtbFile(any<Mtb>())
}
@ParameterizedTest
@ArgumentsSource(Dnpm21MtbFile::class)
fun shouldProcessPostRequest(mtb: Mtb, broadConsent: TtpConsentStatus, shouldProcess: String) {
whenever(consentEvaluator.check(any<Mtb>())).thenReturn(
ConsentEvaluation(
broadConsent,
shouldProcess == "process"
)
)
mockMvc.post("/mtbfile") {
content = objectMapper.writeValueAsString(mtb)
contentType = CustomMediaType.APPLICATION_VND_DNPM_V2_MTB_JSON
}.andExpect {
status {
isAccepted()
}
}
if (shouldProcess == "process") {
verify(requestProcessor, times(1)).processMtbFile(any<Mtb>())
} else {
verify(requestProcessor, times(1)).processDeletion(
anyValueClass(),
org.mockito.kotlin.eq(broadConsent)
)
}
}
verify(requestProcessor, times(1)).processMtbFile(any())
}
@Test
fun shouldProcessMtbFilePostRequestWithRejectedConsent() {
val mtbFile = MtbFile.builder()
.withPatient(
Patient.builder()
.withId("TEST_12345678")
.withBirthDate("2000-08-08")
.withGender(Patient.Gender.MALE)
.build()
)
.withConsent(
Consent.builder()
.withId("1")
.withStatus(Consent.Status.REJECTED)
.withPatient("TEST_12345678")
.build()
)
.withEpisode(
Episode.builder()
.withId("1")
.withPatient("TEST_12345678")
.withPeriod(PeriodStart("2023-08-08"))
.build()
)
.build()
mockMvc.post("/mtbfile") {
content = objectMapper.writeValueAsString(mtbFile)
contentType = MediaType.APPLICATION_JSON
}.andExpect {
status {
isAccepted()
@Test
fun shouldProcessDeleteRequest() {
mockMvc.delete("/mtbfile/TEST_12345678").andExpect {
status {
isAccepted()
}
}
verify(requestProcessor, times(1)).processDeletion(
anyValueClass(),
org.mockito.kotlin.eq(TtpConsentStatus.UNKNOWN_CHECK_FILE)
)
verify(consentEvaluator, times(0)).check(any<Mtb>())
}
val captor = argumentCaptor<String>()
verify(requestProcessor, times(1)).processDeletion(captor.capture())
assertThat(captor.firstValue).isEqualTo("TEST_12345678")
}
}
@Test
fun shouldProcessMtbFileDeleteRequest() {
mockMvc.delete("/mtbfile/TEST_12345678").andExpect {
status {
isAccepted()
}
class Dnpm21MtbFile : ArgProvider(
// No Metadata and no broad consent => delete
Arguments.of(
buildMtb(null),
TtpConsentStatus.BROAD_CONSENT_MISSING_OR_REJECTED,
"delete"
),
// No Metadata and broad consent given => process
Arguments.of(
buildMtb(null),
TtpConsentStatus.BROAD_CONSENT_GIVEN,
"process"
),
// No model project consent and no broad consent => delete
Arguments.of(
buildMtb(MvhMetadata.builder().modelProjectConsent(ModelProjectConsent.builder().build()).build()),
TtpConsentStatus.BROAD_CONSENT_MISSING_OR_REJECTED,
"delete"
),
// No model project consent and broad consent given => process
Arguments.of(
buildMtb(MvhMetadata.builder().modelProjectConsent(ModelProjectConsent.builder().build()).build()),
TtpConsentStatus.BROAD_CONSENT_GIVEN,
"process"
),
// Model project consent given and no broad consent => process
Arguments.of(
buildMtb(
MvhMetadata.builder().modelProjectConsent(
ModelProjectConsent.builder().provisions(
listOf(
Provision.builder().date(Date()).type(ConsentProvision.PERMIT)
.purpose(ModelProjectConsentPurpose.SEQUENCING).build()
)
).build()
).build()
),
TtpConsentStatus.UNKNOWN_CHECK_FILE,
"process"
),
// Model project consent given and broad consent given => process
Arguments.of(
buildMtb(
MvhMetadata.builder().modelProjectConsent(
ModelProjectConsent.builder().provisions(
listOf(
Provision.builder().date(Date()).type(ConsentProvision.PERMIT)
.purpose(ModelProjectConsentPurpose.SEQUENCING).build()
)
).build()
).build()
),
TtpConsentStatus.BROAD_CONSENT_GIVEN,
"process"
)
) {
companion object {
fun buildMtb(metadata: MvhMetadata?): Mtb {
return Mtb.builder()
.patient(
Patient.builder().id("TEST_12345678")
.birthDate(Date.from(Instant.parse("2000-08-08T12:34:56Z"))).gender(
GenderCoding.builder().code(GenderCodingCode.MALE).build()
).build()
)
.metadata(metadata)
.episodesOfCare(
listOf(
MtbEpisodeOfCare.builder().id("1")
.patient(Reference.builder().id("TEST_12345678").build())
.build()
)
)
.build()
}
val captor = argumentCaptor<String>()
verify(requestProcessor, times(1)).processDeletion(captor.capture())
assertThat(captor.firstValue).isEqualTo("TEST_12345678")
}
}
}

View File

@@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@@ -20,16 +20,20 @@
package dev.dnpm.etl.processor.output
import com.fasterxml.jackson.databind.ObjectMapper
import de.ukw.ccc.bwhc.dto.*
import dev.dnpm.etl.processor.CustomMediaType
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.RequestId
import dev.dnpm.etl.processor.config.KafkaProperties
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.pcvolkmer.mv64e.mtb.*
import org.apache.kafka.clients.producer.ProducerRecord
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Nested
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.MethodSource
import org.mockito.ArgumentMatchers.anyString
import org.mockito.Mock
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.*
@@ -37,181 +41,189 @@ import org.springframework.kafka.core.KafkaTemplate
import org.springframework.kafka.support.SendResult
import org.springframework.retry.policy.SimpleRetryPolicy
import org.springframework.retry.support.RetryTemplateBuilder
import java.time.Instant
import java.util.*
import java.util.concurrent.CompletableFuture.completedFuture
import java.util.concurrent.ExecutionException
@ExtendWith(MockitoExtension::class)
class KafkaMtbFileSenderTest {
private lateinit var kafkaTemplate: KafkaTemplate<String, String>
@Nested
inner class BwhcV1Record {
private lateinit var kafkaMtbFileSender: KafkaMtbFileSender
private lateinit var kafkaTemplate: KafkaTemplate<String, String>
private lateinit var objectMapper: ObjectMapper
private lateinit var kafkaMtbFileSender: KafkaMtbFileSender
@BeforeEach
fun setup(
@Mock kafkaTemplate: KafkaTemplate<String, String>
) {
val kafkaProperties = KafkaProperties("testtopic")
val retryTemplate = RetryTemplateBuilder().customPolicy(SimpleRetryPolicy(1)).build()
private lateinit var objectMapper: ObjectMapper
this.objectMapper = ObjectMapper()
this.kafkaTemplate = kafkaTemplate
@BeforeEach
fun setup(
@Mock kafkaTemplate: KafkaTemplate<String, String>
) {
val kafkaProperties = KafkaProperties("testtopic")
val retryTemplate = RetryTemplateBuilder().customPolicy(SimpleRetryPolicy(1)).build()
this.kafkaMtbFileSender = KafkaMtbFileSender(kafkaTemplate, kafkaProperties, retryTemplate, objectMapper)
}
this.objectMapper = ObjectMapper()
this.kafkaTemplate = kafkaTemplate
@ParameterizedTest
@MethodSource("requestWithResponseSource")
fun shouldSendMtbFileRequestAndReturnExpectedState(testData: TestData) {
doAnswer {
if (null != testData.exception) {
throw testData.exception
}
completedFuture(SendResult<String, String>(null, null))
}.whenever(kafkaTemplate).send(anyString(), anyString(), anyString())
val response = kafkaMtbFileSender.send(MtbFileSender.MtbFileRequest("TestID", mtbFile(Consent.Status.ACTIVE)))
assertThat(response.status).isEqualTo(testData.requestStatus)
}
@ParameterizedTest
@MethodSource("requestWithResponseSource")
fun shouldSendDeleteRequestAndReturnExpectedState(testData: TestData) {
doAnswer {
if (null != testData.exception) {
throw testData.exception
}
completedFuture(SendResult<String, String>(null, null))
}.whenever(kafkaTemplate).send(anyString(), anyString(), anyString())
val response = kafkaMtbFileSender.send(MtbFileSender.DeleteRequest("TestID", "PID"))
assertThat(response.status).isEqualTo(testData.requestStatus)
}
@Test
fun shouldSendMtbFileRequestWithCorrectKeyAndBody() {
doAnswer {
completedFuture(SendResult<String, String>(null, null))
}.whenever(kafkaTemplate).send(anyString(), anyString(), anyString())
kafkaMtbFileSender.send(MtbFileSender.MtbFileRequest("TestID", mtbFile(Consent.Status.ACTIVE)))
val captor = argumentCaptor<String>()
verify(kafkaTemplate, times(1)).send(anyString(), captor.capture(), captor.capture())
assertThat(captor.firstValue).isNotNull
assertThat(captor.firstValue).isEqualTo("{\"pid\": \"PID\"}")
assertThat(captor.secondValue).isNotNull
assertThat(captor.secondValue).isEqualTo(objectMapper.writeValueAsString(kafkaRecordData("TestID", Consent.Status.ACTIVE)))
}
@Test
fun shouldSendDeleteRequestWithCorrectKeyAndBody() {
doAnswer {
completedFuture(SendResult<String, String>(null, null))
}.whenever(kafkaTemplate).send(anyString(), anyString(), anyString())
kafkaMtbFileSender.send(MtbFileSender.DeleteRequest("TestID", "PID"))
val captor = argumentCaptor<String>()
verify(kafkaTemplate, times(1)).send(anyString(), captor.capture(), captor.capture())
assertThat(captor.firstValue).isNotNull
assertThat(captor.firstValue).isEqualTo("{\"pid\": \"PID\"}")
assertThat(captor.secondValue).isNotNull
assertThat(captor.secondValue).isEqualTo(objectMapper.writeValueAsString(kafkaRecordData("TestID", Consent.Status.REJECTED)))
}
@ParameterizedTest
@MethodSource("requestWithResponseSource")
fun shouldRetryOnMtbFileKafkaSendError(testData: TestData) {
val kafkaProperties = KafkaProperties("testtopic")
val retryTemplate = RetryTemplateBuilder().customPolicy(SimpleRetryPolicy(3)).build()
this.kafkaMtbFileSender = KafkaMtbFileSender(this.kafkaTemplate, kafkaProperties, retryTemplate, this.objectMapper)
doAnswer {
if (null != testData.exception) {
throw testData.exception
}
completedFuture(SendResult<String, String>(null, null))
}.whenever(kafkaTemplate).send(anyString(), anyString(), anyString())
kafkaMtbFileSender.send(MtbFileSender.MtbFileRequest("TestID", mtbFile(Consent.Status.ACTIVE)))
val expectedCount = when (testData.exception) {
// OK - No Retry
null -> times(1)
// Request failed - Retry max 3 times
else -> times(3)
this.kafkaMtbFileSender = KafkaMtbFileSender(kafkaTemplate, kafkaProperties, retryTemplate, objectMapper)
}
verify(kafkaTemplate, expectedCount).send(anyString(), anyString(), anyString())
}
@ParameterizedTest
@MethodSource("dev.dnpm.etl.processor.output.KafkaMtbFileSenderTest#requestWithResponseSource")
fun shouldSendDeleteRequestAndReturnExpectedState(testData: TestData) {
doAnswer {
if (null != testData.exception) {
throw testData.exception
}
completedFuture(SendResult<String, String>(null, null))
}.whenever(kafkaTemplate).send(any<ProducerRecord<String, String>>())
@ParameterizedTest
@MethodSource("requestWithResponseSource")
fun shouldRetryOnDeleteKafkaSendError(testData: TestData) {
val kafkaProperties = KafkaProperties("testtopic")
val retryTemplate = RetryTemplateBuilder().customPolicy(SimpleRetryPolicy(3)).build()
this.kafkaMtbFileSender = KafkaMtbFileSender(this.kafkaTemplate, kafkaProperties, retryTemplate, this.objectMapper)
doAnswer {
if (null != testData.exception) {
throw testData.exception
}
completedFuture(SendResult<String, String>(null, null))
}.whenever(kafkaTemplate).send(anyString(), anyString(), anyString())
kafkaMtbFileSender.send(MtbFileSender.DeleteRequest("TestID", "PID"))
val expectedCount = when (testData.exception) {
// OK - No Retry
null -> times(1)
// Request failed - Retry max 3 times
else -> times(3)
val response = kafkaMtbFileSender.send(DeleteRequest(TEST_REQUEST_ID, TEST_PATIENT_PSEUDONYM))
assertThat(response.status).isEqualTo(testData.requestStatus)
}
@ParameterizedTest
@MethodSource("dev.dnpm.etl.processor.output.KafkaMtbFileSenderTest#requestWithResponseSource")
fun shouldRetryOnDeleteKafkaSendError(testData: TestData) {
val kafkaProperties = KafkaProperties("testtopic")
val retryTemplate = RetryTemplateBuilder().customPolicy(SimpleRetryPolicy(3)).build()
this.kafkaMtbFileSender = KafkaMtbFileSender(this.kafkaTemplate, kafkaProperties, retryTemplate, this.objectMapper)
doAnswer {
if (null != testData.exception) {
throw testData.exception
}
completedFuture(SendResult<String, String>(null, null))
}.whenever(kafkaTemplate).send(any<ProducerRecord<String, String>>())
kafkaMtbFileSender.send(DeleteRequest(TEST_REQUEST_ID, TEST_PATIENT_PSEUDONYM))
val expectedCount = when (testData.exception) {
// OK - No Retry
null -> times(1)
// Request failed - Retry max 3 times
else -> times(3)
}
verify(kafkaTemplate, expectedCount).send(any<ProducerRecord<String, String>>())
}
}
@Nested
inner class DnpmV2Record {
private lateinit var kafkaTemplate: KafkaTemplate<String, String>
private lateinit var kafkaMtbFileSender: KafkaMtbFileSender
private lateinit var objectMapper: ObjectMapper
@BeforeEach
fun setup(
@Mock kafkaTemplate: KafkaTemplate<String, String>
) {
val kafkaProperties = KafkaProperties("testtopic")
val retryTemplate = RetryTemplateBuilder().customPolicy(SimpleRetryPolicy(1)).build()
this.objectMapper = ObjectMapper()
this.kafkaTemplate = kafkaTemplate
this.kafkaMtbFileSender = KafkaMtbFileSender(kafkaTemplate, kafkaProperties, retryTemplate, objectMapper)
}
@ParameterizedTest
@MethodSource("dev.dnpm.etl.processor.output.KafkaMtbFileSenderTest#requestWithResponseSource")
fun shouldSendMtbFileRequestAndReturnExpectedState(testData: TestData) {
doAnswer {
if (null != testData.exception) {
throw testData.exception
}
completedFuture(SendResult<String, String>(null, null))
}.whenever(kafkaTemplate).send(any<ProducerRecord<String, String>>())
val response = kafkaMtbFileSender.send(DnpmV2MtbFileRequest(TEST_REQUEST_ID, dnpmV2MtbFile()))
assertThat(response.status).isEqualTo(testData.requestStatus)
}
@Test
fun shouldSendMtbFileRequestWithCorrectKeyAndHeaderAndBody() {
doAnswer {
completedFuture(SendResult<String, String>(null, null))
}.whenever(kafkaTemplate).send(any<ProducerRecord<String, String>>())
kafkaMtbFileSender.send(DnpmV2MtbFileRequest(TEST_REQUEST_ID, dnpmV2MtbFile()))
val captor = argumentCaptor<ProducerRecord<String, String>>()
verify(kafkaTemplate, times(1)).send(captor.capture())
assertThat(captor.firstValue.key()).isNotNull
assertThat(captor.firstValue.key()).isEqualTo("{\"pid\": \"PID\"}")
assertThat(captor.firstValue.headers().headers("contentType")).isNotNull
assertThat(captor.firstValue.headers().headers("contentType")?.firstOrNull()?.value()).isEqualTo(CustomMediaType.APPLICATION_VND_DNPM_V2_MTB_JSON_VALUE.toByteArray())
assertThat(captor.firstValue.value()).isNotNull
assertThat(captor.firstValue.value()).isEqualTo(objectMapper.writeValueAsString(dnmpV2kafkaRecordData(TEST_REQUEST_ID)))
}
@ParameterizedTest
@MethodSource("dev.dnpm.etl.processor.output.KafkaMtbFileSenderTest#requestWithResponseSource")
fun shouldRetryOnMtbFileKafkaSendError(testData: TestData) {
val kafkaProperties = KafkaProperties("testtopic")
val retryTemplate = RetryTemplateBuilder().customPolicy(SimpleRetryPolicy(3)).build()
this.kafkaMtbFileSender = KafkaMtbFileSender(this.kafkaTemplate, kafkaProperties, retryTemplate, this.objectMapper)
doAnswer {
if (null != testData.exception) {
throw testData.exception
}
completedFuture(SendResult<String, String>(null, null))
}.whenever(kafkaTemplate).send(any<ProducerRecord<String, String>>())
kafkaMtbFileSender.send(DnpmV2MtbFileRequest(TEST_REQUEST_ID, dnpmV2MtbFile()))
val expectedCount = when (testData.exception) {
// OK - No Retry
null -> times(1)
// Request failed - Retry max 3 times
else -> times(3)
}
verify(kafkaTemplate, expectedCount).send(any<ProducerRecord<String, String>>())
}
verify(kafkaTemplate, expectedCount).send(anyString(), anyString(), anyString())
}
companion object {
fun mtbFile(consentStatus: Consent.Status): MtbFile {
return if (consentStatus == Consent.Status.ACTIVE) {
MtbFile.builder()
.withPatient(
Patient.builder()
.withId("PID")
.withBirthDate("2000-08-08")
.withGender(Patient.Gender.MALE)
.build()
)
.withConsent(
Consent.builder()
.withId("1")
.withStatus(consentStatus)
.withPatient("PID")
.build()
)
.withEpisode(
Episode.builder()
.withId("1")
.withPatient("PID")
.withPeriod(PeriodStart("2023-08-08"))
.build()
)
} else {
MtbFile.builder()
.withConsent(
Consent.builder()
.withStatus(consentStatus)
.withPatient("PID")
.build()
)
}.build()
val TEST_REQUEST_ID = RequestId("TestId")
val TEST_PATIENT_PSEUDONYM = PatientPseudonym("PID")
fun dnpmV2MtbFile(): Mtb {
return Mtb().apply {
this.patient = dev.pcvolkmer.mv64e.mtb.Patient().apply {
this.id = "PID"
this.birthDate = Date.from(Instant.now())
this.gender = GenderCoding().apply {
this.code = GenderCodingCode.MALE
}
}
this.episodesOfCare = listOf(
MtbEpisodeOfCare().apply {
this.id = "1"
this.patient = Reference().apply {
this.id = "PID"
}
this.period = PeriodDate().apply {
this.start = Date.from(Instant.now())
}
}
)
}
}
fun kafkaRecordData(requestId: String, consentStatus: Consent.Status): KafkaMtbFileSender.Data {
return KafkaMtbFileSender.Data(requestId, mtbFile(consentStatus))
fun dnmpV2kafkaRecordData(requestId: RequestId): MtbRequest {
return DnpmV2MtbFileRequest(requestId, dnpmV2MtbFile())
}
data class TestData(val requestStatus: RequestStatus, val exception: Throwable? = null)
@@ -226,4 +238,4 @@ class KafkaMtbFileSenderTest {
}
}
}
}

View File

@@ -0,0 +1,313 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.output
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.kotlin.KotlinModule
import dev.dnpm.etl.processor.CustomMediaType
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.RequestId
import dev.dnpm.etl.processor.config.AppConfigProperties
import dev.dnpm.etl.processor.config.AppConfiguration
import dev.dnpm.etl.processor.config.RestTargetProperties
import dev.dnpm.etl.processor.monitoring.ReportService
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.pcvolkmer.mv64e.mtb.*
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Nested
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.MethodSource
import org.springframework.http.HttpHeaders
import org.springframework.http.HttpMethod
import org.springframework.http.HttpStatus
import org.springframework.http.MediaType
import org.springframework.retry.backoff.NoBackOffPolicy
import org.springframework.retry.policy.SimpleRetryPolicy
import org.springframework.retry.support.RetryTemplateBuilder
import org.springframework.test.web.client.ExpectedCount
import org.springframework.test.web.client.MockRestServiceServer
import org.springframework.test.web.client.match.MockRestRequestMatchers.*
import org.springframework.test.web.client.response.MockRestResponseCreators.withStatus
import org.springframework.web.client.RestTemplate
import java.time.Instant
import java.util.*
class RestDipMtbFileSenderTest {
@Nested
inner class DnpmV2ContentRequest {
private lateinit var mockRestServiceServer: MockRestServiceServer
private lateinit var restMtbFileSender: RestMtbFileSender
private var reportService = ReportService(ObjectMapper().registerModule(KotlinModule.Builder().build()))
@BeforeEach
fun setup() {
val restTemplate = RestTemplate()
val restTargetProperties = RestTargetProperties("http://localhost:9000/api", null, null)
val retryTemplate = RetryTemplateBuilder().customPolicy(SimpleRetryPolicy(1)).build()
this.mockRestServiceServer = MockRestServiceServer.createServer(restTemplate)
this.restMtbFileSender = RestDipMtbFileSender(restTemplate, restTargetProperties, retryTemplate, reportService)
}
@ParameterizedTest
@MethodSource("dev.dnpm.etl.processor.output.RestDipMtbFileSenderTest#mtbFileRequestWithResponseSource")
fun shouldReturnExpectedResponseForDnpmV2MtbFilePost(requestWithResponse: RequestWithResponse) {
this.mockRestServiceServer
.expect(method(HttpMethod.POST))
.andExpect(requestTo("http://localhost:9000/api/mtb/etl/patient-record"))
.andExpect(header(HttpHeaders.CONTENT_TYPE, CustomMediaType.APPLICATION_VND_DNPM_V2_MTB_JSON_VALUE))
.andRespond {
withStatus(requestWithResponse.httpStatus).body(requestWithResponse.body).createResponse(it)
}
val response = restMtbFileSender.send(DnpmV2MtbFileRequest(TEST_REQUEST_ID, dnpmV2MtbFile()))
assertThat(response.status).isEqualTo(requestWithResponse.response.status)
assertThat(response.body).isEqualTo(requestWithResponse.response.body)
}
}
@Nested
inner class DeleteRequest {
private lateinit var mockRestServiceServer: MockRestServiceServer
private lateinit var restMtbFileSender: RestMtbFileSender
private var reportService = ReportService(ObjectMapper().registerModule(KotlinModule.Builder().build()))
@BeforeEach
fun setup() {
val restTemplate = RestTemplate()
val restTargetProperties = RestTargetProperties("http://localhost:9000/api", null, null)
val retryTemplate = RetryTemplateBuilder().customPolicy(SimpleRetryPolicy(1)).build()
this.mockRestServiceServer = MockRestServiceServer.createServer(restTemplate)
this.restMtbFileSender =
RestDipMtbFileSender(restTemplate, restTargetProperties, retryTemplate, reportService)
}
@ParameterizedTest
@MethodSource("dev.dnpm.etl.processor.output.RestDipMtbFileSenderTest#deleteRequestWithResponseSource")
fun shouldReturnExpectedResponseForDelete(requestWithResponse: RequestWithResponse) {
this.mockRestServiceServer
.expect(method(HttpMethod.DELETE))
.andExpect(requestTo("http://localhost:9000/api/mtb/etl/patient/${TEST_PATIENT_PSEUDONYM.value}"))
.andRespond {
withStatus(requestWithResponse.httpStatus).body(requestWithResponse.body).createResponse(it)
}
val response = restMtbFileSender.send(DeleteRequest(TEST_REQUEST_ID, TEST_PATIENT_PSEUDONYM))
assertThat(response.status).isEqualTo(requestWithResponse.response.status)
assertThat(response.body).isEqualTo(requestWithResponse.response.body)
}
@ParameterizedTest
@MethodSource("dev.dnpm.etl.processor.output.RestDipMtbFileSenderTest#deleteRequestWithResponseSource")
fun shouldRetryOnDeleteHttpRequestError(requestWithResponse: RequestWithResponse) {
val restTemplate = RestTemplate()
val restTargetProperties = RestTargetProperties("http://localhost:9000/api", null, null)
val retryTemplate = AppConfiguration().retryTemplate(AppConfigProperties())
retryTemplate.setBackOffPolicy(NoBackOffPolicy())
this.mockRestServiceServer = MockRestServiceServer.createServer(restTemplate)
this.restMtbFileSender =
RestDipMtbFileSender(restTemplate, restTargetProperties, retryTemplate, reportService)
val expectedCount = when (requestWithResponse.httpStatus) {
// OK - No Retry
HttpStatus.OK, HttpStatus.CREATED, HttpStatus.UNPROCESSABLE_ENTITY, HttpStatus.BAD_REQUEST -> ExpectedCount.max(
1
)
// Request failed - Retry max 3 times
else -> ExpectedCount.max(3)
}
this.mockRestServiceServer
.expect(expectedCount, method(HttpMethod.DELETE))
.andExpect(requestTo("http://localhost:9000/api/mtb/etl/patient/${TEST_PATIENT_PSEUDONYM.value}"))
.andRespond {
withStatus(requestWithResponse.httpStatus).body(requestWithResponse.body).createResponse(it)
}
val response = restMtbFileSender.send(DeleteRequest(TEST_REQUEST_ID, TEST_PATIENT_PSEUDONYM))
assertThat(response.status).isEqualTo(requestWithResponse.response.status)
assertThat(response.body).isEqualTo(requestWithResponse.response.body)
}
}
companion object {
data class RequestWithResponse(
val httpStatus: HttpStatus,
val body: String,
val response: MtbFileSender.Response
)
val TEST_REQUEST_ID = RequestId("TestId")
val TEST_PATIENT_PSEUDONYM = PatientPseudonym("PID")
fun dnpmV2MtbFile(): Mtb {
return Mtb().apply {
this.patient = dev.pcvolkmer.mv64e.mtb.Patient().apply {
this.id = "PID"
this.birthDate = Date.from(Instant.now())
this.gender = GenderCoding().apply {
this.code = GenderCodingCode.MALE
}
}
this.episodesOfCare = listOf(
MtbEpisodeOfCare().apply {
this.id = "1"
this.patient = Reference().apply {
this.id = "PID"
}
this.period = PeriodDate().apply {
this.start = Date.from(Instant.now())
}
}
)
}
}
private const val ERROR_RESPONSE_BODY = "Sonstiger Fehler bei der Übertragung"
/**
* Synthetic http responses with related request status
* Also see: https://ibmi-intra.cs.uni-tuebingen.de/display/ZPM/bwHC+REST+API
*/
@JvmStatic
fun mtbFileRequestWithResponseSource(): Set<RequestWithResponse> {
return setOf(
RequestWithResponse(
HttpStatus.OK,
responseBodyWithMaxSeverity(ReportService.Severity.INFO),
MtbFileSender.Response(
RequestStatus.SUCCESS,
responseBodyWithMaxSeverity(ReportService.Severity.INFO)
)
),
RequestWithResponse(
HttpStatus.CREATED,
responseBodyWithMaxSeverity(ReportService.Severity.WARNING),
MtbFileSender.Response(RequestStatus.WARNING, responseBodyWithMaxSeverity(ReportService.Severity.WARNING))
),
RequestWithResponse(
HttpStatus.BAD_REQUEST,
responseBodyWithMaxSeverity(ReportService.Severity.ERROR),
MtbFileSender.Response(RequestStatus.ERROR, responseBodyWithMaxSeverity(ReportService.Severity.ERROR))
),
RequestWithResponse(
HttpStatus.UNPROCESSABLE_ENTITY,
responseBodyWithMaxSeverity(ReportService.Severity.ERROR),
MtbFileSender.Response(RequestStatus.ERROR, responseBodyWithMaxSeverity(ReportService.Severity.ERROR))
),
// Some more errors not mentioned in documentation
RequestWithResponse(
HttpStatus.NOT_FOUND,
ERROR_RESPONSE_BODY,
MtbFileSender.Response(RequestStatus.ERROR, ERROR_RESPONSE_BODY)
),
RequestWithResponse(
HttpStatus.INTERNAL_SERVER_ERROR,
ERROR_RESPONSE_BODY,
MtbFileSender.Response(RequestStatus.ERROR, ERROR_RESPONSE_BODY)
)
)
}
/**
* Synthetic http responses with related request status
* Also see: https://ibmi-intra.cs.uni-tuebingen.de/display/ZPM/bwHC+REST+API
*/
@JvmStatic
fun deleteRequestWithResponseSource(): Set<RequestWithResponse> {
return setOf(
RequestWithResponse(HttpStatus.OK, "", MtbFileSender.Response(RequestStatus.SUCCESS)),
// Some more errors not mentioned in documentation
RequestWithResponse(
HttpStatus.NOT_FOUND,
"what????",
MtbFileSender.Response(RequestStatus.ERROR, ERROR_RESPONSE_BODY)
),
RequestWithResponse(
HttpStatus.INTERNAL_SERVER_ERROR,
"what????",
MtbFileSender.Response(RequestStatus.ERROR, ERROR_RESPONSE_BODY)
)
)
}
fun responseBodyWithMaxSeverity(severity: ReportService.Severity): String {
return when (severity) {
ReportService.Severity.INFO -> """
{
"patient": "PID",
"issues": [
{ "severity": "info", "message": "Info Message" }
]
}
"""
ReportService.Severity.WARNING -> """
{
"patient": "PID",
"issues": [
{ "severity": "info", "message": "Info Message" },
{ "severity": "warning", "message": "Warning Message" }
]
}
"""
ReportService.Severity.ERROR -> """
{
"patient": "PID",
"issues": [
{ "severity": "info", "message": "Info Message" },
{ "severity": "warning", "message": "Warning Message" },
{ "severity": "error", "message": "Error Message" }
]
}
"""
ReportService.Severity.FATAL -> """
{
"patient": "PID",
"issues": [
{ "severity": "info", "message": "Info Message" },
{ "severity": "warning", "message": "Warning Message" },
{ "severity": "error", "message": "Error Message" },
{ "severity": "fatal", "message": "Fatal Message" }
]
}
"""
}
}
}
}

View File

@@ -1,257 +0,0 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.output
import de.ukw.ccc.bwhc.dto.*
import dev.dnpm.etl.processor.config.RestTargetProperties
import dev.dnpm.etl.processor.monitoring.RequestStatus
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.MethodSource
import org.springframework.http.HttpMethod
import org.springframework.http.HttpStatus
import org.springframework.retry.policy.SimpleRetryPolicy
import org.springframework.retry.support.RetryTemplateBuilder
import org.springframework.test.web.client.ExpectedCount
import org.springframework.test.web.client.MockRestServiceServer
import org.springframework.test.web.client.match.MockRestRequestMatchers.method
import org.springframework.test.web.client.match.MockRestRequestMatchers.requestTo
import org.springframework.test.web.client.response.MockRestResponseCreators.withStatus
import org.springframework.web.client.RestTemplate
class RestMtbFileSenderTest {
private lateinit var mockRestServiceServer: MockRestServiceServer
private lateinit var restMtbFileSender: RestMtbFileSender
@BeforeEach
fun setup() {
val restTemplate = RestTemplate()
val restTargetProperties = RestTargetProperties("http://localhost:9000/mtbfile")
val retryTemplate = RetryTemplateBuilder().customPolicy(SimpleRetryPolicy(1)).build()
this.mockRestServiceServer = MockRestServiceServer.createServer(restTemplate)
this.restMtbFileSender = RestMtbFileSender(restTemplate, restTargetProperties, retryTemplate)
}
@ParameterizedTest
@MethodSource("deleteRequestWithResponseSource")
fun shouldReturnExpectedResponseForDelete(requestWithResponse: RequestWithResponse) {
this.mockRestServiceServer.expect {
method(HttpMethod.DELETE)
requestTo("/mtbfile")
}.andRespond {
withStatus(requestWithResponse.httpStatus).body(requestWithResponse.body).createResponse(it)
}
val response = restMtbFileSender.send(MtbFileSender.DeleteRequest("TestID", "PID"))
assertThat(response.status).isEqualTo(requestWithResponse.response.status)
assertThat(response.body).isEqualTo(requestWithResponse.response.body)
}
@ParameterizedTest
@MethodSource("mtbFileRequestWithResponseSource")
fun shouldReturnExpectedResponseForMtbFilePost(requestWithResponse: RequestWithResponse) {
this.mockRestServiceServer.expect {
method(HttpMethod.POST)
requestTo("/mtbfile")
}.andRespond {
withStatus(requestWithResponse.httpStatus).body(requestWithResponse.body).createResponse(it)
}
val response = restMtbFileSender.send(MtbFileSender.MtbFileRequest("TestID", mtbFile))
assertThat(response.status).isEqualTo(requestWithResponse.response.status)
assertThat(response.body).isEqualTo(requestWithResponse.response.body)
}
@ParameterizedTest
@MethodSource("mtbFileRequestWithResponseSource")
fun shouldRetryOnMtbFileHttpRequestError(requestWithResponse: RequestWithResponse) {
val restTemplate = RestTemplate()
val restTargetProperties = RestTargetProperties("http://localhost:9000/mtbfile")
val retryTemplate = RetryTemplateBuilder().customPolicy(SimpleRetryPolicy(3)).build()
this.mockRestServiceServer = MockRestServiceServer.createServer(restTemplate)
this.restMtbFileSender = RestMtbFileSender(restTemplate, restTargetProperties, retryTemplate)
val expectedCount = when (requestWithResponse.httpStatus) {
// OK - No Retry
HttpStatus.OK, HttpStatus.CREATED -> ExpectedCount.max(1)
// Request failed - Retry max 3 times
else -> ExpectedCount.max(3)
}
this.mockRestServiceServer.expect(expectedCount) {
method(HttpMethod.POST)
requestTo("/mtbfile")
}.andRespond {
withStatus(requestWithResponse.httpStatus).body(requestWithResponse.body).createResponse(it)
}
val response = restMtbFileSender.send(MtbFileSender.MtbFileRequest("TestID", mtbFile))
assertThat(response.status).isEqualTo(requestWithResponse.response.status)
assertThat(response.body).isEqualTo(requestWithResponse.response.body)
}
@ParameterizedTest
@MethodSource("deleteRequestWithResponseSource")
fun shouldRetryOnDeleteHttpRequestError(requestWithResponse: RequestWithResponse) {
val restTemplate = RestTemplate()
val restTargetProperties = RestTargetProperties("http://localhost:9000/mtbfile")
val retryTemplate = RetryTemplateBuilder().customPolicy(SimpleRetryPolicy(3)).build()
this.mockRestServiceServer = MockRestServiceServer.createServer(restTemplate)
this.restMtbFileSender = RestMtbFileSender(restTemplate, restTargetProperties, retryTemplate)
val expectedCount = when (requestWithResponse.httpStatus) {
// OK - No Retry
HttpStatus.OK, HttpStatus.CREATED -> ExpectedCount.max(1)
// Request failed - Retry max 3 times
else -> ExpectedCount.max(3)
}
this.mockRestServiceServer.expect(expectedCount) {
method(HttpMethod.DELETE)
requestTo("/mtbfile")
}.andRespond {
withStatus(requestWithResponse.httpStatus).body(requestWithResponse.body).createResponse(it)
}
val response = restMtbFileSender.send(MtbFileSender.DeleteRequest("TestID", "PID"))
assertThat(response.status).isEqualTo(requestWithResponse.response.status)
assertThat(response.body).isEqualTo(requestWithResponse.response.body)
}
companion object {
data class RequestWithResponse(
val httpStatus: HttpStatus,
val body: String,
val response: MtbFileSender.Response
)
private val warningBody = """
{
"patient_id": "PID",
"issues": [
{ "severity": "warning", "message": "Something is not right" }
]
}
""".trimIndent()
private val errorBody = """
{
"patient_id": "PID",
"issues": [
{ "severity": "error", "message": "Something is very bad" }
]
}
""".trimIndent()
val mtbFile: MtbFile = MtbFile.builder()
.withPatient(
Patient.builder()
.withId("PID")
.withBirthDate("2000-08-08")
.withGender(Patient.Gender.MALE)
.build()
)
.withConsent(
Consent.builder()
.withId("1")
.withStatus(Consent.Status.ACTIVE)
.withPatient("PID")
.build()
)
.withEpisode(
Episode.builder()
.withId("1")
.withPatient("PID")
.withPeriod(PeriodStart("2023-08-08"))
.build()
)
.build()
private const val ERROR_RESPONSE_BODY = "Sonstiger Fehler bei der Übertragung"
/**
* Synthetic http responses with related request status
* Also see: https://ibmi-intra.cs.uni-tuebingen.de/display/ZPM/bwHC+REST+API
*/
@JvmStatic
fun mtbFileRequestWithResponseSource(): Set<RequestWithResponse> {
return setOf(
RequestWithResponse(HttpStatus.OK, "{}", MtbFileSender.Response(RequestStatus.SUCCESS, "{}")),
RequestWithResponse(
HttpStatus.CREATED,
warningBody,
MtbFileSender.Response(RequestStatus.WARNING, warningBody)
),
RequestWithResponse(
HttpStatus.BAD_REQUEST,
"??",
MtbFileSender.Response(RequestStatus.ERROR, ERROR_RESPONSE_BODY)
),
RequestWithResponse(
HttpStatus.UNPROCESSABLE_ENTITY,
errorBody,
MtbFileSender.Response(RequestStatus.ERROR, ERROR_RESPONSE_BODY)
),
// Some more errors not mentioned in documentation
RequestWithResponse(
HttpStatus.NOT_FOUND,
"what????",
MtbFileSender.Response(RequestStatus.ERROR, ERROR_RESPONSE_BODY)
),
RequestWithResponse(
HttpStatus.INTERNAL_SERVER_ERROR,
"what????",
MtbFileSender.Response(RequestStatus.ERROR, ERROR_RESPONSE_BODY)
)
)
}
/**
* Synthetic http responses with related request status
* Also see: https://ibmi-intra.cs.uni-tuebingen.de/display/ZPM/bwHC+REST+API
*/
@JvmStatic
fun deleteRequestWithResponseSource(): Set<RequestWithResponse> {
return setOf(
RequestWithResponse(HttpStatus.OK, "", MtbFileSender.Response(RequestStatus.SUCCESS)),
// Some more errors not mentioned in documentation
RequestWithResponse(
HttpStatus.NOT_FOUND,
"what????",
MtbFileSender.Response(RequestStatus.ERROR, ERROR_RESPONSE_BODY)
),
RequestWithResponse(
HttpStatus.INTERNAL_SERVER_ERROR,
"what????",
MtbFileSender.Response(RequestStatus.ERROR, ERROR_RESPONSE_BODY)
)
)
}
}
}

View File

@@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2023 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@@ -19,180 +19,182 @@
package dev.dnpm.etl.processor.pseudonym
import ca.uhn.fhir.context.FhirContext
import com.fasterxml.jackson.databind.ObjectMapper
import de.ukw.ccc.bwhc.dto.*
import dev.dnpm.etl.processor.config.AppConfigProperties
import dev.dnpm.etl.processor.config.GIcsConfigProperties
import dev.dnpm.etl.processor.config.JacksonConfig
import dev.dnpm.etl.processor.consent.MtbFileConsentService
import dev.dnpm.etl.processor.services.ConsentProcessor
import dev.dnpm.etl.processor.services.ConsentProcessorTest
import dev.pcvolkmer.mv64e.mtb.*
import org.assertj.core.api.Assertions.assertThat
import org.hl7.fhir.r4.model.Bundle
import org.junit.jupiter.api.Nested
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.assertThrows
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.ArgumentMatchers
import org.mockito.Mock
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.anyValueClass
import org.mockito.kotlin.doAnswer
import org.mockito.kotlin.whenever
import org.springframework.core.io.ClassPathResource
const val FAKE_MTB_FILE_PATH = "fake_MTBFile.json"
const val CLEAN_PATIENT_ID = "5dad2f0b-49c6-47d8-a952-7b9e9e0f7549"
import java.time.Instant
import java.util.*
@ExtendWith(MockitoExtension::class)
class ExtensionsTest {
private fun fakeMtbFile(): MtbFile {
val mtbFile = ClassPathResource(FAKE_MTB_FILE_PATH).inputStream
return ObjectMapper().readValue(mtbFile, MtbFile::class.java)
fun getObjectMapper(): ObjectMapper {
return JacksonConfig().objectMapper()
}
private fun MtbFile.serialized(): String {
return ObjectMapper().writeValueAsString(this)
}
@Nested
inner class UsingDnpmV2Datamodel {
@Test
fun shouldNotContainCleanPatientId(@Mock pseudonymizeService: PseudonymizeService) {
doAnswer {
it.arguments[0]
"PSEUDO-ID"
}.whenever(pseudonymizeService).patientPseudonym(ArgumentMatchers.anyString())
val FAKE_MTB_FILE_PATH = "mv64e-mtb-fake-patient.json"
val CLEAN_PATIENT_ID = "644bae7a-56f6-4ee8-b02f-c532e65af5b1"
val mtbFile = fakeMtbFile()
mtbFile.pseudonymizeWith(pseudonymizeService)
assertThat(mtbFile.patient.id).isEqualTo("PSEUDO-ID")
assertThat(mtbFile.serialized()).doesNotContain(CLEAN_PATIENT_ID)
}
@Test
fun shouldNotContainAnyUuidAfterRehashingOfIds(@Mock pseudonymizeService: PseudonymizeService) {
doAnswer {
it.arguments[0]
"PSEUDO-ID"
}.whenever(pseudonymizeService).patientPseudonym(ArgumentMatchers.anyString())
doAnswer {
"TESTDOMAIN"
}.whenever(pseudonymizeService).prefix()
val mtbFile = fakeMtbFile()
mtbFile.pseudonymizeWith(pseudonymizeService)
mtbFile.anonymizeContentWith(pseudonymizeService)
val pattern = "\"[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}\"".toRegex().toPattern()
val matcher = pattern.matcher(mtbFile.serialized())
assertThrows<IllegalStateException> {
matcher.find()
matcher.group()
}.also {
assertThat(it.message).isEqualTo("No match found")
private fun fakeMtbFile(): Mtb {
val mtbFile = ClassPathResource(FAKE_MTB_FILE_PATH).inputStream
return getObjectMapper().readValue(mtbFile, Mtb::class.java)
}
private fun Mtb.serialized(): String {
return getObjectMapper().writeValueAsString(this)
}
@Test
fun shouldNotContainCleanPatientId(@Mock pseudonymizeService: PseudonymizeService) {
doAnswer {
it.arguments[0]
"PSEUDO-ID"
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
val mtbFile = fakeMtbFile()
mtbFile.ensureMetaDataIsInitialized()
addConsentData(mtbFile)
mtbFile.pseudonymizeWith(pseudonymizeService)
assertThat(mtbFile.patient.id).isEqualTo("PSEUDO-ID")
assertThat(mtbFile.serialized()).doesNotContain(CLEAN_PATIENT_ID)
}
private fun addConsentData(mtbFile: Mtb) {
val gIcsConfigProperties = GIcsConfigProperties("", "", "")
val appConfigProperties = AppConfigProperties(emptyList())
val bundle = Bundle()
val dummyConsent = ConsentProcessorTest.getDummyGenomDeConsent()
dummyConsent.patient.reference = "Patient/$CLEAN_PATIENT_ID"
bundle.addEntry().resource = dummyConsent
ConsentProcessor(
appConfigProperties,
gIcsConfigProperties,
JacksonConfig().objectMapper(),
FhirContext.forR4(),
MtbFileConsentService()
).embedBroadConsentResources(mtbFile, bundle)
}
@Test
fun shouldNotThrowExceptionOnNullValues(@Mock pseudonymizeService: PseudonymizeService) {
doAnswer {
it.arguments[0]
"PSEUDO-ID"
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
"TESTDOMAIN"
}.whenever(pseudonymizeService).prefix()
val mtbFile = Mtb().apply {
this.patient = dev.pcvolkmer.mv64e.mtb.Patient().apply {
this.id = "PID"
this.birthDate = Date.from(Instant.now())
this.gender = GenderCoding().apply {
this.code = GenderCodingCode.MALE
}
}
this.episodesOfCare = listOf(
MtbEpisodeOfCare().apply {
this.id = "1"
this.patient = Reference().apply {
this.id = "PID"
}
this.period = PeriodDate().apply {
this.start = Date.from(Instant.now())
}
}
)
}
mtbFile.pseudonymizeWith(pseudonymizeService)
mtbFile.anonymizeContentWith(pseudonymizeService)
assertThat(mtbFile.episodesOfCare).hasSize(1)
assertThat(mtbFile.episodesOfCare.map { it.id }).isNotNull
}
@Test
fun shouldNotContainAnyUuidAfterRehashingOfIds(@Mock pseudonymizeService: PseudonymizeService) {
doAnswer {
it.arguments[0]
"PSEUDO-ID"
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
"TESTDOMAIN"
}.whenever(pseudonymizeService).prefix()
val mtbFile = fakeMtbFile()
/**
* replace hex values with random long, so our test does not match false positives
*/
mtbFile.ngsReports.forEach { report ->
report.results.simpleVariants.forEach { simpleVariant ->
simpleVariant.externalIds.forEach { extIdValue ->
extIdValue.value =
Math.random().toLong().toString()
}
}
}
mtbFile.ngsReports.forEach { report ->
report.results.rnaFusions.forEach { simpleVariant ->
simpleVariant.externalIds.forEach { extIdValue ->
extIdValue.value =
Math.random().toLong().toString()
}
simpleVariant.fusionPartner3Prime?.transcriptId?.value =
Math.random().toLong().toString()
simpleVariant.fusionPartner5Prime?.transcriptId?.value =
Math.random().toLong().toString()
simpleVariant.externalIds?.forEach { it ->
it?.value = Math.random().toLong().toString()
}
}
}
mtbFile.pseudonymizeWith(pseudonymizeService)
mtbFile.anonymizeContentWith(pseudonymizeService)
val pattern =
"\"[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}\"".toRegex()
.toPattern()
val input = mtbFile.serialized()
val matcher = pattern.matcher(input)
assertThrows<IllegalStateException> {
matcher.find()
val posSt = "check at pos: " + matcher.start().toString() + ", " + matcher.end()
println(posSt + " with " + matcher.group())
}.also {
assertThat(it.message).isEqualTo("No match found")
}
}
}
@Test
fun shouldRehashIdsWithPrefix(@Mock pseudonymizeService: PseudonymizeService) {
doAnswer {
it.arguments[0]
"PSEUDO-ID"
}.whenever(pseudonymizeService).patientPseudonym(ArgumentMatchers.anyString())
doAnswer {
"TESTDOMAIN"
}.whenever(pseudonymizeService).prefix()
val mtbFile = MtbFile.builder()
.withPatient(
Patient.builder()
.withId("1")
.withBirthDate("2000-08-08")
.withGender(Patient.Gender.MALE)
.build()
)
.withConsent(
Consent.builder()
.withId("1")
.withStatus(Consent.Status.ACTIVE)
.withPatient("123")
.build()
)
.withEpisode(
Episode.builder()
.withId("1")
.withPatient("1")
.withPeriod(PeriodStart("2023-08-08"))
.build()
)
.build()
mtbFile.pseudonymizeWith(pseudonymizeService)
mtbFile.anonymizeContentWith(pseudonymizeService)
assertThat(mtbFile.episode.id)
// TESTDOMAIN<sha256(TESTDOMAIN-1)[0-41]>
.isEqualTo("TESTDOMAIN44e20a53bbbf9f3ae39626d05df7014dcd77d6098")
}
@Test
fun shouldNotThrowExceptionOnNullValues(@Mock pseudonymizeService: PseudonymizeService) {
doAnswer {
it.arguments[0]
"PSEUDO-ID"
}.whenever(pseudonymizeService).patientPseudonym(ArgumentMatchers.anyString())
doAnswer {
"TESTDOMAIN"
}.whenever(pseudonymizeService).prefix()
val mtbFile = MtbFile.builder()
.withPatient(
Patient.builder()
.withId("1")
.withBirthDate("2000-08-08")
.withGender(Patient.Gender.MALE)
.build()
)
.withConsent(
Consent.builder()
.withId("1")
.withStatus(Consent.Status.ACTIVE)
.withPatient("123")
.build()
)
.withEpisode(
Episode.builder()
.withId("1")
.withPatient("1")
.withPeriod(PeriodStart("2023-08-08"))
.build()
)
.withClaims(null)
.withDiagnoses(null)
.withCarePlans(null)
.withClaimResponses(null)
.withEcogStatus(null)
.withFamilyMemberDiagnoses(null)
.withGeneticCounsellingRequests(null)
.withHistologyReevaluationRequests(null)
.withHistologyReports(null)
.withLastGuidelineTherapies(null)
.withMolecularPathologyFindings(null)
.withMolecularTherapies(null)
.withNgsReports(null)
.withPreviousGuidelineTherapies(null)
.withRebiopsyRequests(null)
.withRecommendations(null)
.withResponses(null)
.withStudyInclusionRequests(null)
.withSpecimens(null)
.build()
mtbFile.pseudonymizeWith(pseudonymizeService)
mtbFile.anonymizeContentWith(pseudonymizeService)
assertThat(mtbFile.episode.id).isNotNull()
}
}
}

View File

@@ -19,8 +19,8 @@
package dev.dnpm.etl.processor.pseudonym
import de.ukw.ccc.bwhc.dto.*
import dev.dnpm.etl.processor.config.PseudonymizeConfigProperties
import dev.pcvolkmer.mv64e.mtb.*
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
@@ -29,31 +29,26 @@ import org.mockito.Mock
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.doAnswer
import org.mockito.kotlin.whenever
import java.time.Instant
import java.util.*
@ExtendWith(MockitoExtension::class)
class PseudonymizeServiceTest {
private val mtbFile = MtbFile.builder()
.withPatient(
private val mtbFile = Mtb.builder()
.patient(
Patient.builder()
.withId("123")
.withBirthDate("2000-08-08")
.withGender(Patient.Gender.MALE)
.id("123")
.build()
)
.withConsent(
Consent.builder()
.withId("1")
.withStatus(Consent.Status.ACTIVE)
.withPatient("123")
.build()
)
.withEpisode(
Episode.builder()
.withId("1")
.withPatient("123")
.withPeriod(PeriodStart("2023-08-08"))
.build()
.episodesOfCare(
listOf(
MtbEpisodeOfCare.builder()
.id("1")
.patient(Reference.builder().id("123").build())
.period(PeriodDate.builder().start(Date.from(Instant.parse("2021-01-01T00:00:00.00Z"))).build())
.build()
)
)
.build()
@@ -71,8 +66,8 @@ class PseudonymizeServiceTest {
}
@Test
fun sanitizeFileName(@Mock generator: GpasPseudonymGenerator) {
val result= GpasPseudonymGenerator.sanitizeValue("l://a\\bs;1*2?3>")
fun sanitizeFileName() {
val result = GpasPseudonymGenerator.sanitizeValue("l://a\\bs;1*2?3>")
assertThat(result).isEqualTo("l___a_bs_1_2_3_")
}
@@ -90,4 +85,16 @@ class PseudonymizeServiceTest {
assertThat(mtbFile.patient.id).isEqualTo("UNKNOWN_123")
}
}
@Test
fun shouldReturnDifferentValues() {
val ag = AnonymizingGenerator()
val tans = HashSet<String>()
(1..1000).forEach { i ->
val tan = ag.generateGenomDeTan("12345")
assertThat(tan).hasSize(64)
assertThat(tans.add(tan)).`as`("never the same result!").isTrue
}
}
}

View File

@@ -17,13 +17,12 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.services
package dev.dnpm.etl.processor.security
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.ArgumentCaptor
import org.mockito.ArgumentMatchers.anyLong
import org.mockito.ArgumentMatchers.anyString
import org.mockito.Mock
@@ -96,11 +95,11 @@ class TokenServiceTest {
val actual = this.tokenService.addToken("Test Token")
val captor = ArgumentCaptor.forClass(Token::class.java)
val captor = argumentCaptor<Token>()
verify(tokenRepository, times(1)).save(captor.capture())
assertThat(actual).satisfies(Consumer { assertThat(it.isSuccess).isTrue() })
assertThat(captor.value).satisfies(
assertThat(captor.firstValue).satisfies(
Consumer { assertThat(it.name).isEqualTo("Test Token") },
Consumer { assertThat(it.username).isEqualTo("testtoken") },
Consumer { assertThat(it.password).isEqualTo("{test}verysecret") }
@@ -116,13 +115,13 @@ class TokenServiceTest {
this.tokenService.deleteToken(42)
val stringCaptor = ArgumentCaptor.forClass(String::class.java)
val stringCaptor = argumentCaptor<String>()
verify(userDetailsManager, times(1)).deleteUser(stringCaptor.capture())
assertThat(stringCaptor.value).isEqualTo("testtoken")
assertThat(stringCaptor.firstValue).isEqualTo("testtoken")
val tokenCaptor = ArgumentCaptor.forClass(Token::class.java)
val tokenCaptor = argumentCaptor<Token>()
verify(tokenRepository, times(1)).delete(tokenCaptor.capture())
assertThat(tokenCaptor.value.id).isEqualTo(42)
assertThat(tokenCaptor.firstValue.id).isEqualTo(42)
}
@Test

View File

@@ -0,0 +1,202 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.security
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Nested
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.Mock
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.*
import org.springframework.security.core.session.SessionInformation
import org.springframework.security.core.session.SessionRegistry
import org.springframework.security.oauth2.core.oidc.OidcIdToken
import org.springframework.security.oauth2.core.oidc.user.DefaultOidcUser
import org.springframework.security.oauth2.core.oidc.user.OidcUser
import java.time.Instant
import java.util.*
@ExtendWith(MockitoExtension::class)
class UserRoleServiceTest {
private lateinit var userRoleRepository: UserRoleRepository
private lateinit var sessionRegistry: SessionRegistry
private lateinit var userRoleService: UserRoleService
@BeforeEach
fun setup(
@Mock userRoleRepository: UserRoleRepository,
@Mock sessionRegistry: SessionRegistry
) {
this.userRoleRepository = userRoleRepository
this.sessionRegistry = sessionRegistry
this.userRoleService = UserRoleService(userRoleRepository, sessionRegistry)
}
@Test
fun shouldDelegateFindAllToRepository() {
userRoleService.findAll()
verify(userRoleRepository, times(1)).findAll()
}
@Nested
inner class WithExistingUserRole {
@BeforeEach
fun setup() {
doAnswer { invocation ->
Optional.of(
UserRole(invocation.getArgument(0), "patrick.tester", Role.USER)
)
}.whenever(userRoleRepository).findById(any<Long>())
doAnswer { _ ->
listOf(
dummyPrincipal()
)
}.whenever(sessionRegistry).allPrincipals
}
@Test
fun shouldUpdateUserRole() {
userRoleService.updateUserRole(1, Role.ADMIN)
val userRoleCaptor = argumentCaptor<UserRole>()
verify(userRoleRepository, times(1)).save(userRoleCaptor.capture())
assertThat(userRoleCaptor.firstValue.id).isEqualTo(1L)
assertThat(userRoleCaptor.firstValue.role).isEqualTo(Role.ADMIN)
}
@Test
fun shouldExpireSessionOnUpdate() {
val dummySessions = dummySessions()
whenever(sessionRegistry.getAllSessions(any(), any<Boolean>())).thenReturn(
dummySessions
)
assertThat(dummySessions.filter { it.isExpired }).hasSize(0)
userRoleService.updateUserRole(1, Role.ADMIN)
verify(sessionRegistry, times(1)).getAllSessions(any<OidcUser>(), any<Boolean>())
assertThat(dummySessions.filter { it.isExpired }).hasSize(2)
}
@Test
fun shouldDeleteUserRole() {
userRoleService.deleteUserRole(1)
val userRoleCaptor = argumentCaptor<UserRole>()
verify(userRoleRepository, times(1)).delete(userRoleCaptor.capture())
assertThat(userRoleCaptor.firstValue.id).isEqualTo(1L)
assertThat(userRoleCaptor.firstValue.role).isEqualTo(Role.USER)
}
@Test
fun shouldExpireSessionOnDelete() {
val dummySessions = dummySessions()
whenever(sessionRegistry.getAllSessions(any(), any<Boolean>())).thenReturn(
dummySessions
)
assertThat(dummySessions.filter { it.isExpired }).hasSize(0)
userRoleService.deleteUserRole(1)
verify(sessionRegistry, times(1)).getAllSessions(any<OidcUser>(), any<Boolean>())
assertThat(dummySessions.filter { it.isExpired }).hasSize(2)
}
}
@Nested
inner class WithoutExistingUserRole {
@BeforeEach
fun setup() {
doAnswer { _ ->
Optional.empty<UserRole>()
}.whenever(userRoleRepository).findById(any<Long>())
}
@Test
fun shouldNotUpdateUserRole() {
userRoleService.updateUserRole(1, Role.ADMIN)
verify(userRoleRepository, never()).save(any<UserRole>())
}
@Test
fun shouldNotExpireSessionOnUpdate() {
userRoleService.updateUserRole(1, Role.ADMIN)
verify(sessionRegistry, never()).getAllSessions(any<OidcUser>(), any<Boolean>())
}
@Test
fun shouldNotDeleteUserRole() {
userRoleService.deleteUserRole(1)
verify(userRoleRepository, never()).delete(any<UserRole>())
}
@Test
fun shouldNotExpireSessionOnDelete() {
userRoleService.deleteUserRole(1)
verify(sessionRegistry, never()).getAllSessions(any<OidcUser>(), any<Boolean>())
}
}
companion object {
private fun dummyPrincipal() = DefaultOidcUser(
listOf(),
OidcIdToken(
"anytokenvalue",
Instant.now(),
Instant.now().plusSeconds(10),
mapOf("sub" to "testsub", "preferred_username" to "patrick.tester")
)
)
private fun dummySessions() = listOf(
SessionInformation(
dummyPrincipal(),
"SESSIONID1",
Date.from(Instant.now()),
),
SessionInformation(
dummyPrincipal(),
"SESSIONID2",
Date.from(Instant.now()),
)
)
}
}

View File

@@ -0,0 +1,232 @@
package dev.dnpm.etl.processor.services
import ca.uhn.fhir.context.FhirContext
import com.fasterxml.jackson.databind.ObjectMapper
import dev.dnpm.etl.processor.config.AppConfigProperties
import dev.dnpm.etl.processor.config.GIcsConfigProperties
import dev.dnpm.etl.processor.config.JacksonConfig
import dev.dnpm.etl.processor.consent.ConsentDomain
import dev.dnpm.etl.processor.consent.GicsConsentService
import dev.pcvolkmer.mv64e.mtb.Mtb
import dev.pcvolkmer.mv64e.mtb.MvhSubmissionType
import dev.pcvolkmer.mv64e.mtb.Patient
import org.assertj.core.api.Assertions.assertThat
import org.hl7.fhir.r4.model.Bundle
import org.hl7.fhir.r4.model.CodeableConcept
import org.hl7.fhir.r4.model.Coding
import org.hl7.fhir.r4.model.Consent
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.CsvSource
import org.junit.jupiter.params.provider.ValueSource
import org.mockito.Mock
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.any
import org.mockito.kotlin.doAnswer
import org.mockito.kotlin.eq
import org.mockito.kotlin.whenever
import org.springframework.core.io.ClassPathResource
import java.io.IOException
import java.io.InputStream
import java.time.Instant
import java.time.OffsetDateTime
import java.util.*
@ExtendWith(MockitoExtension::class)
class ConsentProcessorTest {
private lateinit var appConfigProperties: AppConfigProperties
private lateinit var gicsConsentService: GicsConsentService
private lateinit var objectMapper: ObjectMapper
private lateinit var gIcsConfigProperties: GIcsConfigProperties
private lateinit var fhirContext: FhirContext
private lateinit var consentProcessor: ConsentProcessor
@BeforeEach
fun setups(
@Mock gicsConsentService: GicsConsentService,
) {
this.gIcsConfigProperties = GIcsConfigProperties("https://gics.example.com")
val jacksonConfig = JacksonConfig()
this.objectMapper = jacksonConfig.objectMapper()
this.fhirContext = JacksonConfig.fhirContext()
this.gicsConsentService = gicsConsentService
this.appConfigProperties = AppConfigProperties(emptyList())
this.consentProcessor =
ConsentProcessor(
appConfigProperties,
gIcsConfigProperties,
objectMapper,
fhirContext,
gicsConsentService
)
}
@Test
fun consentOk() {
assertThat(consentProcessor.toString()).isNotNull
// prep gICS response
doAnswer { getDummyBroadConsentBundle() }.whenever(gicsConsentService)
.getConsent(any(), any(), eq(ConsentDomain.BROAD_CONSENT))
doAnswer { Bundle() }.whenever(gicsConsentService)
.getConsent(any(), any(), eq(ConsentDomain.MODELLVORHABEN_64E))
val inputMtb = Mtb.builder()
.patient(Patient.builder().id("d611d429-5003-11f0-a144-661e92ac9503").build()).build()
val checkResult = consentProcessor.consentGatedCheckAndTryEmbedding(inputMtb)
assertThat(checkResult).isTrue
assertThat(inputMtb.metadata.researchConsents).hasSize(26)
}
companion object {
fun getDummyGenomDeConsent(): Consent {
val consent = Consent()
consent.id = "consent 1 id"
consent.patient.reference = "Patient/1234-pat1"
consent.provision.setType(
Consent.ConsentProvisionType.fromCode(
"deny"
)
)
consent.provision.period.start =
Date.from(Instant.parse("2025-08-15T00:00:00.00Z"))
consent.provision.period.end =
Date.from(Instant.parse("3000-01-01T00:00:00.00Z"))
val addProvision1 = consent.provision.addProvision()
addProvision1.setType(Consent.ConsentProvisionType.fromCode("permit"))
addProvision1.period.start = Date.from(Instant.parse("2025-08-15T00:00:00.00Z"))
addProvision1.period.end = Date.from(Instant.parse("3000-01-01T00:00:00.00Z"))
addProvision1.code.addLast(
CodeableConcept(
Coding(
"https://ths-greifswald.de/fhir/CodeSystem/gics/Policy/GenomDE_MV",
"Teilnahme",
"Teilnahme am Modellvorhaben und Einwilligung zur Genomsequenzierung"
)
)
)
val addProvision2 = consent.provision.addProvision()
addProvision2.setType(Consent.ConsentProvisionType.fromCode("deny"))
addProvision2.period.start = Date.from(Instant.parse("2025-08-15T00:00:00.00Z"))
addProvision2.period.end = Date.from(Instant.parse("3000-01-01T00:00:00.00Z"))
addProvision2.code.addLast(
CodeableConcept(
Coding(
"https://ths-greifswald.de/fhir/CodeSystem/gics/Policy/GenomDE_MV",
"Rekontaktierung",
"Re-Identifizierung meiner Daten über die Vertrauensstelle beim Robert Koch-Institut und in die erneute Kontaktaufnahme durch meine behandelnde Ärztin oder meinen behandelnden Arzt"
)
)
)
return consent
}
}
@ParameterizedTest
@CsvSource(
"2.16.840.1.113883.3.1937.777.24.5.3.8,urn:oid:2.16.840.1.113883.3.1937.777.24.5.3,2025-08-15T00:00:00+02:00,PERMIT,expect permit",
"2.16.840.1.113883.3.1937.777.24.5.3.8,urn:oid:2.16.840.1.113883.3.1937.777.24.5.3,2025-08-15T00:00:00+02:00,PERMIT,expect permit date is exactly on start",
"2.16.840.1.113883.3.1937.777.24.5.3.8,urn:oid:2.16.840.1.113883.3.1937.777.24.5.3,2055-08-15T00:00:00+02:00,PERMIT,expect permit date is exactly on end",
"2.16.840.1.113883.3.1937.777.24.5.3.8,urn:oid:2.16.840.1.113883.3.1937.777.24.5.3,2021-08-15T00:00:00+02:00,NULL,date is before start",
"2.16.840.1.113883.3.1937.777.24.5.3.8,urn:oid:2.16.840.1.113883.3.1937.777.24.5.3,2060-08-15T00:00:00+02:00,NULL,date is after end",
"2.16.840.1.113883.3.1937.777.24.5.3.27,urn:oid:2.16.840.1.113883.3.1937.777.24.5.3,2025-08-15T00:00:00+02:00,DENY,provision is denied",
"unknownCode,urn:oid:2.16.840.1.113883.3.1937.777.24.5.3,2025-08-15T00:00:00+02:00,NULL,code does not exist - therefore expect NULL",
"2.16.840.1.113883.3.1937.777.24.5.3.8,XXXX,2025-08-15T00:00:00+02:00,NULL,system not found - therefore expect NULL",
)
fun getProvisionTypeByPolicyCode(
code: String?, system: String?, timeStamp: String, expected: String?,
desc: String?
) {
val testData = getDummyBroadConsentBundle()
val requestDate = Date.from(OffsetDateTime.parse(timeStamp).toInstant())
val result: Consent.ConsentProvisionType =
consentProcessor.getProvisionTypeByPolicyCode(testData, code, system, requestDate)
assertThat(result).isNotNull()
assertThat(result).`as`(desc)
.isEqualTo(Consent.ConsentProvisionType.valueOf(expected!!))
}
@Test
fun getProvisionTypeOnEmptyConsent(
) {
val emptyResources = Bundle().addEntry(Bundle.BundleEntryComponent().setResource(Consent()))
val requestDate = Date.from(OffsetDateTime.parse("2025-08-15T00:00:00+02:00").toInstant())
val result: Consent.ConsentProvisionType =
consentProcessor.getProvisionTypeByPolicyCode(
emptyResources,
"anyCode",
"anySystem",
requestDate
)
assertThat(result).isNotNull()
assertThat(result).`as`("empty consent resource - expect NULL")
.isEqualTo(Consent.ConsentProvisionType.NULL)
}
fun getDummyBroadConsentBundle(): Bundle {
val bundle: InputStream?
try {
bundle = ClassPathResource(
"fake_broadConsent_gics_response_permit.json"
).getInputStream()
} catch (e: IOException) {
throw RuntimeException(e)
}
return FhirContext.forR4().newJsonParser()
.parseResource<Bundle>(Bundle::class.java, bundle)
}
@ParameterizedTest
@ValueSource(booleans = [true, false])
fun mvSubmissionTypeIsSet(isTestSubmission: Boolean) {
appConfigProperties.genomDeTestSubmission = isTestSubmission
val fixture =
ConsentProcessor(
appConfigProperties,
gIcsConfigProperties,
objectMapper,
fhirContext,
gicsConsentService
)
doAnswer { getDummyBroadConsentBundle() }.whenever(gicsConsentService)
.getConsent(any(), any(), eq(ConsentDomain.BROAD_CONSENT))
doAnswer {
Bundle().addEntry(
Bundle.BundleEntryComponent().setResource(getDummyGenomDeConsent())
)
}.whenever(gicsConsentService)
.getConsent(any(), any(), eq(ConsentDomain.MODELLVORHABEN_64E))
val inputMtb = Mtb.builder()
.patient(Patient.builder().id("d611d429-5003-11f0-a144-661e92ac9503").build()).build()
val checkResult = fixture.consentGatedCheckAndTryEmbedding(inputMtb)
assertThat(checkResult).isNotNull
if (isTestSubmission)
assertThat(inputMtb.metadata.type).isEqualTo(MvhSubmissionType.TEST)
else {
assertThat(inputMtb.metadata.type).isEqualTo(MvhSubmissionType.INITIAL)
}
}
}

View File

@@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2023 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@@ -22,9 +22,14 @@ package dev.dnpm.etl.processor.services
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.kotlin.KotlinModule
import dev.dnpm.etl.processor.monitoring.ReportService
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.monitoring.asRequestStatus
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.Arguments
import org.junit.jupiter.params.provider.MethodSource
class ReportServiceTest {
@@ -60,6 +65,15 @@ class ReportServiceTest {
assertThat(actual[2].message).isEqualTo("Warning Message")
assertThat(actual[3].severity).isEqualTo(ReportService.Severity.INFO)
assertThat(actual[3].message).isEqualTo("Info Message")
assertThat(actual.asRequestStatus()).isEqualTo(RequestStatus.ERROR)
}
@ParameterizedTest
@MethodSource("testData")
fun shouldParseDataQualityReport(json: String, requestStatus: RequestStatus) {
val actual = this.reportService.deserialize(json)
assertThat(actual.asRequestStatus()).isEqualTo(requestStatus)
}
@Test
@@ -73,4 +87,75 @@ class ReportServiceTest {
assertThat(actual[0].message).isEqualTo("Not parsable data quality report '$invalidResponse'")
}
companion object {
@JvmStatic
fun testData(): Set<Arguments> {
return setOf(
Arguments.of(
"""
{
"patient": "4711",
"issues": [
{ "severity": "info", "message": "Info Message" },
{ "severity": "warning", "message": "Warning Message" },
{ "severity": "error", "message": "Error Message" },
{ "severity": "fatal", "message": "Fatal Message" }
]
}
""".trimIndent(),
RequestStatus.ERROR
),
Arguments.of(
"""
{
"patient": "4711",
"issues": [
{ "severity": "info", "message": "Info Message" },
{ "severity": "warning", "message": "Warning Message" },
{ "severity": "error", "message": "Error Message" }
]
}
""".trimIndent(),
RequestStatus.ERROR
),
Arguments.of(
"""
{
"patient": "4711",
"issues": [
{ "severity": "error", "message": "Error Message" }
{ "severity": "info", "message": "Info Message" }
]
}
""".trimIndent(),
RequestStatus.ERROR
),
Arguments.of(
"""
{
"patient": "4711",
"issues": [
{ "severity": "info", "message": "Info Message" },
{ "severity": "warning", "message": "Warning Message" }
]
}
""".trimIndent(),
RequestStatus.WARNING
),
Arguments.of(
"""
{
"patient": "4711",
"issues": [
{ "severity": "info", "message": "Info Message" }
]
}
""".trimIndent(),
RequestStatus.SUCCESS
)
)
}
}
}

View File

@@ -20,23 +20,30 @@
package dev.dnpm.etl.processor.services
import com.fasterxml.jackson.databind.ObjectMapper
import de.ukw.ccc.bwhc.dto.*
import dev.dnpm.etl.processor.Fingerprint
import dev.dnpm.etl.processor.PatientId
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.config.AppConfigProperties
import dev.dnpm.etl.processor.consent.TtpConsentStatus
import dev.dnpm.etl.processor.monitoring.Request
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.monitoring.RequestType
import dev.dnpm.etl.processor.output.DeleteRequest
import dev.dnpm.etl.processor.output.DnpmV2MtbFileRequest
import dev.dnpm.etl.processor.output.MtbFileSender
import dev.dnpm.etl.processor.output.RestMtbFileSender
import dev.dnpm.etl.processor.pseudonym.PseudonymizeService
import dev.dnpm.etl.processor.randomRequestId
import dev.pcvolkmer.mv64e.mtb.*
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.ArgumentMatchers.anyString
import org.mockito.Mock
import org.mockito.Mockito.*
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.any
import org.mockito.kotlin.anyValueClass
import org.mockito.kotlin.argumentCaptor
import org.mockito.kotlin.whenever
import org.springframework.context.ApplicationEventPublisher
@@ -53,7 +60,7 @@ class RequestProcessorTest {
private lateinit var requestService: RequestService
private lateinit var applicationEventPublisher: ApplicationEventPublisher
private lateinit var appConfigProperties: AppConfigProperties
private lateinit var consentProcessor: ConsentProcessor
private lateinit var requestProcessor: RequestProcessor
@BeforeEach
@@ -62,14 +69,16 @@ class RequestProcessorTest {
@Mock transformationService: TransformationService,
@Mock sender: RestMtbFileSender,
@Mock requestService: RequestService,
@Mock applicationEventPublisher: ApplicationEventPublisher
@Mock applicationEventPublisher: ApplicationEventPublisher,
@Mock consentProcessor: ConsentProcessor
) {
this.pseudonymizeService = pseudonymizeService
this.transformationService = transformationService
this.sender = sender
this.requestService = requestService
this.applicationEventPublisher = applicationEventPublisher
this.appConfigProperties = AppConfigProperties(null)
this.appConfigProperties = AppConfigProperties()
this.consentProcessor = consentProcessor
val objectMapper = ObjectMapper()
@@ -80,7 +89,8 @@ class RequestProcessorTest {
requestService,
objectMapper,
applicationEventPublisher,
appConfigProperties
appConfigProperties,
consentProcessor
)
}
@@ -88,50 +98,45 @@ class RequestProcessorTest {
fun testShouldSendMtbFileDuplicationAndSaveUnknownRequestStatusAtFirst() {
doAnswer {
Request(
id = 1L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "zdlzv5s5ydmd4ktw2v5piohegc4jcyrm6j66bq6tv2uxuerndmga",
type = RequestType.MTB_FILE,
status = RequestStatus.SUCCESS,
processedAt = Instant.parse("2023-08-08T02:00:00Z")
1L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("6vkiti5bk6ikwifpajpt7cygmd3dvw54d6lwfhzlynb3pqtzferq"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS,
Instant.parse("2023-08-08T02:00:00Z")
)
}.`when`(requestService).lastMtbFileRequestForPatientPseudonym(anyString())
}.whenever(requestService).lastMtbFileRequestForPatientPseudonym(anyValueClass())
doAnswer {
false
}.`when`(requestService).isLastRequestWithKnownStatusDeletion(anyString())
}.whenever(requestService).isLastRequestWithKnownStatusDeletion(anyValueClass())
doAnswer {
it.arguments[0] as String
}.`when`(pseudonymizeService).patientPseudonym(any())
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
it.arguments[0]
}.whenever(transformationService).transform(any())
}.whenever(transformationService).transform(any<Mtb>())
val mtbFile = MtbFile.builder()
.withPatient(
whenever(consentProcessor.consentGatedCheckAndTryEmbedding(any())).thenReturn(true)
val mtbFile = Mtb.builder()
.patient(
Patient.builder()
.withId("1")
.withBirthDate("2000-08-08")
.withGender(Patient.Gender.MALE)
.id("123")
.build()
)
.withConsent(
Consent.builder()
.withId("1")
.withStatus(Consent.Status.ACTIVE)
.withPatient("123")
.build()
)
.withEpisode(
Episode.builder()
.withId("1")
.withPatient("1")
.withPeriod(PeriodStart("2023-08-08"))
.build()
.episodesOfCare(
listOf(
MtbEpisodeOfCare.builder()
.id("1")
.patient(Reference.builder().id("123").build())
.period(PeriodDate.builder().start(Date.from(Instant.parse("2023-08-08T02:00:00.00Z"))).build())
.build()
)
)
.build()
@@ -147,50 +152,45 @@ class RequestProcessorTest {
fun testShouldDetectMtbFileDuplicationAndSendDuplicationEvent() {
doAnswer {
Request(
id = 1L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "zdlzv5s5ydmd4ktw2v5piohegc4jcyrm6j66bq6tv2uxuerndmga",
type = RequestType.MTB_FILE,
status = RequestStatus.SUCCESS,
processedAt = Instant.parse("2023-08-08T02:00:00Z")
1L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("4gcjwtjjtcczybsljxepdfpkaeusvd7g3vogfqpmphyffyzfx7dq"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS,
Instant.parse("2023-08-08T02:00:00Z")
)
}.`when`(requestService).lastMtbFileRequestForPatientPseudonym(anyString())
}.whenever(requestService).lastMtbFileRequestForPatientPseudonym(anyValueClass())
doAnswer {
false
}.`when`(requestService).isLastRequestWithKnownStatusDeletion(anyString())
}.whenever(requestService).isLastRequestWithKnownStatusDeletion(anyValueClass())
doAnswer {
it.arguments[0] as String
}.`when`(pseudonymizeService).patientPseudonym(any())
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
it.arguments[0]
}.whenever(transformationService).transform(any())
}.whenever(transformationService).transform(any<Mtb>())
val mtbFile = MtbFile.builder()
.withPatient(
whenever(consentProcessor.consentGatedCheckAndTryEmbedding(any())).thenReturn(true)
val mtbFile = Mtb.builder()
.patient(
Patient.builder()
.withId("1")
.withBirthDate("2000-08-08")
.withGender(Patient.Gender.MALE)
.id("123")
.build()
)
.withConsent(
Consent.builder()
.withId("1")
.withStatus(Consent.Status.ACTIVE)
.withPatient("123")
.build()
)
.withEpisode(
Episode.builder()
.withId("1")
.withPatient("1")
.withPeriod(PeriodStart("2023-08-08"))
.build()
.episodesOfCare(
listOf(
MtbEpisodeOfCare.builder()
.id("1")
.patient(Reference.builder().id("123").build())
.period(PeriodDate.builder().start(Date.from(Instant.parse("2021-01-01T00:00:00.00Z"))).build())
.build()
)
)
.build()
@@ -206,54 +206,49 @@ class RequestProcessorTest {
fun testShouldSendMtbFileAndSendSuccessEvent() {
doAnswer {
Request(
id = 1L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "different",
type = RequestType.MTB_FILE,
status = RequestStatus.SUCCESS,
processedAt = Instant.parse("2023-08-08T02:00:00Z")
1L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("different"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS,
Instant.parse("2023-08-08T02:00:00Z")
)
}.`when`(requestService).lastMtbFileRequestForPatientPseudonym(anyString())
}.whenever(requestService).lastMtbFileRequestForPatientPseudonym(anyValueClass())
doAnswer {
false
}.`when`(requestService).isLastRequestWithKnownStatusDeletion(anyString())
}.whenever(requestService).isLastRequestWithKnownStatusDeletion(anyValueClass())
doAnswer {
MtbFileSender.Response(status = RequestStatus.SUCCESS)
}.`when`(sender).send(any<MtbFileSender.MtbFileRequest>())
}.whenever(sender).send(any<DnpmV2MtbFileRequest>())
doAnswer {
it.arguments[0] as String
}.`when`(pseudonymizeService).patientPseudonym(any())
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
it.arguments[0]
}.whenever(transformationService).transform(any())
}.whenever(transformationService).transform(any<Mtb>())
val mtbFile = MtbFile.builder()
.withPatient(
whenever(consentProcessor.consentGatedCheckAndTryEmbedding(any())).thenReturn(true)
val mtbFile = Mtb.builder()
.patient(
Patient.builder()
.withId("1")
.withBirthDate("2000-08-08")
.withGender(Patient.Gender.MALE)
.id("123")
.build()
)
.withConsent(
Consent.builder()
.withId("1")
.withStatus(Consent.Status.ACTIVE)
.withPatient("123")
.build()
)
.withEpisode(
Episode.builder()
.withId("1")
.withPatient("1")
.withPeriod(PeriodStart("2023-08-08"))
.build()
.episodesOfCare(
listOf(
MtbEpisodeOfCare.builder()
.id("1")
.patient(Reference.builder().id("123").build())
.period(PeriodDate.builder().start(Date.from(Instant.parse("2021-01-01T00:00:00.00Z"))).build())
.build()
)
)
.build()
@@ -269,54 +264,61 @@ class RequestProcessorTest {
fun testShouldSendMtbFileAndSendErrorEvent() {
doAnswer {
Request(
id = 1L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "different",
type = RequestType.MTB_FILE,
status = RequestStatus.SUCCESS,
processedAt = Instant.parse("2023-08-08T02:00:00Z")
1L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("different"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS,
Instant.parse("2023-08-08T02:00:00Z")
)
}.`when`(requestService).lastMtbFileRequestForPatientPseudonym(anyString())
}.whenever(requestService).lastMtbFileRequestForPatientPseudonym(anyValueClass())
doAnswer {
false
}.`when`(requestService).isLastRequestWithKnownStatusDeletion(anyString())
}.whenever(requestService).isLastRequestWithKnownStatusDeletion(anyValueClass())
doAnswer {
MtbFileSender.Response(status = RequestStatus.ERROR)
}.`when`(sender).send(any<MtbFileSender.MtbFileRequest>())
}.whenever(sender).send(any<DnpmV2MtbFileRequest>())
doAnswer {
it.arguments[0] as String
}.`when`(pseudonymizeService).patientPseudonym(any())
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
it.arguments[0]
}.whenever(transformationService).transform(any())
}.whenever(transformationService).transform(any<Mtb>())
val mtbFile = MtbFile.builder()
.withPatient(
whenever(consentProcessor.consentGatedCheckAndTryEmbedding(any())).thenReturn(true)
val mtbFile = Mtb.builder()
.patient(
Patient.builder()
.withId("1")
.withBirthDate("2000-08-08")
.withGender(Patient.Gender.MALE)
.id("123")
.build()
)
.withConsent(
Consent.builder()
.withId("1")
.withStatus(Consent.Status.ACTIVE)
.withPatient("123")
.metadata(
MvhMetadata
.builder()
.modelProjectConsent(
ModelProjectConsent
.builder()
.provisions(
listOf(Provision.builder().type(ConsentProvision.PERMIT).purpose(ModelProjectConsentPurpose.SEQUENCING).build())
).build()
)
.build()
)
.withEpisode(
Episode.builder()
.withId("1")
.withPatient("1")
.withPeriod(PeriodStart("2023-08-08"))
.build()
.episodesOfCare(
listOf(
MtbEpisodeOfCare.builder()
.id("1")
.patient(Reference.builder().id("123").build())
.period(PeriodDate.builder().start(Date.from(Instant.parse("2021-01-01T00:00:00.00Z"))).build())
.build()
)
)
.build()
@@ -332,13 +334,16 @@ class RequestProcessorTest {
fun testShouldSendDeleteRequestAndSaveUnknownRequestStatusAtFirst() {
doAnswer {
"PSEUDONYM"
}.`when`(pseudonymizeService).patientPseudonym(anyString())
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
MtbFileSender.Response(status = RequestStatus.UNKNOWN)
}.`when`(sender).send(any<MtbFileSender.DeleteRequest>())
}.whenever(sender).send(any<DeleteRequest>())
this.requestProcessor.processDeletion("TEST_12345678901")
this.requestProcessor.processDeletion(
TEST_PATIENT_ID,
isConsented = TtpConsentStatus.UNKNOWN_CHECK_FILE
)
val requestCaptor = argumentCaptor<Request>()
verify(requestService, times(1)).save(requestCaptor.capture())
@@ -350,13 +355,16 @@ class RequestProcessorTest {
fun testShouldSendDeleteRequestAndSendSuccessEvent() {
doAnswer {
"PSEUDONYM"
}.`when`(pseudonymizeService).patientPseudonym(anyString())
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
MtbFileSender.Response(status = RequestStatus.SUCCESS)
}.`when`(sender).send(any<MtbFileSender.DeleteRequest>())
}.whenever(sender).send(any<DeleteRequest>())
this.requestProcessor.processDeletion("TEST_12345678901")
this.requestProcessor.processDeletion(
TEST_PATIENT_ID,
isConsented = TtpConsentStatus.UNKNOWN_CHECK_FILE
)
val eventCaptor = argumentCaptor<ResponseEvent>()
verify(applicationEventPublisher, times(1)).publishEvent(eventCaptor.capture())
@@ -368,13 +376,16 @@ class RequestProcessorTest {
fun testShouldSendDeleteRequestAndSendErrorEvent() {
doAnswer {
"PSEUDONYM"
}.`when`(pseudonymizeService).patientPseudonym(anyString())
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
MtbFileSender.Response(status = RequestStatus.ERROR)
}.`when`(sender).send(any<MtbFileSender.DeleteRequest>())
}.whenever(sender).send(any<DeleteRequest>())
this.requestProcessor.processDeletion("TEST_12345678901")
this.requestProcessor.processDeletion(
TEST_PATIENT_ID,
isConsented = TtpConsentStatus.UNKNOWN_CHECK_FILE
)
val eventCaptor = argumentCaptor<ResponseEvent>()
verify(applicationEventPublisher, times(1)).publishEvent(eventCaptor.capture())
@@ -384,9 +395,12 @@ class RequestProcessorTest {
@Test
fun testShouldSendDeleteRequestWithPseudonymErrorAndSaveErrorRequestStatus() {
doThrow(RuntimeException()).`when`(pseudonymizeService).patientPseudonym(anyString())
doThrow(RuntimeException()).whenever(pseudonymizeService).patientPseudonym(anyValueClass())
this.requestProcessor.processDeletion("TEST_12345678901")
this.requestProcessor.processDeletion(
TEST_PATIENT_ID,
isConsented = TtpConsentStatus.UNKNOWN_CHECK_FILE
)
val requestCaptor = argumentCaptor<Request>()
verify(requestService, times(1)).save(requestCaptor.capture())
@@ -400,37 +414,32 @@ class RequestProcessorTest {
doAnswer {
it.arguments[0] as String
}.`when`(pseudonymizeService).patientPseudonym(any())
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
it.arguments[0]
}.whenever(transformationService).transform(any())
}.whenever(transformationService).transform(any<Mtb>())
doAnswer {
MtbFileSender.Response(status = RequestStatus.SUCCESS)
}.`when`(sender).send(any<MtbFileSender.MtbFileRequest>())
}.whenever(sender).send(any<DnpmV2MtbFileRequest>())
val mtbFile = MtbFile.builder()
.withPatient(
whenever(consentProcessor.consentGatedCheckAndTryEmbedding(any())).thenReturn(true)
val mtbFile = Mtb.builder()
.patient(
Patient.builder()
.withId("1")
.withBirthDate("2000-08-08")
.withGender(Patient.Gender.MALE)
.id("123")
.build()
)
.withConsent(
Consent.builder()
.withId("1")
.withStatus(Consent.Status.ACTIVE)
.withPatient("123")
.build()
)
.withEpisode(
Episode.builder()
.withId("1")
.withPatient("1")
.withPeriod(PeriodStart("2023-08-08"))
.build()
.episodesOfCare(
listOf(
MtbEpisodeOfCare.builder()
.id("1")
.patient(Reference.builder().id("123").build())
.period(PeriodDate.builder().start(Date.from(Instant.parse("2021-01-01T00:00:00.00Z"))).build())
.build()
)
)
.build()
@@ -442,4 +451,8 @@ class RequestProcessorTest {
assertThat(eventCaptor.firstValue.status).isEqualTo(RequestStatus.SUCCESS)
}
}
companion object {
val TEST_PATIENT_ID = PatientId("TEST_12345678901")
}
}

View File

@@ -19,6 +19,7 @@
package dev.dnpm.etl.processor.services
import dev.dnpm.etl.processor.*
import dev.dnpm.etl.processor.monitoring.Request
import dev.dnpm.etl.processor.monitoring.RequestRepository
import dev.dnpm.etl.processor.monitoring.RequestStatus
@@ -30,8 +31,9 @@ import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.Mock
import org.mockito.Mockito.*
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.anyValueClass
import org.mockito.kotlin.whenever
import java.time.Instant
import java.util.*
@ExtendWith(MockitoExtension::class)
class RequestServiceTest {
@@ -41,14 +43,14 @@ class RequestServiceTest {
private lateinit var requestService: RequestService
private fun anyRequest() = any(Request::class.java) ?: Request(
id = 0L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_dummy",
pid = "PX",
fingerprint = "dummy",
type = RequestType.MTB_FILE,
status = RequestStatus.SUCCESS,
processedAt = Instant.parse("2023-08-08T02:00:00Z")
0L,
randomRequestId(),
PatientPseudonym("TEST_dummy"),
PatientId("PX"),
Fingerprint("dummy"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS,
Instant.parse("2023-08-08T02:00:00Z")
)
@BeforeEach
@@ -63,34 +65,34 @@ class RequestServiceTest {
fun shouldIndicateLastRequestIsDeleteRequest() {
val requests = listOf(
Request(
id = 1L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "0123456789abcdef1",
type = RequestType.MTB_FILE,
status = RequestStatus.WARNING,
processedAt = Instant.parse("2023-07-07T00:00:00Z")
1L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.MTB_FILE,
RequestStatus.WARNING,
Instant.parse("2023-07-07T00:00:00Z")
),
Request(
id = 2L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "0123456789abcdefd",
type = RequestType.DELETE,
status = RequestStatus.WARNING,
processedAt = Instant.parse("2023-07-07T02:00:00Z")
2L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdefd"),
RequestType.DELETE,
RequestStatus.WARNING,
Instant.parse("2023-07-07T02:00:00Z")
),
Request(
id = 3L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "0123456789abcdef1",
type = RequestType.MTB_FILE,
status = RequestStatus.UNKNOWN,
processedAt = Instant.parse("2023-08-11T00:00:00Z")
3L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.MTB_FILE,
RequestStatus.UNKNOWN,
Instant.parse("2023-08-11T00:00:00Z")
)
)
@@ -103,34 +105,34 @@ class RequestServiceTest {
fun shouldIndicateLastRequestIsNotDeleteRequest() {
val requests = listOf(
Request(
id = 1L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "0123456789abcdef1",
type = RequestType.MTB_FILE,
status = RequestStatus.WARNING,
processedAt = Instant.parse("2023-07-07T00:00:00Z")
1L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.MTB_FILE,
RequestStatus.WARNING,
Instant.parse("2023-07-07T00:00:00Z")
),
Request(
id = 2L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "0123456789abcdef1",
type = RequestType.MTB_FILE,
status = RequestStatus.WARNING,
processedAt = Instant.parse("2023-07-07T02:00:00Z")
2L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.MTB_FILE,
RequestStatus.WARNING,
Instant.parse("2023-07-07T02:00:00Z")
),
Request(
id = 3L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "0123456789abcdef1",
type = RequestType.MTB_FILE,
status = RequestStatus.UNKNOWN,
processedAt = Instant.parse("2023-08-11T00:00:00Z")
3L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.MTB_FILE,
RequestStatus.UNKNOWN,
Instant.parse("2023-08-11T00:00:00Z")
)
)
@@ -143,31 +145,31 @@ class RequestServiceTest {
fun shouldReturnPatientsLastRequest() {
val requests = listOf(
Request(
id = 1L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "0123456789abcdef1",
type = RequestType.DELETE,
status = RequestStatus.SUCCESS,
processedAt = Instant.parse("2023-07-07T02:00:00Z")
1L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.DELETE,
RequestStatus.SUCCESS,
Instant.parse("2023-07-07T02:00:00Z")
),
Request(
id = 1L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678902",
pid = "P2",
fingerprint = "0123456789abcdef2",
type = RequestType.MTB_FILE,
status = RequestStatus.WARNING,
processedAt = Instant.parse("2023-08-08T00:00:00Z")
1L,
randomRequestId(),
PatientPseudonym("TEST_12345678902"),
PatientId("P2"),
Fingerprint("0123456789abcdef2"),
RequestType.MTB_FILE,
RequestStatus.WARNING,
Instant.parse("2023-08-08T00:00:00Z")
)
)
val actual = RequestService.lastMtbFileRequestForPatientPseudonym(requests)
assertThat(actual).isInstanceOf(Request::class.java)
assertThat(actual?.fingerprint).isEqualTo("0123456789abcdef2")
assertThat(actual?.fingerprint).isEqualTo(Fingerprint("0123456789abcdef2"))
}
@Test
@@ -184,16 +186,16 @@ class RequestServiceTest {
doAnswer {
val obj = it.arguments[0] as Request
obj.copy(id = 1L)
}.`when`(requestRepository).save(anyRequest())
}.whenever(requestRepository).save(anyRequest())
val request = Request(
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "0123456789abcdef1",
type = RequestType.DELETE,
status = RequestStatus.SUCCESS,
processedAt = Instant.parse("2023-07-07T02:00:00Z")
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.DELETE,
RequestStatus.SUCCESS,
Instant.parse("2023-07-07T02:00:00Z")
)
requestService.save(request)
@@ -203,23 +205,23 @@ class RequestServiceTest {
@Test
fun allRequestsByPatientPseudonymShouldRequestAllRequestsForPatientPseudonym() {
requestService.allRequestsByPatientPseudonym("TEST_12345678901")
requestService.allRequestsByPatientPseudonym(PatientPseudonym("TEST_12345678901"))
verify(requestRepository, times(1)).findAllByPatientIdOrderByProcessedAtDesc(anyString())
verify(requestRepository, times(1)).findAllByPatientPseudonymOrderByProcessedAtDesc(anyValueClass())
}
@Test
fun lastMtbFileRequestForPatientPseudonymShouldRequestAllRequestsForPatientPseudonym() {
requestService.lastMtbFileRequestForPatientPseudonym("TEST_12345678901")
requestService.lastMtbFileRequestForPatientPseudonym(PatientPseudonym("TEST_12345678901"))
verify(requestRepository, times(1)).findAllByPatientIdOrderByProcessedAtDesc(anyString())
verify(requestRepository, times(1)).findAllByPatientPseudonymOrderByProcessedAtDesc(anyValueClass())
}
@Test
fun isLastRequestDeletionShouldRequestAllRequestsForPatientPseudonym() {
requestService.isLastRequestWithKnownStatusDeletion("TEST_12345678901")
requestService.isLastRequestWithKnownStatusDeletion(PatientPseudonym("TEST_12345678901"))
verify(requestRepository, times(1)).findAllByPatientIdOrderByProcessedAtDesc(anyString())
verify(requestRepository, times(1)).findAllByPatientPseudonymOrderByProcessedAtDesc(anyValueClass())
}
}

View File

@@ -19,8 +19,8 @@
package dev.dnpm.etl.processor.services
import dev.dnpm.etl.processor.*
import dev.dnpm.etl.processor.monitoring.Request
import dev.dnpm.etl.processor.monitoring.RequestRepository
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.monitoring.RequestType
import org.assertj.core.api.Assertions.assertThat
@@ -29,7 +29,6 @@ import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.MethodSource
import org.mockito.ArgumentMatchers.anyString
import org.mockito.Mock
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.*
@@ -40,64 +39,64 @@ import java.util.*
@ExtendWith(MockitoExtension::class)
class ResponseProcessorTest {
private lateinit var requestRepository: RequestRepository
private lateinit var requestService: RequestService
private lateinit var statisticsUpdateProducer: Sinks.Many<Any>
private lateinit var responseProcessor: ResponseProcessor
private val testRequest = Request(
1L,
"TestID1234",
"PSEUDONYM-A",
"1",
"dummyfingerprint",
RequestId("TestID1234"),
PatientPseudonym("PSEUDONYM-A"),
PatientId("1"),
Fingerprint("dummyfingerprint"),
RequestType.MTB_FILE,
RequestStatus.UNKNOWN
)
@BeforeEach
fun setup(
@Mock requestRepository: RequestRepository,
@Mock requestService: RequestService,
@Mock statisticsUpdateProducer: Sinks.Many<Any>
) {
this.requestRepository = requestRepository
this.requestService = requestService
this.statisticsUpdateProducer = statisticsUpdateProducer
this.responseProcessor = ResponseProcessor(requestRepository, statisticsUpdateProducer)
this.responseProcessor = ResponseProcessor(requestService, statisticsUpdateProducer)
}
@Test
fun shouldNotSaveStatusForUnknownRequest() {
doAnswer {
Optional.empty<Request>()
}.whenever(requestRepository).findByUuidEquals(anyString())
}.whenever(requestService).findByUuid(anyValueClass())
val event = ResponseEvent(
"TestID1234",
RequestId("TestID1234"),
Instant.parse("2023-09-09T00:00:00Z"),
RequestStatus.SUCCESS
)
this.responseProcessor.handleResponseEvent(event)
verify(requestRepository, never()).save(any())
verify(requestService, never()).save(any())
}
@Test
fun shouldNotSaveStatusWithUnknownState() {
doAnswer {
Optional.of(testRequest)
}.whenever(requestRepository).findByUuidEquals(anyString())
}.whenever(requestService).findByUuid(anyValueClass())
val event = ResponseEvent(
"TestID1234",
RequestId("TestID1234"),
Instant.parse("2023-09-09T00:00:00Z"),
RequestStatus.UNKNOWN
)
this.responseProcessor.handleResponseEvent(event)
verify(requestRepository, never()).save(any())
verify(requestService, never()).save(any<Request>())
}
@ParameterizedTest
@@ -105,10 +104,10 @@ class ResponseProcessorTest {
fun shouldSaveStatusForKnownRequest(requestStatus: RequestStatus) {
doAnswer {
Optional.of(testRequest)
}.whenever(requestRepository).findByUuidEquals(anyString())
}.whenever(requestService).findByUuid(anyValueClass())
val event = ResponseEvent(
"TestID1234",
RequestId("TestID1234"),
Instant.parse("2023-09-09T00:00:00Z"),
requestStatus
)
@@ -116,7 +115,7 @@ class ResponseProcessorTest {
this.responseProcessor.handleResponseEvent(event)
val captor = argumentCaptor<Request>()
verify(requestRepository, times(1)).save(captor.capture())
verify(requestService, times(1)).save(captor.capture())
assertThat(captor.firstValue).isNotNull
assertThat(captor.firstValue.status).isEqualTo(requestStatus)
}

View File

@@ -19,14 +19,14 @@
package dev.dnpm.etl.processor.services
import com.fasterxml.jackson.databind.ObjectMapper
import de.ukw.ccc.bwhc.dto.Consent
import de.ukw.ccc.bwhc.dto.Diagnosis
import de.ukw.ccc.bwhc.dto.Icd10
import de.ukw.ccc.bwhc.dto.MtbFile
import dev.dnpm.etl.processor.config.JacksonConfig
import dev.pcvolkmer.mv64e.mtb.*
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.hl7.fhir.instance.model.api.IBaseResource
import java.time.Instant
import java.util.Date
class TransformationServiceTest {
@@ -35,61 +35,92 @@ class TransformationServiceTest {
@BeforeEach
fun setup() {
this.service = TransformationService(
ObjectMapper(), listOf(
Transformation.of("consent.status") from Consent.Status.ACTIVE to Consent.Status.REJECTED,
Transformation.of("diagnoses[*].icd10.version") from "2013" to "2014",
JacksonConfig().objectMapper(), listOf(
Transformation.of("diagnoses[*].code.version") from "2013" to "2014",
)
)
}
@Test
fun shouldTransformMtbFile() {
val mtbFile = MtbFile.builder().withDiagnoses(
val mtbFile = Mtb.builder().diagnoses(
listOf(
Diagnosis.builder().withId("1234").withIcd10(Icd10("F79.9").also {
it.version = "2013"
}).build()
MtbDiagnosis.builder().id("1234").code(Coding.builder().code("F79.9").version("2013").build()).build()
)
).build()
val actual = this.service.transform(mtbFile)
assertThat(actual).isNotNull
assertThat(actual.diagnoses[0].icd10.version).isEqualTo("2014")
assertThat(actual.diagnoses[0].code.version).isEqualTo("2014")
}
@Test
fun shouldOnlyTransformGivenValues() {
val mtbFile = MtbFile.builder().withDiagnoses(
val mtbFile = Mtb.builder().diagnoses(
listOf(
Diagnosis.builder().withId("1234").withIcd10(Icd10("F79.9").also {
it.version = "2013"
}).build(),
Diagnosis.builder().withId("5678").withIcd10(Icd10("F79.8").also {
it.version = "2019"
}).build()
MtbDiagnosis.builder().id("1234").code(Coding.builder().code("F79.9").version("2013").build()).build(),
MtbDiagnosis.builder().id("1234").code(Coding.builder().code("F79.8").version("2019").build()).build()
)
).build()
val actual = this.service.transform(mtbFile)
assertThat(actual).isNotNull
assertThat(actual.diagnoses[0].icd10.code).isEqualTo("F79.9")
assertThat(actual.diagnoses[0].icd10.version).isEqualTo("2014")
assertThat(actual.diagnoses[1].icd10.code).isEqualTo("F79.8")
assertThat(actual.diagnoses[1].icd10.version).isEqualTo("2019")
assertThat(actual.diagnoses[0].code.code).isEqualTo("F79.9")
assertThat(actual.diagnoses[0].code.version).isEqualTo("2014")
assertThat(actual.diagnoses[1].code.code).isEqualTo("F79.8")
assertThat(actual.diagnoses[1].code.version).isEqualTo("2019")
}
@Test
fun shouldTransformMtbFileWithConsentEnum() {
val mtbFile = MtbFile.builder().withConsent(
Consent("123", "456", Consent.Status.ACTIVE)
fun shouldTransformConsentValues() {
val mtbFile = Mtb.builder().diagnoses(
listOf(
MtbDiagnosis.builder().id("1234").code(Coding.builder().code("F79.9").version("2013").build()).build(),
MtbDiagnosis.builder().id("1234").code(Coding.builder().code("F79.8").version("2019").build()).build()
)
).build()
val actual = this.service.transform(mtbFile)
assertThat(actual.consent).isNotNull
assertThat(actual.consent.status).isEqualTo(Consent.Status.REJECTED)
assertThat(actual).isNotNull
assertThat(actual.diagnoses[0].code.code).isEqualTo("F79.9")
assertThat(actual.diagnoses[0].code.version).isEqualTo("2014")
assertThat(actual.diagnoses[1].code.code).isEqualTo("F79.8")
assertThat(actual.diagnoses[1].code.version).isEqualTo("2019")
}
}
@Test
fun shouldTransformConsent() {
val mvhMetadata = MvhMetadata.builder().transferTan("transfertan12345").build()
assertThat(mvhMetadata).isNotNull
mvhMetadata.modelProjectConsent =
ModelProjectConsent.builder().date(Date.from(Instant.parse("2025-08-15T00:00:00.00Z")))
.version("1").provisions(
listOf(
Provision.builder().type(ConsentProvision.PERMIT)
.purpose(ModelProjectConsentPurpose.SEQUENCING)
.date(Date.from(Instant.parse("2025-08-15T00:00:00.00Z"))).build(),
Provision.builder().type(ConsentProvision.PERMIT)
.purpose(ModelProjectConsentPurpose.REIDENTIFICATION)
.date(Date.from(Instant.parse("2025-08-15T00:00:00.00Z"))).build(),
Provision.builder().type(ConsentProvision.DENY)
.purpose(ModelProjectConsentPurpose.CASE_IDENTIFICATION)
.date(Date.from(Instant.parse("2025-08-15T00:00:00.00Z"))).build()
)
).build()
val consent = ConsentProcessorTest.getDummyGenomDeConsent()
mvhMetadata.researchConsents = mutableListOf()
mvhMetadata.researchConsents.add(mapOf(consent.id to consent as IBaseResource))
val mtbFile = Mtb.builder().metadata(mvhMetadata).build()
val transformed = service.transform(mtbFile)
assertThat(transformed.metadata.modelProjectConsent.date).isNotNull
}
}

File diff suppressed because one or more lines are too long

File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More