1
0
mirror of https://github.com/pcvolkmer/etl-processor.git synced 2025-07-01 14:12:55 +00:00

169 Commits

Author SHA1 Message Date
ec096d9c81 chore: bump version 2025-04-04 14:57:28 +02:00
a4d0b73d2b docs: update some information in README.md 2025-04-04 14:55:14 +02:00
9307fc0dad docs: change etl image and highlight important information 2025-04-04 14:49:37 +02:00
586d388e57 docs: add info about DNPM:DIP support 2025-04-04 14:36:30 +02:00
7ae34719fd feat: add new MTB endpoint path (#93) 2025-04-04 14:34:31 +02:00
033750eb10 feat: show issue path if available in response body (#92) 2025-04-04 13:59:51 +02:00
befeef3153 feat: use issue severity to create status (#90) 2025-04-03 17:06:03 +02:00
98b971d7db feat: do not retry on validation issues (#89)
This will prevent retry if response is HTTP 400 or HTTP 422.
2025-03-23 13:35:24 +01:00
56a63b276e Code cleanup (#87)
* refactor: Replace usage of Void with Kotlins Unit

* refactor: make ConnectionCheckService a functional interface

* refactor: ignore unused exception

* refactor: use property access syntax

* refactor: use const value for login path
2025-03-23 12:09:34 +01:00
c0ea5fcd51 test: use Europe/Berlin as timezone in tests 2025-03-23 01:05:06 +01:00
d4fd54f51f Merge branch '0.9.x' 2025-03-22 23:58:47 +01:00
d49671f0d4 build: update image name 2025-03-22 23:40:13 +01:00
84868dc22c Merge branch '0.9.x' 2025-03-22 11:26:16 +01:00
4ad6c4bd0a feat: handle and save issue report for non HTTP 2xx responses 2025-03-22 11:04:32 +01:00
9bdd8ba375 chore: update Spring Boot 2025-03-22 10:20:52 +01:00
f027339425 chore: update gradle 2025-03-22 10:20:35 +01:00
3c5639708f chore: highlight selected config tab 2025-03-21 19:26:25 +01:00
639159c677 docs: add information about DNPM:DIP dev environment 2025-03-21 19:15:20 +01:00
38261d6d2c chore: update bwhc-dto-java
This enables use of WHOGrading version 2021.
2025-03-21 19:11:24 +01:00
47ebe46974 feat: add checks for DNPM:DIP backend
Since DNPM:DIP responds with HTTP 404 on API base path, the Kaplan-Meier Config
endpoint will be used to check availability of DNPM:DIP backend API.
2025-03-20 14:39:40 +01:00
f347653be8 refactor: use UriComponentsBuilder to build URL to be used
This prevents problems using trailing slash in remote API URL.
2025-03-20 14:19:25 +01:00
775a7df1ce chore: use API URL to DNPM:DIP 2025-03-20 14:13:21 +01:00
f66b737f11 docs: add example APP_REST_URI for use with dnpm:dip 2025-03-19 15:29:55 +01:00
3a19212a78 chore: update Spring Boot 2025-03-09 09:38:59 +01:00
280fbd445e chore: update Spring Boot 2025-03-09 09:24:51 +01:00
91e2cf5ef1 refactor: use different sender classes for bwHC and DIP 2025-03-08 11:42:14 +01:00
262c54f2e5 fix: use patient pseudonym value 2025-03-08 11:42:14 +01:00
b25e580113 feat: Support POSTing data to dnpm:dip 2025-03-08 11:42:14 +01:00
ff27b7157d chore: update spring boot and dependency management plugin 2025-02-10 19:18:27 +01:00
1e652a7856 test: explicit request URI check and fix use of expect() 2025-02-09 11:19:35 +01:00
74ff9f08a4 chore: update to mockito-kotlin 5.4.0
With this change, `anyValueClass()` from mockito-kotlin
replaces own implementation.
2024-12-23 18:16:37 +01:00
23cc2f365a chore: update dependencies 2024-12-20 10:45:16 +01:00
53b4cf1a95 chore: update dependencies 2024-11-25 21:02:55 +01:00
5ce13e962b build: add description and group to task 2024-11-02 15:08:25 +01:00
3257493b6a build: update HAPI dependencies
This also overrides 'commons-io' due to CVE-2024-47554
2024-11-02 15:08:25 +01:00
2036077c06 build: do not hard code version numbers in dependencies 2024-11-02 15:08:25 +01:00
8ce3aed870 docs: modify changed urls 2024-11-01 15:38:47 +01:00
998989d319 Merge branch '0.9.x' 2024-11-01 15:32:40 +01:00
e95fa2fb12 build: change BP_OCI_SOURCE 2024-11-01 15:06:24 +01:00
1bcc8c13de build: change group name 2024-11-01 14:44:09 +01:00
2fc3299543 build: replace hard coded repo name with variable (#81) 2024-11-01 14:23:26 +01:00
5575867632 chore: update to Spring Boot 3.2.11 (#80) 2024-11-01 14:14:45 +01:00
46ba565c29 chore: update to Spring Boot 3.3.5 (#79) 2024-11-01 14:05:20 +01:00
6cdbd35e64 feat: Allow configuring basic auth for the rest uri (#75) 2024-11-01 13:56:54 +01:00
d258d9081b chore: gPas health check, fetch metadata instead of send invalid gPas request (#73) 2024-11-01 13:54:40 +01:00
eb49ba611b chore: update to Spring Boot 3.3.4 (#78) 2024-11-01 13:51:06 +01:00
efa736f232 chore: update to Spring Boot 3.3.3 (#77) 2024-11-01 13:42:20 +01:00
4a7030e85b chore: update to Spring Boot 3.3.2 (#76) 2024-11-01 13:27:31 +01:00
464c8b8c1d refactor: use dedicated type for path param 2024-07-15 11:59:00 +02:00
3f1bb4f4e2 refactor: rename template attribute to reflect content 2024-07-15 11:51:33 +02:00
370ea87095 refactor: rename db column name to reflect content 2024-07-15 11:44:19 +02:00
c8f6e6efc8 refactor: add types for patient id and pseudonym 2024-07-15 10:31:52 +02:00
c949ec07e5 Merge branch 0.9.x 2024-07-15 08:11:11 +02:00
87658bfa58 chore: update to Spring Boot 3.2.7 2024-07-15 07:58:36 +02:00
99efd6c98a fix: downgrade echarts due to dependency issues 2024-07-15 07:43:44 +02:00
e42d11f125 chore: update webjars dependencies 2024-07-15 07:43:32 +02:00
6e0ec6b95a style: use sans-serif font everywhere 2024-06-25 16:16:13 +02:00
0ff56416dd chore: update dev-compose.yml 2024-06-25 16:13:55 +02:00
3a2f6a2bb6 chore: update oidc client local dev password 2024-06-25 16:05:41 +02:00
3eb9e68786 fix: use versions param for commons-compress 2024-06-25 08:18:02 +02:00
59403d1dba chore: update dependency version definitions
This will use httpclient5 version as defined within spring boot.
2024-06-24 10:18:18 +02:00
9f5ac664af chore: update to spring boot 3.3.1 2024-06-24 10:02:12 +02:00
5867ed9dd3 Merge pull request #67 from CCC-MF/66-update-auf-spring-boot-33x
chore: update to spring boot 3.3.0
2024-06-20 11:37:34 +02:00
4d6d1879e6 chore: update to spring boot 3.3.0
This also adds dependency to org.flywaydb:flyway-database-postgresql
2024-06-20 09:14:17 +02:00
2a34c0efc9 fix: downgrade echarts due to dependency issues 2024-06-19 14:25:58 +02:00
0ee00de5aa chore: update webjars dependencies 2024-06-19 14:16:52 +02:00
baeebdb9b8 Merge branch 0.9.x 2024-05-31 13:31:40 +02:00
8b194e7212 chore: update kotlin version 2024-05-31 12:55:32 +02:00
070100eba0 chore: update spring dependency-management plugin 2024-05-31 12:54:28 +02:00
ce1489d9a1 chore: update spring boot dependencies 2024-05-31 12:54:23 +02:00
ca1e73a0b5 chore: update bwhc-dto-java dependency 2024-05-31 12:54:18 +02:00
041bf459ef fix: add missing 'fatal' severity 2024-05-31 12:52:22 +02:00
c922e27758 fix: handle null values in MtbFile
This should not occur but if, it should not result in NPE except for

* Patient
* Consent
* Episode
2024-05-31 12:51:40 +02:00
4d5c0ce1fb chore: remove println 2024-05-31 12:51:36 +02:00
bb0bbf5a28 chore: update webjars-locator dependency 2024-05-31 12:49:51 +02:00
1b4585d601 docs: fix CVE number in dependency comment 2024-05-31 12:44:23 +02:00
dad3ea80ee chore: update integration test dependency
This mitigates CVE-204-26308 and CVE-2024-25710
2024-05-31 12:44:19 +02:00
01446bdece chore: update GitHub workflow actions 2024-05-31 12:34:57 +02:00
43660a4dcb chore: mark as snapshot version 2024-05-31 12:22:08 +02:00
5320466b6c test: exclude test and integrationTest source set
This replaces filtering for classes that contain 'Test' or 'Tests' in simple name.
2024-05-30 14:29:31 +02:00
263cb02416 test: use Kotlin KArgumentCaptor 2024-05-27 15:22:54 +02:00
0b37fd7091 chore: update kotlin version 2024-05-27 12:33:40 +02:00
bdee969409 chore: update spring dependency-management plugin 2024-05-27 12:33:15 +02:00
4c39920afd build: use compilerOptions instead of kotlinOptions
Config kotlinOptions will be deprecated in Kotlin >= 2.0
2024-05-27 12:31:05 +02:00
5e836c48b0 docs: add information about other credentials for MTBfile endpoint 2024-05-27 12:23:50 +02:00
fb5a3c062c feat: allow access to MTBFile endpoint for non-token users 2024-05-27 12:19:24 +02:00
8fc0609aa4 feat: use RequestId type 2024-05-27 11:23:20 +02:00
a846a8765a chore: update spring boot dependencies 2024-05-27 09:08:08 +02:00
8645becd82 chore: update bwhc-dto-java dependency 2024-05-16 17:59:22 +02:00
011511d5ef feat: use details as alias for message in data quality report 2024-05-16 09:08:21 +02:00
e9839c2731 fix: add missing 'fatal' severity 2024-05-15 17:27:04 +02:00
86bee9e2cf feat: show info if no requests present 2024-05-08 13:08:15 +02:00
f419acb924 test: mock UserRoleService only in nested test class 2024-05-08 10:28:11 +02:00
52171e8ebe test: add test for config SSE endpoint 2024-05-08 09:18:29 +02:00
a2124ba83d test: add test for SSE endpoint 2024-05-07 18:45:07 +02:00
a046203339 test: test statistics json response 2024-05-07 16:57:51 +02:00
b40d41ce8c test: add initial tests for StatisticsRestController 2024-05-07 15:57:35 +02:00
57de96771c test: add test for StatisticsController 2024-05-07 15:48:39 +02:00
3bc148f7ea refactor: move classes into package 'security' 2024-05-07 08:58:00 +02:00
8e6b1ec799 refactor: use whenever() instead of when in tests 2024-05-06 19:11:54 +02:00
8e5f5c73ec test: add tests for UserRoleService 2024-05-06 18:54:28 +02:00
d4f984b138 test: add tests for HomeController 2024-05-06 18:01:08 +02:00
24ebbf3b50 test: add tests to and reorganize ConfigControllerTest 2024-05-06 17:14:22 +02:00
9c6bd64a7e test: ensure correct view is rendered 2024-05-06 13:33:36 +02:00
6567aa803c test: add htmlunit based test for LoginController 2024-05-06 13:26:29 +02:00
e874350712 test: add tests for user role requests 2024-05-06 11:53:54 +02:00
94d7b4c4f0 test: add tests for token requests 2024-05-06 11:42:11 +02:00
107429fda7 fix: handle null values in MtbFile
This should not occur but if, it should not result in NPE except for

* Patient
* Consent
* Episode
2024-05-06 10:43:03 +02:00
26b2f65e67 chore: remove println 2024-05-06 10:03:36 +02:00
e863269a42 build: add jacoco code coverage 2024-05-06 09:44:19 +02:00
4ab95ef11f build: update gradle version 2024-05-03 16:44:41 +02:00
2244ef1b86 build: register task to run all tests at once 2024-05-03 16:37:16 +02:00
c3ddb387e2 test: add some basic arch unit tests 2024-05-02 12:30:12 +02:00
ae5d8341cc refactor: use RequestService in ResponseProcessor 2024-05-02 10:12:40 +02:00
40b2558943 refactor: use RequestService in controllers 2024-05-02 09:58:20 +02:00
9a6a0c6138 refactor: use Fingerprint type instead of plain String 2024-04-30 11:08:27 +02:00
5985327219 refactor: use type alias 2024-04-29 18:40:16 +02:00
06f9e8ace9 test: add initial RequestRepository test 2024-04-29 17:34:27 +02:00
365a651918 chore: update dependency to mockito-kotlin 2024-04-29 16:45:25 +02:00
5fcc24f915 refactor: add additional constructors 2024-04-29 10:11:25 +02:00
3bd7239812 chore: use java records 2024-04-26 12:35:37 +02:00
1672ad8640 chore: update webjars-locator dependency 2024-04-25 09:33:35 +02:00
710aeb1f18 docs: fix CVE number in dependency comment 2024-04-25 09:10:33 +02:00
06d11790b6 chore: update integration test dependency
This mitigates CVE-204-26308 and CVE-2024-25710
2024-04-24 12:06:08 +02:00
959f6889d4 refactor: extract custom SSL gPAS rest template creation 2024-04-24 08:56:31 +02:00
0f5a68660d refactor: move method content and call this method 2024-04-24 08:41:47 +02:00
b809a2da02 refactor: move custom rest template init to config class 2024-04-24 08:22:13 +02:00
effffcfc1a test: add tests with simulated gPAS responses 2024-04-23 15:53:22 +02:00
7b3151d227 fix: custom rest template generation after ssl context creation 2024-04-23 15:52:40 +02:00
26b415f336 docs: add deprecation warnings for config options 2024-04-23 15:02:14 +02:00
bda3c30a74 refactor: use default RestTemplate bean if no custom one required 2024-04-23 14:48:24 +02:00
8779600330 chore: change private method name to fix typo 2024-04-23 14:39:37 +02:00
159fb46009 chore: update GitHub workflow actions 2024-04-22 19:30:46 +02:00
eabbbfbb68 refactor: rename method for custom rest template configuration 2024-04-22 19:13:47 +02:00
4db38ef2f0 refactor: use more convenient way to set basic auth header 2024-04-22 19:11:19 +02:00
ed6d21e920 Merge branch 'v0.9.x' 2024-04-19 09:50:19 +02:00
550bee5ad3 chore: use snapshot version 2024-04-19 09:49:10 +02:00
8313420de5 chore: bump version for new release 2024-04-19 09:42:19 +02:00
1651f446fe chore: update spring boot and other dependencies 2024-04-19 09:41:17 +02:00
bd7dccbd87 Merge pull request #61 from CCC-MF/issue_55
Anzeige der letzten Verbindungsstatusänderung
2024-03-26 10:06:27 +01:00
8ae958b8c4 feat: show information if no output is defined 2024-03-26 09:56:31 +01:00
0f144568e3 style: change tab content background to white 2024-03-25 17:27:08 +01:00
08540e3bd7 feat: add timestamp of last connection check change 2024-03-25 17:24:33 +01:00
43af1aa103 feat: add timestamp to connection check display 2024-03-25 17:09:27 +01:00
056a087065 chore: update spring boot dependencies 2024-03-25 16:12:20 +01:00
a730ce2a53 fix: update spring security due to CVE-2024-22257 2024-03-19 16:40:32 +01:00
12eb1feea6 fix: assign new value from scope function 2024-03-12 18:29:42 +01:00
af714f7b64 fix: ignore possible null values in mtb files 2024-03-12 17:56:25 +01:00
f47b0b7de4 build: bump version 2024-03-12 13:27:45 +01:00
d8ba6b67cb docs: change description of ID anonymization 2024-03-12 13:27:29 +01:00
40b89dd4f1 Merge pull request #60 from CCC-MF/issue_44
feat: salted re-hash IDs within MTB file except patient ID
2024-03-12 13:18:32 +01:00
e3aeee61de feat: salted re-hash IDs within MTB file except patient ID 2024-03-12 13:13:31 +01:00
07e59f9b02 docs: update README.md to mention later kafka record processing 2024-03-09 11:09:44 +01:00
f751d64220 Merge pull request #59 from CCC-MF/issue_58
feat: do not use episode id in kafka record key
2024-03-09 11:00:28 +01:00
299bd56d63 feat: do not use episode id in kafka record key 2024-03-09 10:58:03 +01:00
a0c4d1863f Merge pull request #57 from CCC-MF/issue_56
feat: use requestId from incoming Kafka Record Header
2024-03-08 15:44:39 +01:00
fc1901211d feat: use requestId from incoming Kafka Record Header 2024-03-08 15:42:04 +01:00
bed91439db docs: update etl image 2024-03-07 18:30:44 +01:00
a8e008000e Merge pull request #54 from CCC-MF/issue_53
Anzeige gPAS Verbindungsstatus
2024-03-06 10:54:44 +01:00
a9c771aa99 test: change tests to mock output connection 2024-03-06 10:50:35 +01:00
256d9d4ff0 chore: change wording 2024-03-06 10:08:23 +01:00
41b87835ca feat: add configuration for deprecated config property 2024-03-06 10:02:31 +01:00
3654962294 feat: initial implementation of gPAS connection check 2024-03-06 10:00:17 +01:00
9382da7101 refactor: do not use singleton like rest template object 2024-03-05 17:03:10 +01:00
67ab0ef2be build: next snapshot version 2024-03-05 16:44:17 +01:00
69d796dab4 Merge pull request #52 from CCC-MF/issue_51
Darstellung und Aufteilung der Konfigurationsseite verbessern
2024-03-05 10:27:27 +01:00
4bfe7dc698 style: layout and style changes for config page 2024-03-05 10:24:25 +01:00
0aec5e4479 style: fixed first column width 2024-03-04 17:03:41 +01:00
b1a83510a6 style: fix statistics chart layout 2024-03-04 16:33:52 +01:00
77 changed files with 4137 additions and 913 deletions

View File

@ -21,7 +21,7 @@ jobs:
uses: docker/setup-buildx-action@v3
- name: Login to Docker Hub
uses: docker/login-action@v2
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
@ -30,6 +30,6 @@ jobs:
- name: Execute image build and push
run: |
./gradlew bootBuildImage
docker tag ghcr.io/ccc-mf/etl-processor ghcr.io/ccc-mf/etl-processor:${{ github.ref_name }}
docker push ghcr.io/ccc-mf/etl-processor
docker push ghcr.io/ccc-mf/etl-processor:${{ github.ref_name }}
docker tag ghcr.io/${{ github.repository }} ghcr.io/${{ github.repository }}:${{ github.ref_name }}
docker push ghcr.io/${{ github.repository }}
docker push ghcr.io/${{ github.repository }}:${{ github.ref_name }}

View File

@ -12,7 +12,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v3
- uses: actions/setup-java@v4
with:
java-version: '21'
distribution: 'temurin'
@ -27,7 +27,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v3
- uses: actions/setup-java@v4
with:
java-version: '21'
distribution: 'temurin'

117
README.md
View File

@ -1,6 +1,6 @@
# ETL-Processor for bwHC data [![Run Tests](https://github.com/CCC-MF/etl-processor/actions/workflows/test.yml/badge.svg)](https://github.com/CCC-MF/etl-processor/actions/workflows/test.yml)
# ETL-Processor for DNPM:DIP [![Run Tests](https://github.com/pcvolkmer/etl-processor/actions/workflows/test.yml/badge.svg)](https://github.com/pcvolkmer/etl-processor/actions/workflows/test.yml)
Diese Anwendung versendet ein bwHC-MTB-File an das bwHC-Backend und pseudonymisiert die Patienten-ID.
Diese Anwendung versendet ein bwHC-MTB-File im bwHC-Datenmodell 1.0 an DNPM:DIP und pseudonymisiert die Patienten-ID.
## Einordnung innerhalb einer DNPM-ETL-Strecke
@ -9,7 +9,7 @@ Diese Anwendung erlaubt das Entgegennehmen von HTTP/REST-Anfragen aus dem Onkost
Der Inhalt einer Anfrage, wenn ein bwHC-MTBFile, wird pseudonymisiert und auf Duplikate geprüft.
Duplikate werden verworfen, Änderungen werden weitergeleitet.
Löschanfragen werden immer als Löschanfrage an das bwHC-backend weitergeleitet.
Löschanfragen werden immer als Löschanfrage an DNPM:DIP weitergeleitet.
Zudem ist eine minimalistische Weboberfläche integriert, die einen Einblick in den aktuellen Zustand der Anwendung gewährt.
@ -22,7 +22,17 @@ Die Erkennung von Duplikaten ist normalerweise immer aktiv, kann jedoch über de
### Datenübermittlung über HTTP/REST
Anfragen werden, wenn nicht als Duplikat behandelt, nach der Pseudonymisierung direkt an das bwHC-Backend gesendet.
Anfragen werden, wenn nicht als Duplikat behandelt, nach der Pseudonymisierung direkt an DNPM:DIP gesendet.
Ein HTTP Request kann, angenommen die Installation erfolgte auf dem Host `dnpm.example.com` an nachfolgende URLs gesendet werden:
| HTTP-Request | URL | Consent-Status im Datensatz | Bemerkung |
|--------------|-----------------------------------------|-----------------------------|---------------------------------------------------------------------------------|
| `POST` | `https://dnpm.example.com/mtb` | `ACTIVE` | Die Anwendung verarbeitet den eingehenden Datensatz |
| `POST` | `https://dnpm.example.com/mtb` | `REJECT` | Die Anwendung sendet einen Lösch-Request für die im Datensatz angegebene Pat-ID |
| `DELETE` | `https://dnpm.example.com/mtb/12345678` | - | Die Anwendung sendet einen Lösch-Request für Pat-ID `12345678` |
Anstelle des Pfads `/mtb` kann auch, wie in Version 0.9 und älter üblich, `/mtbfile` verwendet werden.
### Datenübermittlung mit Apache Kafka
@ -33,27 +43,43 @@ Siehe hierzu auch: https://github.com/CCC-MF/kafka-to-bwhc
## Konfiguration
### 🔥 Wichtige Änderungen in Version 0.10
Ab Version 0.10 wird [DNPM:DIP](https://github.com/dnpm-dip) unterstützt und als Standardendpunkt verwendet.
Soll noch das alte bwHC-Backend verwendet werden, so ist die Umgebungsvariable `APP_REST_IS_BWHC` auf `true` zu setzen.
### 🔥 Breaking Changes nach Version 0.10
In Versionen des ETL-Processors **nach Version 0.10** werden die folgenden Konfigurationsoptionen entfernt:
* `APP_PSEUDONYMIZE_GPAS_SSLCALOCATION`: Nutzen Sie hier, wie unter [_Integration eines eigenen Root CA
Zertifikats_](#integration-eines-eigenen-root-ca-zertifikats) beschrieben, das Einbinden eigener Zertifikate.
* `APP_KAFKA_TOPIC`: Nutzen Sie nun die Konfigurationsoption `APP_KAFKA_OUTPUT_TOPIC`
* `APP_KAFKA_RESPONSE_TOPIC`: Nutzen Sie nun die Konfigurationsoption `APP_KAFKA_OUTPUT_RESPONSE_TOPIC`
Der Pfad zum Versenden von MTB-Daten ist nun offiziell `/mtb`.
In Versionen **nach Version 0.10** wird die Unterstützung des Pfads `/mtbfile` entfernt.
### Pseudonymisierung der Patienten-ID
Wenn eine URI zu einer gPAS-Instanz (Version >= 2023.1.0) angegeben ist, wird diese verwendet.
Ist diese nicht gesetzt. wird intern eine Anonymisierung der Patienten-ID vorgenommen.
* `APP_PSEUDONYMIZE_PREFIX`: Standortbezogenes Prefix - `UNKNOWN`, wenn nicht gesetzt
* `APP_PSEUDONYMIZE_PREFIX`: Standortbezogenes Präfix - `UNKNOWN`, wenn nicht gesetzt
* `APP_PSEUDONYMIZE_GENERATOR`: `BUILDIN` oder `GPAS` - `BUILDIN`, wenn nicht gesetzt
**Hinweise**:
* Der alte Konfigurationsparameter `APP_PSEUDONYMIZER` mit den Werten `GPAS` oder `BUILDIN` sollte nicht
mehr verwendet werden.
* Der alte Konfigurationsparameter `APP_PSEUDONYMIZER` mit den Werten `GPAS` oder `BUILDIN` sollte nicht mehr verwendet
werden.
* Die Pseudonymisierung erfolgt im ETL-Prozessor nur für die Patienten-ID.
Andere Referenz-IDs werden nicht anonymisiert.
Dies erfolgt bei Nutzung von **[onkostar-plugin-dnpmexport](https://github.com/CCC-MF/onkostar-plugin-dnpmexport)**
bereits im Plugin selbst.
Andere IDs werden mithilfe des standortbezogenen Präfixes (erneut) anonymisiert, um für den aktuellen Kontext nicht
vergleichbare IDs bereitzustellen.
#### Eingebaute Anonymisierung
Wurde keine oder die Verwendung der eingebauten Anonymisierung konfiguriert, so wird für die Patienten-ID der
entsprechende SHA-256-Hash gebildet und Base64-codiert - hier ohne endende "=" - zuzüglich des konfigurierten Prefixes
entsprechende SHA-256-Hash gebildet und Base64-codiert - hier ohne endende "=" - zuzüglich des konfigurierten Präfixes
als Patienten-Pseudonym verwendet.
#### Pseudonymisierung mit gPAS
@ -65,9 +91,12 @@ Wurde die Verwendung von gPAS konfiguriert, so sind weitere Angaben zu konfiguri
* `APP_PSEUDONYMIZE_GPAS_USERNAME`: gPas Basic-Auth Benutzername
* `APP_PSEUDONYMIZE_GPAS_PASSWORD`: gPas Basic-Auth Passwort
* ~~`APP_PSEUDONYMIZE_GPAS_SSLCALOCATION`~~: **Veraltet** - Root Zertifikat für gPas, falls es dediziert hinzugefügt werden muss.
**Wird in nach Version 0.10 entfernt**
Der Konfigurationsparameter `APP_PSEUDONYMIZE_GPAS_SSLCALOCATION` sollte nicht mehr verwendet werden und wird in einer kommenden Version entfernt.
Stattdessen sollte das Root Zertifikat wie unter [_Integration eines eigenen Root CA Zertifikats_](#integration-eines-eigenen-root-ca-zertifikats) beschrieben eingebunden werden.
Der Konfigurationsparameter `APP_PSEUDONYMIZE_GPAS_SSLCALOCATION` sollte nicht mehr verwendet werden und wird nach
Version 0.10 entfernt.
Stattdessen sollte das Root Zertifikat wie unter [_Integration eines eigenen Root CA
Zertifikats_](#integration-eines-eigenen-root-ca-zertifikats) beschrieben eingebunden werden.
### Anmeldung mit einem Passwort
@ -77,7 +106,7 @@ einem erfolgreichen Login erreichbar sind.
* `APP_SECURITY_ADMIN_USER`: Muss angegeben werden zur Aktivierung der Zugriffsbeschränkung.
* `APP_SECURITY_ADMIN_PASSWORD`: Das Passwort für den Administrator (Empfohlen).
Ein Administrator-Passwort muss inklusive des Encoding-Prefixes vorliegen.
Ein Administrator-Passwort muss inklusive des Encoding-Präfixes vorliegen.
Hier Beispiele für das Beispielpasswort `very-secret`:
@ -134,7 +163,7 @@ Sie bekommen dabei wieder die Standardrolle zugewiesen.
#### Auswirkungen auf den dargestellten Inhalt
Nur Administratoren haben Zugriff auf den Konfigurationsbereich, nur angemeldete Benutzer können die anonymisierte oder
pseudonymisierte Patienten-ID sowie den Qualitätsbericht des bwHC-Backends einsehen.
pseudonymisierte Patienten-ID sowie den Qualitätsbericht von DNPM:DIP einsehen.
Wurde kein Administrator-Account konfiguriert, sind diese Inhalte generell nicht verfügbar.
@ -150,7 +179,7 @@ zur Nutzung des MTB-File-Endpunkts eine HTTP-Basic-Authentifizierung voraussetze
![Tokenverwaltung](docs/tokens.png)
In diesem Fall können den Endpunkt für das Onkostar-Plugin **[onkostar-plugin-dnpmexport](https://github.com/CCC-MF/onkostar-plugin-dnpmexport)** wie folgt konfigurieren:
In diesem Fall kann der Endpunkt für das Onkostar-Plugin **[onkostar-plugin-dnpmexport](https://github.com/CCC-MF/onkostar-plugin-dnpmexport)** wie folgt konfiguriert werden:
```
https://testonkostar:MTg1NTL...NGU4@etl.example.com/mtbfile
@ -158,10 +187,12 @@ https://testonkostar:MTg1NTL...NGU4@etl.example.com/mtbfile
Ist die Verwendung von Tokens aktiv, werden Anfragen ohne die Angabe der Token-Information abgelehnt.
Alternativ kann eine Authentifizierung über Benutzername/Passwort oder OIDC erfolgen.
### Transformation von Werten
In Onkostar kann es vorkommen, dass ein Wert eines Merkmalskatalogs an einem Standort angepasst wurde und dadurch nicht dem Wert entspricht,
der vom bwHC-Backend akzeptiert wird.
der von DNPM:DIP akzeptiert wird.
Diese Anwendung bietet daher die Möglichkeit, eine Transformation vorzunehmen. Hierzu muss der "Pfad" innerhalb des JSON-MTB-Files angegeben werden und
welcher Wert wie ersetzt werden soll.
@ -181,18 +212,23 @@ Werden sowohl REST als auch Kafka-Endpunkt konfiguriert, wird nur der REST-Endpu
#### REST
Folgende Umgebungsvariablen müssen gesetzt sein, damit ein bwHC-MTB-File an das bwHC-Backend gesendet wird:
Folgende Umgebungsvariablen müssen gesetzt sein, damit ein bwHC-MTB-File an DNPM:DIP gesendet wird:
* `APP_REST_URI`: URI der zu benutzenden API der bwHC-Backend-Instanz. z.B.: `http://localhost:9000/bwhc/etl/api`
* `APP_REST_URI`: URI der zu benutzenden API der Backend-Instanz. Zum Beispiel:
* `http://localhost:9000/bwhc/etl/api` für **bwHC Backend**
* `http://localhost:9000/api` für **dnpm:dip**
* `APP_REST_USERNAME`: Basic-Auth-Benutzername für den REST-Endpunkt
* `APP_REST_PASSWORD`: Basic-Auth-Passwort für den REST-Endpunkt
* `APP_REST_IS_BWHC`: `true` für **bwHC Backend**, weglassen oder `false` für **dnpm:dip**
#### Kafka-Topics
Folgende Umgebungsvariablen müssen gesetzt sein, damit ein bwHC-MTB-File an ein Kafka-Topic übermittelt wird:
* `APP_KAFKA_OUTPUT_TOPIC`: Zu verwendendes Topic zum Versenden von Anfragen.
Ersetzt in einer kommenden Version `APP_KAFKA_TOPIC`.
Ersetzt ~~`APP_KAFKA_TOPIC`~~, **welches nach Version 0.10 entfernt wird**.
* `APP_KAFKA_OUTPUT_RESPONSE_TOPIC`: Topic mit Antworten über den Erfolg des Versendens. Standardwert: `APP_KAFKA_TOPIC` mit Anhang "_response".
Ersetzt in einer kommenden Version `APP_KAFKA_RESPONSE_TOPIC`.
Ersetzt ~~`APP_KAFKA_RESPONSE_TOPIC`~~, **welches nach Version 0.10 entfernt wird**.
* `APP_KAFKA_GROUP_ID`: Kafka GroupID des Consumers. Standardwert: `APP_KAFKA_TOPIC` mit Anhang "_group".
* `APP_KAFKA_SERVERS`: Zu verwendende Kafka-Bootstrap-Server als kommagetrennte Liste
@ -200,7 +236,7 @@ Wird keine Rückantwort über Apache Kafka empfangen und es gibt keine weitere M
Weitere Einstellungen können über die Parameter von Spring Kafka konfiguriert werden.
Lässt sich keine Verbindung zu dem bwHC-Backend aufbauen, wird eine Rückantwort mit Status-Code `900` erwartet, welchen es
Lässt sich keine Verbindung zu dem Backend aufbauen, wird eine Rückantwort mit Status-Code `900` erwartet, welchen es
für HTTP nicht gibt.
Wird die Umgebungsvariable `APP_KAFKA_INPUT_TOPIC` gesetzt, kann eine Nachricht auch über dieses Kafka-Topic an den ETL-Prozessor übermittelt werden.
@ -234,19 +270,42 @@ kafka-configs.sh --bootstrap-server localhost:9092 --alter --topic test --add-co
kafka-configs.sh --bootstrap-server localhost:9092 --alter --topic test --add-config cleanup.policy=[delete,compact]
```
Da als Key eines Records die (pseudonymisierte) Patienten-ID und die (anonymisierte) Erkrankungs-ID verwendet wird,
stehen mit obiger Konfiguration der Kafka-Topics nach 10 Sekunden nur noch der jeweils letzte Eintrag für den entsprechenden
Key zur Verfügung.
Da als Key eines Records die (pseudonymisierte) Patienten-ID verwendet wird, stehen mit obiger Konfiguration
der Kafka-Topics nach 10 Sekunden nur noch der jeweils letzte Eintrag für den entsprechenden Key zur Verfügung.
Da der Key sowohl für die Records in Richtung bwHC-Backend für die Rückantwort identisch aufgebaut ist, lassen sich so
Da der Key sowohl für die Records in Richtung DNPM:DIP, als auch für die Rückantwort identisch aufgebaut ist, lassen sich so
auch im Falle eines Consent-Widerspruchs die enthaltenen Daten als auch die Offenlegung durch Verifikationsdaten in der
Antwort effektiv verhindern, da diese nach 10 Sekunden gelöscht werden.
Es steht dann nur noch die jeweils letzten Information zur Verfügung, dass für einen Patienten/eine Erkrankung
ein Consent-Widerspruch erfolgte.
Dieses Vorgehen empfiehlt sich, wenn Sie gespeicherte Records nachgelagert für andere Auswertungen verwenden möchten.
### Antworten und Statusauswertung
Anfragen an das bwHC-Backend aus Versionen bis 0.9.x wurden wie folgt behandelt:
| HTTP-Response | Status |
|----------------|-----------|
| `HTTP 200` | `SUCCESS` |
| `HTTP 201` | `WARNING` |
| `HTTP 400-...` | `ERROR` |
Dies konnte dazu führen, dass zwar mit einem `HTTP 201` geantwortet wurde, aber dennoch in der Issue-Liste die
Severity `error` aufgetaucht ist.
Ab Version 0.10 wird die Issue-Liste der Antwort verwendet und die darion enthaltene höchste Severity-Stufe als Ergebnis verwendet.
| Höchste Severity | Status |
|------------------|-----------|
| `info` | `SUCCESS` |
| `warning` | `WARNING` |
| `error`, `fatal` | `ERROR` |
## Docker-Images
Diese Anwendung ist auch als Docker-Image verfügbar: https://github.com/CCC-MF/etl-processor/pkgs/container/etl-processor
Diese Anwendung ist auch als Docker-Image verfügbar: https://github.com/pcvolkmer/etl-processor/pkgs/container/etl-processor
### Images lokal bauen
@ -305,7 +364,7 @@ auf Docker-Compose mit der gestartet werden kann.
Die Anwendung verarbeitet `X-Forwarded`-HTTP-Header und kann daher auch hinter einem Reverse-Proxy betrieben werden.
Dabei werden, je nachdem welche Header durch den Reverse-Proxy gesendet werden auch Protokoll, Host oder auch Path-Prefix
Dabei werden, je nachdem welche Header durch den Reverse-Proxy gesendet werden auch Protokoll, Host oder auch Path-Präfix
automatisch erkannt und verwendet werden. Dadurch ist z.B. eine abweichende Angabe des Pfads problemlos möglich.
#### Beispiel *Traefik* (mit Docker-Labels):
@ -359,3 +418,5 @@ Die Datei `application-dev.yml` enthält hierzu die Konfiguration für das Profi
Beim Ausführen der Integrationstests wird eine Testdatenbank in einem Docker-Container gestartet.
Siehe hier auch die Klasse `AbstractTestcontainerTest` unter `src/integrationTest`.
Ein einfaches Entwickler-Setup inklusive DNPM:DIP ist mit Hilfe von https://github.com/pcvolkmer/dnpmdip-devenv realisierbar.

View File

@ -1,30 +1,36 @@
import org.gradle.api.tasks.testing.logging.TestLogEvent
import org.jetbrains.kotlin.gradle.dsl.JvmTarget
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
import org.springframework.boot.gradle.tasks.bundling.BootBuildImage
plugins {
war
id("org.springframework.boot") version "3.2.3"
id("io.spring.dependency-management") version "1.1.4"
kotlin("jvm") version "1.9.22"
kotlin("plugin.spring") version "1.9.22"
id("org.springframework.boot") version "3.3.10"
id("io.spring.dependency-management") version "1.1.7"
kotlin("jvm") version "1.9.25"
kotlin("plugin.spring") version "1.9.25"
jacoco
}
group = "de.ukw.ccc"
version = "0.8.0"
group = "dev.dnpm"
version = "0.10.0"
var versions = mapOf(
"bwhc-dto-java" to "0.2.0",
"hapi-fhir" to "6.10.2",
"httpclient5" to "5.2.1",
"mockito-kotlin" to "5.2.1",
"bwhc-dto-java" to "0.4.0",
"hapi-fhir" to "7.6.0",
"commons-compress" to "1.26.2",
"mockito-kotlin" to "5.4.0",
"archunit" to "1.3.0",
// Webjars
"webjars-locator" to "0.52",
"echarts" to "5.4.3",
"htmx.org" to "1.9.10"
"htmx.org" to "1.9.12"
)
java {
sourceCompatibility = JavaVersion.VERSION_21
toolchain {
languageVersion = JavaLanguageVersion.of(21)
}
}
sourceSets {
@ -62,35 +68,47 @@ dependencies {
implementation("org.thymeleaf.extras:thymeleaf-extras-springsecurity6")
implementation("com.fasterxml.jackson.module:jackson-module-kotlin")
implementation("org.springframework.kafka:spring-kafka")
implementation("org.flywaydb:flyway-database-postgresql")
implementation("org.flywaydb:flyway-mysql")
implementation("commons-codec:commons-codec")
implementation("io.projectreactor.kotlin:reactor-kotlin-extensions")
implementation("de.ukw.ccc:bwhc-dto-java:${versions["bwhc-dto-java"]}")
implementation("ca.uhn.hapi.fhir:hapi-fhir-base:${versions["hapi-fhir"]}")
implementation("ca.uhn.hapi.fhir:hapi-fhir-structures-r4:${versions["hapi-fhir"]}")
implementation("org.apache.httpcomponents.client5:httpclient5:${versions["httpclient5"]}")
implementation("org.apache.httpcomponents.client5:httpclient5")
implementation("com.jayway.jsonpath:json-path")
implementation("org.webjars:webjars-locator:0.50")
implementation("org.webjars:webjars-locator:${versions["webjars-locator"]}")
implementation("org.webjars.npm:echarts:${versions["echarts"]}")
implementation("org.webjars.npm:htmx.org:${versions["htmx.org"]}")
runtimeOnly("org.mariadb.jdbc:mariadb-java-client")
runtimeOnly("org.postgresql:postgresql")
developmentOnly("org.springframework.boot:spring-boot-devtools")
developmentOnly("org.springframework.boot:spring-boot-docker-compose")
annotationProcessor("org.springframework.boot:spring-boot-configuration-processor")
providedRuntime("org.springframework.boot:spring-boot-starter-tomcat")
testImplementation("org.springframework.boot:spring-boot-starter-test")
testImplementation("org.springframework.security:spring-security-test")
testImplementation("io.projectreactor:reactor-test")
testImplementation("org.mockito.kotlin:mockito-kotlin:${versions["mockito-kotlin"]}")
integrationTestImplementation("org.testcontainers:junit-jupiter")
integrationTestImplementation("org.testcontainers:postgresql")
integrationTestImplementation("com.tngtech.archunit:archunit:${versions["archunit"]}")
integrationTestImplementation("net.sourceforge.htmlunit:htmlunit")
integrationTestImplementation("org.springframework:spring-webflux")
// Override dependency version from org.testcontainers:junit-jupiter - CVE-2024-26308, CVE-2024-25710
integrationTestImplementation("org.apache.commons:commons-compress:${versions["commons-compress"]}")
}
tasks.withType<KotlinCompile> {
kotlinOptions {
freeCompilerArgs += "-Xjsr305=strict"
jvmTarget = "21"
compilerOptions {
freeCompilerArgs.add("-Xjsr305=strict")
jvmTarget.set(JvmTarget.JVM_21)
}
}
@ -110,8 +128,24 @@ task<Test>("integrationTest") {
shouldRunAfter("test")
}
tasks.register("allTests") {
description = "Run all tests"
group = JavaBasePlugin.VERIFICATION_GROUP
dependsOn(tasks.withType<Test>())
}
tasks.jacocoTestReport {
dependsOn("allTests")
executionData(fileTree(project.rootDir.absolutePath).include("**/build/jacoco/*.exec"))
reports {
xml.required = true
}
}
tasks.named<BootBuildImage>("bootBuildImage") {
imageName.set("ghcr.io/ccc-mf/etl-processor")
imageName.set("ghcr.io/pcvolkmer/etl-processor")
// Binding for CA Certs
bindings.set(listOf(
@ -121,7 +155,7 @@ tasks.named<BootBuildImage>("bootBuildImage") {
environment.set(environment.get() + mapOf(
// Enable this line to embed CA Certs into image on build time
//"BP_EMBED_CERTS" to "true",
"BP_OCI_SOURCE" to "https://github.com/CCC-MF/etl-processor",
"BP_OCI_SOURCE" to "https://github.com/pcvolkmer/etl-processor",
"BP_OCI_LICENSES" to "AGPLv3",
"BP_OCI_DESCRIPTION" to "ETL Processor for bwHC MTB files"
))

View File

@ -18,6 +18,9 @@ services:
APP_KAFKA_GROUP_ID: ${DNPM_KAFKA_GROUP_ID}
APP_KAFKA_RESPONSE_TOPIC: ${DNPM_KAFKA_RESPONSE_TOPIC}
APP_REST_URI: ${DNPM_BWHC_REST_URI}
APP_REST_USERNAME: ${DNPM_BWHC_REST_USERNAME}
APP_REST_PASSWORD: ${DNPM_BWHC_REST_PASSWORD}
APP_REST_IS_BWHC: ${DNPM_BWHC_REST_IS_BWHC}
APP_SECURITY_ADMIN_USER: ${DNPM_ADMIN_USER}
APP_SECURITY_ADMIN_PASSWORD: ${DNPM_ADMIN_PASSWORD}
SPRING_DATASOURCE_URL: ${DNPM_DATASOURCE_URL}

View File

@ -28,6 +28,9 @@ DNPM_DATASOURCE_URL=jdbc:mariadb://dnpm-monitor-db:3306/$DNPM_MARIADB_DB
## TARGET SYSTEMS CONFIG
# in case of direct access to bwhc enter endpoint url here
DNPM_BWHC_REST_URI=
DNPM_BWHC_REST_USERNAME=
DNPM_BWHC_REST_PASSWORD=
DNPM_BWHC_REST_IS_BWHC=false
# produce mtb files to this topic - values 'false' disabling kafka processing
DNPM_KAFKA_TOPIC=false

View File

@ -17,8 +17,9 @@ services:
KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 0@kafka:9093
KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
## Use AKHQ as Kafka web frontend
akhq:
image: tchiotludo/akhq:0.21.0
image: tchiotludo/akhq:0.25.0
environment:
AKHQ_CONFIGURATION: |
akhq:
@ -32,6 +33,8 @@ services:
ports:
- "8084:8080"
## For use with MariaDB
mariadb:
image: mariadb:10
ports:
@ -42,6 +45,7 @@ services:
MARIADB_PASSWORD: dev
MARIADB_ROOT_PASSWORD: dev
## For use with Postgres
# postgres:
# image: postgres:alpine
# ports:

Binary file not shown.

Before

Width:  |  Height:  |  Size: 75 KiB

After

Width:  |  Height:  |  Size: 115 KiB

View File

@ -1,6 +1,6 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-8.6-bin.zip
distributionUrl=https\://services.gradle.org/distributions/gradle-8.13-bin.zip
networkTimeout=10000
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists

View File

@ -0,0 +1,73 @@
package dev.dnpm.etl.processor
import com.tngtech.archunit.core.domain.JavaClasses
import com.tngtech.archunit.core.importer.ClassFileImporter
import com.tngtech.archunit.lang.syntax.ArchRuleDefinition.classes
import com.tngtech.archunit.lang.syntax.ArchRuleDefinition.noClasses
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.springframework.data.repository.Repository
class EtlProcessorArchTest {
private lateinit var noTestClasses: JavaClasses
@BeforeEach
fun setUp() {
this.noTestClasses = ClassFileImporter()
.withImportOption { !(it.contains("/test/") || it.contains("/integrationTest/")) }
.importPackages("dev.dnpm.etl.processor")
}
@Test
fun noClassesInInputPackageShouldDependOnMonitoringPackage() {
val rule = noClasses()
.that()
.resideInAPackage("..input")
.should().dependOnClassesThat()
.resideInAnyPackage("..monitoring")
rule.check(noTestClasses)
}
@Test
fun noClassesInInputPackageShouldDependOnRepositories() {
val rule = noClasses()
.that()
.resideInAPackage("..input")
.should().dependOnClassesThat().haveSimpleNameEndingWith("Repository")
rule.check(noTestClasses)
}
@Test
fun noClassesInOutputPackageShouldDependOnRepositories() {
val rule = noClasses()
.that()
.resideInAPackage("..output")
.should().dependOnClassesThat().haveSimpleNameEndingWith("Repository")
rule.check(noTestClasses)
}
@Test
fun noClassesInWebPackageShouldDependOnRepositories() {
val rule = noClasses()
.that()
.resideInAPackage("..web")
.should().dependOnClassesThat().haveSimpleNameEndingWith("Repository")
rule.check(noTestClasses)
}
@Test
fun repositoryClassNamesShouldEndWithRepository() {
val rule = classes()
.that()
.areInterfaces().and().areAssignableTo(Repository::class.java)
.should().haveSimpleNameEndingWith("Repository")
rule.check(noTestClasses)
}
}

View File

@ -27,8 +27,8 @@ import dev.dnpm.etl.processor.output.RestMtbFileSender
import dev.dnpm.etl.processor.pseudonym.AnonymizingGenerator
import dev.dnpm.etl.processor.pseudonym.GpasPseudonymGenerator
import dev.dnpm.etl.processor.services.RequestProcessor
import dev.dnpm.etl.processor.services.TokenRepository
import dev.dnpm.etl.processor.services.TokenService
import dev.dnpm.etl.processor.security.TokenRepository
import dev.dnpm.etl.processor.security.TokenService
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.Nested
import org.junit.jupiter.api.Test

View File

@ -0,0 +1,30 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor
import org.mockito.ArgumentMatchers
@Suppress("UNCHECKED_CAST")
inline fun <reified T> anyValueClass(): T {
val unboxedClass = T::class.java.declaredFields.first().type
return ArgumentMatchers.any(unboxedClass as Class<T>)
?: T::class.java.getDeclaredMethod("box-impl", unboxedClass)
.invoke(null, null) as T
}

View File

@ -21,13 +21,15 @@ package dev.dnpm.etl.processor.input
import com.fasterxml.jackson.databind.ObjectMapper
import de.ukw.ccc.bwhc.dto.*
import dev.dnpm.etl.processor.anyValueClass
import dev.dnpm.etl.processor.config.AppSecurityConfiguration
import dev.dnpm.etl.processor.security.TokenRepository
import dev.dnpm.etl.processor.security.UserRoleRepository
import dev.dnpm.etl.processor.services.RequestProcessor
import dev.dnpm.etl.processor.services.TokenRepository
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Nested
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.ArgumentMatchers.anyString
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.any
import org.mockito.kotlin.never
@ -37,6 +39,7 @@ import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest
import org.springframework.boot.test.mock.mockito.MockBean
import org.springframework.http.MediaType
import org.springframework.security.oauth2.client.registration.ClientRegistrationRepository
import org.springframework.security.test.web.servlet.request.SecurityMockMvcRequestPostProcessors.anonymous
import org.springframework.security.test.web.servlet.request.SecurityMockMvcRequestPostProcessors.user
import org.springframework.test.context.ContextConfiguration
@ -91,6 +94,19 @@ class MtbFileRestControllerTest {
verify(requestProcessor, times(1)).processMtbFile(any())
}
@Test
fun testShouldGrantPermissionToSendMtbFileToAdminUser() {
mockMvc.post("/mtbfile") {
with(user("onkostarserver").roles("ADMIN"))
contentType = MediaType.APPLICATION_JSON
content = ObjectMapper().writeValueAsString(mtbFile)
}.andExpect {
status { isAccepted() }
}
verify(requestProcessor, times(1)).processMtbFile(any())
}
@Test
fun testShouldDenyPermissionToSendMtbFile() {
mockMvc.post("/mtbfile") {
@ -104,6 +120,19 @@ class MtbFileRestControllerTest {
verify(requestProcessor, never()).processMtbFile(any())
}
@Test
fun testShouldDenyPermissionToSendMtbFileForUser() {
mockMvc.post("/mtbfile") {
with(user("fakeuser").roles("USER"))
contentType = MediaType.APPLICATION_JSON
content = ObjectMapper().writeValueAsString(mtbFile)
}.andExpect {
status { isForbidden() }
}
verify(requestProcessor, never()).processMtbFile(any())
}
@Test
fun testShouldGrantPermissionToDeletePatientData() {
mockMvc.delete("/mtbfile/12345678") {
@ -112,7 +141,7 @@ class MtbFileRestControllerTest {
status { isAccepted() }
}
verify(requestProcessor, times(1)).processDeletion(anyString())
verify(requestProcessor, times(1)).processDeletion(anyValueClass())
}
@Test
@ -123,7 +152,46 @@ class MtbFileRestControllerTest {
status { isUnauthorized() }
}
verify(requestProcessor, never()).processDeletion(anyString())
verify(requestProcessor, never()).processDeletion(anyValueClass())
}
@Nested
@MockBean(UserRoleRepository::class, ClientRegistrationRepository::class)
@TestPropertySource(
properties = [
"app.pseudonymize.generator=BUILDIN",
"app.security.admin-user=admin",
"app.security.admin-password={noop}very-secret",
"app.security.enable-tokens=true",
"app.security.enable-oidc=true"
]
)
inner class WithOidcEnabled {
@Test
fun testShouldGrantPermissionToSendMtbFileToAdminUser() {
mockMvc.post("/mtbfile") {
with(user("onkostarserver").roles("ADMIN"))
contentType = MediaType.APPLICATION_JSON
content = ObjectMapper().writeValueAsString(mtbFile)
}.andExpect {
status { isAccepted() }
}
verify(requestProcessor, times(1)).processMtbFile(any())
}
@Test
fun testShouldGrantPermissionToSendMtbFileToUser() {
mockMvc.post("/mtbfile") {
with(user("onkostarserver").roles("USER"))
contentType = MediaType.APPLICATION_JSON
content = ObjectMapper().writeValueAsString(mtbFile)
}.andExpect {
status { isAccepted() }
}
verify(requestProcessor, times(1)).processMtbFile(any())
}
}
companion object {

View File

@ -0,0 +1,75 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.monitoring
import dev.dnpm.etl.processor.*
import dev.dnpm.etl.processor.output.MtbFileSender
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.autoconfigure.data.jdbc.DataJdbcTest
import org.springframework.boot.test.autoconfigure.jdbc.AutoConfigureTestDatabase
import org.springframework.boot.test.mock.mockito.MockBean
import org.springframework.test.context.TestPropertySource
import org.springframework.test.context.junit.jupiter.SpringExtension
import org.springframework.transaction.annotation.Transactional
import org.testcontainers.junit.jupiter.Testcontainers
import java.time.Instant
@Testcontainers
@ExtendWith(SpringExtension::class)
@DataJdbcTest
@AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE)
@Transactional
@MockBean(MtbFileSender::class)
@TestPropertySource(
properties = [
"app.pseudonymize.generator=buildin",
"app.rest.uri=http://example.com"
]
)
class RequestRepositoryTest : AbstractTestcontainerTest() {
private lateinit var requestRepository: RequestRepository
@BeforeEach
fun setUp(
@Autowired requestRepository: RequestRepository
) {
this.requestRepository = requestRepository
}
@Test
fun shouldSaveRequest() {
val request = Request(
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.MTB_FILE,
RequestStatus.WARNING,
Instant.parse("2023-07-07T00:00:00Z")
)
requestRepository.save(request)
}
}

View File

@ -0,0 +1,136 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.pseudonym
import dev.dnpm.etl.processor.config.GPasConfigProperties
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.assertThrows
import org.springframework.http.HttpHeaders
import org.springframework.http.HttpMethod
import org.springframework.http.HttpStatus
import org.springframework.retry.policy.SimpleRetryPolicy
import org.springframework.retry.support.RetryTemplateBuilder
import org.springframework.test.web.client.MockRestServiceServer
import org.springframework.test.web.client.match.MockRestRequestMatchers.method
import org.springframework.test.web.client.match.MockRestRequestMatchers.requestTo
import org.springframework.test.web.client.response.MockRestResponseCreators.withException
import org.springframework.test.web.client.response.MockRestResponseCreators.withStatus
import org.springframework.web.client.RestTemplate
import java.io.IOException
class GpasPseudonymGeneratorTest {
private lateinit var mockRestServiceServer: MockRestServiceServer
private lateinit var generator: GpasPseudonymGenerator
private lateinit var restTemplate: RestTemplate
@BeforeEach
fun setup() {
val retryTemplate = RetryTemplateBuilder().customPolicy(SimpleRetryPolicy(1)).build()
val gPasConfigProperties = GPasConfigProperties(
"http://localhost/ttp-fhir/fhir/gpas/\$pseudonymizeAllowCreate",
"test",
null,
null,
null
)
this.restTemplate = RestTemplate()
this.mockRestServiceServer = MockRestServiceServer.createServer(restTemplate)
this.generator = GpasPseudonymGenerator(gPasConfigProperties, retryTemplate, restTemplate)
}
@Test
fun shouldReturnExpectedPseudonym() {
this.mockRestServiceServer.expect {
method(HttpMethod.POST)
requestTo("http://localhost/ttp-fhir/fhir/gpas/\$pseudonymizeAllowCreate")
}.andRespond {
withStatus(HttpStatus.OK).body(getDummyResponseBody("1234", "test", "test1234ABCDEF567890"))
.createResponse(it)
}
assertThat(this.generator.generate("ID1234")).isEqualTo("test1234ABCDEF567890")
}
@Test
fun shouldThrowExceptionIfGpasNotAvailable() {
this.mockRestServiceServer.expect {
method(HttpMethod.POST)
requestTo("http://localhost/ttp-fhir/fhir/gpas/\$pseudonymizeAllowCreate")
}.andRespond {
withException(IOException("Simulated IO error")).createResponse(it)
}
assertThrows<PseudonymRequestFailed> { this.generator.generate("ID1234") }
}
@Test
fun shouldThrowExceptionIfGpasDoesNotReturn2xxResponse() {
this.mockRestServiceServer.expect {
method(HttpMethod.POST)
requestTo("http://localhost/ttp-fhir/fhir/gpas/\$pseudonymizeAllowCreate")
}.andRespond {
withStatus(HttpStatus.FOUND)
.header(HttpHeaders.LOCATION, "https://localhost/ttp-fhir/fhir/gpas/\$pseudonymizeAllowCreate")
.createResponse(it)
}
assertThrows<PseudonymRequestFailed> { this.generator.generate("ID1234") }
}
companion object {
fun getDummyResponseBody(original: String, target: String, pseudonym: String) = """{
"resourceType": "Parameters",
"parameter": [
{
"name": "pseudonym",
"part": [
{
"name": "original",
"valueIdentifier": {
"system": "https://ths-greifswald.de/gpas",
"value": "$original"
}
},
{
"name": "target",
"valueIdentifier": {
"system": "https://ths-greifswald.de/gpas",
"value": "$target"
}
},
{
"name": "pseudonym",
"valueIdentifier": {
"system": "https://ths-greifswald.de/gpas",
"value": "$pseudonym"
}
}
]
}
]
}""".trimIndent()
}
}

View File

@ -19,7 +19,7 @@
package dev.dnpm.etl.processor.services
import dev.dnpm.etl.processor.AbstractTestcontainerTest
import dev.dnpm.etl.processor.*
import dev.dnpm.etl.processor.monitoring.Request
import dev.dnpm.etl.processor.monitoring.RequestRepository
import dev.dnpm.etl.processor.monitoring.RequestStatus
@ -37,7 +37,6 @@ import org.springframework.test.context.junit.jupiter.SpringExtension
import org.springframework.transaction.annotation.Transactional
import org.testcontainers.junit.jupiter.Testcontainers
import java.time.Instant
import java.util.*
@Testcontainers
@ExtendWith(SpringExtension::class)
@ -66,7 +65,7 @@ class RequestServiceIntegrationTest : AbstractTestcontainerTest() {
@Test
fun shouldResultInEmptyRequestList() {
val actual = requestService.allRequestsByPatientPseudonym("TEST_12345678901")
val actual = requestService.allRequestsByPatientPseudonym(TEST_PATIENT_PSEUDONYM)
assertThat(actual).isEmpty()
}
@ -76,33 +75,33 @@ class RequestServiceIntegrationTest : AbstractTestcontainerTest() {
this.requestRepository.saveAll(
listOf(
Request(
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "0123456789abcdef1",
type = RequestType.MTB_FILE,
status = RequestStatus.SUCCESS,
processedAt = Instant.parse("2023-07-07T02:00:00Z")
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS,
Instant.parse("2023-07-07T02:00:00Z")
),
// Should be ignored - wrong patient ID -->
Request(
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678902",
pid = "P2",
fingerprint = "0123456789abcdef2",
type = RequestType.MTB_FILE,
status = RequestStatus.WARNING,
processedAt = Instant.parse("2023-08-08T00:00:00Z")
randomRequestId(),
PatientPseudonym("TEST_12345678902"),
PatientId("P2"),
Fingerprint("0123456789abcdef2"),
RequestType.MTB_FILE,
RequestStatus.WARNING,
Instant.parse("2023-08-08T00:00:00Z")
),
// <--
Request(
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P2",
fingerprint = "0123456789abcdee1",
type = RequestType.DELETE,
status = RequestStatus.SUCCESS,
processedAt = Instant.parse("2023-08-08T02:00:00Z")
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P2"),
Fingerprint("0123456789abcdee1"),
RequestType.DELETE,
RequestStatus.SUCCESS,
Instant.parse("2023-08-08T02:00:00Z")
)
)
)
@ -112,18 +111,18 @@ class RequestServiceIntegrationTest : AbstractTestcontainerTest() {
fun shouldResultInSortedRequestList() {
setupTestData()
val actual = requestService.allRequestsByPatientPseudonym("TEST_12345678901")
val actual = requestService.allRequestsByPatientPseudonym(TEST_PATIENT_PSEUDONYM)
assertThat(actual).hasSize(2)
assertThat(actual[0].fingerprint).isEqualTo("0123456789abcdee1")
assertThat(actual[1].fingerprint).isEqualTo("0123456789abcdef1")
assertThat(actual[0].fingerprint).isEqualTo(Fingerprint("0123456789abcdee1"))
assertThat(actual[1].fingerprint).isEqualTo(Fingerprint("0123456789abcdef1"))
}
@Test
fun shouldReturnDeleteRequestAsLastRequest() {
setupTestData()
val actual = requestService.isLastRequestWithKnownStatusDeletion("TEST_12345678901")
val actual = requestService.isLastRequestWithKnownStatusDeletion(TEST_PATIENT_PSEUDONYM)
assertThat(actual).isTrue()
}
@ -132,10 +131,14 @@ class RequestServiceIntegrationTest : AbstractTestcontainerTest() {
fun shouldReturnLastMtbFileRequest() {
setupTestData()
val actual = requestService.lastMtbFileRequestForPatientPseudonym("TEST_12345678901")
val actual = requestService.lastMtbFileRequestForPatientPseudonym(TEST_PATIENT_PSEUDONYM)
assertThat(actual).isNotNull
assertThat(actual?.fingerprint).isEqualTo("0123456789abcdef1")
assertThat(actual?.fingerprint).isEqualTo(Fingerprint("0123456789abcdef1"))
}
companion object {
val TEST_PATIENT_PSEUDONYM = PatientPseudonym("TEST_12345678901")
}
}

View File

@ -19,29 +19,51 @@
package dev.dnpm.etl.processor.web
import com.gargoylesoftware.htmlunit.WebClient
import com.gargoylesoftware.htmlunit.html.HtmlPage
import dev.dnpm.etl.processor.config.AppConfiguration
import dev.dnpm.etl.processor.config.AppSecurityConfiguration
import dev.dnpm.etl.processor.monitoring.ConnectionCheckService
import dev.dnpm.etl.processor.monitoring.ConnectionCheckResult
import dev.dnpm.etl.processor.monitoring.GPasConnectionCheckService
import dev.dnpm.etl.processor.monitoring.RestConnectionCheckService
import dev.dnpm.etl.processor.output.MtbFileSender
import dev.dnpm.etl.processor.pseudonym.Generator
import dev.dnpm.etl.processor.security.Role
import dev.dnpm.etl.processor.services.RequestProcessor
import dev.dnpm.etl.processor.security.TokenService
import dev.dnpm.etl.processor.services.TransformationService
import dev.dnpm.etl.processor.security.UserRoleService
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Nested
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.ArgumentMatchers.anyString
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.argumentCaptor
import org.mockito.kotlin.times
import org.mockito.kotlin.verify
import org.mockito.kotlin.whenever
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest
import org.springframework.boot.test.mock.mockito.MockBean
import org.springframework.http.HttpHeaders
import org.springframework.http.MediaType
import org.springframework.http.MediaType.TEXT_EVENT_STREAM
import org.springframework.security.test.context.support.WithMockUser
import org.springframework.security.test.web.servlet.request.SecurityMockMvcRequestPostProcessors.anonymous
import org.springframework.security.test.web.servlet.request.SecurityMockMvcRequestPostProcessors.user
import org.springframework.test.context.ContextConfiguration
import org.springframework.test.context.TestPropertySource
import org.springframework.test.context.junit.jupiter.SpringExtension
import org.springframework.test.web.servlet.MockMvc
import org.springframework.test.web.servlet.get
import org.springframework.test.web.reactive.server.WebTestClient
import org.springframework.test.web.servlet.*
import org.springframework.test.web.servlet.client.MockMvcWebTestClient
import org.springframework.test.web.servlet.htmlunit.MockMvcWebClientBuilder
import org.springframework.web.context.WebApplicationContext
import reactor.core.publisher.Sinks
import reactor.test.StepVerifier
import java.time.Instant
abstract class MockSink : Sinks.Many<Boolean>
@ -50,47 +72,54 @@ abstract class MockSink : Sinks.Many<Boolean>
@ContextConfiguration(
classes = [
ConfigController::class,
AppConfiguration::class,
AppSecurityConfiguration::class
]
)
@TestPropertySource(
properties = [
"app.pseudonymize.generator=BUILDIN",
"app.security.admin-user=admin",
"app.security.admin-password={noop}very-secret",
"app.security.enable-tokens=true"
"app.pseudonymize.generator=BUILDIN"
]
)
@MockBean(name = "configsUpdateProducer", classes = [MockSink::class])
@MockBean(
Generator::class,
MtbFileSender::class,
ConnectionCheckService::class,
RequestProcessor::class,
TransformationService::class
TransformationService::class,
GPasConnectionCheckService::class,
RestConnectionCheckService::class,
)
class ConfigControllerTest {
private lateinit var mockMvc: MockMvc
private lateinit var webClient: WebClient
private lateinit var requestProcessor: RequestProcessor
private lateinit var connectionCheckUpdateProducer: Sinks.Many<ConnectionCheckResult>
@BeforeEach
fun setup(
@Autowired mockMvc: MockMvc,
@Autowired requestProcessor: RequestProcessor
@Autowired requestProcessor: RequestProcessor,
@Autowired connectionCheckUpdateProducer: Sinks.Many<ConnectionCheckResult>
) {
this.mockMvc = mockMvc
this.webClient = MockMvcWebClientBuilder.mockMvcSetup(mockMvc).build()
this.requestProcessor = requestProcessor
this.connectionCheckUpdateProducer = connectionCheckUpdateProducer
webClient.options.isThrowExceptionOnScriptError = false
}
@Test
fun testShouldShowConfigPageIfLoggedIn() {
fun testShouldRequestConfigPageIfLoggedIn() {
mockMvc.get("/configs") {
with(user("admin").roles("ADMIN"))
accept(MediaType.TEXT_HTML)
}.andExpect {
status { isOk() }
view { name("configs") }
}
}
@ -107,4 +136,228 @@ class ConfigControllerTest {
}
}
@Nested
@TestPropertySource(
properties = [
"app.security.enable-tokens=true",
"app.security.admin-user=admin"
]
)
@MockBean(
TokenService::class
)
inner class WithTokensEnabled {
private lateinit var tokenService: TokenService
@BeforeEach
fun setup(
@Autowired tokenService: TokenService
) {
webClient.options.isThrowExceptionOnScriptError = false
this.tokenService = tokenService
}
@Test
fun testShouldSaveNewToken() {
mockMvc.post("/configs/tokens") {
with(user("admin").roles("ADMIN"))
accept(MediaType.TEXT_HTML)
contentType = MediaType.APPLICATION_FORM_URLENCODED
content = "name=Testtoken"
}.andExpect {
status { is2xxSuccessful() }
view { name("configs/tokens") }
}
val captor = argumentCaptor<String>()
verify(tokenService, times(1)).addToken(captor.capture())
assertThat(captor.firstValue).isEqualTo("Testtoken")
}
@Test
fun testShouldNotSaveTokenWithExstingName() {
whenever(tokenService.addToken(anyString())).thenReturn(Result.failure(RuntimeException("Testfailure")))
mockMvc.post("/configs/tokens") {
with(user("admin").roles("ADMIN"))
accept(MediaType.TEXT_HTML)
contentType = MediaType.APPLICATION_FORM_URLENCODED
content = "name=Testtoken"
}.andExpect {
status { is2xxSuccessful() }
view { name("configs/tokens") }
}
val captor = argumentCaptor<String>()
verify(tokenService, times(1)).addToken(captor.capture())
assertThat(captor.firstValue).isEqualTo("Testtoken")
}
@Test
fun testShouldDeleteToken() {
mockMvc.delete("/configs/tokens/42") {
with(user("admin").roles("ADMIN"))
accept(MediaType.TEXT_HTML)
}.andExpect {
status { is2xxSuccessful() }
view { name("configs/tokens") }
}
val captor = argumentCaptor<Long>()
verify(tokenService, times(1)).deleteToken(captor.capture())
assertThat(captor.firstValue).isEqualTo(42)
}
@Test
@WithMockUser(username = "admin", roles = ["ADMIN"])
fun testShouldRenderConfigPageWithTokens() {
val page = webClient.getPage<HtmlPage>("http://localhost/configs")
assertThat(
page.getElementById("tokens")
).isNotNull
}
}
@Nested
@TestPropertySource(
properties = [
"app.security.enable-tokens=false"
]
)
inner class WithTokensDisabled {
@BeforeEach
fun setup() {
webClient.options.isThrowExceptionOnScriptError = false
}
@Test
@WithMockUser(username = "admin", roles = ["ADMIN"])
fun testShouldRenderConfigPageWithoutTokens() {
val page = webClient.getPage<HtmlPage>("http://localhost/configs")
assertThat(
page.getElementById("tokens")
).isNull()
}
}
@Nested
@TestPropertySource(
properties = [
"app.security.enable-tokens=false",
"app.security.admin-user=admin",
"app.security.admin-password={noop}very-secret"
]
)
@MockBean(
UserRoleService::class
)
inner class WithUserRolesEnabled {
private lateinit var userRoleService: UserRoleService
@BeforeEach
fun setup(
@Autowired userRoleService: UserRoleService
) {
webClient.options.isThrowExceptionOnScriptError = false
this.userRoleService = userRoleService
}
@Test
fun testShouldDeleteUserRole() {
mockMvc.delete("/configs/userroles/42") {
with(user("admin").roles("ADMIN"))
accept(MediaType.TEXT_HTML)
}.andExpect {
status { is2xxSuccessful() }
view { name("configs/userroles") }
}
val captor = argumentCaptor<Long>()
verify(userRoleService, times(1)).deleteUserRole(captor.capture())
assertThat(captor.firstValue).isEqualTo(42)
}
@Test
fun testShouldUpdateUserRole() {
mockMvc.put("/configs/userroles/42") {
with(user("admin").roles("ADMIN"))
accept(MediaType.TEXT_HTML)
contentType = MediaType.APPLICATION_FORM_URLENCODED
content = "role=ADMIN"
}.andExpect {
status { is2xxSuccessful() }
view { name("configs/userroles") }
}
val idCaptor = argumentCaptor<Long>()
val roleCaptor = argumentCaptor<Role>()
verify(userRoleService, times(1)).updateUserRole(idCaptor.capture(), roleCaptor.capture())
assertThat(idCaptor.firstValue).isEqualTo(42)
assertThat(roleCaptor.firstValue).isEqualTo(Role.ADMIN)
}
@Test
@WithMockUser(username = "admin", roles = ["ADMIN"])
fun testShouldRenderConfigPageWithUserRoles() {
val page = webClient.getPage<HtmlPage>("http://localhost/configs")
assertThat(
page.getElementById("userroles")
).isNotNull
}
}
@Nested
inner class WithUserRolesDisabled {
@BeforeEach
fun setup() {
webClient.options.isThrowExceptionOnScriptError = false
}
@Test
fun testShouldRenderConfigPageWithoutUserRoles() {
val page = webClient.getPage<HtmlPage>("http://localhost/configs")
assertThat(
page.getElementById("userroles")
).isNull()
}
}
@Nested
inner class SseTest {
private lateinit var webClient: WebTestClient
@BeforeEach
fun setup(
applicationContext: WebApplicationContext,
) {
this.webClient = MockMvcWebTestClient
.bindToApplicationContext(applicationContext).build()
}
@Test
fun testShouldRequestSSE() {
val expectedEvent = ConnectionCheckResult.GPasConnectionCheckResult(true, Instant.now(), Instant.now())
connectionCheckUpdateProducer.tryEmitNext(expectedEvent)
connectionCheckUpdateProducer.emitComplete { _, _ -> true }
val result = webClient.get().uri("http://localhost/configs/events").accept(TEXT_EVENT_STREAM).exchange()
.expectStatus().isOk()
.expectHeader().contentType(TEXT_EVENT_STREAM)
.returnResult(ConnectionCheckResult.GPasConnectionCheckResult::class.java)
StepVerifier.create(result.responseBody)
.expectNext(expectedEvent)
.expectComplete()
.verify()
}
}
}

View File

@ -0,0 +1,287 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.web
import com.gargoylesoftware.htmlunit.WebClient
import com.gargoylesoftware.htmlunit.html.HtmlPage
import dev.dnpm.etl.processor.*
import dev.dnpm.etl.processor.config.AppConfiguration
import dev.dnpm.etl.processor.config.AppSecurityConfiguration
import dev.dnpm.etl.processor.monitoring.Report
import dev.dnpm.etl.processor.monitoring.Request
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.monitoring.RequestType
import dev.dnpm.etl.processor.services.RequestService
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Nested
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.assertThrows
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.any
import org.mockito.kotlin.whenever
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest
import org.springframework.boot.test.mock.mockito.MockBean
import org.springframework.data.domain.Page
import org.springframework.data.domain.PageImpl
import org.springframework.data.domain.Pageable
import org.springframework.security.test.context.support.WithMockUser
import org.springframework.test.context.ContextConfiguration
import org.springframework.test.context.TestPropertySource
import org.springframework.test.context.junit.jupiter.SpringExtension
import org.springframework.test.web.servlet.MockMvc
import org.springframework.test.web.servlet.get
import org.springframework.test.web.servlet.htmlunit.MockMvcWebClientBuilder
import java.io.IOException
import java.time.Instant
import java.util.*
@WebMvcTest(controllers = [HomeController::class])
@ExtendWith(value = [MockitoExtension::class, SpringExtension::class])
@ContextConfiguration(
classes = [
HomeController::class,
AppConfiguration::class,
AppSecurityConfiguration::class
]
)
@TestPropertySource(
properties = [
"app.pseudonymize.generator=BUILDIN",
"app.security.admin-user=admin",
"app.security.admin-password={noop}very-secret"
]
)
@MockBean(
RequestService::class
)
class HomeControllerTest {
private lateinit var mockMvc: MockMvc
private lateinit var webClient: WebClient
@BeforeEach
fun setup(
@Autowired mockMvc: MockMvc,
@Autowired requestService: RequestService
) {
this.mockMvc = mockMvc
this.webClient = MockMvcWebClientBuilder.mockMvcSetup(mockMvc).build()
whenever(requestService.findAll(any<Pageable>())).thenReturn(Page.empty())
}
@Test
fun testShouldRequestHomePage() {
mockMvc.get("/").andExpect {
status { isOk() }
view { name("index") }
}
}
@Nested
inner class WithRequests {
private lateinit var requestService: RequestService
@BeforeEach
fun setup(
@Autowired requestService: RequestService
) {
this.requestService = requestService
}
@Test
fun testShouldShowHomePage() {
whenever(requestService.findAll(any<Pageable>())).thenReturn(
PageImpl(
listOf(
Request(
2L,
randomRequestId(),
PatientPseudonym("PSEUDO1"),
PatientId("PATIENT1"),
Fingerprint("ashdkasdh"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS
),
Request(
1L,
randomRequestId(),
PatientPseudonym("PSEUDO1"),
PatientId("PATIENT1"),
Fingerprint("asdasdasd"),
RequestType.MTB_FILE,
RequestStatus.ERROR
)
)
)
)
val page = webClient.getPage<HtmlPage>("http://localhost/")
assertThat(page.querySelectorAll("tbody tr")).hasSize(2)
assertThat(page.querySelectorAll("div.notification.info")).isEmpty()
}
@Test
@WithMockUser(username = "admin", roles = ["ADMIN"])
fun testShouldShowRequestDetails() {
val requestId = randomRequestId()
whenever(requestService.findByUuid(anyValueClass())).thenReturn(
Optional.of(
Request(
2L,
requestId,
PatientPseudonym("PSEUDO1"),
PatientId("PATIENT1"),
Fingerprint("ashdkasdh"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS,
Instant.now(),
Report("Test")
)
)
)
val page = webClient.getPage<HtmlPage>("http://localhost/report/${requestId.value}")
assertThat(page.querySelectorAll("tbody tr")).hasSize(1)
assertThat(page.querySelectorAll("div.notification.info")).isEmpty()
}
@Test
@WithMockUser(username = "admin", roles = ["ADMIN"])
fun testShouldShowPatientDetails() {
whenever(requestService.findRequestByPatientId(anyValueClass(), any<Pageable>())).thenReturn(
PageImpl(
listOf(
Request(
2L,
randomRequestId(),
PatientPseudonym("PSEUDO1"),
PatientId("PATIENT1"),
Fingerprint("ashdkasdh"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS
),
Request(
1L,
randomRequestId(),
PatientPseudonym("PSEUDO1"),
PatientId("PATIENT1"),
Fingerprint("asdasdasd"),
RequestType.MTB_FILE,
RequestStatus.ERROR
)
)
)
)
val page = webClient.getPage<HtmlPage>("http://localhost/patient/PSEUDO1")
assertThat(page.querySelectorAll("tbody tr")).hasSize(2)
assertThat(page.querySelectorAll("div.notification.info")).isEmpty()
}
@Test
@WithMockUser(username = "admin", roles = ["ADMIN"])
fun testShouldShowPatientPseudonym() {
whenever(requestService.findRequestByPatientId(anyValueClass(), any<Pageable>())).thenReturn(
PageImpl(
listOf(
Request(
2L,
randomRequestId(),
PatientPseudonym("PSEUDO1"),
PatientId("PATIENT1"),
Fingerprint("ashdkasdh"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS
),
Request(
1L,
randomRequestId(),
PatientPseudonym("PSEUDO1"),
PatientId("PATIENT1"),
Fingerprint("asdasdasd"),
RequestType.MTB_FILE,
RequestStatus.ERROR
)
)
)
)
val page = webClient.getPage<HtmlPage>("http://localhost/patient/PSEUDO1")
assertThat(page.querySelectorAll("h2 > span")).hasSize(1)
assertThat(page.querySelectorAll("h2 > span").first().textContent).isEqualTo("PSEUDO1")
}
}
@Nested
inner class WithoutRequests {
private lateinit var requestService: RequestService
@BeforeEach
fun setup(
@Autowired requestService: RequestService
) {
this.requestService = requestService
whenever(requestService.findAll(any<Pageable>())).thenReturn(Page.empty())
}
@Test
fun testShouldShowHomePage() {
val page = webClient.getPage<HtmlPage>("http://localhost/")
assertThat(page.querySelectorAll("tbody tr")).isEmpty()
assertThat(page.querySelectorAll("div.notification.info")).hasSize(1)
}
@Test
@WithMockUser(username = "admin", roles = ["ADMIN"])
fun testShouldThrowNotFoundExceptionForUnknownReport() {
val requestId = randomRequestId()
whenever(requestService.findByUuid(anyValueClass())).thenReturn(
Optional.empty()
)
assertThrows<IOException> {
webClient.getPage<HtmlPage>("http://localhost/report/${requestId.value}")
}.also {
assertThat(it).hasRootCauseInstanceOf(NotFoundException::class.java)
}
}
@Test
@WithMockUser(username = "admin", roles = ["ADMIN"])
fun testShouldShowEmptyPatientDetails() {
whenever(requestService.findRequestByPatientId(anyValueClass(), any<Pageable>())).thenReturn(Page.empty())
val page = webClient.getPage<HtmlPage>("http://localhost/patient/PSEUDO1")
assertThat(page.querySelectorAll("tbody tr")).isEmpty()
assertThat(page.querySelectorAll("div.notification.info")).hasSize(1)
}
}
}

View File

@ -0,0 +1,88 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.web
import com.gargoylesoftware.htmlunit.WebClient
import com.gargoylesoftware.htmlunit.html.HtmlPage
import dev.dnpm.etl.processor.config.AppConfiguration
import dev.dnpm.etl.processor.config.AppSecurityConfiguration
import dev.dnpm.etl.processor.security.TokenService
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.junit.jupiter.MockitoExtension
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest
import org.springframework.boot.test.mock.mockito.MockBean
import org.springframework.test.context.ContextConfiguration
import org.springframework.test.context.TestPropertySource
import org.springframework.test.context.junit.jupiter.SpringExtension
import org.springframework.test.web.servlet.MockMvc
import org.springframework.test.web.servlet.get
import org.springframework.test.web.servlet.htmlunit.MockMvcWebClientBuilder
@WebMvcTest(controllers = [LoginController::class])
@ExtendWith(value = [MockitoExtension::class, SpringExtension::class])
@ContextConfiguration(
classes = [
LoginController::class,
AppConfiguration::class,
AppSecurityConfiguration::class
]
)
@TestPropertySource(
properties = [
"app.pseudonymize.generator=BUILDIN",
"app.security.admin-user=admin",
"app.security.admin-password={noop}very-secret",
"app.security.enable-tokens=true"
]
)
@MockBean(
TokenService::class,
)
class LoginControllerTest {
private lateinit var mockMvc: MockMvc
private lateinit var webClient: WebClient
@BeforeEach
fun setup(@Autowired mockMvc: MockMvc) {
this.mockMvc = mockMvc
this.webClient = MockMvcWebClientBuilder.mockMvcSetup(mockMvc).build()
}
@Test
fun testShouldRequestLoginPage() {
mockMvc.get("/login").andExpect {
status { isOk() }
view { name("login") }
}
}
@Test
fun testShouldShowLoginForm() {
val page = webClient.getPage<HtmlPage>("http://localhost/login")
assertThat(
page.getElementsByTagName("main").first().firstElementChild.getAttribute("class")
).isEqualTo("login-form")
}
}

View File

@ -0,0 +1,73 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.web
import com.gargoylesoftware.htmlunit.WebClient
import dev.dnpm.etl.processor.config.AppConfiguration
import dev.dnpm.etl.processor.config.AppSecurityConfiguration
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.junit.jupiter.MockitoExtension
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest
import org.springframework.test.context.ContextConfiguration
import org.springframework.test.context.TestPropertySource
import org.springframework.test.context.junit.jupiter.SpringExtension
import org.springframework.test.web.servlet.MockMvc
import org.springframework.test.web.servlet.get
import org.springframework.test.web.servlet.htmlunit.MockMvcWebClientBuilder
@WebMvcTest(controllers = [StatisticsController::class])
@ExtendWith(value = [MockitoExtension::class, SpringExtension::class])
@ContextConfiguration(
classes = [
StatisticsController::class,
AppConfiguration::class,
AppSecurityConfiguration::class
]
)
@TestPropertySource(
properties = [
"app.pseudonymize.generator=BUILDIN",
"app.security.admin-user=admin",
"app.security.admin-password={noop}very-secret"
]
)
class StatisticsControllerTest {
private lateinit var mockMvc: MockMvc
private lateinit var webClient: WebClient
@BeforeEach
fun setup(@Autowired mockMvc: MockMvc) {
this.mockMvc = mockMvc
this.webClient = MockMvcWebClientBuilder.mockMvcSetup(mockMvc).build()
}
@Test
fun testShouldRequestLoginPage() {
mockMvc.get("/statistics").andExpect {
status { isOk() }
view { name("statistics") }
}
}
}

View File

@ -0,0 +1,314 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.web
import dev.dnpm.etl.processor.Fingerprint
import dev.dnpm.etl.processor.PatientId
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.config.AppConfiguration
import dev.dnpm.etl.processor.config.AppSecurityConfiguration
import dev.dnpm.etl.processor.monitoring.CountedState
import dev.dnpm.etl.processor.monitoring.Request
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.monitoring.RequestType
import dev.dnpm.etl.processor.randomRequestId
import dev.dnpm.etl.processor.services.RequestService
import org.hamcrest.Matchers.equalTo
import org.hamcrest.Matchers.hasSize
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Nested
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.doAnswer
import org.mockito.kotlin.whenever
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest
import org.springframework.boot.test.mock.mockito.MockBean
import org.springframework.http.MediaType.TEXT_EVENT_STREAM
import org.springframework.test.context.ContextConfiguration
import org.springframework.test.context.TestPropertySource
import org.springframework.test.context.junit.jupiter.SpringExtension
import org.springframework.test.web.reactive.server.WebTestClient
import org.springframework.test.web.servlet.MockMvc
import org.springframework.test.web.servlet.client.MockMvcWebTestClient
import org.springframework.test.web.servlet.get
import org.springframework.web.context.WebApplicationContext
import reactor.core.publisher.Sinks
import reactor.test.StepVerifier
import java.time.Instant
import java.time.ZoneId
import java.time.temporal.ChronoUnit
@WebMvcTest(controllers = [StatisticsRestController::class])
@ExtendWith(value = [MockitoExtension::class, SpringExtension::class])
@ContextConfiguration(
classes = [
StatisticsRestController::class,
AppConfiguration::class,
AppSecurityConfiguration::class
]
)
@TestPropertySource(
properties = [
"app.pseudonymize.generator=BUILDIN",
"app.security.admin-user=admin",
"app.security.admin-password={noop}very-secret"
]
)
@MockBean(
RequestService::class
)
class StatisticsRestControllerTest {
private lateinit var mockMvc: MockMvc
private lateinit var statisticsUpdateProducer: Sinks.Many<Any>
private lateinit var requestService: RequestService
@BeforeEach
fun setup(
@Autowired mockMvc: MockMvc,
@Autowired statisticsUpdateProducer: Sinks.Many<Any>,
@Autowired requestService: RequestService
) {
this.mockMvc = mockMvc
this.statisticsUpdateProducer = statisticsUpdateProducer
this.requestService = requestService
}
@Nested
inner class RequestStatesTest {
@Test
fun testShouldRequestStatesForMtbFiles() {
doAnswer { _ ->
listOf(
CountedState(42, RequestStatus.WARNING),
CountedState(1, RequestStatus.UNKNOWN)
)
}.whenever(requestService).countStates()
mockMvc.get("/statistics/requeststates").andExpect {
status { isOk() }.also {
jsonPath("$", hasSize<Int>(2))
jsonPath("$[0].name", equalTo(RequestStatus.WARNING.name))
jsonPath("$[0].value", equalTo(42))
jsonPath("$[1].name", equalTo(RequestStatus.UNKNOWN.name))
jsonPath("$[1].value", equalTo(1))
}
}
}
@Test
fun testShouldRequestStatesForDeletes() {
doAnswer { _ ->
listOf(
CountedState(42, RequestStatus.SUCCESS),
CountedState(1, RequestStatus.ERROR)
)
}.whenever(requestService).countDeleteStates()
mockMvc.get("/statistics/requeststates?delete=true").andExpect {
status { isOk() }.also {
jsonPath("$", hasSize<Int>(2))
jsonPath("$[0].name", equalTo(RequestStatus.SUCCESS.name))
jsonPath("$[0].value", equalTo(42))
jsonPath("$[1].name", equalTo(RequestStatus.ERROR.name))
jsonPath("$[1].value", equalTo(1))
}
}
}
}
@Nested
inner class PatientRequestStatesTest {
@Test
fun testShouldRequestPatientStatesForMtbFiles() {
doAnswer { _ ->
listOf(
CountedState(42, RequestStatus.WARNING),
CountedState(1, RequestStatus.UNKNOWN)
)
}.whenever(requestService).findPatientUniqueStates()
mockMvc.get("/statistics/requestpatientstates").andExpect {
status { isOk() }.also {
jsonPath("$", hasSize<Int>(2))
jsonPath("$[0].name", equalTo(RequestStatus.WARNING.name))
jsonPath("$[0].value", equalTo(42))
jsonPath("$[1].name", equalTo(RequestStatus.UNKNOWN.name))
jsonPath("$[1].value", equalTo(1))
}
}
}
@Test
fun testShouldRequestPatientStatesForDeletes() {
doAnswer { _ ->
listOf(
CountedState(42, RequestStatus.SUCCESS),
CountedState(1, RequestStatus.ERROR)
)
}.whenever(requestService).findPatientUniqueDeleteStates()
mockMvc.get("/statistics/requestpatientstates?delete=true").andExpect {
status { isOk() }.also {
jsonPath("$", hasSize<Int>(2))
jsonPath("$[0].name", equalTo(RequestStatus.SUCCESS.name))
jsonPath("$[0].value", equalTo(42))
jsonPath("$[1].name", equalTo(RequestStatus.ERROR.name))
jsonPath("$[1].value", equalTo(1))
}
}
}
}
@Nested
inner class LastMonthStatesTest {
@BeforeEach
fun setup() {
val zoneId = ZoneId.of("Europe/Berlin")
doAnswer { _ ->
listOf(
Request(
1,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS,
Instant.now().atZone(zoneId).truncatedTo(ChronoUnit.DAYS).minus(2, ChronoUnit.DAYS).toInstant()
),
Request(
2,
randomRequestId(),
PatientPseudonym("TEST_12345678902"),
PatientId("P2"),
Fingerprint("0123456789abcdef2"),
RequestType.MTB_FILE,
RequestStatus.WARNING,
Instant.now().atZone(zoneId).truncatedTo(ChronoUnit.DAYS).minus(2, ChronoUnit.DAYS).toInstant()
),
Request(
3,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P2"),
Fingerprint("0123456789abcdee1"),
RequestType.DELETE,
RequestStatus.ERROR,
Instant.now().atZone(zoneId).truncatedTo(ChronoUnit.DAYS).minus(1, ChronoUnit.DAYS).toInstant()
),
Request(
4,
randomRequestId(),
PatientPseudonym("TEST_12345678902"),
PatientId("P2"),
Fingerprint("0123456789abcdef2"),
RequestType.MTB_FILE,
RequestStatus.DUPLICATION,
Instant.now().atZone(zoneId).truncatedTo(ChronoUnit.DAYS).minus(1, ChronoUnit.DAYS).toInstant()
),
Request(
5,
randomRequestId(),
PatientPseudonym("TEST_12345678902"),
PatientId("P2"),
Fingerprint("0123456789abcdef2"),
RequestType.DELETE,
RequestStatus.UNKNOWN,
Instant.now().atZone(zoneId).truncatedTo(ChronoUnit.DAYS).toInstant()
),
)
}.whenever(requestService).findAll()
}
@Test
fun testShouldRequestLastMonthForMtbFiles() {
mockMvc.get("/statistics/requestslastmonth").andExpect {
status { isOk() }.also {
jsonPath("$", hasSize<Int>(31))
}.also {
jsonPath("$[28].nameValues.error", equalTo(0))
jsonPath("$[28].nameValues.warning", equalTo(1))
jsonPath("$[28].nameValues.success", equalTo(1))
jsonPath("$[28].nameValues.duplication", equalTo(0))
jsonPath("$[28].nameValues.unknown", equalTo(0))
jsonPath("$[29].nameValues.error", equalTo(0))
jsonPath("$[29].nameValues.warning", equalTo(0))
jsonPath("$[29].nameValues.success", equalTo(0))
jsonPath("$[29].nameValues.duplication", equalTo(1))
jsonPath("$[29].nameValues.unknown", equalTo(0))
}
}
}
@Test
fun testShouldRequestLastMonthForDeletes() {
mockMvc.get("/statistics/requestslastmonth?delete=true").andExpect {
status { isOk() }.also {
jsonPath("$", hasSize<Int>(31))
}.also {
jsonPath("$[29].nameValues.error", equalTo(1))
jsonPath("$[29].nameValues.warning", equalTo(0))
jsonPath("$[29].nameValues.success", equalTo(0))
jsonPath("$[29].nameValues.duplication", equalTo(0))
jsonPath("$[29].nameValues.unknown", equalTo(0))
jsonPath("$[30].nameValues.error", equalTo(0))
jsonPath("$[30].nameValues.warning", equalTo(0))
jsonPath("$[30].nameValues.success", equalTo(0))
jsonPath("$[30].nameValues.duplication", equalTo(0))
jsonPath("$[30].nameValues.unknown", equalTo(1))
}
}
}
}
@Nested
inner class SseTest {
private lateinit var webClient: WebTestClient
@BeforeEach
fun setup(
applicationContext: WebApplicationContext,
) {
this.webClient = MockMvcWebTestClient
.bindToApplicationContext(applicationContext).build()
}
@Test
fun testShouldRequestSSE() {
statisticsUpdateProducer.emitComplete { _, _ -> true }
val result = webClient.get().uri("http://localhost/statistics/events").accept(TEXT_EVENT_STREAM).exchange()
.expectStatus().isOk()
.expectHeader().contentType(TEXT_EVENT_STREAM)
.returnResult(String::class.java)
StepVerifier.create(result.responseBody)
.expectComplete()
.verify()
}
}
}

View File

@ -23,41 +23,17 @@ import ca.uhn.fhir.context.FhirContext;
import ca.uhn.fhir.parser.IParser;
import dev.dnpm.etl.processor.config.GPasConfigProperties;
import org.apache.commons.lang3.StringUtils;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.client5.http.impl.io.BasicHttpClientConnectionManager;
import org.apache.hc.client5.http.socket.ConnectionSocketFactory;
import org.apache.hc.client5.http.socket.PlainConnectionSocketFactory;
import org.apache.hc.client5.http.ssl.SSLConnectionSocketFactory;
import org.apache.hc.core5.http.config.Registry;
import org.apache.hc.core5.http.config.RegistryBuilder;
import org.hl7.fhir.r4.model.Identifier;
import org.hl7.fhir.r4.model.Parameters;
import org.hl7.fhir.r4.model.Parameters.ParametersParameterComponent;
import org.hl7.fhir.r4.model.StringType;
import org.jetbrains.annotations.NotNull;
import org.jetbrains.annotations.Nullable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.*;
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory;
import org.springframework.retry.support.RetryTemplate;
import org.springframework.web.client.RestTemplate;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManagerFactory;
import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.IOException;
import java.security.KeyManagementException;
import java.security.KeyStore;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.cert.CertificateException;
import java.security.cert.CertificateFactory;
import java.security.cert.X509Certificate;
import java.util.Base64;
public class GpasPseudonymGenerator implements Generator {
private final static FhirContext r4Context = FhirContext.forR4();
@ -67,27 +43,15 @@ public class GpasPseudonymGenerator implements Generator {
private final RetryTemplate retryTemplate;
private final Logger log = LoggerFactory.getLogger(GpasPseudonymGenerator.class);
private SSLContext customSslContext;
private RestTemplate restTemplate;
private final RestTemplate restTemplate;
public GpasPseudonymGenerator(GPasConfigProperties gpasCfg, RetryTemplate retryTemplate) {
public GpasPseudonymGenerator(GPasConfigProperties gpasCfg, RetryTemplate retryTemplate, RestTemplate restTemplate) {
this.retryTemplate = retryTemplate;
this.restTemplate = restTemplate;
this.gPasUrl = gpasCfg.getUri();
this.psnTargetDomain = gpasCfg.getTarget();
httpHeader = getHttpHeaders(gpasCfg.getUsername(), gpasCfg.getPassword());
try {
if (StringUtils.isNotBlank(gpasCfg.getSslCaLocation())) {
customSslContext = getSslContext(gpasCfg.getSslCaLocation());
log.warn(String.format("%s has been initialized with SSL certificate %s. This is deprecated in favor of including Root CA.",
this.getClass().getName(), gpasCfg.getSslCaLocation()));
}
} catch (IOException | KeyManagementException | KeyStoreException | CertificateException |
NoSuchAlgorithmException e) {
throw new RuntimeException(e);
}
log.debug(String.format("%s has been initialized", this.getClass().getName()));
}
@ -97,7 +61,7 @@ public class GpasPseudonymGenerator implements Generator {
var gPasRequestBody = getGpasRequestBody(id);
var responseEntity = getGpasPseudonym(gPasRequestBody);
var gPasPseudonymResult = (Parameters) r4Context.newJsonParser()
.parseResource(responseEntity.getBody());
.parseResource(responseEntity.getBody());
return unwrapPseudonym(gPasPseudonymResult);
}
@ -111,9 +75,9 @@ public class GpasPseudonymGenerator implements Generator {
}
final var identifier = (Identifier) parameters.get().getPart().stream()
.filter(a -> a.getName().equals("pseudonym"))
.findFirst()
.orElseGet(ParametersParameterComponent::new).getValue();
.filter(a -> a.getName().equals("pseudonym"))
.findFirst()
.orElseGet(ParametersParameterComponent::new).getValue();
// pseudonym
return sanitizeValue(identifier.getValue());
@ -139,12 +103,11 @@ public class GpasPseudonymGenerator implements Generator {
HttpEntity<String> requestEntity = new HttpEntity<>(gPasRequestBody, this.httpHeader);
ResponseEntity<String> responseEntity;
var restTemplate = getRestTemplete();
try {
responseEntity = retryTemplate.execute(
ctx -> restTemplate.exchange(gPasUrl, HttpMethod.POST, requestEntity,
String.class));
ctx -> restTemplate.exchange(gPasUrl, HttpMethod.POST, requestEntity,
String.class));
if (responseEntity.getStatusCode().is2xxSuccessful()) {
log.debug("API request succeeded. Response: {}", responseEntity.getStatusCode());
@ -156,16 +119,16 @@ public class GpasPseudonymGenerator implements Generator {
return responseEntity;
} catch (Exception unexpected) {
throw new PseudonymRequestFailed(
"API request due unexpected error unsuccessful gPas unsuccessful.", unexpected);
"API request due unexpected error unsuccessful gPas unsuccessful.", unexpected);
}
}
protected String getGpasRequestBody(String id) {
var requestParameters = new Parameters();
requestParameters.addParameter().setName("target")
.setValue(new StringType().setValue(psnTargetDomain));
.setValue(new StringType().setValue(psnTargetDomain));
requestParameters.addParameter().setName("original")
.setValue(new StringType().setValue(id));
.setValue(new StringType().setValue(id));
final IParser iParser = r4Context.newJsonParser();
return iParser.encodeResourceToString(requestParameters);
}
@ -179,74 +142,7 @@ public class GpasPseudonymGenerator implements Generator {
return headers;
}
String authHeader = gPasUserName + ":" + gPasPassword;
byte[] authHeaderBytes = authHeader.getBytes();
byte[] encodedAuthHeaderBytes = Base64.getEncoder().encode(authHeaderBytes);
String encodedAuthHeader = new String(encodedAuthHeaderBytes);
if (StringUtils.isNotBlank(gPasUserName) && StringUtils.isNotBlank(gPasPassword)) {
headers.set("Authorization", "Basic " + encodedAuthHeader);
}
headers.setBasicAuth(gPasUserName, gPasPassword);
return headers;
}
/**
* Read SSL root certificate and return SSLContext
*
* @param certificateLocation file location to root certificate (PEM)
* @return initialized SSLContext
* @throws IOException file cannot be read
* @throws CertificateException in case we have an invalid certificate of type X.509
* @throws KeyStoreException keystore cannot be initialized
* @throws NoSuchAlgorithmException missing trust manager algorithmus
* @throws KeyManagementException key management failed at init SSLContext
*/
@Nullable
protected SSLContext getSslContext(String certificateLocation)
throws IOException, CertificateException, KeyStoreException, KeyManagementException, NoSuchAlgorithmException {
KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());
FileInputStream fis = new FileInputStream(certificateLocation);
X509Certificate ca = (X509Certificate) CertificateFactory.getInstance("X.509")
.generateCertificate(new BufferedInputStream(fis));
ks.load(null, null);
ks.setCertificateEntry(Integer.toString(1), ca);
TrustManagerFactory tmf = TrustManagerFactory.getInstance(
TrustManagerFactory.getDefaultAlgorithm());
tmf.init(ks);
SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(null, tmf.getTrustManagers(), null);
return sslContext;
}
protected RestTemplate getRestTemplete() {
if (restTemplate != null) {
return restTemplate;
}
if (customSslContext == null) {
restTemplate = new RestTemplate();
return restTemplate;
}
final var sslsf = new SSLConnectionSocketFactory(customSslContext);
final Registry<ConnectionSocketFactory> socketFactoryRegistry = RegistryBuilder.<ConnectionSocketFactory>create()
.register("https", sslsf).register("http", new PlainConnectionSocketFactory()).build();
final BasicHttpClientConnectionManager connectionManager = new BasicHttpClientConnectionManager(
socketFactoryRegistry);
final CloseableHttpClient httpClient = HttpClients.custom()
.setConnectionManager(connectionManager).build();
final HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(
httpClient);
restTemplate = new RestTemplate(requestFactory);
return restTemplate;
}
}

View File

@ -69,6 +69,9 @@ data class GPasConfigProperties(
@ConfigurationProperties(RestTargetProperties.NAME)
data class RestTargetProperties(
val uri: String?,
val username: String?,
val password: String?,
val isBwhc: Boolean = false,
) {
companion object {
const val NAME = "app.rest"

View File

@ -20,21 +20,32 @@
package dev.dnpm.etl.processor.config
import com.fasterxml.jackson.databind.ObjectMapper
import dev.dnpm.etl.processor.monitoring.ConnectionCheckResult
import dev.dnpm.etl.processor.monitoring.ConnectionCheckService
import dev.dnpm.etl.processor.monitoring.GPasConnectionCheckService
import dev.dnpm.etl.processor.monitoring.ReportService
import dev.dnpm.etl.processor.pseudonym.AnonymizingGenerator
import dev.dnpm.etl.processor.pseudonym.Generator
import dev.dnpm.etl.processor.pseudonym.GpasPseudonymGenerator
import dev.dnpm.etl.processor.pseudonym.PseudonymizeService
import dev.dnpm.etl.processor.services.TokenRepository
import dev.dnpm.etl.processor.services.TokenService
import dev.dnpm.etl.processor.security.TokenRepository
import dev.dnpm.etl.processor.security.TokenService
import dev.dnpm.etl.processor.services.Transformation
import dev.dnpm.etl.processor.services.TransformationService
import org.apache.hc.client5.http.impl.classic.HttpClients
import org.apache.hc.client5.http.impl.io.BasicHttpClientConnectionManager
import org.apache.hc.client5.http.socket.ConnectionSocketFactory
import org.apache.hc.client5.http.socket.PlainConnectionSocketFactory
import org.apache.hc.client5.http.ssl.SSLConnectionSocketFactory
import org.apache.hc.core5.http.config.RegistryBuilder
import org.slf4j.LoggerFactory
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty
import org.springframework.boot.context.properties.EnableConfigurationProperties
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import org.springframework.data.jdbc.repository.config.AbstractJdbcConfiguration
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory
import org.springframework.retry.RetryCallback
import org.springframework.retry.RetryContext
import org.springframework.retry.RetryListener
@ -44,7 +55,16 @@ import org.springframework.retry.support.RetryTemplateBuilder
import org.springframework.scheduling.annotation.EnableScheduling
import org.springframework.security.crypto.password.PasswordEncoder
import org.springframework.security.provisioning.InMemoryUserDetailsManager
import org.springframework.web.client.HttpClientErrorException
import org.springframework.web.client.RestTemplate
import reactor.core.publisher.Sinks
import java.io.BufferedInputStream
import java.io.FileInputStream
import java.security.KeyStore
import java.security.cert.CertificateFactory
import java.security.cert.X509Certificate
import javax.net.ssl.SSLContext
import javax.net.ssl.TrustManagerFactory
import kotlin.time.Duration.Companion.seconds
import kotlin.time.toJavaDuration
@ -62,10 +82,27 @@ class AppConfiguration {
private val logger = LoggerFactory.getLogger(AppConfiguration::class.java)
@Bean
fun restTemplate(): RestTemplate {
return RestTemplate()
}
@ConditionalOnProperty(value = ["app.pseudonymize.generator"], havingValue = "GPAS")
@Bean
fun gpasPseudonymGenerator(configProperties: GPasConfigProperties, retryTemplate: RetryTemplate): Generator {
return GpasPseudonymGenerator(configProperties, retryTemplate)
fun gpasPseudonymGenerator(configProperties: GPasConfigProperties, retryTemplate: RetryTemplate, restTemplate: RestTemplate): Generator {
try {
if (!configProperties.sslCaLocation.isNullOrBlank()) {
return GpasPseudonymGenerator(
configProperties,
retryTemplate,
createCustomGpasRestTemplate(configProperties)
)
}
} catch (e: Exception) {
throw RuntimeException(e)
}
return GpasPseudonymGenerator(configProperties, retryTemplate, restTemplate)
}
@ConditionalOnProperty(value = ["app.pseudonymize.generator"], havingValue = "BUILDIN", matchIfMissing = true)
@ -77,8 +114,80 @@ class AppConfiguration {
@ConditionalOnProperty(value = ["app.pseudonymizer"], havingValue = "GPAS")
@ConditionalOnMissingBean
@Bean
fun gpasPseudonymGeneratorOnDeprecatedProperty(configProperties: GPasConfigProperties, retryTemplate: RetryTemplate): Generator {
return GpasPseudonymGenerator(configProperties, retryTemplate)
fun gpasPseudonymGeneratorOnDeprecatedProperty(configProperties: GPasConfigProperties, retryTemplate: RetryTemplate, restTemplate: RestTemplate): Generator {
try {
if (!configProperties.sslCaLocation.isNullOrBlank()) {
return GpasPseudonymGenerator(
configProperties,
retryTemplate,
createCustomGpasRestTemplate(configProperties)
)
}
} catch (e: Exception) {
throw RuntimeException(e)
}
return GpasPseudonymGenerator(configProperties, retryTemplate, restTemplate)
}
private fun createCustomGpasRestTemplate(configProperties: GPasConfigProperties): RestTemplate {
fun getSslContext(certificateLocation: String): SSLContext? {
val ks = KeyStore.getInstance(KeyStore.getDefaultType())
val fis = FileInputStream(certificateLocation)
val ca = CertificateFactory.getInstance("X.509")
.generateCertificate(BufferedInputStream(fis)) as X509Certificate
ks.load(null, null)
ks.setCertificateEntry(1.toString(), ca)
val tmf = TrustManagerFactory.getInstance(
TrustManagerFactory.getDefaultAlgorithm()
)
tmf.init(ks)
val sslContext = SSLContext.getInstance("TLS")
sslContext.init(null, tmf.trustManagers, null)
return sslContext
}
fun getCustomRestTemplate(customSslContext: SSLContext): RestTemplate {
val sslsf = SSLConnectionSocketFactory(customSslContext)
val socketFactoryRegistry = RegistryBuilder.create<ConnectionSocketFactory>()
.register("https", sslsf).register("http", PlainConnectionSocketFactory()).build()
val connectionManager = BasicHttpClientConnectionManager(
socketFactoryRegistry
)
val httpClient = HttpClients.custom()
.setConnectionManager(connectionManager).build()
val requestFactory = HttpComponentsClientHttpRequestFactory(
httpClient
)
return RestTemplate(requestFactory)
}
try {
if (!configProperties.sslCaLocation.isNullOrBlank()) {
val customSslContext = getSslContext(configProperties.sslCaLocation)
logger.warn(
String.format(
"%s has been initialized with SSL certificate %s. This is deprecated in favor of including Root CA.",
this.javaClass.name, configProperties.sslCaLocation
)
)
if (customSslContext != null) {
return getCustomRestTemplate(customSslContext)
}
}
} catch (e: Exception) {
throw RuntimeException(e)
}
throw RuntimeException("Custom SSL configuration for gPAS not usable")
}
@ConditionalOnProperty(value = ["app.pseudonymizer"], havingValue = "BUILDIN")
@ -116,6 +225,8 @@ class AppConfiguration {
fun retryTemplate(configProperties: AppConfigProperties): RetryTemplate {
return RetryTemplateBuilder()
.notRetryOn(IllegalArgumentException::class.java)
.notRetryOn(HttpClientErrorException.BadRequest::class.java)
.notRetryOn(HttpClientErrorException.UnprocessableEntity::class.java)
.exponentialBackoff(2.seconds.toJavaDuration(), 1.25, 5.seconds.toJavaDuration())
.customPolicy(SimpleRetryPolicy(configProperties.maxRetryAttempts))
.withListener(object : RetryListener {
@ -142,9 +253,34 @@ class AppConfiguration {
}
@Bean
fun configsUpdateProducer(): Sinks.Many<Boolean> {
return Sinks.many().multicast().directBestEffort()
fun connectionCheckUpdateProducer(): Sinks.Many<ConnectionCheckResult> {
return Sinks.many().multicast().onBackpressureBuffer()
}
@ConditionalOnProperty(value = ["app.pseudonymize.generator"], havingValue = "GPAS")
@Bean
fun gPasConnectionCheckService(
restTemplate: RestTemplate,
gPasConfigProperties: GPasConfigProperties,
connectionCheckUpdateProducer: Sinks.Many<ConnectionCheckResult>
): ConnectionCheckService {
return GPasConnectionCheckService(restTemplate, gPasConfigProperties, connectionCheckUpdateProducer)
}
@ConditionalOnProperty(value = ["app.pseudonymizer"], havingValue = "GPAS")
@ConditionalOnMissingBean
@Bean
fun gPasConnectionCheckServiceOnDeprecatedProperty(
restTemplate: RestTemplate,
gPasConfigProperties: GPasConfigProperties,
connectionCheckUpdateProducer: Sinks.Many<ConnectionCheckResult>
): ConnectionCheckService {
return GPasConnectionCheckService(restTemplate, gPasConfigProperties, connectionCheckUpdateProducer)
}
@Bean
fun jdbcConfiguration(): AbstractJdbcConfiguration {
return AppJdbcConfiguration()
}
}

View File

@ -0,0 +1,25 @@
package dev.dnpm.etl.processor.config
import dev.dnpm.etl.processor.Fingerprint
import org.springframework.context.annotation.Configuration
import org.springframework.core.convert.converter.Converter
import org.springframework.data.jdbc.repository.config.AbstractJdbcConfiguration
@Configuration
class AppJdbcConfiguration : AbstractJdbcConfiguration() {
override fun userConverters(): MutableList<*> {
return mutableListOf(StringToFingerprintConverter(), FingerprintToStringConverter())
}
}
class StringToFingerprintConverter : Converter<String, Fingerprint> {
override fun convert(source: String): Fingerprint {
return Fingerprint(source)
}
}
class FingerprintToStringConverter : Converter<Fingerprint, String> {
override fun convert(source: Fingerprint): String {
return source.value
}
}

View File

@ -21,6 +21,7 @@ package dev.dnpm.etl.processor.config
import com.fasterxml.jackson.databind.ObjectMapper
import dev.dnpm.etl.processor.input.KafkaInputListener
import dev.dnpm.etl.processor.monitoring.ConnectionCheckResult
import dev.dnpm.etl.processor.monitoring.ConnectionCheckService
import dev.dnpm.etl.processor.monitoring.KafkaConnectionCheckService
import dev.dnpm.etl.processor.output.KafkaMtbFileSender
@ -105,8 +106,11 @@ class AppKafkaConfiguration {
}
@Bean
fun connectionCheckService(consumerFactory: ConsumerFactory<String, String>, configsUpdateProducer: Sinks.Many<Boolean>): ConnectionCheckService {
return KafkaConnectionCheckService(consumerFactory.createConsumer(), configsUpdateProducer)
fun kafkaConnectionCheckService(
consumerFactory: ConsumerFactory<String, String>,
connectionCheckUpdateProducer: Sinks.Many<ConnectionCheckResult>
): ConnectionCheckService {
return KafkaConnectionCheckService(consumerFactory.createConsumer(), connectionCheckUpdateProducer)
}
}

View File

@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@ -19,10 +19,13 @@
package dev.dnpm.etl.processor.config
import dev.dnpm.etl.processor.monitoring.ConnectionCheckResult
import dev.dnpm.etl.processor.monitoring.ConnectionCheckService
import dev.dnpm.etl.processor.monitoring.ReportService
import dev.dnpm.etl.processor.monitoring.RestConnectionCheckService
import dev.dnpm.etl.processor.output.MtbFileSender
import dev.dnpm.etl.processor.output.RestMtbFileSender
import dev.dnpm.etl.processor.output.RestBwhcMtbFileSender
import dev.dnpm.etl.processor.output.RestDipMtbFileSender
import org.slf4j.LoggerFactory
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty
@ -47,28 +50,29 @@ class AppRestConfiguration {
private val logger = LoggerFactory.getLogger(AppRestConfiguration::class.java)
@Bean
fun restTemplate(): RestTemplate {
return RestTemplate()
}
@Bean
fun restMtbFileSender(
restTemplate: RestTemplate,
restTargetProperties: RestTargetProperties,
retryTemplate: RetryTemplate
retryTemplate: RetryTemplate,
reportService: ReportService,
): MtbFileSender {
logger.info("Selected 'RestMtbFileSender'")
return RestMtbFileSender(restTemplate, restTargetProperties, retryTemplate)
if (restTargetProperties.isBwhc) {
logger.info("Selected 'RestBwhcMtbFileSender'")
return RestBwhcMtbFileSender(restTemplate, restTargetProperties, retryTemplate, reportService)
}
logger.info("Selected 'RestDipMtbFileSender'")
return RestDipMtbFileSender(restTemplate, restTargetProperties, retryTemplate, reportService)
}
@Bean
fun connectionCheckService(
fun restConnectionCheckService(
restTemplate: RestTemplate,
restTargetProperties: RestTargetProperties,
configsUpdateProducer: Sinks.Many<Boolean>
connectionCheckUpdateProducer: Sinks.Many<ConnectionCheckResult>
): ConnectionCheckService {
return RestConnectionCheckService(restTemplate, restTargetProperties, configsUpdateProducer)
return RestConnectionCheckService(restTemplate, restTargetProperties, connectionCheckUpdateProducer)
}
}

View File

@ -21,7 +21,7 @@ package dev.dnpm.etl.processor.config
import dev.dnpm.etl.processor.security.UserRole
import dev.dnpm.etl.processor.security.UserRoleRepository
import dev.dnpm.etl.processor.services.UserRoleService
import dev.dnpm.etl.processor.security.UserRoleService
import org.slf4j.LoggerFactory
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty
import org.springframework.boot.context.properties.EnableConfigurationProperties
@ -44,6 +44,8 @@ import org.springframework.security.web.SecurityFilterChain
import java.util.*
private const val LOGIN_PATH = "/login"
@Configuration
@EnableConfigurationProperties(
value = [
@ -89,7 +91,7 @@ class AppSecurityConfiguration(
http {
authorizeRequests {
authorize("/configs/**", hasRole("ADMIN"))
authorize("/mtbfile/**", hasAnyRole("MTBFILE"))
authorize("/mtbfile/**", hasAnyRole("MTBFILE", "ADMIN", "USER"))
authorize("/report/**", hasAnyRole("ADMIN", "USER"))
authorize("*.css", permitAll)
authorize("*.ico", permitAll)
@ -104,15 +106,15 @@ class AppSecurityConfiguration(
realmName = "ETL-Processor"
}
formLogin {
loginPage = "/login"
loginPage = LOGIN_PATH
}
oauth2Login {
loginPage = "/login"
loginPage = LOGIN_PATH
}
sessionManagement {
sessionConcurrency {
maximumSessions = 1
expiredUrl = "/login?expired"
expiredUrl = "$LOGIN_PATH?expired"
}
sessionFixation {
newSession()
@ -147,7 +149,7 @@ class AppSecurityConfiguration(
http {
authorizeRequests {
authorize("/configs/**", hasRole("ADMIN"))
authorize("/mtbfile/**", hasAnyRole("MTBFILE"))
authorize("/mtbfile/**", hasAnyRole("MTBFILE", "ADMIN"))
authorize("/report/**", hasRole("ADMIN"))
authorize(anyRequest, permitAll)
}
@ -155,7 +157,7 @@ class AppSecurityConfiguration(
realmName = "ETL-Processor"
}
formLogin {
loginPage = "/login"
loginPage = LOGIN_PATH
}
csrf { disable() }
}

View File

@ -22,6 +22,8 @@ package dev.dnpm.etl.processor.input
import com.fasterxml.jackson.databind.ObjectMapper
import de.ukw.ccc.bwhc.dto.Consent
import de.ukw.ccc.bwhc.dto.MtbFile
import dev.dnpm.etl.processor.PatientId
import dev.dnpm.etl.processor.RequestId
import dev.dnpm.etl.processor.services.RequestProcessor
import org.apache.kafka.clients.consumer.ConsumerRecord
import org.slf4j.LoggerFactory
@ -35,12 +37,28 @@ class KafkaInputListener(
override fun onMessage(data: ConsumerRecord<String, String>) {
val mtbFile = objectMapper.readValue(data.value(), MtbFile::class.java)
val patientId = PatientId(mtbFile.patient.id)
val firstRequestIdHeader = data.headers().headers("requestId")?.firstOrNull()
val requestId = if (null != firstRequestIdHeader) {
RequestId(String(firstRequestIdHeader.value()))
} else {
RequestId("")
}
if (mtbFile.consent.status == Consent.Status.ACTIVE) {
logger.debug("Accepted MTB File for processing")
requestProcessor.processMtbFile(mtbFile)
if (requestId.isBlank()) {
requestProcessor.processMtbFile(mtbFile)
} else {
requestProcessor.processMtbFile(mtbFile, requestId)
}
} else {
logger.debug("Accepted MTB File and process deletion")
requestProcessor.processDeletion(mtbFile.patient.id)
if (requestId.isBlank()) {
requestProcessor.processDeletion(patientId)
} else {
requestProcessor.processDeletion(patientId, requestId)
}
}
}
}

View File

@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@ -21,13 +21,14 @@ package dev.dnpm.etl.processor.input
import de.ukw.ccc.bwhc.dto.Consent
import de.ukw.ccc.bwhc.dto.MtbFile
import dev.dnpm.etl.processor.PatientId
import dev.dnpm.etl.processor.services.RequestProcessor
import org.slf4j.LoggerFactory
import org.springframework.http.ResponseEntity
import org.springframework.web.bind.annotation.*
@RestController
@RequestMapping(path = ["mtbfile"])
@RequestMapping(path = ["mtbfile", "mtb"])
class MtbFileRestController(
private val requestProcessor: RequestProcessor,
) {
@ -40,21 +41,22 @@ class MtbFileRestController(
}
@PostMapping
fun mtbFile(@RequestBody mtbFile: MtbFile): ResponseEntity<Void> {
fun mtbFile(@RequestBody mtbFile: MtbFile): ResponseEntity<Unit> {
if (mtbFile.consent.status == Consent.Status.ACTIVE) {
logger.debug("Accepted MTB File for processing")
requestProcessor.processMtbFile(mtbFile)
} else {
logger.debug("Accepted MTB File and process deletion")
requestProcessor.processDeletion(mtbFile.patient.id)
val patientId = PatientId(mtbFile.patient.id)
requestProcessor.processDeletion(patientId)
}
return ResponseEntity.accepted().build()
}
@DeleteMapping(path = ["{patientId}"])
fun deleteData(@PathVariable patientId: String): ResponseEntity<Void> {
fun deleteData(@PathVariable patientId: String): ResponseEntity<Unit> {
logger.debug("Accepted patient ID to process deletion")
requestProcessor.processDeletion(patientId)
requestProcessor.processDeletion(PatientId(patientId))
return ResponseEntity.accepted().build()
}

View File

@ -20,46 +20,89 @@
package dev.dnpm.etl.processor.monitoring
import dev.dnpm.etl.processor.config.GPasConfigProperties
import dev.dnpm.etl.processor.config.RestTargetProperties
import jakarta.annotation.PostConstruct
import org.apache.kafka.clients.consumer.Consumer
import org.apache.kafka.common.errors.TimeoutException
import org.springframework.beans.factory.annotation.Qualifier
import org.springframework.http.HttpStatus
import org.springframework.http.*
import org.springframework.scheduling.annotation.Scheduled
import org.springframework.web.client.RestTemplate
import org.springframework.web.util.UriComponentsBuilder
import reactor.core.publisher.Sinks
import java.time.Instant
import kotlin.time.Duration.Companion.seconds
import kotlin.time.toJavaDuration
interface ConnectionCheckService {
fun interface ConnectionCheckService {
fun connectionAvailable(): Boolean
fun connectionAvailable(): ConnectionCheckResult
}
interface OutputConnectionCheckService : ConnectionCheckService
sealed class ConnectionCheckResult {
abstract val available: Boolean
abstract val timestamp: Instant
abstract val lastChange: Instant
data class KafkaConnectionCheckResult(
override val available: Boolean,
override val timestamp: Instant,
override val lastChange: Instant
) : ConnectionCheckResult()
data class RestConnectionCheckResult(
override val available: Boolean,
override val timestamp: Instant,
override val lastChange: Instant
) : ConnectionCheckResult()
data class GPasConnectionCheckResult(
override val available: Boolean,
override val timestamp: Instant,
override val lastChange: Instant
) : ConnectionCheckResult()
}
class KafkaConnectionCheckService(
private val consumer: Consumer<String, String>,
@Qualifier("configsUpdateProducer")
private val configsUpdateProducer: Sinks.Many<Boolean>
) : ConnectionCheckService {
private var connectionAvailable: Boolean = false
@Qualifier("connectionCheckUpdateProducer")
private val connectionCheckUpdateProducer: Sinks.Many<ConnectionCheckResult>
) : OutputConnectionCheckService {
private var result = ConnectionCheckResult.KafkaConnectionCheckResult(false, Instant.now(), Instant.now())
@PostConstruct
@Scheduled(cron = "0 * * * * *")
fun check() {
connectionAvailable = try {
null != consumer.listTopics(5.seconds.toJavaDuration())
} catch (e: TimeoutException) {
false
result = try {
val available = null != consumer.listTopics(5.seconds.toJavaDuration())
ConnectionCheckResult.KafkaConnectionCheckResult(
available,
Instant.now(),
if (result.available == available) { result.lastChange } else { Instant.now() }
)
} catch (_: TimeoutException) {
ConnectionCheckResult.KafkaConnectionCheckResult(
false,
Instant.now(),
if (!result.available) { result.lastChange } else { Instant.now() }
)
}
configsUpdateProducer.emitNext(connectionAvailable, Sinks.EmitFailureHandler.FAIL_FAST)
connectionCheckUpdateProducer.emitNext(
result,
Sinks.EmitFailureHandler.FAIL_FAST
)
}
override fun connectionAvailable(): Boolean {
return this.connectionAvailable
override fun connectionAvailable(): ConnectionCheckResult.KafkaConnectionCheckResult {
return this.result
}
}
@ -67,27 +110,101 @@ class KafkaConnectionCheckService(
class RestConnectionCheckService(
private val restTemplate: RestTemplate,
private val restTargetProperties: RestTargetProperties,
@Qualifier("configsUpdateProducer")
private val configsUpdateProducer: Sinks.Many<Boolean>
) : ConnectionCheckService {
@Qualifier("connectionCheckUpdateProducer")
private val connectionCheckUpdateProducer: Sinks.Many<ConnectionCheckResult>
) : OutputConnectionCheckService {
private var connectionAvailable: Boolean = false
private var result = ConnectionCheckResult.RestConnectionCheckResult(false, Instant.now(), Instant.now())
@PostConstruct
@Scheduled(cron = "0 * * * * *")
fun check() {
connectionAvailable = try {
restTemplate.getForEntity(
restTargetProperties.uri?.replace("/etl/api", "").toString(),
result = try {
val available = restTemplate.getForEntity(
if (restTargetProperties.isBwhc) {
UriComponentsBuilder.fromUriString(restTargetProperties.uri.toString()).path("").toUriString()
} else {
UriComponentsBuilder.fromUriString(restTargetProperties.uri.toString())
.pathSegment("mtb")
.pathSegment("kaplan-meier")
.pathSegment("config")
.toUriString()
},
String::class.java
).statusCode == HttpStatus.OK
} catch (e: Exception) {
false
ConnectionCheckResult.RestConnectionCheckResult(
available,
Instant.now(),
if (result.available == available) { result.lastChange } else { Instant.now() }
)
} catch (_: Exception) {
ConnectionCheckResult.RestConnectionCheckResult(
false,
Instant.now(),
if (!result.available) { result.lastChange } else { Instant.now() }
)
}
configsUpdateProducer.emitNext(connectionAvailable, Sinks.EmitFailureHandler.FAIL_FAST)
connectionCheckUpdateProducer.emitNext(
result,
Sinks.EmitFailureHandler.FAIL_FAST
)
}
override fun connectionAvailable(): Boolean {
return this.connectionAvailable
override fun connectionAvailable(): ConnectionCheckResult.RestConnectionCheckResult {
return this.result
}
}
class GPasConnectionCheckService(
private val restTemplate: RestTemplate,
private val gPasConfigProperties: GPasConfigProperties,
@Qualifier("connectionCheckUpdateProducer")
private val connectionCheckUpdateProducer: Sinks.Many<ConnectionCheckResult>
) : ConnectionCheckService {
private var result = ConnectionCheckResult.GPasConnectionCheckResult(false, Instant.now(), Instant.now())
@PostConstruct
@Scheduled(cron = "0 * * * * *")
fun check() {
result = try {
val uri = UriComponentsBuilder.fromUriString(
gPasConfigProperties.uri?.replace("/\$pseudonymizeAllowCreate", "/metadata").toString()
).build().toUri()
val headers = HttpHeaders()
headers.contentType = MediaType.APPLICATION_JSON
if (!gPasConfigProperties.username.isNullOrBlank() && !gPasConfigProperties.password.isNullOrBlank()) {
headers.setBasicAuth(gPasConfigProperties.username, gPasConfigProperties.password)
}
val available = restTemplate.exchange(
uri,
HttpMethod.GET,
HttpEntity<Void>(headers),
Void::class.java
).statusCode == HttpStatus.OK
ConnectionCheckResult.GPasConnectionCheckResult(
available,
Instant.now(),
if (result.available == available) { result.lastChange } else { Instant.now() }
)
} catch (_: Exception) {
ConnectionCheckResult.GPasConnectionCheckResult(
false,
Instant.now(),
if (!result.available) { result.lastChange } else { Instant.now() }
)
}
connectionCheckUpdateProducer.emitNext(
result,
Sinks.EmitFailureHandler.FAIL_FAST
)
}
override fun connectionAvailable(): ConnectionCheckResult.GPasConnectionCheckResult {
return this.result
}
}

View File

@ -19,11 +19,15 @@
package dev.dnpm.etl.processor.monitoring
import com.fasterxml.jackson.annotation.JsonAlias
import com.fasterxml.jackson.annotation.JsonIgnoreProperties
import com.fasterxml.jackson.annotation.JsonValue
import com.fasterxml.jackson.core.JsonParseException
import com.fasterxml.jackson.databind.JsonMappingException
import com.fasterxml.jackson.databind.ObjectMapper
import dev.dnpm.etl.processor.monitoring.ReportService.Issue
import dev.dnpm.etl.processor.monitoring.ReportService.Severity
import java.util.Optional
class ReportService(
private val objectMapper: ObjectMapper
@ -54,11 +58,25 @@ class ReportService(
private data class DataQualityReport(val issues: List<Issue>)
@JsonIgnoreProperties(ignoreUnknown = true)
data class Issue(val severity: Severity, val message: String)
data class Issue(
val severity: Severity,
@JsonAlias("details") val message: String,
val path: Optional<String> = Optional.empty()
)
enum class Severity(@JsonValue val value: String) {
FATAL("fatal"),
ERROR("error"),
WARNING("warning"),
INFO("info")
}
}
fun List<Issue>.asRequestStatus(): RequestStatus {
val severity = this.minOfOrNull { it.severity }
return when (severity) {
Severity.FATAL, Severity.ERROR -> RequestStatus.ERROR
Severity.WARNING -> RequestStatus.WARNING
else -> RequestStatus.SUCCESS
}
}

View File

@ -19,10 +19,12 @@
package dev.dnpm.etl.processor.monitoring
import dev.dnpm.etl.processor.*
import org.springframework.data.annotation.Id
import org.springframework.data.domain.Page
import org.springframework.data.domain.Pageable
import org.springframework.data.jdbc.repository.query.Query
import org.springframework.data.relational.core.mapping.Column
import org.springframework.data.relational.core.mapping.Embedded
import org.springframework.data.relational.core.mapping.Table
import org.springframework.data.repository.CrudRepository
@ -30,26 +32,48 @@ import org.springframework.data.repository.PagingAndSortingRepository
import java.time.Instant
import java.util.*
typealias RequestId = UUID
@Table("request")
data class Request(
@Id val id: Long? = null,
val uuid: String = RequestId.randomUUID().toString(),
val patientId: String,
val pid: String,
val fingerprint: String,
val uuid: RequestId = randomRequestId(),
val patientPseudonym: PatientPseudonym,
val pid: PatientId,
@Column("fingerprint")
val fingerprint: Fingerprint,
val type: RequestType,
var status: RequestStatus,
var processedAt: Instant = Instant.now(),
@Embedded.Nullable var report: Report? = null
)
) {
constructor(
uuid: RequestId,
patientPseudonym: PatientPseudonym,
pid: PatientId,
fingerprint: Fingerprint,
type: RequestType,
status: RequestStatus
) :
this(null, uuid, patientPseudonym, pid, fingerprint, type, status, Instant.now())
constructor(
uuid: RequestId,
patientPseudonym: PatientPseudonym,
pid: PatientId,
fingerprint: Fingerprint,
type: RequestType,
status: RequestStatus,
processedAt: Instant
) :
this(null, uuid, patientPseudonym, pid, fingerprint, type, status, processedAt)
}
@JvmRecord
data class Report(
val description: String,
val dataQualityReport: String = ""
)
@JvmRecord
data class CountedState(
val count: Int,
val status: RequestStatus,
@ -57,17 +81,17 @@ data class CountedState(
interface RequestRepository : CrudRepository<Request, Long>, PagingAndSortingRepository<Request, Long> {
fun findAllByPatientIdOrderByProcessedAtDesc(patientId: String): List<Request>
fun findAllByPatientPseudonymOrderByProcessedAtDesc(patientId: PatientPseudonym): List<Request>
fun findByUuidEquals(uuid: String): Optional<Request>
fun findByUuidEquals(uuid: RequestId): Optional<Request>
fun findRequestByPatientId(patientId: String, pageable: Pageable): Page<Request>
fun findRequestByPatientPseudonym(patientPseudonym: PatientPseudonym, pageable: Pageable): Page<Request>
@Query("SELECT count(*) AS count, status FROM request WHERE type = 'MTB_FILE' GROUP BY status ORDER BY status, count DESC;")
fun countStates(): List<CountedState>
@Query("SELECT count(*) AS count, status FROM (" +
"SELECT status, rank() OVER (PARTITION BY patient_id ORDER BY processed_at DESC) AS rank FROM request " +
"SELECT status, rank() OVER (PARTITION BY patient_pseudonym ORDER BY processed_at DESC) AS rank FROM request " +
"WHERE type = 'MTB_FILE' AND status NOT IN ('DUPLICATION') " +
") rank WHERE rank = 1 GROUP BY status ORDER BY status, count DESC;")
fun findPatientUniqueStates(): List<CountedState>
@ -76,7 +100,7 @@ interface RequestRepository : CrudRepository<Request, Long>, PagingAndSortingRep
fun countDeleteStates(): List<CountedState>
@Query("SELECT count(*) AS count, status FROM (" +
"SELECT status, rank() OVER (PARTITION BY patient_id ORDER BY processed_at DESC) AS rank FROM request " +
"SELECT status, rank() OVER (PARTITION BY patient_pseudonym ORDER BY processed_at DESC) AS rank FROM request " +
"WHERE type = 'DELETE'" +
") rank WHERE rank = 1 GROUP BY status ORDER BY status, count DESC;")
fun findPatientUniqueDeleteStates(): List<CountedState>

View File

@ -22,6 +22,7 @@ package dev.dnpm.etl.processor.output
import com.fasterxml.jackson.databind.ObjectMapper
import de.ukw.ccc.bwhc.dto.Consent
import de.ukw.ccc.bwhc.dto.MtbFile
import dev.dnpm.etl.processor.RequestId
import dev.dnpm.etl.processor.config.KafkaProperties
import dev.dnpm.etl.processor.monitoring.RequestStatus
import org.slf4j.LoggerFactory
@ -62,7 +63,7 @@ class KafkaMtbFileSender(
val dummyMtbFile = MtbFile.builder()
.withConsent(
Consent.builder()
.withPatient(request.patientId)
.withPatient(request.patientId.value)
.withStatus(Consent.Status.REJECTED)
.build()
)
@ -94,13 +95,12 @@ class KafkaMtbFileSender(
}
private fun key(request: MtbFileSender.MtbFileRequest): String {
return "{\"pid\": \"${request.mtbFile.patient.id}\", " +
"\"eid\": \"${request.mtbFile.episode.id}\"}"
return "{\"pid\": \"${request.mtbFile.patient.id}\"}"
}
private fun key(request: MtbFileSender.DeleteRequest): String {
return "{\"pid\": \"${request.patientId}\"}"
return "{\"pid\": \"${request.patientId.value}\"}"
}
data class Data(val requestId: String, val content: MtbFile)
data class Data(val requestId: RequestId, val content: MtbFile)
}

View File

@ -20,6 +20,8 @@
package dev.dnpm.etl.processor.output
import de.ukw.ccc.bwhc.dto.MtbFile
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.RequestId
import dev.dnpm.etl.processor.monitoring.RequestStatus
import org.springframework.http.HttpStatusCode
@ -32,9 +34,9 @@ interface MtbFileSender {
data class Response(val status: RequestStatus, val body: String = "")
data class MtbFileRequest(val requestId: String, val mtbFile: MtbFile)
data class MtbFileRequest(val requestId: RequestId, val mtbFile: MtbFile)
data class DeleteRequest(val requestId: String, val patientId: String)
data class DeleteRequest(val requestId: RequestId, val patientId: PatientPseudonym)
}

View File

@ -0,0 +1,51 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.output
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.config.RestTargetProperties
import dev.dnpm.etl.processor.monitoring.ReportService
import org.springframework.retry.support.RetryTemplate
import org.springframework.web.client.RestTemplate
import org.springframework.web.util.UriComponentsBuilder
class RestBwhcMtbFileSender(
restTemplate: RestTemplate,
private val restTargetProperties: RestTargetProperties,
retryTemplate: RetryTemplate,
reportService: ReportService,
) : RestMtbFileSender(restTemplate, restTargetProperties, retryTemplate, reportService) {
override fun sendUrl(): String {
return UriComponentsBuilder
.fromUriString(restTargetProperties.uri.toString())
.pathSegment("MTBFile")
.toUriString()
}
override fun deleteUrl(patientId: PatientPseudonym): String {
return UriComponentsBuilder
.fromUriString(restTargetProperties.uri.toString())
.pathSegment("Patient")
.pathSegment(patientId.value)
.toUriString()
}
}

View File

@ -0,0 +1,55 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.output
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.config.RestTargetProperties
import dev.dnpm.etl.processor.monitoring.ReportService
import org.springframework.retry.support.RetryTemplate
import org.springframework.web.client.RestTemplate
import org.springframework.web.util.UriComponentsBuilder
class RestDipMtbFileSender(
restTemplate: RestTemplate,
private val restTargetProperties: RestTargetProperties,
retryTemplate: RetryTemplate,
reportService: ReportService
) : RestMtbFileSender(restTemplate, restTargetProperties, retryTemplate, reportService) {
override fun sendUrl(): String {
return UriComponentsBuilder
.fromUriString(restTargetProperties.uri.toString())
.pathSegment("mtb")
.pathSegment("etl")
.pathSegment("patient-record")
.toUriString()
}
override fun deleteUrl(patientId: PatientPseudonym): String {
return UriComponentsBuilder
.fromUriString(restTargetProperties.uri.toString())
.pathSegment("mtb")
.pathSegment("etl")
.pathSegment("patient")
.pathSegment(patientId.value)
.toUriString()
}
}

View File

@ -21,48 +21,57 @@ package dev.dnpm.etl.processor.output
import dev.dnpm.etl.processor.config.RestTargetProperties
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.monitoring.ReportService
import dev.dnpm.etl.processor.monitoring.asRequestStatus
import org.slf4j.LoggerFactory
import org.springframework.http.HttpEntity
import org.springframework.http.HttpHeaders
import org.springframework.http.MediaType
import org.springframework.retry.support.RetryTemplate
import org.springframework.web.client.RestClientException
import org.springframework.web.client.RestClientResponseException
import org.springframework.web.client.RestTemplate
class RestMtbFileSender(
abstract class RestMtbFileSender(
private val restTemplate: RestTemplate,
private val restTargetProperties: RestTargetProperties,
private val retryTemplate: RetryTemplate
private val retryTemplate: RetryTemplate,
private val reportService: ReportService
) : MtbFileSender {
private val logger = LoggerFactory.getLogger(RestMtbFileSender::class.java)
abstract fun sendUrl(): String
abstract fun deleteUrl(patientId: PatientPseudonym): String
override fun send(request: MtbFileSender.MtbFileRequest): MtbFileSender.Response {
try {
return retryTemplate.execute<MtbFileSender.Response, Exception> {
val headers = HttpHeaders()
headers.contentType = MediaType.APPLICATION_JSON
val headers = getHttpHeaders()
val entityReq = HttpEntity(request.mtbFile, headers)
val response = restTemplate.postForEntity(
"${restTargetProperties.uri}/MTBFile",
sendUrl(),
entityReq,
String::class.java
)
if (!response.statusCode.is2xxSuccessful) {
logger.warn("Error sending to remote system: {}", response.body)
return@execute MtbFileSender.Response(
response.statusCode.asRequestStatus(),
reportService.deserialize(response.body).asRequestStatus(),
"Status-Code: ${response.statusCode.value()}"
)
}
logger.debug("Sent file via RestMtbFileSender")
return@execute MtbFileSender.Response(response.statusCode.asRequestStatus(), response.body.orEmpty())
return@execute MtbFileSender.Response(reportService.deserialize(response.body).asRequestStatus(), response.body.orEmpty())
}
} catch (e: IllegalArgumentException) {
logger.error("Not a valid URI to export to: '{}'", restTargetProperties.uri!!)
} catch (e: RestClientException) {
} catch (e: RestClientResponseException) {
logger.info(restTargetProperties.uri!!.toString())
logger.error("Cannot send data to remote system", e)
logger.error("Request data not accepted by remote system", e)
return MtbFileSender.Response(reportService.deserialize(e.responseBodyAsString).asRequestStatus(), e.responseBodyAsString)
}
return MtbFileSender.Response(RequestStatus.ERROR, "Sonstiger Fehler bei der Übertragung")
}
@ -70,11 +79,10 @@ class RestMtbFileSender(
override fun send(request: MtbFileSender.DeleteRequest): MtbFileSender.Response {
try {
return retryTemplate.execute<MtbFileSender.Response, Exception> {
val headers = HttpHeaders()
headers.contentType = MediaType.APPLICATION_JSON
val headers = getHttpHeaders()
val entityReq = HttpEntity(null, headers)
restTemplate.delete(
"${restTargetProperties.uri}/Patient/${request.patientId}",
deleteUrl(request.patientId),
entityReq,
String::class.java
)
@ -94,4 +102,18 @@ class RestMtbFileSender(
return this.restTargetProperties.uri.orEmpty()
}
private fun getHttpHeaders(): HttpHeaders {
val username = restTargetProperties.username
val password = restTargetProperties.password
val headers = HttpHeaders()
headers.contentType = MediaType.APPLICATION_JSON
if (username.isNullOrBlank() || password.isNullOrBlank()) {
return headers
}
headers.setBasicAuth(username, password)
return headers
}
}

View File

@ -19,6 +19,8 @@
package dev.dnpm.etl.processor.pseudonym
import dev.dnpm.etl.processor.PatientId
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.config.PseudonymizeConfigProperties
class PseudonymizeService(
@ -26,11 +28,15 @@ class PseudonymizeService(
private val configProperties: PseudonymizeConfigProperties
) {
fun patientPseudonym(patientId: String): String {
fun patientPseudonym(patientId: PatientId): PatientPseudonym {
return when (generator) {
is GpasPseudonymGenerator -> generator.generate(patientId)
else -> "${configProperties.prefix}_${generator.generate(patientId)}"
is GpasPseudonymGenerator -> PatientPseudonym(generator.generate(patientId.value))
else -> PatientPseudonym("${configProperties.prefix}_${generator.generate(patientId.value)}")
}
}
fun prefix(): String {
return configProperties.prefix
}
}

View File

@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2023 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@ -20,34 +20,207 @@
package dev.dnpm.etl.processor.pseudonym
import de.ukw.ccc.bwhc.dto.MtbFile
import dev.dnpm.etl.processor.PatientId
import org.apache.commons.codec.digest.DigestUtils
/** Replaces patient ID with generated patient pseudonym
*
* @param pseudonymizeService The pseudonymizeService to be used
*
* @return The MTB file containing patient pseudonymes
*/
infix fun MtbFile.pseudonymizeWith(pseudonymizeService: PseudonymizeService) {
val patientPseudonym = pseudonymizeService.patientPseudonym(this.patient.id)
val patientPseudonym = pseudonymizeService.patientPseudonym(PatientId(this.patient.id)).value
this.episode.patient = patientPseudonym
this.carePlans.forEach { it.patient = patientPseudonym }
this.episode?.patient = patientPseudonym
this.carePlans?.forEach { it.patient = patientPseudonym }
this.patient.id = patientPseudonym
this.claims.forEach { it.patient = patientPseudonym }
this.consent.patient = patientPseudonym
this.claimResponses.forEach { it.patient = patientPseudonym }
this.diagnoses.forEach { it.patient = patientPseudonym }
this.ecogStatus.forEach { it.patient = patientPseudonym }
this.familyMemberDiagnoses.forEach { it.patient = patientPseudonym }
this.geneticCounsellingRequests.forEach { it.patient = patientPseudonym }
this.histologyReevaluationRequests.forEach { it.patient = patientPseudonym }
this.histologyReports.forEach {
this.claims?.forEach { it.patient = patientPseudonym }
this.consent?.patient = patientPseudonym
this.claimResponses?.forEach { it.patient = patientPseudonym }
this.diagnoses?.forEach { it.patient = patientPseudonym }
this.ecogStatus?.forEach { it.patient = patientPseudonym }
this.familyMemberDiagnoses?.forEach { it.patient = patientPseudonym }
this.geneticCounsellingRequests?.forEach { it.patient = patientPseudonym }
this.histologyReevaluationRequests?.forEach { it.patient = patientPseudonym }
this.histologyReports?.forEach {
it.patient = patientPseudonym
it.tumorMorphology.patient = patientPseudonym
it.tumorMorphology?.patient = patientPseudonym
}
this.lastGuidelineTherapies?.forEach { it.patient = patientPseudonym }
this.molecularPathologyFindings?.forEach { it.patient = patientPseudonym }
this.molecularTherapies?.forEach { molecularTherapy -> molecularTherapy.history.forEach { it.patient = patientPseudonym } }
this.ngsReports?.forEach { it.patient = patientPseudonym }
this.previousGuidelineTherapies?.forEach { it.patient = patientPseudonym }
this.rebiopsyRequests?.forEach { it.patient = patientPseudonym }
this.recommendations?.forEach { it.patient = patientPseudonym }
this.responses?.forEach { it.patient = patientPseudonym }
this.studyInclusionRequests?.forEach { it.patient = patientPseudonym }
this.specimens?.forEach { it.patient = patientPseudonym }
}
/**
* Creates new hash of content IDs with given prefix except for patient IDs
*
* @param pseudonymizeService The pseudonymizeService to be used
*
* @return The MTB file containing rehashed content IDs
*/
infix fun MtbFile.anonymizeContentWith(pseudonymizeService: PseudonymizeService) {
val prefix = pseudonymizeService.prefix()
fun anonymize(id: String): String {
val hash = DigestUtils.sha256Hex("$prefix-$id").substring(0, 41).lowercase()
return "$prefix$hash"
}
this.episode?.apply {
id = id?.let {
anonymize(it)
}
}
this.carePlans?.onEach { carePlan ->
carePlan?.apply {
id = id?.let { anonymize(it) }
diagnosis = diagnosis?.let { anonymize(it) }
geneticCounsellingRequest = geneticCounsellingRequest?.let { anonymize(it) }
rebiopsyRequests = rebiopsyRequests.map { it?.let { anonymize(it) } }
recommendations = recommendations.map { it?.let { anonymize(it) } }
studyInclusionRequests = studyInclusionRequests.map { it?.let { anonymize(it) } }
}
}
this.claims?.onEach { claim ->
claim?.apply {
id = id?.let { anonymize(it) }
therapy = therapy?.let { anonymize(it) }
}
}
this.claimResponses?.onEach { claimResponse ->
claimResponse?.apply {
id = id?.let { anonymize(it) }
claim = claim?.let { anonymize(it) }
}
}
this.consent?.apply {
id = id?.let { anonymize(it) }
}
this.diagnoses?.onEach { diagnosis ->
diagnosis?.apply {
id = id?.let { anonymize(it) }
histologyResults = histologyResults?.map { it?.let { anonymize(it) } }
}
}
this.ecogStatus?.onEach { ecogStatus ->
ecogStatus?.apply {
id = id?.let { anonymize(it) }
}
}
this.familyMemberDiagnoses?.onEach { familyMemberDiagnosis ->
familyMemberDiagnosis?.apply {
id = id?.let { anonymize(it) }
}
}
this.geneticCounsellingRequests?.onEach { geneticCounsellingRequest ->
geneticCounsellingRequest?.apply {
id = id?.let { anonymize(it) }
}
}
this.histologyReevaluationRequests?.onEach { histologyReevaluationRequest ->
histologyReevaluationRequest?.apply {
id = id?.let { anonymize(it) }
specimen = specimen?.let { anonymize(it) }
}
}
this.histologyReports?.onEach { histologyReport ->
histologyReport?.apply {
id = id?.let { anonymize(it) }
specimen = specimen?.let { anonymize(it) }
tumorMorphology?.apply {
id = id?.let { anonymize(it) }
specimen = specimen?.let { anonymize(it) }
}
tumorCellContent?.apply {
id = id?.let { anonymize(it) }
specimen = specimen?.let { anonymize(it) }
}
}
}
this.lastGuidelineTherapies?.onEach { lastGuidelineTherapy ->
lastGuidelineTherapy?.apply {
id = id?.let { anonymize(it) }
diagnosis = diagnosis?.let { anonymize(it) }
}
}
this.molecularPathologyFindings?.onEach { molecularPathologyFinding ->
molecularPathologyFinding?.apply {
id = id?.let { anonymize(it) }
specimen = specimen?.let { anonymize(it) }
}
}
this.molecularTherapies?.onEach { molecularTherapy ->
molecularTherapy?.apply {
history?.onEach { history ->
history?.apply {
id = id?.let { anonymize(it) }
basedOn = basedOn?.let { anonymize(it) }
}
}
}
}
this.ngsReports?.onEach { ngsReport ->
ngsReport?.apply {
id = id?.let { anonymize(it) }
specimen = specimen?.let { anonymize(it) }
tumorCellContent?.apply {
id = id?.let { anonymize(it) }
specimen = specimen?.let { anonymize(it) }
}
simpleVariants?.onEach { simpleVariant ->
simpleVariant?.apply {
id = id?.let { anonymize(it) }
}
}
}
}
this.previousGuidelineTherapies?.onEach { previousGuidelineTherapy ->
previousGuidelineTherapy?.apply {
id = id?.let { anonymize(it) }
diagnosis = diagnosis?.let { anonymize(it) }
medication.forEach { medication ->
medication?.apply {
id = id?.let { anonymize(it) }
}
}
}
}
this.rebiopsyRequests?.onEach { rebiopsyRequest ->
rebiopsyRequest?.apply {
id = id?.let { anonymize(it) }
specimen = specimen?.let { anonymize(it) }
}
}
this.recommendations?.onEach { recommendation ->
recommendation?.apply {
id = id?.let { anonymize(it) }
diagnosis = diagnosis?.let { anonymize(it) }
ngsReport = ngsReport?.let { anonymize(it) }
}
}
this.responses?.onEach { response ->
response?.apply {
id = id?.let { anonymize(it) }
therapy = therapy?.let { anonymize(it) }
}
}
this.studyInclusionRequests?.onEach { studyInclusionRequest ->
studyInclusionRequest?.apply {
id = id?.let { anonymize(it) }
reason = reason?.let { anonymize(it) }
}
}
this.specimens?.onEach { specimen ->
specimen?.apply {
id = id?.let { anonymize(it) }
}
}
this.lastGuidelineTherapies.forEach { it.patient = patientPseudonym }
this.molecularPathologyFindings.forEach { it.patient = patientPseudonym }
this.molecularTherapies.forEach { molecularTherapy -> molecularTherapy.history.forEach { it.patient = patientPseudonym } }
this.ngsReports.forEach { it.patient = patientPseudonym }
this.previousGuidelineTherapies.forEach { it.patient = patientPseudonym }
this.rebiopsyRequests.forEach { it.patient = patientPseudonym }
this.recommendations.forEach { it.patient = patientPseudonym }
this.recommendations.forEach { it.patient = patientPseudonym }
this.responses.forEach { it.patient = patientPseudonym }
this.studyInclusionRequests.forEach { it.patient = patientPseudonym }
this.specimens.forEach { it.patient = patientPseudonym }
}

View File

@ -17,7 +17,7 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.services
package dev.dnpm.etl.processor.security
import jakarta.annotation.PostConstruct
import org.springframework.data.annotation.Id

View File

@ -17,11 +17,8 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.services
package dev.dnpm.etl.processor.security
import dev.dnpm.etl.processor.security.Role
import dev.dnpm.etl.processor.security.UserRole
import dev.dnpm.etl.processor.security.UserRoleRepository
import org.springframework.data.repository.findByIdOrNull
import org.springframework.security.core.session.SessionRegistry
import org.springframework.security.oauth2.core.oidc.user.OidcUser

View File

@ -21,6 +21,7 @@ package dev.dnpm.etl.processor.services
import com.fasterxml.jackson.databind.ObjectMapper
import de.ukw.ccc.bwhc.dto.MtbFile
import dev.dnpm.etl.processor.*
import dev.dnpm.etl.processor.config.AppConfigProperties
import dev.dnpm.etl.processor.monitoring.Report
import dev.dnpm.etl.processor.monitoring.Request
@ -28,6 +29,7 @@ import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.monitoring.RequestType
import dev.dnpm.etl.processor.output.MtbFileSender
import dev.dnpm.etl.processor.pseudonym.PseudonymizeService
import dev.dnpm.etl.processor.pseudonym.anonymizeContentWith
import dev.dnpm.etl.processor.pseudonym.pseudonymizeWith
import org.apache.commons.codec.binary.Base32
import org.apache.commons.codec.digest.DigestUtils
@ -48,21 +50,27 @@ class RequestProcessor(
) {
fun processMtbFile(mtbFile: MtbFile) {
val requestId = UUID.randomUUID().toString()
val pid = mtbFile.patient.id
processMtbFile(mtbFile, randomRequestId())
}
fun processMtbFile(mtbFile: MtbFile, requestId: RequestId) {
val pid = PatientId(mtbFile.patient.id)
mtbFile pseudonymizeWith pseudonymizeService
mtbFile anonymizeContentWith pseudonymizeService
val request = MtbFileSender.MtbFileRequest(requestId, transformationService.transform(mtbFile))
val patientPseudonym = PatientPseudonym(request.mtbFile.patient.id)
requestService.save(
Request(
uuid = requestId,
patientId = request.mtbFile.patient.id,
pid = pid,
fingerprint = fingerprint(request.mtbFile),
status = RequestStatus.UNKNOWN,
type = RequestType.MTB_FILE
requestId,
patientPseudonym,
pid,
fingerprint(request.mtbFile),
RequestType.MTB_FILE,
RequestStatus.UNKNOWN
)
)
@ -85,7 +93,7 @@ class RequestProcessor(
Instant.now(),
responseStatus.status,
when (responseStatus.status) {
RequestStatus.WARNING -> Optional.of(responseStatus.body)
RequestStatus.ERROR, RequestStatus.WARNING -> Optional.of(responseStatus.body)
else -> Optional.empty()
}
)
@ -93,29 +101,33 @@ class RequestProcessor(
}
private fun isDuplication(pseudonymizedMtbFile: MtbFile): Boolean {
val patientPseudonym = PatientPseudonym(pseudonymizedMtbFile.patient.id)
val lastMtbFileRequestForPatient =
requestService.lastMtbFileRequestForPatientPseudonym(pseudonymizedMtbFile.patient.id)
val isLastRequestDeletion = requestService.isLastRequestWithKnownStatusDeletion(pseudonymizedMtbFile.patient.id)
requestService.lastMtbFileRequestForPatientPseudonym(patientPseudonym)
val isLastRequestDeletion = requestService.isLastRequestWithKnownStatusDeletion(patientPseudonym)
return null != lastMtbFileRequestForPatient
&& !isLastRequestDeletion
&& lastMtbFileRequestForPatient.fingerprint == fingerprint(pseudonymizedMtbFile)
}
fun processDeletion(patientId: String) {
val requestId = UUID.randomUUID().toString()
fun processDeletion(patientId: PatientId) {
processDeletion(patientId, randomRequestId())
}
fun processDeletion(patientId: PatientId, requestId: RequestId) {
try {
val patientPseudonym = pseudonymizeService.patientPseudonym(patientId)
requestService.save(
Request(
uuid = requestId,
patientId = patientPseudonym,
pid = patientId,
fingerprint = fingerprint(patientPseudonym),
status = RequestStatus.UNKNOWN,
type = RequestType.DELETE
requestId,
patientPseudonym,
patientId,
fingerprint(patientPseudonym.value),
RequestType.DELETE,
RequestStatus.UNKNOWN
)
)
@ -137,9 +149,9 @@ class RequestProcessor(
requestService.save(
Request(
uuid = requestId,
patientId = "???",
patientPseudonym = emptyPatientPseudonym(),
pid = patientId,
fingerprint = "",
fingerprint = Fingerprint.empty(),
status = RequestStatus.ERROR,
type = RequestType.DELETE,
report = Report("Fehler bei der Pseudonymisierung")
@ -148,14 +160,16 @@ class RequestProcessor(
}
}
private fun fingerprint(mtbFile: MtbFile): String {
private fun fingerprint(mtbFile: MtbFile): Fingerprint {
return fingerprint(objectMapper.writeValueAsString(mtbFile))
}
private fun fingerprint(s: String): String {
return Base32().encodeAsString(DigestUtils.sha256(s))
.replace("=", "")
.lowercase()
private fun fingerprint(s: String): Fingerprint {
return Fingerprint(
Base32().encodeAsString(DigestUtils.sha256(s))
.replace("=", "")
.lowercase()
)
}
}

View File

@ -19,11 +19,13 @@
package dev.dnpm.etl.processor.services
import dev.dnpm.etl.processor.monitoring.Request
import dev.dnpm.etl.processor.monitoring.RequestRepository
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.monitoring.RequestType
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.RequestId
import dev.dnpm.etl.processor.monitoring.*
import org.springframework.data.domain.Page
import org.springframework.data.domain.Pageable
import org.springframework.stereotype.Service
import java.util.*
@Service
class RequestService(
@ -32,15 +34,32 @@ class RequestService(
fun save(request: Request) = requestRepository.save(request)
fun allRequestsByPatientPseudonym(patientPseudonym: String) = requestRepository
.findAllByPatientIdOrderByProcessedAtDesc(patientPseudonym)
fun findAll(): Iterable<Request> = requestRepository.findAll()
fun lastMtbFileRequestForPatientPseudonym(patientPseudonym: String) =
fun findAll(pageable: Pageable): Page<Request> = requestRepository.findAll(pageable)
fun findByUuid(uuid: RequestId): Optional<Request> =
requestRepository.findByUuidEquals(uuid)
fun findRequestByPatientId(patientPseudonym: PatientPseudonym, pageable: Pageable): Page<Request> = requestRepository.findRequestByPatientPseudonym(patientPseudonym, pageable)
fun allRequestsByPatientPseudonym(patientPseudonym: PatientPseudonym) = requestRepository
.findAllByPatientPseudonymOrderByProcessedAtDesc(patientPseudonym)
fun lastMtbFileRequestForPatientPseudonym(patientPseudonym: PatientPseudonym) =
Companion.lastMtbFileRequestForPatientPseudonym(allRequestsByPatientPseudonym(patientPseudonym))
fun isLastRequestWithKnownStatusDeletion(patientPseudonym: String) =
fun isLastRequestWithKnownStatusDeletion(patientPseudonym: PatientPseudonym) =
Companion.isLastRequestWithKnownStatusDeletion(allRequestsByPatientPseudonym(patientPseudonym))
fun countStates(): Iterable<CountedState> = requestRepository.countStates()
fun countDeleteStates(): Iterable<CountedState> = requestRepository.countDeleteStates()
fun findPatientUniqueStates(): List<CountedState> = requestRepository.findPatientUniqueStates()
fun findPatientUniqueDeleteStates(): List<CountedState> = requestRepository.findPatientUniqueDeleteStates()
companion object {
fun lastMtbFileRequestForPatientPseudonym(allRequests: List<Request>) = allRequests

View File

@ -19,8 +19,8 @@
package dev.dnpm.etl.processor.services
import dev.dnpm.etl.processor.RequestId
import dev.dnpm.etl.processor.monitoring.Report
import dev.dnpm.etl.processor.monitoring.RequestRepository
import dev.dnpm.etl.processor.monitoring.RequestStatus
import org.slf4j.LoggerFactory
import org.springframework.context.event.EventListener
@ -31,7 +31,7 @@ import java.util.*
@Service
class ResponseProcessor(
private val requestRepository: RequestRepository,
private val requestService: RequestService,
private val statisticsUpdateProducer: Sinks.Many<Any>
) {
@ -39,7 +39,7 @@ class ResponseProcessor(
@EventListener(classes = [ResponseEvent::class])
fun handleResponseEvent(event: ResponseEvent) {
requestRepository.findByUuidEquals(event.requestUuid).ifPresentOrElse({
requestService.findByUuid(event.requestUuid).ifPresentOrElse({
it.processedAt = event.timestamp
it.status = event.status
@ -76,7 +76,7 @@ class ResponseProcessor(
}
}
requestRepository.save(it)
requestService.save(it)
statisticsUpdateProducer.emitNext("", Sinks.EmitFailureHandler.FAIL_FAST)
}, {
@ -87,7 +87,7 @@ class ResponseProcessor(
}
data class ResponseEvent(
val requestUuid: String,
val requestUuid: RequestId,
val timestamp: Instant,
val status: RequestStatus,
val body: Optional<String> = Optional.empty()

View File

@ -22,6 +22,7 @@ package dev.dnpm.etl.processor.services.kafka
import com.fasterxml.jackson.annotation.JsonAlias
import com.fasterxml.jackson.annotation.JsonProperty
import com.fasterxml.jackson.databind.ObjectMapper
import dev.dnpm.etl.processor.RequestId
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.output.asRequestStatus
import dev.dnpm.etl.processor.services.ResponseEvent
@ -47,7 +48,7 @@ class KafkaResponseProcessor(
Optional.empty()
}.ifPresentOrElse({ responseBody ->
val event = ResponseEvent(
responseBody.requestId,
RequestId(responseBody.requestId),
Instant.ofEpochMilli(data.timestamp()),
responseBody.statusCode.asRequestStatus(),
when (responseBody.statusCode.asRequestStatus()) {

View File

@ -0,0 +1,49 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor
import java.util.*
class Fingerprint(val value: String) {
override fun hashCode() = value.hashCode()
override fun equals(other: Any?) = other is Fingerprint && other.value == value
companion object {
fun empty() = Fingerprint("")
}
}
@JvmInline
value class RequestId(val value: String) {
fun isBlank() = value.isBlank()
}
fun randomRequestId() = RequestId(UUID.randomUUID().toString())
@JvmInline
value class PatientId(val value: String)
@JvmInline
value class PatientPseudonym(val value: String)
fun emptyPatientPseudonym() = PatientPseudonym("")

View File

@ -19,15 +19,18 @@
package dev.dnpm.etl.processor.web
import dev.dnpm.etl.processor.monitoring.ConnectionCheckResult
import dev.dnpm.etl.processor.monitoring.ConnectionCheckService
import dev.dnpm.etl.processor.monitoring.GPasConnectionCheckService
import dev.dnpm.etl.processor.monitoring.OutputConnectionCheckService
import dev.dnpm.etl.processor.output.MtbFileSender
import dev.dnpm.etl.processor.pseudonym.Generator
import dev.dnpm.etl.processor.security.Role
import dev.dnpm.etl.processor.security.UserRole
import dev.dnpm.etl.processor.services.Token
import dev.dnpm.etl.processor.services.TokenService
import dev.dnpm.etl.processor.security.Token
import dev.dnpm.etl.processor.security.TokenService
import dev.dnpm.etl.processor.services.TransformationService
import dev.dnpm.etl.processor.services.UserRoleService
import dev.dnpm.etl.processor.security.UserRoleService
import org.springframework.beans.factory.annotation.Qualifier
import org.springframework.http.MediaType
import org.springframework.http.codec.ServerSentEvent
@ -40,22 +43,29 @@ import reactor.core.publisher.Sinks
@Controller
@RequestMapping(path = ["configs"])
class ConfigController(
@Qualifier("configsUpdateProducer")
private val configsUpdateProducer: Sinks.Many<Boolean>,
@Qualifier("connectionCheckUpdateProducer")
private val connectionCheckUpdateProducer: Sinks.Many<ConnectionCheckResult>,
private val transformationService: TransformationService,
private val pseudonymGenerator: Generator,
private val mtbFileSender: MtbFileSender,
private val connectionCheckService: ConnectionCheckService,
private val connectionCheckServices: List<ConnectionCheckService>,
private val tokenService: TokenService?,
private val userRoleService: UserRoleService?
) {
@GetMapping
fun index(model: Model): String {
val outputConnectionAvailable =
connectionCheckServices.filterIsInstance<OutputConnectionCheckService>().firstOrNull()?.connectionAvailable()
val gPasConnectionAvailable =
connectionCheckServices.filterIsInstance<GPasConnectionCheckService>().firstOrNull()?.connectionAvailable()
model.addAttribute("pseudonymGenerator", pseudonymGenerator.javaClass.simpleName)
model.addAttribute("mtbFileSender", mtbFileSender.javaClass.simpleName)
model.addAttribute("mtbFileEndpoint", mtbFileSender.endpoint())
model.addAttribute("connectionAvailable", connectionCheckService.connectionAvailable())
model.addAttribute("outputConnectionAvailable", outputConnectionAvailable)
model.addAttribute("gPasConnectionAvailable", gPasConnectionAvailable)
model.addAttribute("tokensEnabled", tokenService != null)
if (tokenService != null) {
model.addAttribute("tokens", tokenService.findAll())
@ -73,11 +83,14 @@ class ConfigController(
return "configs"
}
@GetMapping(params = ["connectionAvailable"])
fun connectionAvailable(model: Model): String {
@GetMapping(params = ["outputConnectionAvailable"])
fun outputConnectionAvailable(model: Model): String {
val outputConnectionAvailable =
connectionCheckServices.filterIsInstance<OutputConnectionCheckService>().first().connectionAvailable()
model.addAttribute("mtbFileSender", mtbFileSender.javaClass.simpleName)
model.addAttribute("mtbFileEndpoint", mtbFileSender.endpoint())
model.addAttribute("connectionAvailable", connectionCheckService.connectionAvailable())
model.addAttribute("outputConnectionAvailable", outputConnectionAvailable)
if (tokenService != null) {
model.addAttribute("tokensEnabled", true)
model.addAttribute("tokens", tokenService.findAll())
@ -85,7 +98,25 @@ class ConfigController(
model.addAttribute("tokens", listOf<Token>())
}
return "configs/connectionAvailable"
return "configs/outputConnectionAvailable"
}
@GetMapping(params = ["gPasConnectionAvailable"])
fun gPasConnectionAvailable(model: Model): String {
val gPasConnectionAvailable =
connectionCheckServices.filterIsInstance<GPasConnectionCheckService>().firstOrNull()?.connectionAvailable()
model.addAttribute("mtbFileSender", mtbFileSender.javaClass.simpleName)
model.addAttribute("mtbFileEndpoint", mtbFileSender.endpoint())
model.addAttribute("gPasConnectionAvailable", gPasConnectionAvailable)
if (tokenService != null) {
model.addAttribute("tokensEnabled", true)
model.addAttribute("tokens", tokenService.findAll())
} else {
model.addAttribute("tokens", listOf<Token>())
}
return "configs/gPasConnectionAvailable"
}
@PostMapping(path = ["tokens"])
@ -96,10 +127,11 @@ class ConfigController(
} else {
model.addAttribute("tokensEnabled", true)
val result = tokenService.addToken(name)
if (result.isSuccess) {
model.addAttribute("newTokenValue", result.getOrDefault(""))
result.onSuccess {
model.addAttribute("newTokenValue", it)
model.addAttribute("success", true)
} else {
}
result.onFailure {
model.addAttribute("success", false)
}
model.addAttribute("tokens", tokenService.findAll())
@ -151,10 +183,17 @@ class ConfigController(
}
@GetMapping(path = ["events"], produces = [MediaType.TEXT_EVENT_STREAM_VALUE])
@ResponseBody
fun events(): Flux<ServerSentEvent<Any>> {
return configsUpdateProducer.asFlux().map {
return connectionCheckUpdateProducer.asFlux().map {
val event = when (it) {
is ConnectionCheckResult.KafkaConnectionCheckResult -> "output-connection-check"
is ConnectionCheckResult.RestConnectionCheckResult -> "output-connection-check"
is ConnectionCheckResult.GPasConnectionCheckResult -> "gpas-connection-check"
}
ServerSentEvent.builder<Any>()
.event("connection-available").id("none").data("")
.event(event).id("none").data(it)
.build()
}
}

View File

@ -20,9 +20,10 @@
package dev.dnpm.etl.processor.web
import dev.dnpm.etl.processor.NotFoundException
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.RequestId
import dev.dnpm.etl.processor.monitoring.ReportService
import dev.dnpm.etl.processor.monitoring.RequestId
import dev.dnpm.etl.processor.monitoring.RequestRepository
import dev.dnpm.etl.processor.services.RequestService
import org.springframework.data.domain.Pageable
import org.springframework.data.domain.Sort
import org.springframework.data.web.PageableDefault
@ -35,7 +36,7 @@ import org.springframework.web.bind.annotation.RequestMapping
@Controller
@RequestMapping(path = ["/"])
class HomeController(
private val requestRepository: RequestRepository,
private val requestService: RequestService,
private val reportService: ReportService
) {
@ -44,20 +45,20 @@ class HomeController(
@PageableDefault(page = 0, size = 20, sort = ["processedAt"], direction = Sort.Direction.DESC) pageable: Pageable,
model: Model
): String {
val requests = requestRepository.findAll(pageable)
val requests = requestService.findAll(pageable)
model.addAttribute("requests", requests)
return "index"
}
@GetMapping(path = ["patient/{patientId}"])
@GetMapping(path = ["patient/{patientPseudonym}"])
fun byPatient(
@PathVariable patientId: String,
@PathVariable patientPseudonym: PatientPseudonym,
@PageableDefault(page = 0, size = 20, sort = ["processedAt"], direction = Sort.Direction.DESC) pageable: Pageable,
model: Model
): String {
val requests = requestRepository.findRequestByPatientId(patientId, pageable)
model.addAttribute("patientId", patientId)
val requests = requestService.findRequestByPatientId(patientPseudonym, pageable)
model.addAttribute("patientPseudonym", patientPseudonym.value)
model.addAttribute("requests", requests)
return "index"
@ -65,7 +66,7 @@ class HomeController(
@GetMapping(path = ["/report/{id}"])
fun report(@PathVariable id: RequestId, model: Model): String {
val request = requestRepository.findByUuidEquals(id.toString()).orElse(null) ?: throw NotFoundException()
val request = requestService.findByUuid(id).orElse(null) ?: throw NotFoundException()
model.addAttribute("request", request)
model.addAttribute("issues", reportService.deserialize(request.report?.dataQualityReport))

View File

@ -19,9 +19,9 @@
package dev.dnpm.etl.processor.web
import dev.dnpm.etl.processor.monitoring.RequestRepository
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.monitoring.RequestType
import dev.dnpm.etl.processor.services.RequestService
import org.springframework.beans.factory.annotation.Qualifier
import org.springframework.http.MediaType
import org.springframework.http.codec.ServerSentEvent
@ -41,15 +41,15 @@ import java.time.temporal.ChronoUnit
class StatisticsRestController(
@Qualifier("statisticsUpdateProducer")
private val statisticsUpdateProducer: Sinks.Many<Any>,
private val requestRepository: RequestRepository
private val requestService: RequestService
) {
@GetMapping(path = ["requeststates"])
fun requestStates(@RequestParam(required = false, defaultValue = "false") delete: Boolean): List<NameValue> {
val states = if (delete) {
requestRepository.countDeleteStates()
requestService.countDeleteStates()
} else {
requestRepository.countStates()
requestService.countStates()
}
return states
@ -79,7 +79,7 @@ class StatisticsRestController(
}
val formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.of("Europe/Berlin"))
val data = requestRepository.findAll()
val data = requestService.findAll()
.filter { it.type == requestType }
.filter { it.processedAt.isAfter(Instant.now().minus(30, ChronoUnit.DAYS)) }
.groupBy { formatter.format(it.processedAt) }
@ -115,9 +115,9 @@ class StatisticsRestController(
@GetMapping(path = ["requestpatientstates"])
fun requestPatientStates(@RequestParam(required = false, defaultValue = "false") delete: Boolean): List<NameValue> {
val states = if (delete) {
requestRepository.findPatientUniqueDeleteStates()
requestService.findPatientUniqueDeleteStates()
} else {
requestRepository.findPatientUniqueStates()
requestService.findPatientUniqueStates()
}
return states.map {
@ -134,7 +134,6 @@ class StatisticsRestController(
@GetMapping(path = ["events"], produces = [MediaType.TEXT_EVENT_STREAM_VALUE])
fun updater(): Flux<ServerSentEvent<Any>> {
return statisticsUpdateProducer.asFlux().flatMap {
println(it)
Flux.fromIterable(
listOf(
ServerSentEvent.builder<Any>()

View File

@ -3,17 +3,34 @@ spring:
compose:
file: ./dev-compose.yml
security:
oauth2:
client:
registration:
custom:
client-name: App-Dev
client-id: app-dev
client-secret: very-secret-ae3f7a-5a9f-1190
scope:
- openid
provider:
custom:
issuer-uri: https://dnpm.dev/auth/realms/intern
user-name-attribute: name
app:
#rest:
# uri: http://localhost:9000/bwhc/etl/api
kafka:
topic: test
response-topic: test_response
servers: localhost:9094
#security:
# admin-user: admin
# admin-password: "{noop}very-secret"
rest:
uri: http://localhost:9000/bwhc/etl/api
#kafka:
# topic: test
# response-topic: test_response
# servers: localhost:9094
security:
admin-user: admin
admin-password: "{noop}very-secret"
enable-oidc: "true"
server:
port: 8000

View File

@ -0,0 +1 @@
ALTER TABLE request RENAME COLUMN patient_id TO patient_pseudonym;

View File

@ -0,0 +1 @@
ALTER TABLE request RENAME COLUMN patient_id TO patient_pseudonym;

View File

@ -22,6 +22,10 @@
--bg-gray-op: rgba(112, 128, 144, .35);
}
* {
font-family: sans-serif;
}
html {
background: linear-gradient(-5deg, var(--bg-blue-op), transparent 10em);
min-height: 100vh;
@ -30,7 +34,6 @@ html {
body {
margin: 0 0 5em 0;
font-family: sans-serif;
font-size: .8rem;
color: var(--text);
@ -257,6 +260,10 @@ form.samplecode-input input:focus-visible {
display: block;
}
.userrole-form {
display: inline-block;
}
.userrole-form form {
margin: 0;
padding: 0;
@ -321,6 +328,15 @@ table {
font-family: sans-serif;
}
table.config-table td:first-child {
width: 24em;
min-width: fit-content;
}
table.config-table td > button:last-of-type {
float: right;
}
.border > table {
padding: 0;
border: none;
@ -490,7 +506,7 @@ td.clipboard.clipped {
.btn:active,
.btn:hover {
filter: drop-shadow(1px 1px 1px gray) var(--dark);
filter: drop-shadow(0px 1px 1px var(--bg-gray)) var(--dark);
}
.btn:active {
@ -555,15 +571,24 @@ input.inline:focus-visible {
font-weight: bold;
}
.chart {
width: calc(100% - 2.4rem - 4px);
height: 320px;
display: inline-block;
.charts {
display: grid;
grid-gap: 1em;
grid-template:
"a b" 28em
"c c" 28em / 1fr 1fr;
}
.chart-50pc {
width: calc(50% - 2.4rem - 4px);
.charts > .grid-left {
grid-area: a;
}
.charts > .grid-right {
grid-area: b;
}
.charts > .grid-full {
grid-area: c;
}
.connection-display {
@ -571,7 +596,7 @@ input.inline:focus-visible {
grid-template-columns: 10em 16em 10em;
place-items: center;
width: fit-content;
margin: 1em 0;
margin: 1em auto;
}
.connection-display > * {
@ -597,6 +622,10 @@ input.inline:focus-visible {
text-align: center;
}
.notification.info {
color: var(--bg-blue);
}
.notification.success {
color: var(--bg-green);
}
@ -609,6 +638,34 @@ input.inline:focus-visible {
color: var(--bg-red);
}
.tab {
padding: 1em;
border: none;
border-radius: 3px 3px 0 0;
cursor: pointer;
transition: all 0.2s;
font-weight: bold;
}
.tab:hover,
.tab.active {
background: var(--bg-gray);
color: white;
}
.tabcontent {
border: 2px solid var(--bg-gray);
border-radius: 0 .5em .5em .5em;
display: none;
padding: 1em;
background: white;
}
.tabcontent.active {
display: block;
}
a.reload {
display: none;
position: absolute;
@ -640,4 +697,14 @@ a.reload {
.no-token {
padding: 1em;
background: var(--bg-red-op);
}
.issue-message {
font-family: monospace;
font-weight: bolder;
}
.issue-path {
font-family: monospace;
line-height: 1rem;
}

View File

@ -10,90 +10,116 @@
<main>
<h1>Konfiguration</h1>
<section>
<h2>🔧 Allgemeine Konfiguration</h2>
<table>
<thead>
<tr>
<th>Name</th>
<th>Wert</th>
</tr>
</thead>
<tbody>
<div class="tabs">
<button class="tab active" onclick="selectTab(this, 'common');">Allgemeine Informationen</button>
<button class="tab" onclick="selectTab(this, 'security');">Sicherheit</button>
<button class="tab" onclick="selectTab(this, 'transformation');">Transformationen</button>
</div>
<div id="common" class="tabcontent active">
<section>
<h2>🔧 Allgemeine Konfiguration</h2>
<table class="config-table">
<thead>
<tr>
<td>Pseudonym erzeugt über</td>
<td>[[ ${pseudonymGenerator} ]]</td>
<th>Name</th>
<th>Wert</th>
</tr>
</thead>
<tbody>
<tr>
<td>Pseudonym erzeugt über</td>
<td>[[ ${pseudonymGenerator} ]]</td>
</tr>
<tr>
<td>MTBFile-Sender</td>
<td>[[ ${mtbFileSender} ]]</td>
</tr>
<tr>
<td th:if="${mtbFileSender.startsWith('Rest')}">REST-Endpunkt</td>
<td th:if="${mtbFileSender.startsWith('Kafka')}">Kafka-Broker und Topics</td>
<td>[[ ${mtbFileEndpoint} ]]</td>
</tr>
</tbody>
</table>
</section>
<section hx-ext="sse" th:sse-connect="@{/configs/events}">
<div th:insert="~{configs/gPasConnectionAvailable.html}" th:hx-get="@{/configs?gPasConnectionAvailable}" hx-trigger="sse:gpas-connection-check">
</div>
</section>
<section hx-ext="sse" th:sse-connect="@{/configs/events}">
<div th:insert="~{configs/outputConnectionAvailable.html}" th:hx-get="@{/configs?outputConnectionAvailable}" hx-trigger="sse:output-connection-check">
</div>
</section>
</div>
<div id="security" class="tabcontent">
<section th:insert="~{configs/tokens.html}">
</section>
<section th:insert="~{configs/userroles.html}">
</section>
</div>
<div id="transformation" class="tabcontent">
<section>
<h2><span th:if="${not transformations.isEmpty()}"></span><span th:if="${transformations.isEmpty()}"></span> Transformationen</h2>
<h3>Syntax</h3>
Hier einige Beispiele zum Syntax des JSON-Path
<ul>
<li style="padding: 0.6rem 0;"><span class="bg-path">diagnoses[*].icdO3T.version</span>: Ersetze die ICD-O3T-Version in allen Diagnosen, z.B. zur Version der deutschen Übersetzung</li>
<li style="padding: 0.6rem 0;"><span class="bg-path">patient.gender</span>: Ersetze das Geschlecht des Patienten, z.B. in das von bwHC verlangte Format</li>
</ul>
<h3>Konfigurierte Transformationen</h3>
<th:block th:if="${transformations.isEmpty()}">
<p>
Keine konfigurierten Transformationen.
</p>
</th:block>
<th:block th:if="${not transformations.isEmpty()}">
<p>
Hier sehen Sie eine Übersicht der konfigurierten Transformationen.
</p>
<table class="config-table">
<thead>
<tr>
<td>MTBFile-Sender</td>
<td>[[ ${mtbFileSender} ]]</td>
<th>JSON-Path</th>
<th>Transformation von &rArr; nach</th>
</tr>
<tr>
<td th:if="${mtbFileSender.startsWith('Rest')}">REST-Endpunkt</td>
<td th:if="${mtbFileSender.startsWith('Kafka')}">Kafka-Broker und Topics</td>
<td>[[ ${mtbFileEndpoint} ]]</td>
</thead>
<tbody>
<tr th:each="transformation : ${transformations}">
<td>
<span class="bg-path" title="Ersetze Wert(e) an dieser Stelle im MTB-File">[[ ${transformation.path} ]]</span>
</td>
<td>
<span class="bg-from" title="Ersetze immer dann, wenn dieser Wert enthalten ist">[[ ${transformation.existingValue} ]]</span>
<strong>&rArr;</strong>
<span class="bg-to" title="Ersetze durch diesen Wert">[[ ${transformation.newValue} ]]</span>
</td>
</tr>
</tbody>
</table>
</section>
<section th:insert="~{configs/tokens.html}">
</section>
<section th:insert="~{configs/userroles.html}">
</section>
<section hx-ext="sse" th:sse-connect="@{/configs/events}">
<div th:insert="~{configs/connectionAvailable.html}" th:hx-get="@{/configs?connectionAvailable}" hx-trigger="sse:connection-available">
</div>
</section>
<section>
<h2><span th:if="${not transformations.isEmpty()}"></span><span th:if="${transformations.isEmpty()}"></span> Transformationen</h2>
<h3>Syntax</h3>
Hier einige Beispiele zum Syntax des JSON-Path
<ul>
<li style="padding: 0.6rem 0;"><span class="bg-path">diagnoses[*].icdO3T.version</span>: Ersetze die ICD-O3T-Version in allen Diagnosen, z.B. zur Version der deutschen Übersetzung</li>
<li style="padding: 0.6rem 0;"><span class="bg-path">patient.gender</span>: Ersetze das Geschlecht des Patienten, z.B. in das von bwHC verlangte Format</li>
</ul>
<h3>Konfigurierte Transformationen</h3>
<th:block th:if="${transformations.isEmpty()}">
<p>
Keine konfigurierten Transformationen.
</p>
</th:block>
<th:block th:if="${not transformations.isEmpty()}">
<p>
Hier sehen Sie eine Übersicht der konfigurierten Transformationen.
</p>
<table>
<thead>
<tr>
<th>JSON-Path</th>
<th>Transformation von &rArr; nach</th>
</tr>
</thead>
<tbody>
<tr th:each="transformation : ${transformations}">
<td>
<span class="bg-path" title="Ersetze Wert(e) an dieser Stelle im MTB-File">[[ ${transformation.path} ]]</span>
</td>
<td>
<span class="bg-from" title="Ersetze immer dann, wenn dieser Wert enthalten ist">[[ ${transformation.existingValue} ]]</span>
<strong>&rArr;</strong>
<span class="bg-to" title="Ersetze durch diesen Wert">[[ ${transformation.newValue} ]]</span>
</td>
</tr>
</tbody>
</table>
</th:block>
</section>
</tbody>
</table>
</th:block>
</section>
</div>
</main>
<script th:src="@{/scripts.js}"></script>
<script th:src="@{/webjars/htmx.org/dist/htmx.min.js}"></script>
<script th:src="@{/webjars/htmx.org/dist/ext/sse.js}"></script>
<script>
function selectTab(self, elem) {
Array.from(document.getElementsByClassName('tab')).forEach(e => e.className = 'tab');
self.className = 'tab active';
Array.from(document.getElementsByClassName('tabcontent')).forEach(e => e.className = 'tabcontent');
document.getElementById(elem).className = 'tabcontent active';
}
</script>
</body>
</html>

View File

@ -1,16 +0,0 @@
<h2><span th:if="${connectionAvailable}"></span><span th:if="${not(connectionAvailable)}"></span> Verbindung zum bwHC-Backend</h2>
<div>
Verbindung über <code>[[ ${mtbFileSender} ]]</code>. Die Verbindung ist aktuell
<strong th:if="${connectionAvailable}" style="color: green">verfügbar.</strong>
<strong th:if="${not(connectionAvailable)}" style="color: red">nicht verfügbar.</strong>
</div>
<div class="connection-display border">
<img th:src="@{/server.png}" alt="ETL-Processor" />
<span class="connection" th:classappend="${connectionAvailable ? 'available' : ''}"></span>
<img th:if="${mtbFileSender.startsWith('Rest')}" th:src="@{/server.png}" alt="bwHC-Backend" />
<img th:if="${mtbFileSender.startsWith('Kafka')}" th:src="@{/kafka.png}" alt="Kafka-Broker" />
<span>ETL-Processor</span>
<span></span>
<span th:if="${mtbFileSender.startsWith('Rest')}">bwHC-Backend</span>
<span th:if="${mtbFileSender.startsWith('Kafka')}">Kafka-Broker</span>
</div>

View File

@ -0,0 +1,24 @@
<th:block th:if="${gPasConnectionAvailable == null}">
<h2><span>🟦</span> gPAS nicht konfiguriert - Patienten-IDs werden intern anonymisiert</h2>
</th:block>
<th:block th:if="${gPasConnectionAvailable != null}">
<h2><span th:if="${gPasConnectionAvailable.available}"></span><span th:if="${not(gPasConnectionAvailable.available)}"></span> Verbindung zu gPAS</h2>
<div>
Stand: <time style="font-weight: bold" th:datetime="${#temporals.formatISO(gPasConnectionAvailable.timestamp)}" th:text="${#temporals.formatISO(gPasConnectionAvailable.timestamp)}"></time>
&nbsp;|&nbsp;
Letzte Änderung: <time style="font-weight: bold" th:datetime="${#temporals.formatISO(gPasConnectionAvailable.lastChange)}" th:text="${#temporals.formatISO(gPasConnectionAvailable.lastChange)}"></time>
</div>
<div>
<span>Die Verbindung ist aktuell</span>
<strong th:if="${gPasConnectionAvailable.available}" style="color: green">verfügbar.</strong>
<strong th:if="${not(gPasConnectionAvailable.available)}" style="color: red">nicht verfügbar.</strong>
</div>
<div class="connection-display border">
<img th:src="@{/server.png}" alt="ETL-Processor" />
<span class="connection" th:classappend="${gPasConnectionAvailable.available ? 'available' : ''}"></span>
<img th:src="@{/server.png}" alt="gPAS" />
<span>ETL-Processor</span>
<span></span>
<span>gPAS</span>
</div>
</th:block>

View File

@ -0,0 +1,27 @@
<th:block th:if="${outputConnectionAvailable == null}">
<h2><span>🟦</span> Keine Ausgabenkonfiguration</h2>
</th:block>
<th:block th:if="${outputConnectionAvailable != null}">
<h2><span th:if="${outputConnectionAvailable.available}"></span><span th:if="${not(outputConnectionAvailable.available)}"></span> MTB-File Verbindung</h2>
<div>
Stand: <time style="font-weight: bold" th:datetime="${#temporals.formatISO(outputConnectionAvailable.timestamp)}" th:text="${#temporals.formatISO(outputConnectionAvailable.timestamp)}"></time>
&nbsp;|&nbsp;
Letzte Änderung: <time style="font-weight: bold" th:datetime="${#temporals.formatISO(outputConnectionAvailable.lastChange)}" th:text="${#temporals.formatISO(outputConnectionAvailable.lastChange)}"></time>
</div>
<div>
Verbindung über <code>[[ ${mtbFileSender} ]]</code>. Die Verbindung ist aktuell
<strong th:if="${outputConnectionAvailable.available}" style="color: green">verfügbar.</strong>
<strong th:if="${not(outputConnectionAvailable.available)}" style="color: red">nicht verfügbar.</strong>
</div>
<div class="connection-display border">
<img th:src="@{/server.png}" alt="ETL-Processor" />
<span class="connection" th:classappend="${outputConnectionAvailable.available ? 'available' : ''}"></span>
<img th:if="${mtbFileSender.startsWith('Rest')}" th:src="@{/server.png}" alt="bwHC-Backend" />
<img th:if="${mtbFileSender.startsWith('Kafka')}" th:src="@{/kafka.png}" alt="Kafka-Broker" />
<span>ETL-Processor</span>
<span></span>
<span th:if="${mtbFileSender.startsWith('RestBwhc')}">bwHC-Backend</span>
<span th:if="${mtbFileSender.startsWith('RestDip')}">DNPM:DIP-Backend</span>
<span th:if="${mtbFileSender.startsWith('Kafka')}">Kafka-Broker</span>
</div>
</th:block>

View File

@ -7,19 +7,20 @@
<h2><span></span> Tokens</h2>
<div class="border">
<div th:if="${tokens.isEmpty()}">Noch keine Tokens vorhanden.</div>
<table th:if="${not tokens.isEmpty()}">
<table th:if="${not tokens.isEmpty()}" class="config-table">
<thead>
<tr>
<th>Name</th>
<th>Erstellt</th>
<th></th>
</tr>
</thead>
<tbody>
<tr th:each="token : ${tokens}">
<td>[[ ${token.name} ]]</td>
<td><time th:datetime="${token.createdAt}">[[ ${token.createdAt} ]]</time></td>
<td><button class="btn btn-red" th:hx-delete="@{/configs/tokens/{id}(id=${token.id})}" hx-target="#tokens">Löschen</button></td>
<td>
<time th:datetime="${token.createdAt}">[[ ${token.createdAt} ]]</time>
<button class="btn btn-red" th:hx-delete="@{/configs/tokens/{id}(id=${token.id})}" hx-target="#tokens">Löschen</button>
</td>
</tr>
</tbody>
</table>

View File

@ -7,12 +7,11 @@
<h2><span></span> Benutzerberechtigungen</h2>
<div class="border">
<div th:if="${userRoles.isEmpty()}">Noch keine Benutzerberechtigungen vorhanden.</div>
<table th:if="${not userRoles.isEmpty()}">
<table th:if="${not userRoles.isEmpty()}" class="config-table">
<thead>
<tr>
<th>Benutzername</th>
<th>Rolle</th>
<th></th>
</tr>
</thead>
<tbody>
@ -29,8 +28,6 @@
<button class="btn btn-blue" th:disabled="${#authorization.authentication.getName() == userRole.username}">Übernehmen</button>
</form>
</div>
</td>
<td>
<button class="btn btn-red" th:hx-delete="@{/configs/userroles/{id}(id=${userRole.id})}" hx-target="#userroles" th:disabled="${#authorization.authentication.getName() == userRole.username}">Löschen</button>
</td>
</tr>

View File

@ -12,26 +12,30 @@
<h1>Alle Anfragen<a id="reload-notify" class="reload" title="Neue Anfragen" th:href="@{/}"></a></h1>
<div>
<h2 th:if="${patientId != null}">
Betreffend Patienten-Pseudonym <span class="monospace" th:text="${patientId}">***</span>
<a class="btn btn-blue" th:if="${patientId != null}" th:href="@{/}">Alle anzeigen</a>
<h2 th:if="${patientPseudonym != null}">
Betreffend Patienten-Pseudonym <span class="monospace" th:text="${patientPseudonym}">***</span>
<a class="btn btn-blue" th:if="${patientPseudonym != null}" th:href="@{/}">Alle anzeigen</a>
</h2>
</div>
<div class="border">
<div th:if="${patientId == null}" class="page-control">
<div class="border" th:if="${requests.totalElements == 0}">
<div class="notification info">Noch keine Anfragen eingegangen</div>
</div>
<div class="border" th:if="${requests.totalElements > 0}">
<div th:if="${patientPseudonym == null}" class="page-control">
<a id="first-page-link" th:href="@{/(page=${0})}" title="Zum Anfang: Taste W" th:if="${not requests.isFirst()}">&larrb;</a><a th:if="${requests.isFirst()}">&larrb;</a>
<a id="prev-page-link" th:href="@{/(page=${requests.getNumber() - 1})}" title="Seite zurück: Taste A" th:if="${not requests.isFirst()}">&larr;</a><a th:if="${requests.isFirst()}">&larr;</a>
<span>Seite [[ ${requests.getNumber() + 1} ]] von [[ ${requests.getTotalPages()} ]]</span>
<a id="next-page-link" th:href="@{/(page=${requests.getNumber() + 1})}" title="Seite vor: Taste D" th:if="${not requests.isLast()}">&rarr;</a><a th:if="${requests.isLast()}">&rarr;</a>
<a id="last-page-link" th:href="@{/(page=${requests.getTotalPages() - 1})}" title="Zum Ende: Taste S" th:if="${not requests.isLast()}">&rarrb;</a><a th:if="${requests.isLast()}">&rarrb;</a>
</div>
<div th:if="${patientId != null}" class="page-control">
<a id="first-page-link" th:href="@{/patient/{patientId}(patientId=${patientId},page=${0})}" title="Zum Anfang: Taste W" th:if="${not requests.isFirst()}">&larrb;</a><a th:if="${requests.isFirst()}">&larrb;</a>
<a id="prev-page-link" th:href="@{/patient/{patientId}(patientId=${patientId},page=${requests.getNumber() - 1})}" title="Seite zurück: Taste A" th:if="${not requests.isFirst()}">&larr;</a><a th:if="${requests.isFirst()}">&larr;</a>
<div th:if="${patientPseudonym != null}" class="page-control">
<a id="first-page-link" th:href="@{/patient/{patientPseudonym}(patientPseudonym=${patientPseudonym},page=${0})}" title="Zum Anfang: Taste W" th:if="${not requests.isFirst()}">&larrb;</a><a th:if="${requests.isFirst()}">&larrb;</a>
<a id="prev-page-link" th:href="@{/patient/{patientPseudonym}(patientPseudonym=${patientPseudonym},page=${requests.getNumber() - 1})}" title="Seite zurück: Taste A" th:if="${not requests.isFirst()}">&larr;</a><a th:if="${requests.isFirst()}">&larr;</a>
<span>Seite [[ ${requests.getNumber() + 1} ]] von [[ ${requests.getTotalPages()} ]]</span>
<a id="next-page-link" th:href="@{/patient/{patientId}(patientId=${patientId},page=${requests.getNumber() + 1})}" title="Seite vor: Taste D" th:if="${not requests.isLast()}">&rarr;</a><a th:if="${requests.isLast()}">&rarr;</a>
<a id="last-page-link" th:href="@{/patient/{patientId}(patientId=${patientId},page=${requests.getTotalPages() - 1})}" title="Zum Ende: Taste S" th:if="${not requests.isLast()}">&rarrb;</a><a th:if="${requests.isLast()}">&rarrb;</a>
<a id="next-page-link" th:href="@{/patient/{patientPseudonym}(patientPseudonym=${patientPseudonym},page=${requests.getNumber() + 1})}" title="Seite vor: Taste D" th:if="${not requests.isLast()}">&rarr;</a><a th:if="${requests.isLast()}">&rarr;</a>
<a id="last-page-link" th:href="@{/patient/{patientPseudonym}(patientPseudonym=${patientPseudonym},page=${requests.getTotalPages() - 1})}" title="Zum Ende: Taste S" th:if="${not requests.isLast()}">&rarrb;</a><a th:if="${requests.isLast()}">&rarrb;</a>
</div>
<table class="paged">
<thead>
@ -57,11 +61,11 @@
<th:block sec:authorize="not (hasRole('USER') or hasRole('ADMIN'))">[[ ${request.uuid} ]]</th:block>
</td>
<td><time th:datetime="${request.processedAt}">[[ ${request.processedAt} ]]</time></td>
<td class="patient-id" th:if="${patientId != null}" sec:authorize="hasRole('USER') or hasRole('ADMIN')">
[[ ${request.patientId} ]]
<td class="patient-id" th:if="${patientPseudonym != null}" sec:authorize="hasRole('USER') or hasRole('ADMIN')">
[[ ${request.patientPseudonym} ]]
</td>
<td class="patient-id" th:if="${patientId == null}" sec:authorize="hasRole('USER') or hasRole('ADMIN')">
<a th:href="@{/patient/{pid}(pid=${request.patientId})}">[[ ${request.patientId} ]]</a>
<td class="patient-id" th:if="${patientPseudonym == null}" sec:authorize="hasRole('USER') or hasRole('ADMIN')">
<a th:href="@{/patient/{pid}(pid=${request.patientPseudonym})}">[[ ${request.patientPseudonym} ]]</a>
</td>
<td class="patient-id" sec:authorize="not (hasRole('USER') or hasRole('ADMIN'))">***</td>
</tr>

View File

@ -31,7 +31,7 @@
<td th:style="${request.type.value == 'delete'} ? 'color: red;'"><small>[[ ${request.type} ]]</small></td>
<td>[[ ${request.uuid} ]]</td>
<td><time th:datetime="${request.processedAt}">[[ ${request.processedAt} ]]</time></td>
<td class="patient-id" sec:authorize="authenticated">[[ ${request.patientId} ]]</td>
<td class="patient-id" sec:authorize="authenticated">[[ ${request.patientPseudonym} ]]</td>
<td class="patient-id" sec:authorize="not authenticated">***</td>
</tr>
</tbody>
@ -47,7 +47,7 @@
<thead>
<tr>
<th>Schweregrad</th>
<th>Beschreibung</th>
<th>Beschreibung und Pfad</th>
</tr>
</thead>
<tbody>
@ -55,7 +55,12 @@
<td th:if="${issue.severity.value == 'info'}" class="bg-blue"><small>[[ ${issue.severity} ]]</small></td>
<td th:if="${issue.severity.value == 'warning'}" class="bg-yellow"><small>[[ ${issue.severity} ]]</small></td>
<td th:if="${issue.severity.value == 'error'}" class="bg-red"><small>[[ ${issue.severity} ]]</small></td>
<td>[[ ${issue.message} ]]</td>
<td th:if="${issue.severity.value == 'fatal'}" class="bg-red"><small>[[ ${issue.severity} ]]</small></td>
<td>
<div class="issue-message">[[ ${issue.message} ]]</div>
<div class="issue-path" th:if="${issue.path.isPresent()}">[[ ${issue.path.get()} ]]</div>
<div class="issue-path" th:if="${issue.path.isEmpty()}"><i>Keine Angabe</i></div>
</td>
</tr>
</tbody>
</table>

View File

@ -18,11 +18,11 @@
<p>
Anfragen zur Aktualisierung von Patientendaten durch Übermittlung eines MTB-Files.
</p>
<div>
<div id="piechart1" class="chart chart-50pc"></div>
<div id="piechart2" class="chart chart-50pc"></div>
<div class="charts">
<div id="piechart1" class="chart grid-left"></div>
<div id="piechart2" class="chart grid-right"></div>
<div id="barchart" class="chart grid-full"></div>
</div>
<div id="barchart" class="chart"></div>
</section>
<section>
@ -30,11 +30,11 @@
<p>
Anfragen zur Löschung von Patientendaten, wenn kein Consent vorliegt.
</p>
<div>
<div id="piechartdel1" class="chart chart-50pc"></div>
<div id="piechartdel2" class="chart chart-50pc"></div>
<div class="charts">
<div id="piechartdel1" class="chart grid-left"></div>
<div id="piechartdel2" class="chart grid-right"></div>
<div id="barchartdel" class="chart grid-full"></div>
</div>
<div id="barchartdel" class="chart"></div>
</section>
</main>

View File

@ -0,0 +1,20 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor

View File

@ -25,15 +25,19 @@ import de.ukw.ccc.bwhc.dto.MtbFile
import de.ukw.ccc.bwhc.dto.Patient
import dev.dnpm.etl.processor.services.RequestProcessor
import org.apache.kafka.clients.consumer.ConsumerRecord
import org.apache.kafka.common.header.internals.RecordHeader
import org.apache.kafka.common.header.internals.RecordHeaders
import org.apache.kafka.common.record.TimestampType
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.ArgumentMatchers.anyString
import org.mockito.Mock
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.any
import org.mockito.kotlin.anyValueClass
import org.mockito.kotlin.times
import org.mockito.kotlin.verify
import java.util.*
@ExtendWith(MockitoExtension::class)
class KafkaInputListenerTest {
@ -73,7 +77,36 @@ class KafkaInputListenerTest {
kafkaInputListener.onMessage(ConsumerRecord("testtopic", 0, 0, "", this.objectMapper.writeValueAsString(mtbFile)))
verify(requestProcessor, times(1)).processDeletion(anyString())
verify(requestProcessor, times(1)).processDeletion(anyValueClass())
}
@Test
fun shouldProcessMtbFileRequestWithExistingRequestId() {
val mtbFile = MtbFile.builder()
.withPatient(Patient.builder().withId("DUMMY_12345678").build())
.withConsent(Consent.builder().withStatus(Consent.Status.ACTIVE).build())
.build()
val headers = RecordHeaders(listOf(RecordHeader("requestId", UUID.randomUUID().toString().toByteArray())))
kafkaInputListener.onMessage(
ConsumerRecord("testtopic", 0, 0, -1L, TimestampType.NO_TIMESTAMP_TYPE, -1, -1, "", this.objectMapper.writeValueAsString(mtbFile), headers, Optional.empty())
)
verify(requestProcessor, times(1)).processMtbFile(any(), anyValueClass())
}
@Test
fun shouldProcessDeleteRequestWithExistingRequestId() {
val mtbFile = MtbFile.builder()
.withPatient(Patient.builder().withId("DUMMY_12345678").build())
.withConsent(Consent.builder().withStatus(Consent.Status.REJECTED).build())
.build()
val headers = RecordHeaders(listOf(RecordHeader("requestId", UUID.randomUUID().toString().toByteArray())))
kafkaInputListener.onMessage(
ConsumerRecord("testtopic", 0, 0, -1L, TimestampType.NO_TIMESTAMP_TYPE, -1, -1, "", this.objectMapper.writeValueAsString(mtbFile), headers, Optional.empty())
)
verify(requestProcessor, times(1)).processDeletion(anyValueClass(), anyValueClass())
}
}

View File

@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@ -22,8 +22,8 @@ package dev.dnpm.etl.processor.input
import com.fasterxml.jackson.databind.ObjectMapper
import de.ukw.ccc.bwhc.dto.*
import dev.dnpm.etl.processor.services.RequestProcessor
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Nested
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.Mock
@ -31,7 +31,7 @@ import org.mockito.Mockito.times
import org.mockito.Mockito.verify
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.any
import org.mockito.kotlin.argumentCaptor
import org.mockito.kotlin.anyValueClass
import org.springframework.http.MediaType
import org.springframework.test.web.servlet.MockMvc
import org.springframework.test.web.servlet.delete
@ -41,24 +41,122 @@ import org.springframework.test.web.servlet.setup.MockMvcBuilders
@ExtendWith(MockitoExtension::class)
class MtbFileRestControllerTest {
private lateinit var mockMvc: MockMvc
private lateinit var requestProcessor: RequestProcessor
private val objectMapper = ObjectMapper()
@BeforeEach
fun setup(
@Mock requestProcessor: RequestProcessor
) {
this.requestProcessor = requestProcessor
val controller = MtbFileRestController(requestProcessor)
this.mockMvc = MockMvcBuilders.standaloneSetup(controller).build()
@Nested
inner class BwhcRequests {
private lateinit var mockMvc: MockMvc
private lateinit var requestProcessor: RequestProcessor
@BeforeEach
fun setup(
@Mock requestProcessor: RequestProcessor
) {
this.requestProcessor = requestProcessor
val controller = MtbFileRestController(requestProcessor)
this.mockMvc = MockMvcBuilders.standaloneSetup(controller).build()
}
@Test
fun shouldProcessPostRequest() {
mockMvc.post("/mtbfile") {
content = objectMapper.writeValueAsString(bwhcMtbFileContent(Consent.Status.ACTIVE))
contentType = MediaType.APPLICATION_JSON
}.andExpect {
status {
isAccepted()
}
}
verify(requestProcessor, times(1)).processMtbFile(any())
}
@Test
fun shouldProcessPostRequestWithRejectedConsent() {
mockMvc.post("/mtbfile") {
content = objectMapper.writeValueAsString(bwhcMtbFileContent(Consent.Status.REJECTED))
contentType = MediaType.APPLICATION_JSON
}.andExpect {
status {
isAccepted()
}
}
verify(requestProcessor, times(1)).processDeletion(anyValueClass())
}
@Test
fun shouldProcessDeleteRequest() {
mockMvc.delete("/mtbfile/TEST_12345678").andExpect {
status {
isAccepted()
}
}
verify(requestProcessor, times(1)).processDeletion(anyValueClass())
}
}
@Test
fun shouldProcessMtbFilePostRequest() {
val mtbFile = MtbFile.builder()
@Nested
inner class BwhcRequestsWithAlias {
private lateinit var mockMvc: MockMvc
private lateinit var requestProcessor: RequestProcessor
@BeforeEach
fun setup(
@Mock requestProcessor: RequestProcessor
) {
this.requestProcessor = requestProcessor
val controller = MtbFileRestController(requestProcessor)
this.mockMvc = MockMvcBuilders.standaloneSetup(controller).build()
}
@Test
fun shouldProcessPostRequest() {
mockMvc.post("/mtb") {
content = objectMapper.writeValueAsString(bwhcMtbFileContent(Consent.Status.ACTIVE))
contentType = MediaType.APPLICATION_JSON
}.andExpect {
status {
isAccepted()
}
}
verify(requestProcessor, times(1)).processMtbFile(any())
}
@Test
fun shouldProcessPostRequestWithRejectedConsent() {
mockMvc.post("/mtb") {
content = objectMapper.writeValueAsString(bwhcMtbFileContent(Consent.Status.REJECTED))
contentType = MediaType.APPLICATION_JSON
}.andExpect {
status {
isAccepted()
}
}
verify(requestProcessor, times(1)).processDeletion(anyValueClass())
}
@Test
fun shouldProcessDeleteRequest() {
mockMvc.delete("/mtb/TEST_12345678").andExpect {
status {
isAccepted()
}
}
verify(requestProcessor, times(1)).processDeletion(anyValueClass())
}
}
companion object {
fun bwhcMtbFileContent(consentStatus: Consent.Status) = MtbFile.builder()
.withPatient(
Patient.builder()
.withId("TEST_12345678")
@ -69,7 +167,7 @@ class MtbFileRestControllerTest {
.withConsent(
Consent.builder()
.withId("1")
.withStatus(Consent.Status.ACTIVE)
.withStatus(consentStatus)
.withPatient("TEST_12345678")
.build()
)
@ -81,70 +179,5 @@ class MtbFileRestControllerTest {
.build()
)
.build()
mockMvc.post("/mtbfile") {
content = objectMapper.writeValueAsString(mtbFile)
contentType = MediaType.APPLICATION_JSON
}.andExpect {
status {
isAccepted()
}
}
verify(requestProcessor, times(1)).processMtbFile(any())
}
@Test
fun shouldProcessMtbFilePostRequestWithRejectedConsent() {
val mtbFile = MtbFile.builder()
.withPatient(
Patient.builder()
.withId("TEST_12345678")
.withBirthDate("2000-08-08")
.withGender(Patient.Gender.MALE)
.build()
)
.withConsent(
Consent.builder()
.withId("1")
.withStatus(Consent.Status.REJECTED)
.withPatient("TEST_12345678")
.build()
)
.withEpisode(
Episode.builder()
.withId("1")
.withPatient("TEST_12345678")
.withPeriod(PeriodStart("2023-08-08"))
.build()
)
.build()
mockMvc.post("/mtbfile") {
content = objectMapper.writeValueAsString(mtbFile)
contentType = MediaType.APPLICATION_JSON
}.andExpect {
status {
isAccepted()
}
}
val captor = argumentCaptor<String>()
verify(requestProcessor, times(1)).processDeletion(captor.capture())
assertThat(captor.firstValue).isEqualTo("TEST_12345678")
}
@Test
fun shouldProcessMtbFileDeleteRequest() {
mockMvc.delete("/mtbfile/TEST_12345678").andExpect {
status {
isAccepted()
}
}
val captor = argumentCaptor<String>()
verify(requestProcessor, times(1)).processDeletion(captor.capture())
assertThat(captor.firstValue).isEqualTo("TEST_12345678")
}
}
}

View File

@ -21,6 +21,8 @@ package dev.dnpm.etl.processor.output
import com.fasterxml.jackson.databind.ObjectMapper
import de.ukw.ccc.bwhc.dto.*
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.RequestId
import dev.dnpm.etl.processor.config.KafkaProperties
import dev.dnpm.etl.processor.monitoring.RequestStatus
import org.assertj.core.api.Assertions.assertThat
@ -72,7 +74,7 @@ class KafkaMtbFileSenderTest {
completedFuture(SendResult<String, String>(null, null))
}.whenever(kafkaTemplate).send(anyString(), anyString(), anyString())
val response = kafkaMtbFileSender.send(MtbFileSender.MtbFileRequest("TestID", mtbFile(Consent.Status.ACTIVE)))
val response = kafkaMtbFileSender.send(MtbFileSender.MtbFileRequest(TEST_REQUEST_ID, mtbFile(Consent.Status.ACTIVE)))
assertThat(response.status).isEqualTo(testData.requestStatus)
}
@ -86,7 +88,7 @@ class KafkaMtbFileSenderTest {
completedFuture(SendResult<String, String>(null, null))
}.whenever(kafkaTemplate).send(anyString(), anyString(), anyString())
val response = kafkaMtbFileSender.send(MtbFileSender.DeleteRequest("TestID", "PID"))
val response = kafkaMtbFileSender.send(MtbFileSender.DeleteRequest(TEST_REQUEST_ID, TEST_PATIENT_PSEUDONYM))
assertThat(response.status).isEqualTo(testData.requestStatus)
}
@ -96,14 +98,14 @@ class KafkaMtbFileSenderTest {
completedFuture(SendResult<String, String>(null, null))
}.whenever(kafkaTemplate).send(anyString(), anyString(), anyString())
kafkaMtbFileSender.send(MtbFileSender.MtbFileRequest("TestID", mtbFile(Consent.Status.ACTIVE)))
kafkaMtbFileSender.send(MtbFileSender.MtbFileRequest(TEST_REQUEST_ID, mtbFile(Consent.Status.ACTIVE)))
val captor = argumentCaptor<String>()
verify(kafkaTemplate, times(1)).send(anyString(), captor.capture(), captor.capture())
assertThat(captor.firstValue).isNotNull
assertThat(captor.firstValue).isEqualTo("{\"pid\": \"PID\", \"eid\": \"1\"}")
assertThat(captor.firstValue).isEqualTo("{\"pid\": \"PID\"}")
assertThat(captor.secondValue).isNotNull
assertThat(captor.secondValue).isEqualTo(objectMapper.writeValueAsString(kafkaRecordData("TestID", Consent.Status.ACTIVE)))
assertThat(captor.secondValue).isEqualTo(objectMapper.writeValueAsString(kafkaRecordData(TEST_REQUEST_ID, Consent.Status.ACTIVE)))
}
@Test
@ -112,14 +114,14 @@ class KafkaMtbFileSenderTest {
completedFuture(SendResult<String, String>(null, null))
}.whenever(kafkaTemplate).send(anyString(), anyString(), anyString())
kafkaMtbFileSender.send(MtbFileSender.DeleteRequest("TestID", "PID"))
kafkaMtbFileSender.send(MtbFileSender.DeleteRequest(TEST_REQUEST_ID, TEST_PATIENT_PSEUDONYM))
val captor = argumentCaptor<String>()
verify(kafkaTemplate, times(1)).send(anyString(), captor.capture(), captor.capture())
assertThat(captor.firstValue).isNotNull
assertThat(captor.firstValue).isEqualTo("{\"pid\": \"PID\"}")
assertThat(captor.secondValue).isNotNull
assertThat(captor.secondValue).isEqualTo(objectMapper.writeValueAsString(kafkaRecordData("TestID", Consent.Status.REJECTED)))
assertThat(captor.secondValue).isEqualTo(objectMapper.writeValueAsString(kafkaRecordData(TEST_REQUEST_ID, Consent.Status.REJECTED)))
}
@ParameterizedTest
@ -136,7 +138,7 @@ class KafkaMtbFileSenderTest {
completedFuture(SendResult<String, String>(null, null))
}.whenever(kafkaTemplate).send(anyString(), anyString(), anyString())
kafkaMtbFileSender.send(MtbFileSender.MtbFileRequest("TestID", mtbFile(Consent.Status.ACTIVE)))
kafkaMtbFileSender.send(MtbFileSender.MtbFileRequest(TEST_REQUEST_ID, mtbFile(Consent.Status.ACTIVE)))
val expectedCount = when (testData.exception) {
// OK - No Retry
@ -162,7 +164,7 @@ class KafkaMtbFileSenderTest {
completedFuture(SendResult<String, String>(null, null))
}.whenever(kafkaTemplate).send(anyString(), anyString(), anyString())
kafkaMtbFileSender.send(MtbFileSender.DeleteRequest("TestID", "PID"))
kafkaMtbFileSender.send(MtbFileSender.DeleteRequest(TEST_REQUEST_ID, TEST_PATIENT_PSEUDONYM))
val expectedCount = when (testData.exception) {
// OK - No Retry
@ -175,6 +177,9 @@ class KafkaMtbFileSenderTest {
}
companion object {
val TEST_REQUEST_ID = RequestId("TestId")
val TEST_PATIENT_PSEUDONYM = PatientPseudonym("PID")
fun mtbFile(consentStatus: Consent.Status): MtbFile {
return if (consentStatus == Consent.Status.ACTIVE) {
MtbFile.builder()
@ -210,7 +215,7 @@ class KafkaMtbFileSenderTest {
}.build()
}
fun kafkaRecordData(requestId: String, consentStatus: Consent.Status): KafkaMtbFileSender.Data {
fun kafkaRecordData(requestId: RequestId, consentStatus: Consent.Status): KafkaMtbFileSender.Data {
return KafkaMtbFileSender.Data(requestId, mtbFile(consentStatus))
}

View File

@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@ -19,12 +19,18 @@
package dev.dnpm.etl.processor.output
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.kotlin.KotlinModule
import de.ukw.ccc.bwhc.dto.*
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.RequestId
import dev.dnpm.etl.processor.config.RestTargetProperties
import dev.dnpm.etl.processor.monitoring.ReportService
import dev.dnpm.etl.processor.monitoring.RequestStatus
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.Arguments
import org.junit.jupiter.params.provider.MethodSource
import org.springframework.http.HttpMethod
import org.springframework.http.HttpStatus
@ -37,34 +43,37 @@ import org.springframework.test.web.client.match.MockRestRequestMatchers.request
import org.springframework.test.web.client.response.MockRestResponseCreators.withStatus
import org.springframework.web.client.RestTemplate
class RestMtbFileSenderTest {
class RestBwhcMtbFileSenderTest {
private lateinit var mockRestServiceServer: MockRestServiceServer
private lateinit var restMtbFileSender: RestMtbFileSender
private var reportService = ReportService(ObjectMapper().registerModule(KotlinModule.Builder().build()))
@BeforeEach
fun setup() {
val restTemplate = RestTemplate()
val restTargetProperties = RestTargetProperties("http://localhost:9000/mtbfile")
val restTargetProperties = RestTargetProperties("http://localhost:9000/mtbfile", null, null)
val retryTemplate = RetryTemplateBuilder().customPolicy(SimpleRetryPolicy(1)).build()
this.mockRestServiceServer = MockRestServiceServer.createServer(restTemplate)
this.restMtbFileSender = RestMtbFileSender(restTemplate, restTargetProperties, retryTemplate)
this.restMtbFileSender =
RestBwhcMtbFileSender(restTemplate, restTargetProperties, retryTemplate, reportService)
}
@ParameterizedTest
@MethodSource("deleteRequestWithResponseSource")
fun shouldReturnExpectedResponseForDelete(requestWithResponse: RequestWithResponse) {
this.mockRestServiceServer.expect {
method(HttpMethod.DELETE)
requestTo("/mtbfile")
}.andRespond {
withStatus(requestWithResponse.httpStatus).body(requestWithResponse.body).createResponse(it)
}
this.mockRestServiceServer
.expect(method(HttpMethod.DELETE))
.andExpect(requestTo("http://localhost:9000/mtbfile/Patient/${TEST_PATIENT_PSEUDONYM.value}"))
.andRespond {
withStatus(requestWithResponse.httpStatus).body(requestWithResponse.body).createResponse(it)
}
val response = restMtbFileSender.send(MtbFileSender.DeleteRequest("TestID", "PID"))
val response = restMtbFileSender.send(MtbFileSender.DeleteRequest(TEST_REQUEST_ID, TEST_PATIENT_PSEUDONYM))
assertThat(response.status).isEqualTo(requestWithResponse.response.status)
assertThat(response.body).isEqualTo(requestWithResponse.response.body)
}
@ -72,14 +81,14 @@ class RestMtbFileSenderTest {
@ParameterizedTest
@MethodSource("mtbFileRequestWithResponseSource")
fun shouldReturnExpectedResponseForMtbFilePost(requestWithResponse: RequestWithResponse) {
this.mockRestServiceServer.expect {
method(HttpMethod.POST)
requestTo("/mtbfile")
}.andRespond {
withStatus(requestWithResponse.httpStatus).body(requestWithResponse.body).createResponse(it)
}
this.mockRestServiceServer
.expect(method(HttpMethod.POST))
.andExpect(requestTo("http://localhost:9000/mtbfile/MTBFile"))
.andRespond {
withStatus(requestWithResponse.httpStatus).body(requestWithResponse.body).createResponse(it)
}
val response = restMtbFileSender.send(MtbFileSender.MtbFileRequest("TestID", mtbFile))
val response = restMtbFileSender.send(MtbFileSender.MtbFileRequest(TEST_REQUEST_ID, mtbFile))
assertThat(response.status).isEqualTo(requestWithResponse.response.status)
assertThat(response.body).isEqualTo(requestWithResponse.response.body)
}
@ -88,11 +97,12 @@ class RestMtbFileSenderTest {
@MethodSource("mtbFileRequestWithResponseSource")
fun shouldRetryOnMtbFileHttpRequestError(requestWithResponse: RequestWithResponse) {
val restTemplate = RestTemplate()
val restTargetProperties = RestTargetProperties("http://localhost:9000/mtbfile")
val restTargetProperties = RestTargetProperties("http://localhost:9000/mtbfile", null, null)
val retryTemplate = RetryTemplateBuilder().customPolicy(SimpleRetryPolicy(3)).build()
this.mockRestServiceServer = MockRestServiceServer.createServer(restTemplate)
this.restMtbFileSender = RestMtbFileSender(restTemplate, restTargetProperties, retryTemplate)
this.restMtbFileSender =
RestBwhcMtbFileSender(restTemplate, restTargetProperties, retryTemplate, reportService)
val expectedCount = when (requestWithResponse.httpStatus) {
// OK - No Retry
@ -101,14 +111,14 @@ class RestMtbFileSenderTest {
else -> ExpectedCount.max(3)
}
this.mockRestServiceServer.expect(expectedCount) {
method(HttpMethod.POST)
requestTo("/mtbfile")
}.andRespond {
withStatus(requestWithResponse.httpStatus).body(requestWithResponse.body).createResponse(it)
}
this.mockRestServiceServer
.expect(expectedCount, method(HttpMethod.POST))
.andExpect(requestTo("http://localhost:9000/mtbfile/MTBFile"))
.andRespond {
withStatus(requestWithResponse.httpStatus).body(requestWithResponse.body).createResponse(it)
}
val response = restMtbFileSender.send(MtbFileSender.MtbFileRequest("TestID", mtbFile))
val response = restMtbFileSender.send(MtbFileSender.MtbFileRequest(TEST_REQUEST_ID, mtbFile))
assertThat(response.status).isEqualTo(requestWithResponse.response.status)
assertThat(response.body).isEqualTo(requestWithResponse.response.body)
}
@ -117,11 +127,12 @@ class RestMtbFileSenderTest {
@MethodSource("deleteRequestWithResponseSource")
fun shouldRetryOnDeleteHttpRequestError(requestWithResponse: RequestWithResponse) {
val restTemplate = RestTemplate()
val restTargetProperties = RestTargetProperties("http://localhost:9000/mtbfile")
val restTargetProperties = RestTargetProperties("http://localhost:9000/mtbfile", null, null)
val retryTemplate = RetryTemplateBuilder().customPolicy(SimpleRetryPolicy(3)).build()
this.mockRestServiceServer = MockRestServiceServer.createServer(restTemplate)
this.restMtbFileSender = RestMtbFileSender(restTemplate, restTargetProperties, retryTemplate)
this.restMtbFileSender =
RestBwhcMtbFileSender(restTemplate, restTargetProperties, retryTemplate, reportService)
val expectedCount = when (requestWithResponse.httpStatus) {
// OK - No Retry
@ -130,14 +141,14 @@ class RestMtbFileSenderTest {
else -> ExpectedCount.max(3)
}
this.mockRestServiceServer.expect(expectedCount) {
method(HttpMethod.DELETE)
requestTo("/mtbfile")
}.andRespond {
withStatus(requestWithResponse.httpStatus).body(requestWithResponse.body).createResponse(it)
}
this.mockRestServiceServer
.expect(expectedCount, method(HttpMethod.DELETE))
.andExpect(requestTo("http://localhost:9000/mtbfile/Patient/${TEST_PATIENT_PSEUDONYM.value}"))
.andRespond {
withStatus(requestWithResponse.httpStatus).body(requestWithResponse.body).createResponse(it)
}
val response = restMtbFileSender.send(MtbFileSender.DeleteRequest("TestID", "PID"))
val response = restMtbFileSender.send(MtbFileSender.DeleteRequest(TEST_REQUEST_ID, TEST_PATIENT_PSEUDONYM))
assertThat(response.status).isEqualTo(requestWithResponse.response.status)
assertThat(response.body).isEqualTo(requestWithResponse.response.body)
}
@ -149,23 +160,8 @@ class RestMtbFileSenderTest {
val response: MtbFileSender.Response
)
private val warningBody = """
{
"patient_id": "PID",
"issues": [
{ "severity": "warning", "message": "Something is not right" }
]
}
""".trimIndent()
private val errorBody = """
{
"patient_id": "PID",
"issues": [
{ "severity": "error", "message": "Something is very bad" }
]
}
""".trimIndent()
val TEST_REQUEST_ID = RequestId("TestId")
val TEST_PATIENT_PSEUDONYM = PatientPseudonym("PID")
val mtbFile: MtbFile = MtbFile.builder()
.withPatient(
@ -200,31 +196,44 @@ class RestMtbFileSenderTest {
@JvmStatic
fun mtbFileRequestWithResponseSource(): Set<RequestWithResponse> {
return setOf(
RequestWithResponse(HttpStatus.OK, "{}", MtbFileSender.Response(RequestStatus.SUCCESS, "{}")),
RequestWithResponse(
HttpStatus.OK,
responseBodyWithMaxSeverity(ReportService.Severity.INFO),
MtbFileSender.Response(
RequestStatus.SUCCESS,
responseBodyWithMaxSeverity(ReportService.Severity.INFO)
)
),
RequestWithResponse(
HttpStatus.CREATED,
warningBody,
MtbFileSender.Response(RequestStatus.WARNING, warningBody)
responseBodyWithMaxSeverity(ReportService.Severity.WARNING),
MtbFileSender.Response(
RequestStatus.WARNING,
responseBodyWithMaxSeverity(ReportService.Severity.WARNING)
)
),
RequestWithResponse(
HttpStatus.BAD_REQUEST,
"??",
MtbFileSender.Response(RequestStatus.ERROR, ERROR_RESPONSE_BODY)
responseBodyWithMaxSeverity(ReportService.Severity.ERROR),
MtbFileSender.Response(RequestStatus.ERROR, responseBodyWithMaxSeverity(ReportService.Severity.ERROR))
),
RequestWithResponse(
HttpStatus.UNPROCESSABLE_ENTITY,
errorBody,
MtbFileSender.Response(RequestStatus.ERROR, ERROR_RESPONSE_BODY)
responseBodyWithMaxSeverity(ReportService.Severity.FATAL),
MtbFileSender.Response(
RequestStatus.ERROR,
responseBodyWithMaxSeverity(ReportService.Severity.FATAL)
)
),
// Some more errors not mentioned in documentation
RequestWithResponse(
HttpStatus.NOT_FOUND,
"what????",
ERROR_RESPONSE_BODY,
MtbFileSender.Response(RequestStatus.ERROR, ERROR_RESPONSE_BODY)
),
RequestWithResponse(
HttpStatus.INTERNAL_SERVER_ERROR,
"what????",
ERROR_RESPONSE_BODY,
MtbFileSender.Response(RequestStatus.ERROR, ERROR_RESPONSE_BODY)
)
)
@ -251,6 +260,52 @@ class RestMtbFileSenderTest {
)
)
}
fun responseBodyWithMaxSeverity(severity: ReportService.Severity): String {
return when (severity) {
ReportService.Severity.INFO -> """
{
"patient": "PID",
"issues": [
{ "severity": "info", "message": "Info Message" }
]
}
"""
ReportService.Severity.WARNING -> """
{
"patient": "PID",
"issues": [
{ "severity": "info", "message": "Info Message" },
{ "severity": "warning", "message": "Warning Message" }
]
}
"""
ReportService.Severity.ERROR -> """
{
"patient": "PID",
"issues": [
{ "severity": "info", "message": "Info Message" },
{ "severity": "warning", "message": "Warning Message" },
{ "severity": "error", "message": "Error Message" }
]
}
"""
ReportService.Severity.FATAL -> """
{
"patient": "PID",
"issues": [
{ "severity": "info", "message": "Info Message" },
{ "severity": "warning", "message": "Warning Message" },
{ "severity": "error", "message": "Error Message" },
{ "severity": "fatal", "message": "Fatal Message" }
]
}
"""
}
}
}

View File

@ -0,0 +1,314 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.output
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.kotlin.KotlinModule
import de.ukw.ccc.bwhc.dto.*
import dev.dnpm.etl.processor.PatientPseudonym
import dev.dnpm.etl.processor.RequestId
import dev.dnpm.etl.processor.config.AppConfigProperties
import dev.dnpm.etl.processor.config.AppConfiguration
import dev.dnpm.etl.processor.config.RestTargetProperties
import dev.dnpm.etl.processor.monitoring.ReportService
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.output.RestBwhcMtbFileSenderTest.Companion
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.MethodSource
import org.springframework.http.HttpMethod
import org.springframework.http.HttpStatus
import org.springframework.retry.backoff.NoBackOffPolicy
import org.springframework.retry.policy.SimpleRetryPolicy
import org.springframework.retry.support.RetryTemplateBuilder
import org.springframework.test.web.client.ExpectedCount
import org.springframework.test.web.client.MockRestServiceServer
import org.springframework.test.web.client.match.MockRestRequestMatchers.method
import org.springframework.test.web.client.match.MockRestRequestMatchers.requestTo
import org.springframework.test.web.client.response.MockRestResponseCreators.withStatus
import org.springframework.web.client.RestTemplate
class RestDipMtbFileSenderTest {
private lateinit var mockRestServiceServer: MockRestServiceServer
private lateinit var restMtbFileSender: RestMtbFileSender
private var reportService = ReportService(ObjectMapper().registerModule(KotlinModule.Builder().build()))
@BeforeEach
fun setup() {
val restTemplate = RestTemplate()
val restTargetProperties = RestTargetProperties("http://localhost:9000/api", null, null, false)
val retryTemplate = RetryTemplateBuilder().customPolicy(SimpleRetryPolicy(1)).build()
this.mockRestServiceServer = MockRestServiceServer.createServer(restTemplate)
this.restMtbFileSender = RestDipMtbFileSender(restTemplate, restTargetProperties, retryTemplate, reportService)
}
@ParameterizedTest
@MethodSource("deleteRequestWithResponseSource")
fun shouldReturnExpectedResponseForDelete(requestWithResponse: RequestWithResponse) {
this.mockRestServiceServer
.expect(method(HttpMethod.DELETE))
.andExpect(requestTo("http://localhost:9000/api/mtb/etl/patient/${TEST_PATIENT_PSEUDONYM.value}"))
.andRespond {
withStatus(requestWithResponse.httpStatus).body(requestWithResponse.body).createResponse(it)
}
val response = restMtbFileSender.send(MtbFileSender.DeleteRequest(TEST_REQUEST_ID, TEST_PATIENT_PSEUDONYM))
assertThat(response.status).isEqualTo(requestWithResponse.response.status)
assertThat(response.body).isEqualTo(requestWithResponse.response.body)
}
@ParameterizedTest
@MethodSource("mtbFileRequestWithResponseSource")
fun shouldReturnExpectedResponseForMtbFilePost(requestWithResponse: RequestWithResponse) {
this.mockRestServiceServer
.expect(method(HttpMethod.POST))
.andExpect(requestTo("http://localhost:9000/api/mtb/etl/patient-record"))
.andRespond {
withStatus(requestWithResponse.httpStatus).body(requestWithResponse.body).createResponse(it)
}
val response = restMtbFileSender.send(MtbFileSender.MtbFileRequest(TEST_REQUEST_ID, mtbFile))
assertThat(response.status).isEqualTo(requestWithResponse.response.status)
assertThat(response.body).isEqualTo(requestWithResponse.response.body)
}
@ParameterizedTest
@MethodSource("mtbFileRequestWithResponseSource")
fun shouldRetryOnMtbFileHttpRequestError(requestWithResponse: RequestWithResponse) {
val restTemplate = RestTemplate()
val restTargetProperties = RestTargetProperties("http://localhost:9000/api", null, null, false)
val retryTemplate = AppConfiguration().retryTemplate(AppConfigProperties("http://localhost:9000"))
retryTemplate.setBackOffPolicy(NoBackOffPolicy())
this.mockRestServiceServer = MockRestServiceServer.createServer(restTemplate)
this.restMtbFileSender =
RestDipMtbFileSender(restTemplate, restTargetProperties, retryTemplate, reportService)
val expectedCount = when (requestWithResponse.httpStatus) {
// OK - No Retry
HttpStatus.OK, HttpStatus.CREATED, HttpStatus.UNPROCESSABLE_ENTITY, HttpStatus.BAD_REQUEST -> ExpectedCount.max(
1
)
// Request failed - Retry max 3 times
else -> ExpectedCount.max(3)
}
this.mockRestServiceServer
.expect(expectedCount, method(HttpMethod.POST))
.andExpect(requestTo("http://localhost:9000/api/mtb/etl/patient-record"))
.andRespond {
withStatus(requestWithResponse.httpStatus).body(requestWithResponse.body).createResponse(it)
}
val response = restMtbFileSender.send(MtbFileSender.MtbFileRequest(TEST_REQUEST_ID, mtbFile))
assertThat(response.status).isEqualTo(requestWithResponse.response.status)
assertThat(response.body).isEqualTo(requestWithResponse.response.body)
}
@ParameterizedTest
@MethodSource("deleteRequestWithResponseSource")
fun shouldRetryOnDeleteHttpRequestError(requestWithResponse: RequestWithResponse) {
val restTemplate = RestTemplate()
val restTargetProperties = RestTargetProperties("http://localhost:9000/api", null, null, false)
val retryTemplate = AppConfiguration().retryTemplate(AppConfigProperties("http://localhost:9000"))
retryTemplate.setBackOffPolicy(NoBackOffPolicy())
this.mockRestServiceServer = MockRestServiceServer.createServer(restTemplate)
this.restMtbFileSender =
RestDipMtbFileSender(restTemplate, restTargetProperties, retryTemplate, reportService)
val expectedCount = when (requestWithResponse.httpStatus) {
// OK - No Retry
HttpStatus.OK, HttpStatus.CREATED, HttpStatus.UNPROCESSABLE_ENTITY, HttpStatus.BAD_REQUEST -> ExpectedCount.max(
1
)
// Request failed - Retry max 3 times
else -> ExpectedCount.max(3)
}
this.mockRestServiceServer
.expect(expectedCount, method(HttpMethod.DELETE))
.andExpect(requestTo("http://localhost:9000/api/mtb/etl/patient/${TEST_PATIENT_PSEUDONYM.value}"))
.andRespond {
withStatus(requestWithResponse.httpStatus).body(requestWithResponse.body).createResponse(it)
}
val response = restMtbFileSender.send(MtbFileSender.DeleteRequest(TEST_REQUEST_ID, TEST_PATIENT_PSEUDONYM))
assertThat(response.status).isEqualTo(requestWithResponse.response.status)
assertThat(response.body).isEqualTo(requestWithResponse.response.body)
}
companion object {
data class RequestWithResponse(
val httpStatus: HttpStatus,
val body: String,
val response: MtbFileSender.Response
)
val TEST_REQUEST_ID = RequestId("TestId")
val TEST_PATIENT_PSEUDONYM = PatientPseudonym("PID")
val mtbFile: MtbFile = MtbFile.builder()
.withPatient(
Patient.builder()
.withId("PID")
.withBirthDate("2000-08-08")
.withGender(Patient.Gender.MALE)
.build()
)
.withConsent(
Consent.builder()
.withId("1")
.withStatus(Consent.Status.ACTIVE)
.withPatient("PID")
.build()
)
.withEpisode(
Episode.builder()
.withId("1")
.withPatient("PID")
.withPeriod(PeriodStart("2023-08-08"))
.build()
)
.build()
private const val ERROR_RESPONSE_BODY = "Sonstiger Fehler bei der Übertragung"
/**
* Synthetic http responses with related request status
* Also see: https://ibmi-intra.cs.uni-tuebingen.de/display/ZPM/bwHC+REST+API
*/
@JvmStatic
fun mtbFileRequestWithResponseSource(): Set<RequestWithResponse> {
return setOf(
RequestWithResponse(
HttpStatus.OK,
responseBodyWithMaxSeverity(ReportService.Severity.INFO),
MtbFileSender.Response(
RequestStatus.SUCCESS,
responseBodyWithMaxSeverity(ReportService.Severity.INFO)
)
),
RequestWithResponse(
HttpStatus.CREATED,
responseBodyWithMaxSeverity(ReportService.Severity.WARNING),
MtbFileSender.Response(RequestStatus.WARNING, responseBodyWithMaxSeverity(ReportService.Severity.WARNING))
),
RequestWithResponse(
HttpStatus.BAD_REQUEST,
responseBodyWithMaxSeverity(ReportService.Severity.ERROR),
MtbFileSender.Response(RequestStatus.ERROR, responseBodyWithMaxSeverity(ReportService.Severity.ERROR))
),
RequestWithResponse(
HttpStatus.UNPROCESSABLE_ENTITY,
responseBodyWithMaxSeverity(ReportService.Severity.ERROR),
MtbFileSender.Response(RequestStatus.ERROR, responseBodyWithMaxSeverity(ReportService.Severity.ERROR))
),
// Some more errors not mentioned in documentation
RequestWithResponse(
HttpStatus.NOT_FOUND,
ERROR_RESPONSE_BODY,
MtbFileSender.Response(RequestStatus.ERROR, ERROR_RESPONSE_BODY)
),
RequestWithResponse(
HttpStatus.INTERNAL_SERVER_ERROR,
ERROR_RESPONSE_BODY,
MtbFileSender.Response(RequestStatus.ERROR, ERROR_RESPONSE_BODY)
)
)
}
/**
* Synthetic http responses with related request status
* Also see: https://ibmi-intra.cs.uni-tuebingen.de/display/ZPM/bwHC+REST+API
*/
@JvmStatic
fun deleteRequestWithResponseSource(): Set<RequestWithResponse> {
return setOf(
RequestWithResponse(HttpStatus.OK, "", MtbFileSender.Response(RequestStatus.SUCCESS)),
// Some more errors not mentioned in documentation
RequestWithResponse(
HttpStatus.NOT_FOUND,
"what????",
MtbFileSender.Response(RequestStatus.ERROR, ERROR_RESPONSE_BODY)
),
RequestWithResponse(
HttpStatus.INTERNAL_SERVER_ERROR,
"what????",
MtbFileSender.Response(RequestStatus.ERROR, ERROR_RESPONSE_BODY)
)
)
}
fun responseBodyWithMaxSeverity(severity: ReportService.Severity): String {
return when (severity) {
ReportService.Severity.INFO -> """
{
"patient": "PID",
"issues": [
{ "severity": "info", "message": "Info Message" }
]
}
"""
ReportService.Severity.WARNING -> """
{
"patient": "PID",
"issues": [
{ "severity": "info", "message": "Info Message" },
{ "severity": "warning", "message": "Warning Message" }
]
}
"""
ReportService.Severity.ERROR -> """
{
"patient": "PID",
"issues": [
{ "severity": "info", "message": "Info Message" },
{ "severity": "warning", "message": "Warning Message" },
{ "severity": "error", "message": "Error Message" }
]
}
"""
ReportService.Severity.FATAL -> """
{
"patient": "PID",
"issues": [
{ "severity": "info", "message": "Info Message" },
{ "severity": "warning", "message": "Warning Message" },
{ "severity": "error", "message": "Error Message" },
{ "severity": "fatal", "message": "Fatal Message" }
]
}
"""
}
}
}
}

View File

@ -20,13 +20,14 @@
package dev.dnpm.etl.processor.pseudonym
import com.fasterxml.jackson.databind.ObjectMapper
import de.ukw.ccc.bwhc.dto.MtbFile
import de.ukw.ccc.bwhc.dto.*
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.assertThrows
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.ArgumentMatchers
import org.mockito.Mock
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.anyValueClass
import org.mockito.kotlin.doAnswer
import org.mockito.kotlin.whenever
import org.springframework.core.io.ClassPathResource
@ -51,7 +52,7 @@ class ExtensionsTest {
doAnswer {
it.arguments[0]
"PSEUDO-ID"
}.whenever(pseudonymizeService).patientPseudonym(ArgumentMatchers.anyString())
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
val mtbFile = fakeMtbFile()
@ -61,4 +62,137 @@ class ExtensionsTest {
assertThat(mtbFile.serialized()).doesNotContain(CLEAN_PATIENT_ID)
}
@Test
fun shouldNotContainAnyUuidAfterRehashingOfIds(@Mock pseudonymizeService: PseudonymizeService) {
doAnswer {
it.arguments[0]
"PSEUDO-ID"
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
"TESTDOMAIN"
}.whenever(pseudonymizeService).prefix()
val mtbFile = fakeMtbFile()
mtbFile.pseudonymizeWith(pseudonymizeService)
mtbFile.anonymizeContentWith(pseudonymizeService)
val pattern = "\"[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}\"".toRegex().toPattern()
val matcher = pattern.matcher(mtbFile.serialized())
assertThrows<IllegalStateException> {
matcher.find()
matcher.group()
}.also {
assertThat(it.message).isEqualTo("No match found")
}
}
@Test
fun shouldRehashIdsWithPrefix(@Mock pseudonymizeService: PseudonymizeService) {
doAnswer {
it.arguments[0]
"PSEUDO-ID"
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
"TESTDOMAIN"
}.whenever(pseudonymizeService).prefix()
val mtbFile = MtbFile.builder()
.withPatient(
Patient.builder()
.withId("1")
.withBirthDate("2000-08-08")
.withGender(Patient.Gender.MALE)
.build()
)
.withConsent(
Consent.builder()
.withId("1")
.withStatus(Consent.Status.ACTIVE)
.withPatient("123")
.build()
)
.withEpisode(
Episode.builder()
.withId("1")
.withPatient("1")
.withPeriod(PeriodStart("2023-08-08"))
.build()
)
.build()
mtbFile.pseudonymizeWith(pseudonymizeService)
mtbFile.anonymizeContentWith(pseudonymizeService)
assertThat(mtbFile.episode.id)
// TESTDOMAIN<sha256(TESTDOMAIN-1)[0-41]>
.isEqualTo("TESTDOMAIN44e20a53bbbf9f3ae39626d05df7014dcd77d6098")
}
@Test
fun shouldNotThrowExceptionOnNullValues(@Mock pseudonymizeService: PseudonymizeService) {
doAnswer {
it.arguments[0]
"PSEUDO-ID"
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
"TESTDOMAIN"
}.whenever(pseudonymizeService).prefix()
val mtbFile = MtbFile.builder()
.withPatient(
Patient.builder()
.withId("1")
.withBirthDate("2000-08-08")
.withGender(Patient.Gender.MALE)
.build()
)
.withConsent(
Consent.builder()
.withId("1")
.withStatus(Consent.Status.ACTIVE)
.withPatient("123")
.build()
)
.withEpisode(
Episode.builder()
.withId("1")
.withPatient("1")
.withPeriod(PeriodStart("2023-08-08"))
.build()
)
.withClaims(null)
.withDiagnoses(null)
.withCarePlans(null)
.withClaimResponses(null)
.withEcogStatus(null)
.withFamilyMemberDiagnoses(null)
.withGeneticCounsellingRequests(null)
.withHistologyReevaluationRequests(null)
.withHistologyReports(null)
.withLastGuidelineTherapies(null)
.withMolecularPathologyFindings(null)
.withMolecularTherapies(null)
.withNgsReports(null)
.withPreviousGuidelineTherapies(null)
.withRebiopsyRequests(null)
.withRecommendations(null)
.withResponses(null)
.withStudyInclusionRequests(null)
.withSpecimens(null)
.build()
mtbFile.pseudonymizeWith(pseudonymizeService)
mtbFile.anonymizeContentWith(pseudonymizeService)
assertThat(mtbFile.episode.id).isNotNull()
}
}

View File

@ -17,13 +17,12 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.services
package dev.dnpm.etl.processor.security
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.ArgumentCaptor
import org.mockito.ArgumentMatchers.anyLong
import org.mockito.ArgumentMatchers.anyString
import org.mockito.Mock
@ -96,11 +95,11 @@ class TokenServiceTest {
val actual = this.tokenService.addToken("Test Token")
val captor = ArgumentCaptor.forClass(Token::class.java)
val captor = argumentCaptor<Token>()
verify(tokenRepository, times(1)).save(captor.capture())
assertThat(actual).satisfies(Consumer { assertThat(it.isSuccess).isTrue() })
assertThat(captor.value).satisfies(
assertThat(captor.firstValue).satisfies(
Consumer { assertThat(it.name).isEqualTo("Test Token") },
Consumer { assertThat(it.username).isEqualTo("testtoken") },
Consumer { assertThat(it.password).isEqualTo("{test}verysecret") }
@ -116,13 +115,13 @@ class TokenServiceTest {
this.tokenService.deleteToken(42)
val stringCaptor = ArgumentCaptor.forClass(String::class.java)
val stringCaptor = argumentCaptor<String>()
verify(userDetailsManager, times(1)).deleteUser(stringCaptor.capture())
assertThat(stringCaptor.value).isEqualTo("testtoken")
assertThat(stringCaptor.firstValue).isEqualTo("testtoken")
val tokenCaptor = ArgumentCaptor.forClass(Token::class.java)
val tokenCaptor = argumentCaptor<Token>()
verify(tokenRepository, times(1)).delete(tokenCaptor.capture())
assertThat(tokenCaptor.value.id).isEqualTo(42)
assertThat(tokenCaptor.firstValue.id).isEqualTo(42)
}
@Test

View File

@ -0,0 +1,202 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2024 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package dev.dnpm.etl.processor.security
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Nested
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.Mock
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.*
import org.springframework.security.core.session.SessionInformation
import org.springframework.security.core.session.SessionRegistry
import org.springframework.security.oauth2.core.oidc.OidcIdToken
import org.springframework.security.oauth2.core.oidc.user.DefaultOidcUser
import org.springframework.security.oauth2.core.oidc.user.OidcUser
import java.time.Instant
import java.util.*
@ExtendWith(MockitoExtension::class)
class UserRoleServiceTest {
private lateinit var userRoleRepository: UserRoleRepository
private lateinit var sessionRegistry: SessionRegistry
private lateinit var userRoleService: UserRoleService
@BeforeEach
fun setup(
@Mock userRoleRepository: UserRoleRepository,
@Mock sessionRegistry: SessionRegistry
) {
this.userRoleRepository = userRoleRepository
this.sessionRegistry = sessionRegistry
this.userRoleService = UserRoleService(userRoleRepository, sessionRegistry)
}
@Test
fun shouldDelegateFindAllToRepository() {
userRoleService.findAll()
verify(userRoleRepository, times(1)).findAll()
}
@Nested
inner class WithExistingUserRole {
@BeforeEach
fun setup() {
doAnswer { invocation ->
Optional.of(
UserRole(invocation.getArgument(0), "patrick.tester", Role.USER)
)
}.whenever(userRoleRepository).findById(any<Long>())
doAnswer { _ ->
listOf(
dummyPrincipal()
)
}.whenever(sessionRegistry).allPrincipals
}
@Test
fun shouldUpdateUserRole() {
userRoleService.updateUserRole(1, Role.ADMIN)
val userRoleCaptor = argumentCaptor<UserRole>()
verify(userRoleRepository, times(1)).save(userRoleCaptor.capture())
assertThat(userRoleCaptor.firstValue.id).isEqualTo(1L)
assertThat(userRoleCaptor.firstValue.role).isEqualTo(Role.ADMIN)
}
@Test
fun shouldExpireSessionOnUpdate() {
val dummySessions = dummySessions()
whenever(sessionRegistry.getAllSessions(any(), any<Boolean>())).thenReturn(
dummySessions
)
assertThat(dummySessions.filter { it.isExpired }).hasSize(0)
userRoleService.updateUserRole(1, Role.ADMIN)
verify(sessionRegistry, times(1)).getAllSessions(any<OidcUser>(), any<Boolean>())
assertThat(dummySessions.filter { it.isExpired }).hasSize(2)
}
@Test
fun shouldDeleteUserRole() {
userRoleService.deleteUserRole(1)
val userRoleCaptor = argumentCaptor<UserRole>()
verify(userRoleRepository, times(1)).delete(userRoleCaptor.capture())
assertThat(userRoleCaptor.firstValue.id).isEqualTo(1L)
assertThat(userRoleCaptor.firstValue.role).isEqualTo(Role.USER)
}
@Test
fun shouldExpireSessionOnDelete() {
val dummySessions = dummySessions()
whenever(sessionRegistry.getAllSessions(any(), any<Boolean>())).thenReturn(
dummySessions
)
assertThat(dummySessions.filter { it.isExpired }).hasSize(0)
userRoleService.deleteUserRole(1)
verify(sessionRegistry, times(1)).getAllSessions(any<OidcUser>(), any<Boolean>())
assertThat(dummySessions.filter { it.isExpired }).hasSize(2)
}
}
@Nested
inner class WithoutExistingUserRole {
@BeforeEach
fun setup() {
doAnswer { _ ->
Optional.empty<UserRole>()
}.whenever(userRoleRepository).findById(any<Long>())
}
@Test
fun shouldNotUpdateUserRole() {
userRoleService.updateUserRole(1, Role.ADMIN)
verify(userRoleRepository, never()).save(any<UserRole>())
}
@Test
fun shouldNotExpireSessionOnUpdate() {
userRoleService.updateUserRole(1, Role.ADMIN)
verify(sessionRegistry, never()).getAllSessions(any<OidcUser>(), any<Boolean>())
}
@Test
fun shouldNotDeleteUserRole() {
userRoleService.deleteUserRole(1)
verify(userRoleRepository, never()).delete(any<UserRole>())
}
@Test
fun shouldNotExpireSessionOnDelete() {
userRoleService.deleteUserRole(1)
verify(sessionRegistry, never()).getAllSessions(any<OidcUser>(), any<Boolean>())
}
}
companion object {
private fun dummyPrincipal() = DefaultOidcUser(
listOf(),
OidcIdToken(
"anytokenvalue",
Instant.now(),
Instant.now().plusSeconds(10),
mapOf("sub" to "testsub", "preferred_username" to "patrick.tester")
)
)
private fun dummySessions() = listOf(
SessionInformation(
dummyPrincipal(),
"SESSIONID1",
Date.from(Instant.now()),
),
SessionInformation(
dummyPrincipal(),
"SESSIONID2",
Date.from(Instant.now()),
)
)
}
}

View File

@ -1,7 +1,7 @@
/*
* This file is part of ETL-Processor
*
* Copyright (c) 2023 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
* Copyright (c) 2025 Comprehensive Cancer Center Mainfranken, Datenintegrationszentrum Philipps-Universität Marburg and Contributors
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
@ -22,9 +22,14 @@ package dev.dnpm.etl.processor.services
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.kotlin.KotlinModule
import dev.dnpm.etl.processor.monitoring.ReportService
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.monitoring.asRequestStatus
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.Arguments
import org.junit.jupiter.params.provider.MethodSource
class ReportServiceTest {
@ -43,20 +48,32 @@ class ReportServiceTest {
"issues": [
{ "severity": "info", "message": "Info Message" },
{ "severity": "warning", "message": "Warning Message" },
{ "severity": "error", "message": "Error Message" }
{ "severity": "error", "message": "Error Message" },
{ "severity": "fatal", "message": "Fatal Message" }
]
}
""".trimIndent()
val actual = this.reportService.deserialize(json)
assertThat(actual).hasSize(3)
assertThat(actual[0].severity).isEqualTo(ReportService.Severity.ERROR)
assertThat(actual[0].message).isEqualTo("Error Message")
assertThat(actual[1].severity).isEqualTo(ReportService.Severity.WARNING)
assertThat(actual[1].message).isEqualTo("Warning Message")
assertThat(actual[2].severity).isEqualTo(ReportService.Severity.INFO)
assertThat(actual[2].message).isEqualTo("Info Message")
assertThat(actual).hasSize(4)
assertThat(actual[0].severity).isEqualTo(ReportService.Severity.FATAL)
assertThat(actual[0].message).isEqualTo("Fatal Message")
assertThat(actual[1].severity).isEqualTo(ReportService.Severity.ERROR)
assertThat(actual[1].message).isEqualTo("Error Message")
assertThat(actual[2].severity).isEqualTo(ReportService.Severity.WARNING)
assertThat(actual[2].message).isEqualTo("Warning Message")
assertThat(actual[3].severity).isEqualTo(ReportService.Severity.INFO)
assertThat(actual[3].message).isEqualTo("Info Message")
assertThat(actual.asRequestStatus()).isEqualTo(RequestStatus.ERROR)
}
@ParameterizedTest
@MethodSource("testData")
fun shouldParseDataQualityReport(json: String, requestStatus: RequestStatus) {
val actual = this.reportService.deserialize(json)
assertThat(actual.asRequestStatus()).isEqualTo(requestStatus)
}
@Test
@ -70,4 +87,75 @@ class ReportServiceTest {
assertThat(actual[0].message).isEqualTo("Not parsable data quality report '$invalidResponse'")
}
companion object {
@JvmStatic
fun testData(): Set<Arguments> {
return setOf(
Arguments.of(
"""
{
"patient": "4711",
"issues": [
{ "severity": "info", "message": "Info Message" },
{ "severity": "warning", "message": "Warning Message" },
{ "severity": "error", "message": "Error Message" },
{ "severity": "fatal", "message": "Fatal Message" }
]
}
""".trimIndent(),
RequestStatus.ERROR
),
Arguments.of(
"""
{
"patient": "4711",
"issues": [
{ "severity": "info", "message": "Info Message" },
{ "severity": "warning", "message": "Warning Message" },
{ "severity": "error", "message": "Error Message" }
]
}
""".trimIndent(),
RequestStatus.ERROR
),
Arguments.of(
"""
{
"patient": "4711",
"issues": [
{ "severity": "error", "message": "Error Message" }
{ "severity": "info", "message": "Info Message" }
]
}
""".trimIndent(),
RequestStatus.ERROR
),
Arguments.of(
"""
{
"patient": "4711",
"issues": [
{ "severity": "info", "message": "Info Message" },
{ "severity": "warning", "message": "Warning Message" }
]
}
""".trimIndent(),
RequestStatus.WARNING
),
Arguments.of(
"""
{
"patient": "4711",
"issues": [
{ "severity": "info", "message": "Info Message" }
]
}
""".trimIndent(),
RequestStatus.SUCCESS
)
)
}
}
}

View File

@ -21,6 +21,7 @@ package dev.dnpm.etl.processor.services
import com.fasterxml.jackson.databind.ObjectMapper
import de.ukw.ccc.bwhc.dto.*
import dev.dnpm.etl.processor.*
import dev.dnpm.etl.processor.config.AppConfigProperties
import dev.dnpm.etl.processor.monitoring.Request
import dev.dnpm.etl.processor.monitoring.RequestStatus
@ -32,16 +33,15 @@ import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.ArgumentMatchers.anyString
import org.mockito.Mock
import org.mockito.Mockito.*
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.any
import org.mockito.kotlin.anyValueClass
import org.mockito.kotlin.argumentCaptor
import org.mockito.kotlin.whenever
import org.springframework.context.ApplicationEventPublisher
import java.time.Instant
import java.util.*
@ExtendWith(MockitoExtension::class)
@ -88,24 +88,24 @@ class RequestProcessorTest {
fun testShouldSendMtbFileDuplicationAndSaveUnknownRequestStatusAtFirst() {
doAnswer {
Request(
id = 1L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "xrysxpozhbs2lnrjgf3yq4fzj33kxr7xr5c2cbuskmelfdmckl3a",
type = RequestType.MTB_FILE,
status = RequestStatus.SUCCESS,
processedAt = Instant.parse("2023-08-08T02:00:00Z")
1L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("zdlzv5s5ydmd4ktw2v5piohegc4jcyrm6j66bq6tv2uxuerndmga"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS,
Instant.parse("2023-08-08T02:00:00Z")
)
}.`when`(requestService).lastMtbFileRequestForPatientPseudonym(anyString())
}.whenever(requestService).lastMtbFileRequestForPatientPseudonym(anyValueClass())
doAnswer {
false
}.`when`(requestService).isLastRequestWithKnownStatusDeletion(anyString())
}.whenever(requestService).isLastRequestWithKnownStatusDeletion(anyValueClass())
doAnswer {
it.arguments[0] as String
}.`when`(pseudonymizeService).patientPseudonym(any())
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
it.arguments[0]
@ -147,24 +147,24 @@ class RequestProcessorTest {
fun testShouldDetectMtbFileDuplicationAndSendDuplicationEvent() {
doAnswer {
Request(
id = 1L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "xrysxpozhbs2lnrjgf3yq4fzj33kxr7xr5c2cbuskmelfdmckl3a",
type = RequestType.MTB_FILE,
status = RequestStatus.SUCCESS,
processedAt = Instant.parse("2023-08-08T02:00:00Z")
1L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("zdlzv5s5ydmd4ktw2v5piohegc4jcyrm6j66bq6tv2uxuerndmga"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS,
Instant.parse("2023-08-08T02:00:00Z")
)
}.`when`(requestService).lastMtbFileRequestForPatientPseudonym(anyString())
}.whenever(requestService).lastMtbFileRequestForPatientPseudonym(anyValueClass())
doAnswer {
false
}.`when`(requestService).isLastRequestWithKnownStatusDeletion(anyString())
}.whenever(requestService).isLastRequestWithKnownStatusDeletion(anyValueClass())
doAnswer {
it.arguments[0] as String
}.`when`(pseudonymizeService).patientPseudonym(any())
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
it.arguments[0]
@ -206,28 +206,28 @@ class RequestProcessorTest {
fun testShouldSendMtbFileAndSendSuccessEvent() {
doAnswer {
Request(
id = 1L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "different",
type = RequestType.MTB_FILE,
status = RequestStatus.SUCCESS,
processedAt = Instant.parse("2023-08-08T02:00:00Z")
1L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("different"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS,
Instant.parse("2023-08-08T02:00:00Z")
)
}.`when`(requestService).lastMtbFileRequestForPatientPseudonym(anyString())
}.whenever(requestService).lastMtbFileRequestForPatientPseudonym(anyValueClass())
doAnswer {
false
}.`when`(requestService).isLastRequestWithKnownStatusDeletion(anyString())
}.whenever(requestService).isLastRequestWithKnownStatusDeletion(anyValueClass())
doAnswer {
MtbFileSender.Response(status = RequestStatus.SUCCESS)
}.`when`(sender).send(any<MtbFileSender.MtbFileRequest>())
}.whenever(sender).send(any<MtbFileSender.MtbFileRequest>())
doAnswer {
it.arguments[0] as String
}.`when`(pseudonymizeService).patientPseudonym(any())
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
it.arguments[0]
@ -269,28 +269,28 @@ class RequestProcessorTest {
fun testShouldSendMtbFileAndSendErrorEvent() {
doAnswer {
Request(
id = 1L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "different",
type = RequestType.MTB_FILE,
status = RequestStatus.SUCCESS,
processedAt = Instant.parse("2023-08-08T02:00:00Z")
1L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("different"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS,
Instant.parse("2023-08-08T02:00:00Z")
)
}.`when`(requestService).lastMtbFileRequestForPatientPseudonym(anyString())
}.whenever(requestService).lastMtbFileRequestForPatientPseudonym(anyValueClass())
doAnswer {
false
}.`when`(requestService).isLastRequestWithKnownStatusDeletion(anyString())
}.whenever(requestService).isLastRequestWithKnownStatusDeletion(anyValueClass())
doAnswer {
MtbFileSender.Response(status = RequestStatus.ERROR)
}.`when`(sender).send(any<MtbFileSender.MtbFileRequest>())
}.whenever(sender).send(any<MtbFileSender.MtbFileRequest>())
doAnswer {
it.arguments[0] as String
}.`when`(pseudonymizeService).patientPseudonym(any())
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
it.arguments[0]
@ -332,13 +332,13 @@ class RequestProcessorTest {
fun testShouldSendDeleteRequestAndSaveUnknownRequestStatusAtFirst() {
doAnswer {
"PSEUDONYM"
}.`when`(pseudonymizeService).patientPseudonym(anyString())
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
MtbFileSender.Response(status = RequestStatus.UNKNOWN)
}.`when`(sender).send(any<MtbFileSender.DeleteRequest>())
}.whenever(sender).send(any<MtbFileSender.DeleteRequest>())
this.requestProcessor.processDeletion("TEST_12345678901")
this.requestProcessor.processDeletion(TEST_PATIENT_ID)
val requestCaptor = argumentCaptor<Request>()
verify(requestService, times(1)).save(requestCaptor.capture())
@ -350,13 +350,13 @@ class RequestProcessorTest {
fun testShouldSendDeleteRequestAndSendSuccessEvent() {
doAnswer {
"PSEUDONYM"
}.`when`(pseudonymizeService).patientPseudonym(anyString())
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
MtbFileSender.Response(status = RequestStatus.SUCCESS)
}.`when`(sender).send(any<MtbFileSender.DeleteRequest>())
}.whenever(sender).send(any<MtbFileSender.DeleteRequest>())
this.requestProcessor.processDeletion("TEST_12345678901")
this.requestProcessor.processDeletion(TEST_PATIENT_ID)
val eventCaptor = argumentCaptor<ResponseEvent>()
verify(applicationEventPublisher, times(1)).publishEvent(eventCaptor.capture())
@ -368,13 +368,13 @@ class RequestProcessorTest {
fun testShouldSendDeleteRequestAndSendErrorEvent() {
doAnswer {
"PSEUDONYM"
}.`when`(pseudonymizeService).patientPseudonym(anyString())
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
MtbFileSender.Response(status = RequestStatus.ERROR)
}.`when`(sender).send(any<MtbFileSender.DeleteRequest>())
}.whenever(sender).send(any<MtbFileSender.DeleteRequest>())
this.requestProcessor.processDeletion("TEST_12345678901")
this.requestProcessor.processDeletion(TEST_PATIENT_ID)
val eventCaptor = argumentCaptor<ResponseEvent>()
verify(applicationEventPublisher, times(1)).publishEvent(eventCaptor.capture())
@ -384,9 +384,9 @@ class RequestProcessorTest {
@Test
fun testShouldSendDeleteRequestWithPseudonymErrorAndSaveErrorRequestStatus() {
doThrow(RuntimeException()).`when`(pseudonymizeService).patientPseudonym(anyString())
doThrow(RuntimeException()).whenever(pseudonymizeService).patientPseudonym(anyValueClass())
this.requestProcessor.processDeletion("TEST_12345678901")
this.requestProcessor.processDeletion(TEST_PATIENT_ID)
val requestCaptor = argumentCaptor<Request>()
verify(requestService, times(1)).save(requestCaptor.capture())
@ -400,7 +400,7 @@ class RequestProcessorTest {
doAnswer {
it.arguments[0] as String
}.`when`(pseudonymizeService).patientPseudonym(any())
}.whenever(pseudonymizeService).patientPseudonym(anyValueClass())
doAnswer {
it.arguments[0]
@ -408,7 +408,7 @@ class RequestProcessorTest {
doAnswer {
MtbFileSender.Response(status = RequestStatus.SUCCESS)
}.`when`(sender).send(any<MtbFileSender.MtbFileRequest>())
}.whenever(sender).send(any<MtbFileSender.MtbFileRequest>())
val mtbFile = MtbFile.builder()
.withPatient(
@ -442,4 +442,8 @@ class RequestProcessorTest {
assertThat(eventCaptor.firstValue.status).isEqualTo(RequestStatus.SUCCESS)
}
companion object {
val TEST_PATIENT_ID = PatientId("TEST_12345678901")
}
}

View File

@ -19,6 +19,7 @@
package dev.dnpm.etl.processor.services
import dev.dnpm.etl.processor.*
import dev.dnpm.etl.processor.monitoring.Request
import dev.dnpm.etl.processor.monitoring.RequestRepository
import dev.dnpm.etl.processor.monitoring.RequestStatus
@ -30,8 +31,9 @@ import org.junit.jupiter.api.extension.ExtendWith
import org.mockito.Mock
import org.mockito.Mockito.*
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.anyValueClass
import org.mockito.kotlin.whenever
import java.time.Instant
import java.util.*
@ExtendWith(MockitoExtension::class)
class RequestServiceTest {
@ -41,14 +43,14 @@ class RequestServiceTest {
private lateinit var requestService: RequestService
private fun anyRequest() = any(Request::class.java) ?: Request(
id = 0L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_dummy",
pid = "PX",
fingerprint = "dummy",
type = RequestType.MTB_FILE,
status = RequestStatus.SUCCESS,
processedAt = Instant.parse("2023-08-08T02:00:00Z")
0L,
randomRequestId(),
PatientPseudonym("TEST_dummy"),
PatientId("PX"),
Fingerprint("dummy"),
RequestType.MTB_FILE,
RequestStatus.SUCCESS,
Instant.parse("2023-08-08T02:00:00Z")
)
@BeforeEach
@ -63,34 +65,34 @@ class RequestServiceTest {
fun shouldIndicateLastRequestIsDeleteRequest() {
val requests = listOf(
Request(
id = 1L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "0123456789abcdef1",
type = RequestType.MTB_FILE,
status = RequestStatus.WARNING,
processedAt = Instant.parse("2023-07-07T00:00:00Z")
1L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.MTB_FILE,
RequestStatus.WARNING,
Instant.parse("2023-07-07T00:00:00Z")
),
Request(
id = 2L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "0123456789abcdefd",
type = RequestType.DELETE,
status = RequestStatus.WARNING,
processedAt = Instant.parse("2023-07-07T02:00:00Z")
2L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdefd"),
RequestType.DELETE,
RequestStatus.WARNING,
Instant.parse("2023-07-07T02:00:00Z")
),
Request(
id = 3L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "0123456789abcdef1",
type = RequestType.MTB_FILE,
status = RequestStatus.UNKNOWN,
processedAt = Instant.parse("2023-08-11T00:00:00Z")
3L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.MTB_FILE,
RequestStatus.UNKNOWN,
Instant.parse("2023-08-11T00:00:00Z")
)
)
@ -103,34 +105,34 @@ class RequestServiceTest {
fun shouldIndicateLastRequestIsNotDeleteRequest() {
val requests = listOf(
Request(
id = 1L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "0123456789abcdef1",
type = RequestType.MTB_FILE,
status = RequestStatus.WARNING,
processedAt = Instant.parse("2023-07-07T00:00:00Z")
1L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.MTB_FILE,
RequestStatus.WARNING,
Instant.parse("2023-07-07T00:00:00Z")
),
Request(
id = 2L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "0123456789abcdef1",
type = RequestType.MTB_FILE,
status = RequestStatus.WARNING,
processedAt = Instant.parse("2023-07-07T02:00:00Z")
2L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.MTB_FILE,
RequestStatus.WARNING,
Instant.parse("2023-07-07T02:00:00Z")
),
Request(
id = 3L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "0123456789abcdef1",
type = RequestType.MTB_FILE,
status = RequestStatus.UNKNOWN,
processedAt = Instant.parse("2023-08-11T00:00:00Z")
3L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.MTB_FILE,
RequestStatus.UNKNOWN,
Instant.parse("2023-08-11T00:00:00Z")
)
)
@ -143,31 +145,31 @@ class RequestServiceTest {
fun shouldReturnPatientsLastRequest() {
val requests = listOf(
Request(
id = 1L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "0123456789abcdef1",
type = RequestType.DELETE,
status = RequestStatus.SUCCESS,
processedAt = Instant.parse("2023-07-07T02:00:00Z")
1L,
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.DELETE,
RequestStatus.SUCCESS,
Instant.parse("2023-07-07T02:00:00Z")
),
Request(
id = 1L,
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678902",
pid = "P2",
fingerprint = "0123456789abcdef2",
type = RequestType.MTB_FILE,
status = RequestStatus.WARNING,
processedAt = Instant.parse("2023-08-08T00:00:00Z")
1L,
randomRequestId(),
PatientPseudonym("TEST_12345678902"),
PatientId("P2"),
Fingerprint("0123456789abcdef2"),
RequestType.MTB_FILE,
RequestStatus.WARNING,
Instant.parse("2023-08-08T00:00:00Z")
)
)
val actual = RequestService.lastMtbFileRequestForPatientPseudonym(requests)
assertThat(actual).isInstanceOf(Request::class.java)
assertThat(actual?.fingerprint).isEqualTo("0123456789abcdef2")
assertThat(actual?.fingerprint).isEqualTo(Fingerprint("0123456789abcdef2"))
}
@Test
@ -184,16 +186,16 @@ class RequestServiceTest {
doAnswer {
val obj = it.arguments[0] as Request
obj.copy(id = 1L)
}.`when`(requestRepository).save(anyRequest())
}.whenever(requestRepository).save(anyRequest())
val request = Request(
uuid = UUID.randomUUID().toString(),
patientId = "TEST_12345678901",
pid = "P1",
fingerprint = "0123456789abcdef1",
type = RequestType.DELETE,
status = RequestStatus.SUCCESS,
processedAt = Instant.parse("2023-07-07T02:00:00Z")
randomRequestId(),
PatientPseudonym("TEST_12345678901"),
PatientId("P1"),
Fingerprint("0123456789abcdef1"),
RequestType.DELETE,
RequestStatus.SUCCESS,
Instant.parse("2023-07-07T02:00:00Z")
)
requestService.save(request)
@ -203,23 +205,23 @@ class RequestServiceTest {
@Test
fun allRequestsByPatientPseudonymShouldRequestAllRequestsForPatientPseudonym() {
requestService.allRequestsByPatientPseudonym("TEST_12345678901")
requestService.allRequestsByPatientPseudonym(PatientPseudonym("TEST_12345678901"))
verify(requestRepository, times(1)).findAllByPatientIdOrderByProcessedAtDesc(anyString())
verify(requestRepository, times(1)).findAllByPatientPseudonymOrderByProcessedAtDesc(anyValueClass())
}
@Test
fun lastMtbFileRequestForPatientPseudonymShouldRequestAllRequestsForPatientPseudonym() {
requestService.lastMtbFileRequestForPatientPseudonym("TEST_12345678901")
requestService.lastMtbFileRequestForPatientPseudonym(PatientPseudonym("TEST_12345678901"))
verify(requestRepository, times(1)).findAllByPatientIdOrderByProcessedAtDesc(anyString())
verify(requestRepository, times(1)).findAllByPatientPseudonymOrderByProcessedAtDesc(anyValueClass())
}
@Test
fun isLastRequestDeletionShouldRequestAllRequestsForPatientPseudonym() {
requestService.isLastRequestWithKnownStatusDeletion("TEST_12345678901")
requestService.isLastRequestWithKnownStatusDeletion(PatientPseudonym("TEST_12345678901"))
verify(requestRepository, times(1)).findAllByPatientIdOrderByProcessedAtDesc(anyString())
verify(requestRepository, times(1)).findAllByPatientPseudonymOrderByProcessedAtDesc(anyValueClass())
}
}

View File

@ -19,8 +19,8 @@
package dev.dnpm.etl.processor.services
import dev.dnpm.etl.processor.*
import dev.dnpm.etl.processor.monitoring.Request
import dev.dnpm.etl.processor.monitoring.RequestRepository
import dev.dnpm.etl.processor.monitoring.RequestStatus
import dev.dnpm.etl.processor.monitoring.RequestType
import org.assertj.core.api.Assertions.assertThat
@ -29,7 +29,6 @@ import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.MethodSource
import org.mockito.ArgumentMatchers.anyString
import org.mockito.Mock
import org.mockito.junit.jupiter.MockitoExtension
import org.mockito.kotlin.*
@ -40,64 +39,64 @@ import java.util.*
@ExtendWith(MockitoExtension::class)
class ResponseProcessorTest {
private lateinit var requestRepository: RequestRepository
private lateinit var requestService: RequestService
private lateinit var statisticsUpdateProducer: Sinks.Many<Any>
private lateinit var responseProcessor: ResponseProcessor
private val testRequest = Request(
1L,
"TestID1234",
"PSEUDONYM-A",
"1",
"dummyfingerprint",
RequestId("TestID1234"),
PatientPseudonym("PSEUDONYM-A"),
PatientId("1"),
Fingerprint("dummyfingerprint"),
RequestType.MTB_FILE,
RequestStatus.UNKNOWN
)
@BeforeEach
fun setup(
@Mock requestRepository: RequestRepository,
@Mock requestService: RequestService,
@Mock statisticsUpdateProducer: Sinks.Many<Any>
) {
this.requestRepository = requestRepository
this.requestService = requestService
this.statisticsUpdateProducer = statisticsUpdateProducer
this.responseProcessor = ResponseProcessor(requestRepository, statisticsUpdateProducer)
this.responseProcessor = ResponseProcessor(requestService, statisticsUpdateProducer)
}
@Test
fun shouldNotSaveStatusForUnknownRequest() {
doAnswer {
Optional.empty<Request>()
}.whenever(requestRepository).findByUuidEquals(anyString())
}.whenever(requestService).findByUuid(anyValueClass())
val event = ResponseEvent(
"TestID1234",
RequestId("TestID1234"),
Instant.parse("2023-09-09T00:00:00Z"),
RequestStatus.SUCCESS
)
this.responseProcessor.handleResponseEvent(event)
verify(requestRepository, never()).save(any())
verify(requestService, never()).save(any())
}
@Test
fun shouldNotSaveStatusWithUnknownState() {
doAnswer {
Optional.of(testRequest)
}.whenever(requestRepository).findByUuidEquals(anyString())
}.whenever(requestService).findByUuid(anyValueClass())
val event = ResponseEvent(
"TestID1234",
RequestId("TestID1234"),
Instant.parse("2023-09-09T00:00:00Z"),
RequestStatus.UNKNOWN
)
this.responseProcessor.handleResponseEvent(event)
verify(requestRepository, never()).save(any())
verify(requestService, never()).save(any<Request>())
}
@ParameterizedTest
@ -105,10 +104,10 @@ class ResponseProcessorTest {
fun shouldSaveStatusForKnownRequest(requestStatus: RequestStatus) {
doAnswer {
Optional.of(testRequest)
}.whenever(requestRepository).findByUuidEquals(anyString())
}.whenever(requestService).findByUuid(anyValueClass())
val event = ResponseEvent(
"TestID1234",
RequestId("TestID1234"),
Instant.parse("2023-09-09T00:00:00Z"),
requestStatus
)
@ -116,7 +115,7 @@ class ResponseProcessorTest {
this.responseProcessor.handleResponseEvent(event)
val captor = argumentCaptor<Request>()
verify(requestRepository, times(1)).save(captor.capture())
verify(requestService, times(1)).save(captor.capture())
assertThat(captor.firstValue).isNotNull
assertThat(captor.firstValue.status).isEqualTo(requestStatus)
}

File diff suppressed because one or more lines are too long