etl-processor

pcvolkmer/etl-processor

Fork 0

mirror of https://github.com/pcvolkmer/etl-processor.git synced 2025-04-19 17:26:51 +00:00

Commit Graph

Select branches

Hide Pull Requests

0.9.x

master

#1

#101

#104

#105

#107

#109

#11

#13

#14

#16

#19

#21

#26

#27

#28

#30

#32

#35

#4

#40

#41

#45

#46

#47

#48

#5

#50

#52

#54

#57

#59

#60

#61

#67

#70

#70

#71

#71

#72

#72

#73

#74

#74

#75

#76

#77

#78

#79

#8

#80

#81

#82

#83

#85

#87

#89

#9

#90

#92

#93

#95

#96

#99

v0.1.0

v0.1.1

v0.1.2

v0.10.0

v0.2.0

v0.3.0

v0.3.1

v0.4.0

v0.5.0

v0.7.0

v0.8.0

v0.9.0

v0.9.1

v0.9.2

v0.9.3

v0.9.4

v0.9.5

v0.9.6

v0.9.7

v0.9.8

v0.9.9

0eee1908df

Merge pull request #13 from CCC-MF/issue_12 Paul-Christian Volkmer 2023-10-05 12:41:49 +02:00
ffea9343c8 Issue #12: Change README.md to show transformation config names as env var Paul-Christian Volkmer 2023-10-05 12:36:37 +02:00
eb24995ed9 Issue #12: Log transformation count applied on application start Paul-Christian Volkmer 2023-10-05 12:35:29 +02:00
4196664060 Issue #12: Transform MTBFile objects by using transformation rules Paul-Christian Volkmer 2023-10-05 12:09:56 +02:00
2824951e5e Issue #12: Add information about transformation rules in README.md Paul-Christian Volkmer 2023-10-05 11:45:42 +02:00
1e1db1c4d9 Issue #12: Add application config for transformation configuration Paul-Christian Volkmer 2023-10-05 11:37:10 +02:00
7440fe1e23 Issue #12: Basic implementation of transformation service Paul-Christian Volkmer 2023-10-05 10:51:49 +02:00
3f5c5e28fa chore: update Spring Boot dependencies Paul-Christian Volkmer 2023-09-26 09:26:59 +02:00
6397b2a019 chore: pump version to dev version snapshot Paul-Christian Volkmer 2023-09-26 09:26:39 +02:00
bf8f87b261

fix: removed gaps system from GPAS pseudonym value. Also added clean up method, which will replace filename invalid characters witch '_'. (#11) jlidke 2023-09-04 15:41:22 +02:00
2f32834de0 Release 0.1.2 v0.1.2 Paul-Christian Volkmer 2023-08-30 13:45:06 +02:00
79709caa39 Merge remote-tracking branch 'origin/add-docker-build' Paul-Christian Volkmer 2023-08-30 13:27:56 +02:00
c52509054d chore: Add kafka-clients dependency with fixed version to mitigate CVEs Paul-Christian Volkmer 2023-08-30 13:26:05 +02:00
8fd587c2a3 chore: remove unused HealthCheck.java Jakub Lidke 2023-08-30 11:54:44 +02:00
edafe30a4b chore: added log msg to GpasPseudonymGenerator Jakub Lidke 2023-08-30 11:51:08 +02:00
e24be0d325 chore: cleanup deployment docker-compose.yaml and env-sample.env. added 'DNPM' prefix for better integration into productive environment. Jakub Lidke 2023-08-30 11:50:24 +02:00
5e93e834ad Remove comment to use host alias Paul-Christian Volkmer 2023-08-30 10:24:48 +02:00
5e5bd579fb test: * added additional external host 'localhost', now we can connect without additional host alias. * added akhq to dev-compose.yml Jakub Lidke 2023-08-28 12:47:09 +02:00
a24f869c84 Update dependencies Paul-Christian Volkmer 2023-08-30 10:03:04 +02:00
635985bfd1 chore: remove previous build via Dockerfile. Fix security issue: CVE-2023-34453, CVE-2023-34454, CVE-2023-34455, CVE-2022-1471 Jakub Lidke 2023-08-28 14:27:28 +02:00
25143745c4 chore: added deployment docker-compose.yaml and env-sample.env added. Jakub Lidke 2023-08-28 12:54:14 +02:00
532254593f test: * added additional external host 'localhost', now we can connect without additional host alias. * added akhq to dev-compose.yml Jakub Lidke 2023-08-28 12:47:09 +02:00
01ff53ab23 chore: deployment environment has maria db entries Jakub Lidke 2023-08-25 13:42:07 +02:00
9643c80cc5 build: locally build docker image has license entry,now Jakub Lidke 2023-08-25 13:39:42 +02:00
aa40da4995 chore: dev kafka is available via localhost, now. Jakub Lidke 2023-08-25 13:11:10 +02:00
da26b5a2c8 Merge branch 'master' into add-docker-build Jakub Lidke 2023-08-25 12:59:38 +02:00
bbea48322f chore: added deployment port mapping for monitoring access Jakub Lidke 2023-08-25 12:50:29 +02:00
480f165c7b chore: add deployment docker-compose.yaml and fitting env-sample.env file Jakub Lidke 2023-08-24 13:48:46 +02:00
3d2c73ff8f doc: gPas Version requirement added Jakub Lidke 2023-08-24 13:01:29 +02:00
9921e1e684 Throw PseudonymRequestFailed exception with error message Paul-Christian Volkmer 2023-08-19 11:45:21 +02:00
5bd26b894c Add information about key based retention config Paul-Christian Volkmer 2023-08-18 22:15:10 +02:00
8dc82225a4

Issue #7: Send and expect requestId in record body, not in record key (#8) Paul-Christian Volkmer 2023-08-16 15:25:46 +02:00
2eb5cc61b9 Change Kafka response body JSON alias Paul-Christian Volkmer 2023-08-15 10:58:17 +02:00
78b2287163 Add information about Kafka retention time Paul-Christian Volkmer 2023-08-15 08:51:23 +02:00
66dc96680d Update dev config and added related information into README.md Paul-Christian Volkmer 2023-08-15 01:09:22 +02:00
64b8636145 Update Apache Kafka service config for KRaft mode Paul-Christian Volkmer 2023-08-15 00:49:01 +02:00
2e7ef25a49 Update project version and versions in gradle file Paul-Christian Volkmer 2023-08-12 23:16:17 +02:00
7186a45f6c Add link to onkostar-plugin-dnpmexport Paul-Christian Volkmer 2023-08-12 22:27:20 +02:00
72295202ec Code cleanup Paul-Christian Volkmer 2023-08-12 22:19:29 +02:00
bc48a7217e Add more information about usage in an ETl process Paul-Christian Volkmer 2023-08-11 14:37:48 +02:00
0e1034d964 New version and add status badge to README.md v0.1.1 Paul-Christian Volkmer 2023-08-11 09:47:20 +02:00
6ecb439007

Issue #3: Detect the request type of request with last known status (#5) Paul-Christian Volkmer 2023-08-11 09:22:54 +02:00
cb9c590472

Issue #2: Do not serialize JSON string as custom string (#4) Paul-Christian Volkmer 2023-08-11 09:13:45 +02:00
044d01534b Build and deploy docker image on new release v0.1.0 Paul-Christian Volkmer 2023-08-10 20:17:10 +02:00
4dde13e79a Run tests on each tag Paul-Christian Volkmer 2023-08-10 18:59:04 +02:00
55153d8050 Add information about docker image Paul-Christian Volkmer 2023-08-10 18:55:33 +02:00
25ec557c25 Change condition when to run docker job Paul-Christian Volkmer 2023-08-10 18:44:33 +02:00
2264d85bd1 Run docker workflow after test workflow Paul-Christian Volkmer 2023-08-10 18:29:25 +02:00
2ec5e27a40 Full setup for docker build Paul-Christian Volkmer 2023-08-10 18:07:25 +02:00
d24d9a7fd0 Add docker deploy workflow Paul-Christian Volkmer 2023-08-10 17:46:11 +02:00
1e29ecc891 Fix event name in workflow file Paul-Christian Volkmer 2023-08-10 17:32:02 +02:00
b956eba6c7 Add workflow to run tests on push or pull request Paul-Christian Volkmer 2023-08-10 16:44:26 +02:00
002b0618cf Add tests for KafkaMtbFileSender Paul-Christian Volkmer 2023-08-10 13:35:35 +02:00
2b42a4d262 Tests for RestMtbFileSender Paul-Christian Volkmer 2023-08-10 12:11:39 +02:00
501762d451 Add test logging Paul-Christian Volkmer 2023-08-09 18:32:03 +02:00
1a640ff9df Decouple request and response processing Paul-Christian Volkmer 2023-08-09 18:15:20 +02:00
7f048e2483 Do not append custom prefix to gPAS pseudonym Paul-Christian Volkmer 2023-08-09 12:26:57 +02:00
47830ed9f7 Use single MtbFileSender Paul-Christian Volkmer 2023-08-09 10:34:23 +02:00
13bfa0018d Change endpoint configuration to select single endpoint Paul-Christian Volkmer 2023-08-09 10:20:20 +02:00
7739afad1f Handle MTB File with rejected consent as deletion request Paul-Christian Volkmer 2023-08-09 08:13:27 +02:00
6ad6ee13a1 Ignore unknown properties in DataQualityResponse Paul-Christian Volkmer 2023-08-08 19:16:59 +02:00
536ecbbd56 Add tests for error response status Paul-Christian Volkmer 2023-08-08 18:52:03 +02:00
422441a3b3 Add tests for RequestProcessor Paul-Christian Volkmer 2023-08-08 16:46:02 +02:00
b75328b74d Move integration tests into own source-set Paul-Christian Volkmer 2023-08-08 15:16:58 +02:00
4051b5094c Keep database testcontainer alive until all tests are done Paul-Christian Volkmer 2023-08-08 14:58:10 +02:00
bcc23f6b14 Add RequestService to handle access to requests Paul-Christian Volkmer 2023-08-08 14:50:12 +02:00
1fc09d691e Rename test class to match applications main class name Paul-Christian Volkmer 2023-08-08 13:34:53 +02:00
3039b4b2a7 Add basic Testcontainers test setup Paul-Christian Volkmer 2023-08-08 13:23:18 +02:00
3dea664999 Update Spring Boot dependencies Paul-Christian Volkmer 2023-08-08 12:46:04 +02:00
459ad59c1d Do not detect duplicates after deletion request Paul-Christian Volkmer 2023-08-04 11:43:23 +02:00
b14f2c1794 Add information about 'no connection' responses Paul-Christian Volkmer 2023-08-03 16:18:16 +02:00
ec76c775d9 Explicit producer topic configuration Paul-Christian Volkmer 2023-08-03 16:04:31 +02:00
3dcee41569 Implement delete request using Apache Kafka Paul-Christian Volkmer 2023-08-03 15:14:49 +02:00
ac91620651 Use Map as status body since it contains JSON Paul-Christian Volkmer 2023-08-03 12:59:53 +02:00
577509e6f2 Map 'status_code' and 'status code' to same data value Paul-Christian Volkmer 2023-08-02 16:53:09 +02:00
7f8b21efd2 Handle not parsable data quality reports Paul-Christian Volkmer 2023-08-02 16:23:16 +02:00
70d4fa2f0f Use duplication fingerprinting based on MTB file requests only Paul-Christian Volkmer 2023-08-02 16:10:53 +02:00
35cb258b13 Do not return specific status code based on remote status code Paul-Christian Volkmer 2023-08-02 16:10:18 +02:00
a075f73162 feat: add Dockerfile for build within docker environment and run application within a container. Jakub Lidke 2023-08-02 15:19:38 +02:00
51cf7a7917 Add processor to handle responses from Kafka topic Paul-Christian Volkmer 2023-08-02 11:52:11 +02:00
db631fbd8e Added more information about pseudonym generators Paul-Christian Volkmer 2023-08-01 16:06:57 +02:00
857bd157a0 Modified config property to select pseudonym generator Paul-Christian Volkmer 2023-08-01 16:06:40 +02:00
ccf6908223 Added statistics update using SSE for delete requests Paul-Christian Volkmer 2023-08-01 13:54:26 +02:00
7a7ea8c96d Extract request processing Paul-Christian Volkmer 2023-08-01 12:39:15 +02:00
11ed3a6296 Show request type in addition to state and ID Paul-Christian Volkmer 2023-08-01 08:53:43 +02:00
9c9760c3ec Add delete requests to be shown in statistics page Paul-Christian Volkmer 2023-08-01 08:45:30 +02:00
4f7f5e4d89 Add request type to be saved with each request Paul-Christian Volkmer 2023-08-01 08:39:30 +02:00
a1e56f1596 Handle delete requests by using REST request to bwHC Paul-Christian Volkmer 2023-07-31 19:24:10 +02:00
e1d05ef574 Mark delete requests with errors or success as expected Paul-Christian Volkmer 2023-07-31 19:23:12 +02:00
9af0e159b0 Enable patient data deletion Paul-Christian Volkmer 2023-07-31 17:08:44 +02:00
aed5f15d2d Add request ID to Kafka key Paul-Christian Volkmer 2023-07-31 10:28:54 +02:00
2d140d94b2 Add Apache Kafka to dev compose file Paul-Christian Volkmer 2023-07-31 09:15:22 +02:00
c8b2e50d47

Merge pull request #1 from CCC-MF/add-gPas-support Paul-Christian Volkmer 2023-07-31 09:07:14 +02:00
361bba5b65 Code polish for PR Paul-Christian Volkmer 2023-07-31 09:05:51 +02:00
aa5b35970a Code polish for PR Paul-Christian Volkmer 2023-07-27 23:16:19 +02:00
90c5b81c2b feat: patient pid may be replaced with gPAS pseudonym, now. Jakub Lidke 2023-07-27 13:01:58 +02:00
79d83ef04a fix: if no prefix is set, _ is also not added as prefix to patient pseudonym. Jakub Lidke 2023-07-27 12:18:17 +02:00
469c58fe26 chore: update .gitignore do not checkin gpas files Jakub Lidke 2023-07-27 11:50:49 +02:00
cf8c5a8692 fix: wait for kafka to accept message and return success than Jakub Lidke 2023-07-27 11:49:31 +02:00
e9e7139ca4 test: add kafka test/dev environment docker-compose.dev.yml; add README_TEST_WITH_GPAS.md Jakub Lidke 2023-07-27 11:25:32 +02:00