Reconciliation+−⟲ SFTP Server write only inotify S3 s3 copy Trusted source S3 events for created files SQS/Kafka EC2 processor cron + s3 sync 6hours Check file hash -> skip if proceeded S3 versioning, get the newest file to proceed File has rows, batching + commit transaction Idempotent unique key for each rows Rollback: soft delete + Mark and Sweep whole day Rerun the data Data finalization by day: is_frozen = true No re-uploading Link to original
+−⟲ SFTP Server write only inotify S3 s3 copy Trusted source S3 events for created files SQS/Kafka EC2 processor cron + s3 sync 6hours Check file hash -> skip if proceeded S3 versioning, get the newest file to proceed File has rows, batching + commit transaction Idempotent unique key for each rows Rollback: soft delete + Mark and Sweep whole day Rerun the data Data finalization by day: is_frozen = true No re-uploading