Responsibilities include:
- Developed a spark based application in java which monitors 100s of data ingestion jobs periodically for data completeness, timeouts and successful completions.
- Used Delta tables to fetch summaries for past job runs. Further airflow ensures monitoring job runs periodically.
- Grafana displays metrics as a time series graph, showing all metrics for jobs instantly saving hours of work.