Satellite data processing and anomaly monitoring tools

🧊Full-stack Flask application

• Developed a full-stack Flask application and deployed on a linux server for real-time product monitoring, incorporating advanced file processing to extract and analyze remote sensing products, including L0, L1, and L2.
• The tool facilitates the comparison of product availability between the EUMETSAT Data Lake and the data received from ground stations, ensuring seamless integration and synchronization of satellite data for accurate monitoring and analysis.
• The tool can also monitor each L0, L1 and L2 product file size and compare them with the expected size with respect to requirement documents.
• Designed and implemented a NoSQL Elasticsearch database to efficiently store and manage structured log data for advanced querying and analysis.
• Automated data ingestion with scheduled tasks to ensure continuous updates, keeping the database synchronized with the latest logs.
• Built dynamic dashboards with interactive and static visualizations, leveraging processed log data to enhance system monitoring and decision-making.
• Created and configured customized Kibana dashboards using high-level Vega scripting, integrated with Elasticsearch to monitor and visualize size of L0, L1, and L2 products. The dashboards include interactive filters, time-series visualizations, and alerting features to detect size deviations and support timely troubleshooting.
• Integrated end-to-end workflows, from raw data extraction and transformation to database storage and real-time analytics, optimizing operational efficiency.

🧊 Telemetry packet analysis tool

• Developed a Jupyter Notebook-based tool to analyze telemetry packets from L0 satellite products, focusing on the detection of missing or anomalous packets. The tool aids in identifying unusual patterns and behaviors in specific products. Integrated the solution with the tm-packet-analyser utility to extract detailed packet-level information for in-depth analysis.