Publications

RescueNet: a high resolution UAV semantic segmentation dataset for natural disaster damage assessment

Maryam Rahnemoonfar, Tashnim Chowdhury, and Robin Murphy

Recent advancements in computer vision and deep learning techniques have facilitated notable progress in scene understanding, thereby assisting rescue teams in achieving precise damage assessment. In this paper, we present RescueNet, a meticulously curated high-resolution post-disaster dataset that includes detailed classification and semantic segmentation annotations. This dataset aims to facilitate comprehensive scene understanding in the aftermath of natural disasters. RescueNet comprises post-disaster images collected after Hurricane Michael, obtained using Unmanned Aerial Vehicles (UAVs) from multiple impacted regions. The uniqueness of RescueNet lies in its provision of high-resolution post-disaster imagery, accompanied by comprehensive annotations for each image. Unlike existing datasets that offer annotations limited to specific scene elements such as buildings, RescueNet provides pixel-level annotations for all classes, including buildings, roads, pools, trees, and more. Furthermore, we evaluate the utility of the dataset by implementing state-of-the-art segmentation models on RescueNet, demonstrating its value in enhancing existing methodologies for natural disaster damage assessment.

Presented In : Scientific data (Nature)

Publication Date : 20 December, 2023

Link : https://www.nature.com/articles/s41597-023-02799-4 

 

Flood-ResNet50: Optimized Deep Learning Model for Efficient Flood Detection on Edge Device

Md Azim Khan, Nadeem Ahmed, Joyce Padela, Muhammad Shehrose Raza, Aryya Gangopadhyay, Jianwu Wang, James Foulds, Carl Busart, Robert F. Erbacher

Floods are highly destructive natural disasters that result in significant economic losses and endanger human and wildlife lives. Efficiently monitoring Flooded areas through the utilization of deep learning models can contribute to mitigating these risks. This study focuses on the deployment of deep learning models specifically designed for classifying flooded and non-flooded in UAV images. In consideration of computational costs, we propose modified version of ResNet50 called Flood-ResNet50. By incorporating additional layers and leveraging transfer learning techniques, Flood-ResNet50 achieves comparable performance to larger models like VGG16/19, AlexNet, DenseNet161, EfficientNetB7, Swin(small), andvision transformer. Experimental results demonstrate that the proposed modification of ResNet50, incorporating additional layers, achieves a classification accuracy of 96.43%, F1 score of 86.36%, Recall of 81.11%, Precision of 92.41%, model size 98MB and FLOPs 4.3 billions for the FloodNet dataset. When deployed on edge devices such as the Jetson Nano, our model demonstrates faster inference speed (820 ms), higher throughput (39.02 fps), and lower average power consumption (6.9 W) compared to larger ResNet101 and ResNet152 models.

Published at : International Conference on Machine Learning and Applications (ICMLA), IEEE, 2023

 

Gradient Inversion Attacks on Acoustic Signals: Revealing Security Risks in Audio Recognition Systems

Pretom Roy Ovi, Aryya Gangopadhyay

With a greater emphasis on data confidentiality and legislation, distributed training and collaborative machine learning algorithms are being developed to protect sensitive private data. Gradient exchange has become a widely used practice in those multi-node machine learning systems. But with the advent of gradient inversion attacks, it is already established that private training data can be revealed from the gradients. Gradient inversion attacks covertly spy on gradient updates and backtrack from the gradients to obtain information about the raw data. Although this attack has been widely studied in computer vision and natural language processing tasks, understanding the impact of this attack on acoustic signals still requires a comprehensive investigation. To the best of our knowledge, we are the first to explore gradient inversion attacks on acoustic signals by extracting the speakers’ voices from an audio recognition system. Here, we design a new application of gradient inversion attack to retrieve the audio signal used for training the deep learning model, irrespective of whether the audio has undergone conversion into melspectrogram or MFCC representations prior to feed to neural network. Experimental results demonstrate the capability of our attack method to extract the input vectors of the audio data from the gradients, which highlight the security risks in revealing the sensitive audio data from highly secured systems.

Presenting at : IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2024

 

Skip-WaveNet: A Wavelet based Multi-scale Architecture to Trace Firn Layers in Radar Echograms

Debvrat Varshney, Masoud Yari, Oluwanisola Ibikunle, Jilu Li, John Paden, Maryam Rahnemoonfar, Aryya Gangopadhyay

Echograms created from airborne radar sensors capture the profile of firn layers present on top of an ice sheet. Accurate tracking of these layers is essential to calculate the snow accumulation rates, which are required to investigate the contribution of polar ice cap melt to sea level rise. However, automatically processing the radar echograms to detect the underlying firn layers is a challenging problem. In our work, we develop wavelet-based multi-scale deep learning architectures for these radar echograms to improve firn layer detection. We show that wavelet based architectures improve the optimal dataset scale (ODS) and optimal image scale (OIS) F-scores by 3.99% and 3.7%, respectively, over the non-wavelet architecture. Further, our proposed Skip-WaveNet architecture generates new wavelets in each iteration, achieves higher generalizability as compared to state-of-the-art firn layer detection networks, and estimates layer depths with a mean absolute error of 3.31 pixels and 94.3% average precision. Such a network can be used by scientists to trace firn layers, calculate the annual snow accumulation rates, estimate the resulting surface mass balance of the ice sheet, and help project global sea level rise.

Link to the paper : https://arxiv.org/abs/2310.19574

 

A Reliable Technique for Moving Target Detection using mm Wave FMCW Radars in Adversarial Environmental Conditions 

Debjyoti Chowdhury, Nikhitha Vikram Melige, Biplab Pal, Aryya Gangopadhyay

This paper introduces a novel and computationally inexpensive technique for moving target detection in challenging outdoor settings, utilizing millimeter-wave frequency-modulated continuous-wave (FMCW) radars. Departing from conventional learning-based methods, the approach employs robust digital signal processing (DSP) methodologies, including wavelet transform, FIR filtering, and peak detection, to address variations in reflective data caused by dynamic environmental conditions.The paper underscores the superiority of DSP-based approach over deep learning methods when facing extreme data variations in changing environments.
Accepted by : MDPI, Electronics ( ISSN 2079-9292 )
Link to the paper: https://www.mdpi.com/2079-9292/12/24/5030

 

SAM-VQA: Supervised Attention-Based Visual Question Answering Model for Post-Disaster Damage Assessment on Remote Sensing Imagery, Accepted for IEEE Transaction on Geoscience and Remote Sensing, 2023 (Impact Factor 8.2)

Argho Sarkar, Tashnim Chowdhury,  Robin Roberson Murphy, Aryya Gangopadhyay and Maryam Rahnemoonfar

This study introduces a visual question answering (VQA) framework called supervised attention-based VQA (SAM-VQA) to assess post-disaster damage efficiently using drone imagery, improving decision-making in disaster response and recovery.

 

A Novel ROS2 QoS Policy-enabled Synchronizing Middleware for Co-simulation of Heterogeneous Multi-Robot Systems

Emon Dey, Mikolaj Walczak, Mohammad Saeid Anwar, Nirmalya Roy

A tailored Data Distribution Service (DDS) QoS policy has been suggested to reduce packet loss and transmission latency among ground and aerial agents. At simulation and system levels, the middleware met mission-critical applications’ reliability and high-fidelity criteria with low-latency communication.

 

SynchroSim: An Integrated Co-simulation Middleware for Heterogeneous Multi-robot System

Emon Dey, Jumman Hossain, Nirmalya Roy, Carl Bussart

Here we propose a velocity difference-driven adjustable window size approach with a view to reducing packet loss probability. It takes into account the respective velocities of deployed agents to calculate a suitable window size before transmitting data between them.

 

HeteroEdge: Addressing Asymmetry in Heterogeneous Collaborative Autonomous Systems

Mohammad Saeid Anwar, Emon Dey, Maloy Kumar Devnath, Indrajeet Ghosh, Naima Khan, Jade Freeman, Timothy Gregory, Niranjan Suri, Kasthuri Jayaraja, Sreenivasan Ramasamy Ramamurthy, Nirmalya Roy

Here, we consider a testbed comprising two Unmanned Ground Vehicles (UGVs) and two NVIDIA Jetson devices and posit a self-adaptive optimization framework that is capable of navigating the workload of multiple tasks (storage, processing, computation, transmission, inference) collaboratively on multiple heterogenous nodes for multiple tasks simultaneously. The self-adaptive optimization framework involves compressing and masking the input image frames, identifying similar frames, and profiling the devices for various tasks to obtain the boundary conditions for the optimization framework.

 

Reg-Tune: A Regression-Focused Fine-Tuning Approach for Profiling Low Energy Consumption and Latency

Arnab Neelim Mazumder, Farshad Safavi, Maryam Rahnemoonfar, Tinoosh Mohsenin

Reg-Tune is a regression-based profiling approach to quickly determine the trend of different metrics in relation to hardware deployment of neural networks on tinyML platforms like FPGAs and edge devices.

 

Reg-TuneV2: Hardware-Aware and Multi-Objective Regression-Based Fine-Tuning Approach for DNNs on Embedded Platforms

Arnab Neelim Mazumder, Farshad Safavi, Maryam Rahnemoonfar, Tinoosh Mohsenin

Reg-TuneV2 is a systematic approach to fine-tune DNNs for hardware deployment by considering multiple objectives, including accuracy, power, and latency contours. In addition, this approach uses metric learning to achieve smaller and better-suited configurations for deployment.

 

Mixed Quantization enabled to Tackle Gradient Leakage Attacks in Federated Learning. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023

Ovi Pretom Roy, Emon Dey, Nirmalya Roy, Aryya Gangopadhyay.

Federated Learning (FL) faces vulnerabilities to gradient inversion attacks, which can covertly retrieve sensitive data from model gradients, leading to potential privacy breaches; however, the proposed mixed quantization-enabled FL scheme effectively addresses these issues, providing improved robustness and outperforming existing defense mechanisms in empirical evaluations.

 

Confident federated learning to tackle label flipped data poisoning attacks, In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications IV (Vol. 12113, pp. 189-198). SPIE.

Ovi Pretom Roy, Aryya Gangopadhyay, Busart Carl and Robert F. Erbacher.

This paper introduces “Confident Federated Learning,” a framework that validates label quality by identifying and excluding mislabeled samples during local training, effectively preventing data poisoning attacks in Federated Learning, with validation results showing over 85% accuracy.

 

An Online Continuous Semantic Segmentation Framework With Minimal Labeling Efforts

Masud Ahmed; Zahid Hasan; Tim Yingling; Eric O’Leary; Sanjay Purushotham; Suya You; Nirmalya Roy

In this work, we propose a novel hard sample mining strategy for an active domain adaptation based semantic segmentation network, with the aim of automatically selecting a small subset of labeled target data to fine-tune the network.

 

Benchmarking domain adaptation for semantic segmentation

Masud Ahmed; Zahid Hasan; Khan, Naima ; Sanjay Purushotham; Aryya Gangopadhyay, Suya You; Nirmalya Roy

In this research work, we study domain adaptation based semantic segmentation approaches by benchmarking different data transformation approaches on source-only and single-source domain adaptation setups.

 

Dr. Aryya Gangopadhyay

  1. Hasib-Al Rashid, Pretom Roy Ovi, Carl Busart, Aryya Gangopadhyay, Tinoosh Mohsenin, Tinym2net: A flexible system algorithm co-designed multimodal learning framework for tiny devices. ArXiv Journal, 9th February 2022.
  2. Jangho Lee, Yingxi Rona Shi, Changjie Cai, Pubu Ciren, Jianwu Wang, Aryya Gangopadhyay, Zhibo Zhang, Machine learning based algorithms for global dust aerosol detection from satellite images: inter-comparisons and evaluation. Remote sensing journal, 28th January 2021
  3. Jianwu Wang, Matthias K Gobbert, Zhibo Zhang, Aryya Gangopadhyay, Team-based online multidisciplinary education on big data+ high-performance computing+ atmospheric sciences. Advances in Software Engineering, Education, and e-Learning: Proceedings from FECS’20, FCS’20, SERP’20, and EEE’20. 2021

Dr. Nirmalya Roy