The growing demand for air travel requires technological advancements in air traffic management as well as mechanisms for monitoring and ensuring safe and efficient operations. In terminal airspaces, predictive models of future movements and traffic flows can help with proactive planning and efficient coordination; however, varying airport topologies, and interactions with other agents, among other factors, make accurate predictions challenging. Datadriven predictive models have shown promise for handling numerous variables to enable various downstream tasks, including collision risk assessment, taxi-out time prediction, departure metering, and emission estimations. While data-driven methods have shown improvements in these tasks, prior works lack large-scale curated surface movement datasets within the public domain and the development of generalizable trajectory forecasting models. In response to this, we propose two contributions: (1) Amelia-48 dataset, a large surface movement dataset collected using the System Wide Information Management (SWIM) Surface Movement Event Service (SMES). With data collection beginning in December 2022, the Phase1 Amelia-48 dataset provides more than a year’s worth of SMES data (∼30TB) and covers 48 airports within the US National Airspace System. In addition to releasing this data in the public domain, we also provide post-processing scripts and associated airport maps to enable research in the forecasting domain and beyond. (2) Amelia-TF model, a transformer-based next-token-prediction large multi-agent multi-airport trajectory forecasting model trained on 292 days or 9.4 billion tokens of position data encompassing 10 different airports with varying topology. The open-sourced Amelia-TF model is validated on unseen airports with experiments showcasing the different prediction horizon lengths, ego-agent selection strategies, and training recipes to demonstrate the generalization capabilities.

As autonomous driving technology matures, safety and robustness of its key components, including trajectory prediction, is vital. Though real-world datasets, such as Waymo Open Motion, provide realistic recorded scenarios for model development, they often lack truly safety-critical situations. Rather than utilizing unrealistic simulation or dangerous realworld testing, we instead propose a framework to characterize such datasets and find hidden safety-relevant scenarios within. Our approach expands the spectrum of safety-relevance, allowing us to study trajectory prediction models under a safety-informed, distribution shift setting. We contribute a generalized scenario characterization method, a novel scoring scheme to find subtly-avoided risky scenarios and an evaluation of trajectory prediction models in this setting. We further contribute a remediation strategy, achieving a 10% average reduction in prediction collision rates.

The fast-growing demand for fully autonomous robots in shared spaces calls for the development of trustworthy agents that can safely and seamlessly navigate in crowded environments. Recent models for motion prediction show promise in characterizing social interactions in such environments. Still, adapting them for navigation is challenging as they often suffer from generalization failures. Prompted by this, we propose Social Robot Tree Search (SoRTS), an algorithm for safe robot navigation in social domains. SoRTS aims to augment existing socially aware motion prediction models for long-horizon navigation using Monte Carlo Tree Search. We use social navigation in general aviation as a case study to evaluate our approach and further the research in full-scale aerial autonomy. In doing so, we introduce X-PlaneROS, a highfidelity aerial simulator that enables human-robot interaction. We use X-PlaneROS to conduct a first-of-its-kind user study where 26 FAA-certified pilots interact with a human pilot, our algorithm, and its ablation. Our results, supported by statistical evidence, show that SoRTS exhibits a comparable performance to competent human pilots, significantly outperforming its ablation. Finally, we complement these results with a broad set of self-play experiments to showcase our algorithm’s performance in scenarios with increasing complexity.

We propose developing an integrated system to keep autonomous unmanned aircraft safely separated and behave as expected in conjunction with manned traffic. The main goal is to achieve safe manned-unmanned vehicle teaming to improve system performance, have each (robot/human) teammate learn from each other in various aircraft operations, and reduce the manning needs of manned aircraft. The proposed system anticipates and reacts to other aircraft using natural language instructions and can serve as a co-pilot or operate entirely autonomously. We point out the main technical challenges where improvements on current state-of-the-art are needed to enable Visual Flight Rules to fully autonomous aerial operations, bringing insights to these critical areas. Furthermore, we present an interactive demonstration in a prototypical scenario with one AI pilot and one human pilot sharing the same terminal airspace, interacting with each other using language, and landing safely on the same runway. We also show a demonstration of a vision-only aircraft detection system.

As intelligent robots across domains start collaborating with humans in shared environments, e.g., urban settings and airspace, algorithms that enable them to reason over human motion and intent are important to ensure seamless and safe interplay. Even beyond robotics, other domains, e.g., surveillance and sports analysis, may also benefit from this type of algorithms. In our work, we study human intent by focusing on the problem of predicting trajectories in dynamic environments. We are further interested in designing methods that are able to generalize across domains. Specifically, we target domains where navigation guidelines are relatively strictly defined yet not necessarily marked in their physical environments. We hypothesize that within these domains, in the short-term, agents tend to exhibit motion patterns that reveal important context information related to the agent's general direction, admissible motions, intermediate goals and social influences. From this intuition, we propose Social-PatteRNN, a new recurrent generative model that exploits motion patterns to encode the aforesaid context information and use it as conditioning signal for predicting trajectories. We assess our approach across three different problem domains: human motion in crowds, human motion in sports and aircraft motion in terminal airspace. Finally, we show that our approach achieves state-of-the-art results across these domains.

Generalisation to unseen contexts remains a challenge for embodied navigation agents. In the context of semantic audio-visual navigation (SAVi) tasks, the notion of generalisation should include both generalising to unseen indoor visual scenes as well as generalising to unheard sounding objects. However, previous SAVi task definitions do not include evaluation conditions on truly novel sounding objects, resorting instead to evaluating agents on unheard sound clips of known objects; meanwhile, previous SAVi methods do not include explicit mechanisms for incorporating domain knowledge about object and region semantics. These weaknesses limit the development and assessment of models' abilities to generalise their learned experience. In this work, we introduce the use of knowledge-driven scene priors in the semantic audio-visual embodied navigation task: we combine semantic information from our novel knowledge graph that encodes object-region relations, spatial knowledge from dual Graph Encoder Networks, and background knowledge from a series of pre-training tasks -- all within a reinforcement learning framework for audio-visual navigation. We also define a new audio-visual navigation sub-task, where agents are evaluated on novel sounding objects, as opposed to unheard clips of known objects. We show improvements over strong baselines in generalisation to unseen regions and novel sounding objects, within the Habitat-Matterport3D simulation environment, under the SoundSpaces task.

Recent advances in the areas of multimodal machine learning and artificial intelligence (AI) have led to the development of challenging tasks at the intersection of Computer Vision, Natural Language Processing, and Embodied AI. Whereas many approaches and previous survey pursuits have characterised one or two of these dimensions, there has not been a holistic analysis at the center of all three. Moreover, even when combinations of these topics are considered, more focus is placed on describing, e.g., current architectural methods, as opposed to also illustrating high-level challenges and opportunities for the field. In this survey paper, we discuss Embodied Vision-Language Planning (EVLP) tasks, a family of prominent embodied navigation and manipulation problems that jointly use computer vision and natural language. We propose a taxonomy to unify these tasks and provide an in-depth analysis and comparison of the new and current algorithmic approaches, metrics, simulated environments, as well as the datasets used for EVLP tasks. Finally, we present the core challenges that we believe new EVLP works should seek to address, and we advocate for task construction that enables model generalizability and furthers real-world deployment.

Deep learning-based frameworks have been widely used in object recognition, perception and autonomous navigation tasks, showing outstanding feature extraction capabilities. Nevertheless, the effectiveness of such detectors usually depends on large amounts of training data. For specific object-recognition tasks, it is often difficult and time-consuming to gather enough valuable data. Data Augmentation has been broadly adopted to overcome these difficulties, as it allows to increase the training data and introduce variation in qualitative elements like color, illumination, distortion and orientation. In this paper, we leverage on the object detection framework YOLOv2 to evaluate the behavior of an obstacle detection system for an autonomous boat designed for the International RoboBoat Competition. We are focused on how the overall performance of a model changes with different augmentation techniques. Thus, we analyze the features that the network learns by using geometric and pixel-wise transformations to augment our data. Our instances of interest are buoys and sea markers, thus to generate training data comprising these classes, we simulated the aquatic surface of the boat and collected data from the COCO dataset. Finally, we discuss that significant generalization is achieved in the learning process of our experiments using different augmentation techniques.

This paper proposes an approach to segment point clouds with high levels of vertical sparsity, which are typically generated by low-end LiDARs, like the Velodyne VLP-16. Special consideration is given–but not limited–to the identification of ground points. The approach addresses two important issues: the fact that the sparsity of points makes it hard to infer an object’s structure, and the difficulty of obtaining and annotating data for testing and training. The first issue is tackled by using a lightweight Convolutional Neural Network (CNN) with Recurrent CRF for point-wise class prediction. To address the second issue, which arises from the need to train this network, the proposed approach extracts sparse examples from dense point clouds available in public datasets. The approach was tested using down-sampled data from the KITTI dataset, which contains annotated examples of the object instances car, pedestrian and cyclist. However, since the ground class is often missing from annotated datasets, and given its significance in this work, a ground annotation algorithm was developed and used to automatically label the sparse data and add it to the other labeled classes available from the dataset. The sparse data obtained was used to train and test the CNN to characterize its prediction accuracy. Additionally, the network was tested using data collected with a low-end sensor. The experiments show that the system achieves accurate predictions in real time that are comparable to those reported from denser point clouds.

Wheelchairs are one of the most important auxiliary instruments for people with mobility impairments. Accurate detection and tracking of these devices could bring a number of improvements in automated services that aim to assist, monitor and provide better accessibility to allow wheelchair users to participate in community life. In this paper, we present a Deep Learning-based wheelchair detection and tracking system to address some of the limitations of previous approaches, which include detecting different types of wheelchairs in cluttered environments and from different viewing angles. We explore region-based Convolutional Neural Networks (R-CNN), in particular Faster R-CNN, as it has become one of the top performers for object detection tasks. We evaluate the performance of different training techniques using two Faster R-CNN frameworks and different backbone network structures. Furthermore, we present how we empirically addressed some of the preceding limitations by applying specific data augmentation techniques and constraints to our model. We demonstrate that using a region- based implementation outperforms previous approaches in terms of overall robustness, accuracy and flexibility.