We introduce ScenarioCharacterization, an open-source framework for automated, dataset-agnostic profiling of driving scenarios in trajectory datasets. Our framework is packaged as a modular, configuration-driven pipeline of three layers: a dataset adapter that maps custom datasets onto an open Scenario representation, a characterizer that performs feature extraction, behavior probing, and criticality scoring at scenario and agent levels, and an analysis layer for scenario visualization and feature, score, and probe analyses.

Because the layers communicate only through Pydantic-validated schemas composed via configurations, a new dataset can easily plug in without rewriting the characterization and analysis stack. This technical report describes the design and APIs, shows example outputs on Waymo Open Motion, Argoverse2, and nuPlan, and discusses downstream uses of the approach.

Verification and validation of autonomous driving (AD) systems and components are of increasing importance, as such technology grows in real-world prevalence. Safety-critical scenario generation is a key approach to robustify AD policies through closed-loop training. However, existing approaches for scenario generation rely on simplistic objectives, resulting in overly-aggressive or non-reactive adversarial behaviors.

To generate diverse adversarial yet realistic scenarios, we propose Skill-Enabled Adversary Learning (SEAL), a scenario perturbation approach which leverages learned scoring functions and adversarial, human-like skills. SEAL-perturbed scenarios are more realistic than SOTA baselines, leading to improved ego task success, by more than 20%, across real-world, in-distribution, and out-of-distribution scenarios.

Demand for air travel is rising, straining existing aviation infrastructure. In the US, more than 90% of airport control towers are understaffed, falling short of FAA and union standards. This, in part, has contributed to an uptick in near-misses and safety-critical events, highlighting the need for advancements in air traffic management technologies to ensure safe and efficient operations.

Data-driven predictive models for terminal airspace show promise in addressing these challenges; however, the lack of large-scale surface movement datasets in the public domain has hindered the development of scalable and generalizable approaches.

To address this, we introduce Amelia-42, a first-of-its-kind large collection of raw airport surface movement reports streamed through the FAA’s System Wide Information Management (SWIM) Program, comprising over two years of trajectory data (∼9.19TB) across 42 US airports. We open-source tools to process this data into clean tabular position reports. We also release Amelia42-Mini, a fully processed 15-day sample per airport, on HuggingFace for ease of use. We also present a trajectory forecasting benchmark consisting of Amelia10-Bench, an accessible experiment family using 292 days from 10 airports, as well as Amelia-TF, a transformer-based baseline for multi-agent trajectory forecasting.

As autonomous driving technology matures, the safety and robustness of its key components, including trajectory prediction, are vital. Though real-world datasets, such as Waymo Open Motion, provide realistic recorded scenarios for model development, they often lack truly safety-critical situations. Rather than utilizing unrealistic simulation or dangerous real-world testing, we instead propose a framework to characterize such datasets and find the safety-relevant scenarios hidden within them. Our approach expands the spectrum of safety-relevance, allowing us to study trajectory prediction models under a safety-informed, distribution shift setting.

We contribute a generalized scenario characterization method, a novel scoring scheme to find subtly-avoided risky scenarios and an evaluation of trajectory prediction models in this setting. We further contribute a remediation strategy, achieving a 10% average reduction in prediction collision rates.

The fast-growing demand for fully autonomous robots in shared spaces calls for the development of trustworthy agents that can safely and seamlessly navigate in crowded environments. Recent models for motion prediction show promise in characterizing social interactions in such environments. Still, adapting them for navigation is challenging as they often suffer from generalization failures. Prompted by this, we propose Social Robot Tree Search (SoRTS), an algorithm for safe robot navigation in social domains. SoRTS aims to augment existing socially aware motion prediction models for long-horizon navigation using Monte Carlo Tree Search.

We use social navigation in general aviation as a case study to evaluate our approach and further research in full-scale aerial autonomy. In doing so, we introduce X-PlaneROS, a high-fidelity aerial simulator that enables human-robot interaction. We use X-PlaneROS to conduct a first-of-its-kind user study where 26 FAA-certified pilots interact with a human pilot, our algorithm, and its ablation. Our results, supported by statistical evidence, show that SoRTS exhibits a comparable performance to competent human pilots, significantly outperforming its ablation. Finally, we complement these results with a broad set of self-play experiments to showcase our algorithm’s performance in scenarios with increasing complexity.

We propose developing an integrated system to keep autonomous unmanned aircraft safely separated and behaving as expected in conjunction with manned traffic. The main goal is to achieve safe manned-unmanned vehicle teaming to improve system performance, have each teammate (robot or human) learn from the other in various aircraft operations, and reduce the manning needs of manned aircraft. The proposed system anticipates and reacts to other aircraft using natural language instructions and can serve as a co-pilot or operate entirely autonomously. We point out the main technical challenges where improvements on the current state of the art are needed to enable fully autonomous aerial operations under Visual Flight Rules, offering insights into these critical areas. Furthermore, we present an interactive demonstration in a prototypical scenario with one AI pilot and one human pilot sharing the same terminal airspace, interacting with each other using language, and landing safely on the same runway. We also show a demonstration of a vision-only aircraft detection system.

As intelligent robots across domains start collaborating with humans in shared environments, e.g., urban settings and airspace, algorithms that enable them to reason over human motion and intent are important to ensure seamless and safe interplay. Even beyond robotics, other domains, e.g., surveillance and sports analysis, may also benefit from this type of algorithm. In our work, we study human intent by focusing on the problem of predicting trajectories in dynamic environments. We are further interested in designing methods that are able to generalize across domains. Specifically, we target domains where navigation guidelines are relatively strictly defined yet not necessarily marked in their physical environments. We hypothesize that within these domains, in the short-term, agents tend to exhibit motion patterns that reveal important context information related to the agent's general direction, admissible motions, intermediate goals and social influences. From this intuition, we propose Social-PatteRNN, a new recurrent generative model that exploits motion patterns to encode the aforesaid context information and use it as a conditioning signal for predicting trajectories. We assess our approach across three different problem domains: human motion in crowds, human motion in sports and aircraft motion in terminal airspace. Finally, we show that our approach achieves state-of-the-art results across these domains.

Generalisation to unseen contexts remains a challenge for embodied navigation agents. In the context of semantic audio-visual navigation (SAVi) tasks, the notion of generalisation should include both generalising to unseen indoor visual scenes as well as generalising to unheard sounding objects. However, previous SAVi task definitions do not include evaluation conditions on truly novel sounding objects, resorting instead to evaluating agents on unheard sound clips of known objects; meanwhile, previous SAVi methods do not include explicit mechanisms for incorporating domain knowledge about object and region semantics. These weaknesses limit the development and assessment of models' abilities to generalise their learned experience. In this work, we introduce the use of knowledge-driven scene priors in the semantic audio-visual embodied navigation task: we combine semantic information from our novel knowledge graph that encodes object-region relations, spatial knowledge from dual Graph Encoder Networks, and background knowledge from a series of pre-training tasks -- all within a reinforcement learning framework for audio-visual navigation. We also define a new audio-visual navigation sub-task, where agents are evaluated on novel sounding objects, as opposed to unheard clips of known objects. We show improvements over strong baselines in generalisation to unseen regions and novel sounding objects, within the Habitat-Matterport3D simulation environment, under the SoundSpaces task.

Recent advances in the areas of multimodal machine learning and artificial intelligence (AI) have led to the development of challenging tasks at the intersection of Computer Vision, Natural Language Processing, and Embodied AI. Whereas many approaches and previous survey pursuits have characterized one or two of these dimensions, there has not been a holistic analysis at the intersection of all three. Moreover, even when combinations of these topics are considered, more focus is placed on describing, e.g., current architectural methods, as opposed to also illustrating high-level challenges and opportunities for the field. In this survey paper, we discuss Embodied Vision-Language Planning (EVLP) tasks, a family of prominent embodied navigation and manipulation problems that jointly use computer vision and natural language. We propose a taxonomy to unify these tasks and provide an in-depth analysis and comparison of the new and current algorithmic approaches, metrics, simulated environments, as well as the datasets used for EVLP tasks. Finally, we present the core challenges that we believe new EVLP works should seek to address, and we advocate for task construction that enables model generalizability and furthers real-world deployment.

Deep learning-based frameworks have been widely used in object recognition, perception and autonomous navigation tasks, showing outstanding feature extraction capabilities. Nevertheless, the effectiveness of such detectors usually depends on large amounts of training data. For specific object-recognition tasks, it is often difficult and time-consuming to gather enough valuable data. Data Augmentation has been broadly adopted to overcome these difficulties, as it allows us to increase the training data and introduce variation in qualitative elements like color, illumination, distortion and orientation. In this paper, we leverage the object detection framework YOLOv2 to evaluate the behavior of an obstacle detection system for an autonomous boat designed for the International RoboBoat Competition. We focus on how the overall performance of a model changes with different augmentation techniques. Thus, we analyze the features that the network learns by using geometric and pixel-wise transformations to augment our data. Our instances of interest are buoys and sea markers, thus to generate training data comprising these classes, we simulated the aquatic surface of the boat and collected data from the COCO dataset. Finally, we show that significant generalization is achieved in the learning process of our experiments using different augmentation techniques.

This paper proposes an approach to segment point clouds with high levels of vertical sparsity, which are typically generated by low-end LiDARs, like the Velodyne VLP-16. Special consideration is given–but not limited–to the identification of ground points. The approach addresses two important issues: the fact that the sparsity of points makes it hard to infer an object’s structure, and the difficulty of obtaining and annotating data for testing and training. The first issue is tackled by using a lightweight Convolutional Neural Network (CNN) with Recurrent CRF for point-wise class prediction. To address the second issue, which arises from the need to train this network, the proposed approach extracts sparse examples from dense point clouds available in public datasets. The approach was tested using down-sampled data from the KITTI dataset, which contains annotated examples of the object instances car, pedestrian and cyclist. However, since the ground class is often missing from annotated datasets, and given its significance in this work, a ground annotation algorithm was developed and used to automatically label the sparse data and add it to the other labeled classes available from the dataset. The sparse data obtained was used to train and test the CNN to characterize its prediction accuracy. Additionally, the network was tested using data collected with a low-end sensor. The experiments show that the system achieves accurate predictions in real time that are comparable to those reported from denser point clouds.

Wheelchairs are one of the most important auxiliary instruments for people with mobility impairments. Accurate detection and tracking of these devices could bring a number of improvements in automated services that aim to assist, monitor and provide better accessibility to allow wheelchair users to participate in community life. In this paper, we present a Deep Learning-based wheelchair detection and tracking system to address some of the limitations of previous approaches, which include detecting different types of wheelchairs in cluttered environments and from different viewing angles. We explore region-based Convolutional Neural Networks (R-CNN), in particular Faster R-CNN, as it has become one of the top performers for object detection tasks. We evaluate the performance of different training techniques using two Faster R-CNN frameworks and different backbone network structures. Furthermore, we show how we empirically addressed some of the preceding limitations by applying specific data augmentation techniques and constraints to our model. We demonstrate that a region- based implementation outperforms previous approaches in terms of overall robustness, accuracy and flexibility.