MISSION STATEMENT AND GOALS
Project homepage: https://bifold.berlin/
This BIFOLD project will carry out the research around neural-symbolic learning and reasoning with the following topics. The first topic is how to develop efficient and scalable algorithms to enable query answering over visual stream data generated by cameras and lidars in traffic monitoring and autonomous vehicles. While DNNs can accurately extract bounding boxes, visual relationships, key points and object trajectories from video frames or lidar point clouds to answer rich online/offline analytical queries over such streams, however, naively employing DNNs on every video frame or lidar scan are extremely computing-intensive and costly. For instance, cities now deploy tens of thousands of CCTV cameras (London alone has over 500,000 CCTV) and a self-driving car can generate several GBs of data per second from 4-6 cameras and 2-4 lidars but a heavy duty workstation can only process around 10-1000 video frames per second with object detections and multi-object tracking pipelines. Hence, we will extend graphical probabilistic models of SSR to capture semantic and casual relationships of objects extracted via DNNs in key video frames or lidar scans so that we can develop optimisation algorithms that can exploit such relationships to accelerate visual stream ingestion and query answering. The advantage of employing our neural-symbolic reasoning and learning approach in this context is the explainability and interpretability inherently offered by the semantic reasoning capabilities.
To automate the process of constructing if-then-rules for such graphical probabilistic models, the second topic will be how to develop scalable algorithms for learning reasoning program structures using the counter-parts from sum-product networks, probabilistic circuits or neural network architecture search to meets certain optimization goals, e.g. accuracy, throughput and bandwidth. The core elements to build such reasoning program structures are semantic representations of objects, events, hierarchical abstractions, spatial-temporal relationships and affordances. Hence, algorithms to be developed will exploit commonsense and domain specific knowledge graphs (Event Calculus, ConceptNet, ATOMIC, VisualComet, YAGO, etc) as guiding/template structures for the learning processes. The main processing load of the algorithms that facilitate neural-symbolic reasoning in the above topics lies in the model-counting processes which need to repeatedly execute SAT, SMT and ILP/MIP solvers or the likes to enumerate or to sample for satisfied models to derive the probability of the underlying graphical probabilistic model. Hence, the third topic is studying graph neural networks and reinforcement learning techniques to accelerate the continually solving processes by remembering previous successful search branches. The expected outcome of this topic will pay the way for solving the problem on how to optimise the data fusion engines to be developed in the process. Subsequently, the next topic of interest is to develop algorithms to continuously change storage layouts, execution plans and cost metrics/models to adapt with the changes in data distributions and structures from incoming data streams. Such adaptive optimization algorithms are the core building blocks to continue our development Fed4Edge which aims to enable effective coordination among autonomous data fusion engines tailored for ROS and edge/fog computing devices used in connected vehicles (vehicular networks, robots in factory floors and drones with 5G).
Topics: