TON-ViT: A Neuro-Symbolic AI Based on Task Oriented Network with a Vision Transformer

Yupeng Zhuo, Nina Jiang, Andrew W. Kirkpatrick, Kyle Couperus, Oanh Tran, Jonah Beck, Deanna Devane, Ross Candelore, Jessica McKee, Chad Gorbatkin, Eleanor Birch, Christopher Colombo, Bradley Duerstock, Juan Wachs*

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

The objective of this paper is to present a neuro-symbolic AI based technique to represent field-medicine knowledge, referred as to TON-ViT. TON-ViT integrates a Deep Learning Model with an explicit symbolic manipulation, a task graph. This task graph describes the steps of each trauma resuscitation as denoted by a verb and noun pair. Through this representation, symbolic processing and manipulation on task graphs, we can find stereotypical procedures, regardless of style of the performer. Furthermore, we can use this technique to find differences in styles, errors, shortcuts and generate procedures never seen before. When used in combination with a transformer, it can help recognize actions in egocentric vision datasets. Last, through symbolic manipulations on the graph, it is possible to generate medical knowledge which the model has not seen before. We present preliminary results after testing the TON-ViT with the Trauma Thompson Dataset.

Original languageEnglish
Title of host publicationMedical Image Understanding and Analysis - 27th Annual Conference, MIUA 2023, Proceedings
EditorsGordon Waiter, Georgios Leontidis, Teresa Morris, Tryphon Lambrou, Nir Oren, Sharon Gordon
PublisherSpringer Science and Business Media Deutschland GmbH
Pages157-170
Number of pages14
ISBN (Print)9783031485923
DOIs
StatePublished - 2024
Externally publishedYes
Event27th Annual Conference on Medical Image Understanding and Analysis, MIUA 2023 - Aberdeen, United Kingdom
Duration: 19 Jul 202321 Jul 2023

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume14122 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference27th Annual Conference on Medical Image Understanding and Analysis, MIUA 2023
Country/TerritoryUnited Kingdom
CityAberdeen
Period19/07/2321/07/23

Keywords

  • knowledge graph
  • medical procedures
  • neuro-symbolic AI
  • semantic understanding
  • task graph
  • vision transformer

Fingerprint

Dive into the research topics of 'TON-ViT: A Neuro-Symbolic AI Based on Task Oriented Network with a Vision Transformer'. Together they form a unique fingerprint.

Cite this