Recent studies featuring Tom Franken in iScience and Neuron add new knowledge to how the brain’s assignment of borders to foreground surfaces aids in visual processing.
Tom Franken, MD, PhD, sheds new light on how the brain parses complex visual input with two recently published studies.
The papers in iScience and Neuron deepen the knowledge of how border ownership (BOS) neurons assign borders to foreground objects and how this aids in predicting future visual input. A better understanding of these processes could eventually help unravel diseases such as schizophrenia and autism spectrum disorder.
The iScience paper investigates how the brain predicts what people see next as humans must constantly anticipate future events, such as when catching a ball or navigating a crowd. What’s surprising, according to Franken, is that brain-like BOS signals emerged spontaneously in an artificial neural network called PredNet, which is trained not to identify objects but simply to predict all pixels of the next frame in a natural video.
“Our findings suggest that BOS neurons, a hallmark of the ‘what-stream’ in the primate brain, which is specialized in identifying objects in visual scenes, may be much more important in processing dynamic visual information than previously recognized,” said Franken, who is an assistant professor in the Department of Neuroscience at WashU Medicine. “They may be important to predict future visual input.”
The iScience work is a collaboration with Ralf Wessel, PhD, a WashU physics professor, whose graduate student Zeyuan Ye led the research. The team is investigating how artificial neural networks and primate brains process and predict video imagery through a $427,625 grant from the National Institutes of Health.
The partnership was sparked by a meeting in late 2022 when Franken shared puzzling data on BOS neuron behavior with Wessel and Ye, who were already working with PredNet. A shared curiosity about how BOS neurons might support visual stability led to this result.
“This collaboration was key,” Franken said. “For projects like these, one needs to combine different types of expertise. I had no experience working with artificial neural networks. The Wessel lab’s expertise was critical.”
Neuron paper research builds off postdoc work
With the Neuron paper, Franken and John Reynolds, PhD, a professor at the Salk Institute for Biological Studies, advance insight into how BOS neurons in the brain distinguish between foreground and background. It has been a mystery what information these neurons rely on, as it requires cues that lie far outside their direct sensory range.
A hypothesis proposes the existence of “grouping cells” in a higher brain area that computes persistent proto-object representations and has the properties to endow cells in lower brain areas with selectivity for border ownership.
It is unknown if these grouping cells exist, but Franken and Reynolds identify such a grouping signal in the brain’s visual area 4 (V4).
“We find that several of the signal’s properties are consistent with the hypothesis,” Franken said. “It is fast; it survives brief interruption of the scene; it persists for hundreds of milliseconds if everything except for the local border disappears; and this persistent signal is transferred to other brain cells with eye movements.”
The discovery is significant. Feedback signals in the brain, sent along connections from higher to lower areas, are thought to be disrupted in disorders like schizophrenia and autism spectrum disorder. The grouping signal identified in this study could be an important component of feedback between visual areas, opening new avenues for research that may result in a better understanding of these disorders.
Franken recorded the brain activity reported in the paper while a postdoctoral researcher in Reynolds’ lab at the Salk Institute.
“Dr. Reynolds has been very inspiring to me as a mentor,” Franken said. “First, he has triggered my interest in understanding how the brain performs complex operations to make sense of the visual world. In our daily experience, vision seems so trivial — it doesn’t seem to cost us any effort or energy to make sense of the world through vision. However, the underlying brain computations are enormously complex (up to one-third of our brain’s cortex is devoted to vision) and remain poorly understood.
“Second, his approach to mentorship truly engages and challenges the mentee. He encourages his trainees to think about big, important questions and be creative and bold in tackling them. I try to do the same in my lab.”
Moving forward, the Franken Lab is building on both studies. They are investigating how BOS neurons contribute to video prediction by combining experiments in the brain and artificial neural networks. Simultaneously, they’re examining how grouping signals in V4 interact with BOS neurons by concurrently recording both types of brain activity.