A high-tech aviary at Pennovation Works, equipped with eight computer-vision cameras and 24 high-precision microphones to record the behavior of 10 male and 10 female brown-headed cowbirds, is allowing biologists, physicists and computer scientists to make advancements in our understanding of animal behavior, neuroscience and machine learning. The group began recording in the aviary during this past spring’s breeding season; the cameras and microphones recorded 10 hours a day for 100 days.

The goal of the aviary, originally envisioned by Marc Schmidt, a professor of biology in Penn’s School of Arts and Sciences, is to use the latest in machine learning technology to answer questions about animals’ social behavior that can be addressed in no other way. 

The scientists working on the aviary are collecting data and then developing the tools and algorithms to parse it carefully to make new discoveries. According to Kostas Daniilidis, the Ruth Yalom Stone Professor in the School of Engineering and Applied Science’s department of computer and information science, “It gives us the chance to work on translating two-dimensional data to three-dimensional data, and it also gives us a chance to use AI tools to recognize complex poses in the birds.”

A grant from the National Science Foundation’s Major Research Instrumentation Program funded building the facility. Vijay Balasubramanian, the Cathy and Marc Lasry Professor of Physics in SAS, who had pursued computational work in neuroscience at various levels, is also involved. “The thing I find very interesting about this project is it’s an attempt to completely analyze a developing social network,” he noted. “As a physicist I would like to understand the formation of the collective behavior.”

Cowbirds are a gregarious species that dwell in groups, but they also form breeding pairs each season. One aim of the work is to see how different interactions between the birds give rise to these stable bonds. 

Marc Badger, a postdoctoral researcher in Dr. Daniilidis’s group, is working to craft algorithms capable of discerning different poses of the birds based on their silhouettes. 

Dr. Schmidt would like to expand the experimental work on brain circuits associated with singing and reproductive behavior that he’s conducted in the lab to the aviary. The group is also exploring applications for this kind of observatory so the setup could be used by other research groups pursuing different scientific questions on a variety of species.

For the complete story, visit https://penntoday.upenn.edu/news/smart-aviary-poised-break-new-ground-behavioral-research