Sunday, April 14, 2013

The Lay of the Land

The goal of this project will be, ultimately, to turn brain signals from the complex and data-rich visual cortex into a symphony, so that we can hear the visual cortex at work while subjects undergoing neuroimaging are viewing a visual stimulus.

My background is in studying decision-making processes on environmental issues using fMRI, for my PhD in E-IPER, The Interdisciplinary Program in Environment and Resources. However, analyzing the visual cortex has a number of idiosyncrasies and complexities compared to the regions I normally study, involved in emotional decision-making and reward/valuation analysis. The visual cortex is one of the most studied and complex areas of the brain, allowing the roles of its regions to be clearly and precisely articulated (as opposed to regions associated with more nebulous concepts like theory of mind, empathy, or cost-benefit analysis). To elicit a map of what bits of grey do what, stimuli that are often geometric in nature: checkerboard patterns of rotating wedges, and shrinking and expanding rings, are often used. The goal of this first piece will be to break down a data set when someone is viewing one of these retinotopic mapping stimulus sets, and form a piece that viewers can listen to as they watch the stimulus themselves. If this works, we can try it with more visually complex and engaging stimuli.

The process of creating this music will go roughly as follows:

  1. Obtain a retinotopic data set.
  2. Isolate regions of interest (ROIs) in the visual cortex, that have distinct reactions to different aspects of the stimulus (i.e., the way the lines are oriented, the scale of the objects, the colors, etc.)
  3. Obtain time courses for each ROI. In other words, I will know what the strength of the fMRI signal was for each of these ROIs, and how it changed over the duration of the stimulus presentation.
  4. Convert the time courses into MIDI notes or control changes (CCs). There are a number of interpretive liberties at this stage which we'll go into later.
  5. Arrange instruments and effects within a DAW that will play these notes and respond to these CCs.
  6. Present the stimulus set to an audience visually, as they hear the arrangement.
As luck would have it, the other course I'm taking this quarter is Psych 204B, Computational Neuroimaging: Analysis Methods, taught by members of Stanford's Vision and Perception Lab. Last week I spent a little time meeting with them and getting an idea of what might be necessary in order to obtain a retinotopic data set. I was led to Mr. Vista, a MATLAB interface for obtaining functional and anatomical data from fMRI data sets. On this website there is also a very clean (i.e., not noisy) sample data set. However, in order to dump out time courses - i.e., what's lighting up when - I need to learn Mr. Vista, learn exactly what to look for in the neuroanatomy of the visual cortex, and then hand-draw the ROIs on the data set. If this were a well-studied area for my lab, i.e., the nucleus accumbens (reward center) or anterior cingulate cortex (detects errors between prediction/expectation and outcomes, cognitive dissonance), we would have predefined ROIs and a Python script that would dump out the time courses. However, since we don't and the 204B group would like us to analyze a data set using Mr. Vista regardless, I'll be learning the program and the neuroanatomy in order to do this.

An interesting side story I learned is that we can actually derive more data than just signal strength. In my lab, only signal strength matters - however, there are at least 2 more dimensions that vision scientists care about in the data, and these can be derived for each ROI using Mr. Vista. This is very exciting for our purposes, as we can control multiple aspects of each instrument: for example, the pitch, velocity, and overlaid effects. We can also control other modulations of each instrument by equations involving these 3 dimensions, in addition to equations involving regions picked out in other analyses, like PCA (which can get signals not just of specific ROIs, but of distinct networks of neurons that are working together during the task, across the entire brain).

The other intriguing detail I learned - which Kevin Weiner of the vision lab had initially thought would be a disadvantage - is the presence of an echo signal. There are stratified regions of the visual cortex that pass information along across different streams, that become more and more specialized. Apparently some of these regions will basically have almost identical signal patterns, only delayed in time. Imagine a phrase being played by the cellos first, then followed a moment later by the violins, and you'll begin to grasp the kind of structures and impressions of intentionality that could fall out of this kind of a data set when you render it to audio.

Ultimately we'll be converting the time courses we get in CSV form into MIDI notes. To do this I'll be using this CSV to MIDI converter. Tonight I tested it and it worked with a little editing of the batch file and installation of the latest Java development kit. Over the coming week, I'll be building a Signal-to-CSV converter in the statistical program R that will make a bland data set into a CSV that is ready for the MIDI converter. This will involve defining channels, note durations, tempo, and most importantly, the ability to easily convert the raw signal numbers into a MIDI scale. I'd like to put in place packages, so that, say you want the potential notes to be output in F Blues scale rather than all possible notes, we auto-allot the signal across just the notes that comprise F Blues. I would have various command streams to output the initial CSV in a new file that specifies which conversion is used for each column. This doesn't just have to be used for defining scales: say I enjoy a specific VST which has amazing percussive sounds in 17 widely and unevenly distributed notes, but I don't much care for the rest? I would find out which MIDI notes these are, and specify them in an array. The program reads the length of the array and then auto-splices and assigns all signal values amidst those 17. Similarly, if you want to cut the high end out of the possible values for a MIDI CC, you just end the array early - say, 0-95.

Over the next week I'll not only be working on this converter but also reading up on the visual cortex and downloading Mr. Vista. Here's the first primer I'm tackling: Visual Field Maps in the Human Cortex. This should give me an idea of how to chop up the ROIs. Hopefully next week I'll be at a stage where I can actually define these ROIs in Mr. Vista and output a workable data set. Creating an underlying Signal-to-CSV converter that can quickly define scales and other presets for each column will take some time as well, so if Mr. Vista is a bit slow to pick up, I'll still have plenty to chew on. With the signal converter lined up, it should be straightforward to send time courses right to the arrangement stage.

Stay tuned!

No comments:

Post a Comment