Support

Agenda

• \(D(\Psi) >> D(\Phi)\)
• 4 realms and their interactions
• You say you want a revolution…
• Fellow travelers…
• Closing the loop
• Take homes
• Our (essential) open science future

But why?

By Pieter Brueghel the Elder - bAGKOdJfvfAhYQ at Google Cultural Institute zoom level Scaled down from second-highest, Public Domain, Link

The ‘Rumsfeldian’ space of knowledge

The ‘Rumsfeldian’ space of knowledge

Oh give me a ’ome…

• Genome
• Proteome
• Metabolome
• Connectome
• Behaviorome?

“Behavior is the linchpin of the most vexing problems in public health, and a better understanding of behavior is fundamental to achieving positive health outcomes, from prenatal development throughout adulthood.”

Gilmore, Adolph, & Tamis-LeMonda, 2019

“Behavior contributes to the progression or prevention of disease, defines a disorder or marks recovery, and provides mechanisms for therapeutic intervention.”

Gilmore, Adolph, & Tamis-LeMonda, 2019

What to do about it?

Nested causality

• Mind (\(M\)) within nervous system (\(N\))
• Nervous system (\(N\)) within body (\(N\))
• Body (\(B\)) within world (\(W\))

Mutual coupling of dynamic states

\(\dot{M} = f(M,N)\)

\(\dot{N} = f(N,B)\)

\(\dot{B} = f(B,N,W)\)

\(\dot{W} = f(W,B)\)

Measure

• \(W\), \(B\), \(N\) directly
• At varied spatial & temporal scales

Sejnowski, Churchland, & Movshon, 2014

Measure

• \(M\) indirectly
• Only via \(N\), \(B\)

\(M\) highly salient, but…

By Krauss - Own work, CC BY-SA 4.0, Link

“How should an experimenter proceed when faced with a black box?”

Ashby, 1956/2015, 6/2

“What properties of the Box’s contents are discoverable and what are fundamentally not discoverable?”

Ashby, 1956/2015, 6/2

“What methods should be used if the Box is to be investigated efficiently?”

Ashby, 1956/2015, 6/2

“The primary data of any investigation of a Black Box consists of a sequence of values of the vector with two components: (input state, output state).”

Ashby, 1956/2015, 6/5

“From this there follows the fundamental deduction that all fundamental knowledge obtainable from a Black Box (of given input and output) is such as can be obtained by re-coding the protocol [sequence of input/output measurements].”

Ashby, 1956/2015, 6/5

“What is being suggested now is not that Black Box behavior is somewhat like real objects, but that the real objects are in fact all Black Boxes, and that we have in fact been operating with Black Boxes all our lives.”

Ashby, 1956/2015, 6/18

“The theory of the Black Box is simply the study of the relations between the experimenter and his environment, when special attention is given to the flow of information.”

Ashby, 1956/2015, 6/18

B.F. Skinner

\(S \rightarrow R\) psychology

• \(S\) (stimulus) -> \(R\) (response)
• \(S\) and \(R\) observable

• \(S\) is the effect of some \(W_i\) on \(B\)
• \(R\) is some change in \(B\) (\(\dot{B_i}\)) that is observable in \(W\)

Noam Chomsky

\(S \rightarrow C \rightarrow R\)

• \(S\) (stimulus) -> \(C\) (cognition/computation) -> \(R\) (response)
• \(C\) inferred from structure in \(S\)->\(R\)
• \(C\) elaborate, probably innate
• \(S\) alone insufficient to explain \(C\), especially language (Chomsky, 1959)

The cognitive ‘revolution’

• \(C\) (\(M\)) viewed in algorithmic terms
• Formal operations on information structures
• Developed in parallel with computer science, AI

The cognitive (neuroscience) ‘revolution’

• Developed alongside cognitive science
• \(C (M) \leftrightarrow N\)
• What \(N_i\) compute functions of \(C (M)\)?
• Assume 1:1 relationship between \(N\) and \(C (M)\)

The problem with revolutions…

Linear/open-loop causal chains

• \(S \rightarrow R\)
• \(S \rightarrow C \rightarrow R\)

vs. closed-loop causal chains

\(R\) (actions) create effects in the World (\(W\))

\(W\) states induce sensations (\(S\))

\(S\) affect \(C\) (computations/cognitions)

\(C\) affect \(A\)

…

Consequences…

• Properties of \(W\) downplayed, ignored
• Why are \(S\) or \(C\) causal but not \(R\)?
• Variability in behavior \(R\) only noise
• Origins of \(R\), \(C\), \(A\)?
• \(W \leftrightarrow B \leftrightarrow N\) reflect ontogenetic and phylogenetic history

“Reports that say that something hasn’t happened are always interesting to me, because as we know, there are known knowns; there are things we know we know.”

https://en.wikipedia.org/wiki/There_are_known_knowns

“We also know there are known unknowns; that is to say we know there are some things we do not know.”

https://en.wikipedia.org/wiki/There_are_known_knowns

“But there are also unknown unknowns—the ones we don’t know we don’t know. And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones.”

https://en.wikipedia.org/wiki/There_are_known_knowns

The ‘unknown-known’ of cybernetics

Alan Turing’s letter to Ross Ashby (~1946)

“In working on the ACE [Automatic Computing Engine] I am more interested in the possibility of producing models of the action of the brain than in the practical applications to computing.”

Turing’s letter to Ashby, 1946

“The ACE will be used, as you suggest,…in an entirely undisciplined manner, similar to the action of the lower centres, although the reflexes are extremely complicated.”

Turing’s letter to Ashby, 1946

“The disciplined action carries with it the disagreeable feature, which you mentioned, that it will be entirely uncritical when anything goes wrong.”

Turing’s letter to Ashby, 1946

“It will also be necessarily be devoid of anything that could be called originality.”

Turing’s letter to Ashby, 1946

“It would be quite possible for the machine to try out variations in behaviour and accept or reject them in the manner you describe, and I have been hoping to make the machine do this…”

Turing’s letter to Ashby, 1946

Ashby’s homeostat

In neuroscience

Algorithms vs. Dynamics (Van Gelder, 1995)

• How to regulate the speed of a steam engine?

By Nicolás Pérez, CC BY-SA 3.0, Link

1. Measure the speed of the flywheel.
2. Compare the actual speed against the desired speed.
3. If there is no discrepancy, return to step 1. Otherwise,
    a. measure the current steam pressure;
    b. calculate the desired alteration in steam pressure;
    c. calculate the necessary throttle valve adjustment.
4. Make the throttle valve adjustment.
5. Return to step 1. 

By R. Routledge - Image from “Discoveries & Inventions of the Nineteenth Century” by R. Routledge, 13th edition, published 1900., Public Domain, Link

David Marr & J.J. Gibson

What if we took Marr too literally?

• Computational theories matter
• Algorithms matter
• Hardware matters

Biological computing

• Constrained by space, time, energy

25 W vs. ?? MW

Biological computing

• Computes with chemistry (when possible), and ‘wires’ (when necessary)

Biological computing

• Engages in real-time behaviors with existential consequences (e.g., ingestion, defense, reproduction, locomotion, pursuit)
• Interacts with and in dynamic environments

Biological computing

• Separate neural ‘streams’ for \(S \leftrightarrow C \leftrightarrow A\)

Am I my connectome?

“What we know from C. elegans and the crustacean stomatogastric ganglion, with networks involving very few neurons, is that we haven’t a clue how they work even with the essential addition of electrophysiology.”

Costandi, 2012

“So, while SS’s memorable (if pukeworthy) ‘I am more than my genome, I am my connectome’ may be true in a grossly superficial way, they are not going resolve connectional issues in any useful way using their current approaches.”

Costandi, 2012

“So, while SS’s memorable (if pukeworthy) ‘I am more than my genome, I am my connectome’ may be true in a grossly superficial way, they are not going resolve connectional issues in any useful way using their current approaches.”

Costandi, 2012

Jonas & Kording, 2017

“We show that [classic analytic neuroscience] approaches reveal interesting structure in the data but do not meaningfully describe the hierarchy of information processing in the microprocessor.”

Jonas & Kording, 2017

“This suggests current analytic approaches in neuroscience may fall short of producing meaningful understanding of neural systems, regardless of the amount of data.”

Jonas & Kording, 2017

Perceptual Control Theory (PCT)

William T. Powers

Powers, 1973

Powers, 1973

Tenets of PCT

• Perception is the source of controlled variables
• Actions control perception with respect to some referent signal
• Nested hierarchies of perceptual control systems

Wang et al. 2012

Related approaches

• Dynamic Systems Theory (DST) (Van Gelder)
• Ecological Psychology (e.g., Gibson, Turvey)
• Forward models (e.g., Shadmehr & Mussa-Ivaldi; Wolpert)
• Cognitive control/executive functions
• Reafference, Helmholtzian ‘unconscious inference’

What is being controlled?

• Control illustrates the purposive dimension of behavior (Tolman)
• Perceptual states (PCT) vs. actions (Ecological psychologists, forward modelers)
• Mappings from \(S \rightarrow R\) and \(C \rightarrow C\) (cognitive control/executive functions)

What do animals control?

Internal states

• Hunger, thirst, activity/arousal level, temperature, …

• Mental states (mood, thought processes)?

External states

• Direction/position/motion of sensory organs
• Body postures, movements in space
• Emissions (e.g., sound)
• Secretions

Visuo-motor pursuit

Powers, 1973

Challenges

• Deep ‘theories of tasks’
• Amenable to control variable analyses

Powers, 1973

Challenges

• But which tasks?

Krakauer et al., 2017

Criteria for task selection

• Ethological relevance
• Transparent control variables
• Measurable state spaces
• Rich ‘computational’ theory of the task

Peripheral control points

• Vision: direction of gaze, eyes open/closed, pupil diameter, lens thickness
• Audition: head position, ears covered/un-, acoustic/stapedius reflex, outer hair cells in cochlea

Locomotion

• Foundational example of “knowing” (Turvey)
• Systematic, often rhythmic changes in body configuration
• Produce forces that propel organism
• Control variables: speed, heading direction, posture

Embedded within locomotion

• Perceiving a ‘locomotable’ trajectory
• Perceiving stable body posture

Perceiving a ‘locomotable’ trajectory

• From vision/audition -> surface layout
• From kinesthesia -> body orientation relative to contact surface

Constraints on ‘locomotable’ trajectory

• Wide enough to permit passage of body (geometric properties)
• In general direction of desired heading (geometric properties)
• Direction toward or away depending on valence of target(s)

Fajen & Warren 2003

Constraints on ‘locomotable’ trajectory

• With suitable supporting surface (friction, support rigidity, size)
• With sufficient ‘clearance’ to avoid collision or injury

Controlling speed

• From proprioception -> energy expenditure, limb movement rate, respiration and heart rate
• From vision -> optical flow (as a function of distance)
• From audition -> change in heard azimuth, elevation
• From vestibular sense -> \(\int{\ddot{x}dx}\)

Perceiving stable body posture

• From vestibular sense -> linear and angular head accelerations (gravity +)
• From kinesthesia -> body configuration and Center of Mass (CoM)
• From kinesthesia -> base of support

Perceiving surface layout (visually)

• Stabilize gaze direction (despite head/body movements)
• Regulate binocular fixation
• Sample visual field
• Regulate optical focus

Stabilizing gaze direction

• Vestibulo-ocular Response (VOR)
• Optokinetic Nystagmus
• Pursuit
• Saccades & microsaccades

Controlling posture

• Righting reflex (relative to gravity)
• Myotatic reflexes (relative to target joint angle configuration)

Challenges

• Characterizing states
• Characterizing state transitions
• Is bigger better?

“Rhymes with B” -> H

Inspired by Ashby, 1956/2015

Challenges

• Dimensionality of input stream (\(S\)) state spaces
• Energy (EM, mechanical, thermal) or material fields in \(W\)

Plenoptic function (Adelson & Bergen, 1991)

\(P(\theta,\phi,\bar{\lambda},t,V_{x},V_{y},V_{z})\).

\((V_{x},V_{y},V_{z})\): locations in \(W\)

\((\theta,\phi)\): optic axis, usually aligned with \(V_{z}\)

\(\bar{\lambda}\): wavelength distribution

Challenges

• Most biologically meaningful events are multimodal
• Handling mixtures of (within or between mode) perceptual signals?

e.g., object distance from vision (monocular, binocular, motion cues, …)

object distance from audition, olfaction, proprioception, or

body posture from vision, vestibular proprioception, muscle/joint proprioception

\(P(t)=\sum_{i=1}^n w_{i}(t)S_{i}(t)\)

\(\sum_{i=1}^n w_{i}(t) = 1\)

Challenges

• Characterizing statistics of the perceptual environment…
• Natural scene statistics

Geisler, 2008

Challenges

• States of perceptual environment across ontogenetic development…

Gilmore et al., 2015

Gilmore et al., 2015

Visual motion statistics differ across age due to

• Changes in body posture (crawling vs. walking)
• Locomotor speed
• Increasingly stable gaze

Adolph, 2013

Even earlier ontogenetic origins

• Spontaneous activity
• In retina (e.g., Shatz)
• In muscles (twitching; Blumberg)

Challenges

• Behaviors sequential, multivariate
• Mixtures of control laws…

Crawl/walk to mom + keep balance + stabilize gaze + look around + …

\(A(t)=\sum_{i=1}^n w_{i}(t)F_{i}\)

\(\sum_{i=1}^n w_{i}(t) = 1\)

Challenges

• Temporal delays
• Info from simultaneous events propagates at speeds differing by 100x
• Need theory of ‘psycho/neural’ relativity?
• Comparable challenges for output streams

Challenges

• Access to sufficiently rich datasets with multiple state variables
• Clear specification of controlled systems & control variables
• Widespread open sharing
• Imagine easy ‘meta-analyses’ of control variables & dynamical relationships

Krakauer et al., 2017

Despite the challenges

• Many (most?) are ‘known-unknowns’

Different ways to characterize Black Boxes

(Maunsell)

(Wang et al. 2012)

Complicate before simplifying?

And yet…

1. An organism in some state of relative constancy (\(\dot{B}<\epsilon\)) is under the influence of one or more control systems.
2. \(\dot{B}=f(B,N,W)\)
3. For every \(R_i\) (\(\dot{B_i}\)) there is some \(S_i\) (\(\dot{B_i}\) or \(\dot{W_i}\)) …

Is there a reproducibility crisis in science?

• Yes, a significant crisis
• Yes, a slight crisis
• No crisis
• Don’t know

Have you failed to reproduce an experiment from your lab or someone else’s?

Why this matters

“The advancement of detailed and diverse knowledge about the development of the world’s children is essential for improving the health and well-being of humanity…”

“We regard scientific integrity, transparency, and openness as essential for the conduct of research and its application to practice and policy…”

https://www.srcd.org/about-us/policy-scientific-integrity-transparency-and-openness

Open science will accelerate discovery

• Open sharing of materials, analysis code, & data
• Causal/dynamic models in discoverable formats
• Video + physio + other temporally dense time series?
• Databrary.org

An openly-shared ‘behaviorome’

rogilmore@psu.edu
https://gilmore-lab.github.io
https://gilmore-lab.github.io/2019-03-25-cne/
@rogilmore

sessionInfo()

## R version 3.5.2 (2018-12-20)
## Platform: x86_64-apple-darwin15.6.0 (64-bit)
## Running under: macOS Mojave 10.14.3
## 
## Matrix products: default
## BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] igraph_1.2.2       databraryapi_0.1.4
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.0       revealjs_0.9     digest_0.6.18    assertthat_0.2.0
##  [5] R6_2.4.0         magrittr_1.5     evaluate_0.13    httr_1.4.0      
##  [9] stringi_1.3.1    curl_3.3         keyring_1.1.0    rmarkdown_1.11  
## [13] tools_3.5.2      stringr_1.4.0    xfun_0.4         yaml_2.2.0      
## [17] compiler_3.5.2   pkgconfig_2.0.2  htmltools_0.3.6  knitr_1.21