The whole elephant




  • \(D(\Psi) >> D(\Phi)\)
  • 4 realms and their interactions
  • You say you want a revolution…
  • Fellow travelers…
  • Closing the loop
  • Take homes
  • Our (essential) open science future

\(D(\Psi) >> D(\Phi)\)

But why?

Pieter Bruegel the Elder - The Tower of Babel (Vienna) - Google Art Project.jpg
By Pieter Brueghel the Elder - bAGKOdJfvfAhYQ at Google Cultural Institute zoom level Scaled down from second-highest, Public Domain, Link

The ‘Rumsfeldian’ space of knowledge

known-knowns known-unknowns
unknown-knowns unknown-unknowns

The ‘Rumsfeldian’ space of knowledge

known-knowns known-unknowns
unknown-knowns unknown-unknowns

Oh give me a ’ome…

Behavior is the linchpin of the most vexing problems in public health, and a better understanding of behavior is fundamental to achieving positive health outcomes, from prenatal development throughout adulthood.

Gilmore, Adolph, & Tamis-LeMonda, 2019

Behavior contributes to the progression or prevention of disease, defines a disorder or marks recovery, and provides mechanisms for therapeutic intervention.

Gilmore, Adolph, & Tamis-LeMonda, 2019

What to do about it?

4 realms and their interactions

Realm Domain
\(W\) The world
\(B\) The body
\(N\) The nervous system
\(M\) The mind

Nested causality

  • Mind (\(M\)) within nervous system (\(N\))
  • Nervous system (\(N\)) within body (\(N\))
  • Body (\(B\)) within world (\(W\))

Mutual coupling of dynamic states

\(\dot{M} = f(M,N)\)

\(\dot{N} = f(N,B)\)

\(\dot{B} = f(B,N,W)\)

\(\dot{W} = f(W,B)\)


  • \(W\), \(B\), \(N\) directly
  • At varied spatial & temporal scales

Sejnowski, Churchland, & Movshon, 2014


  • \(M\) indirectly
  • Only via \(N\), \(B\)

\(M\) highly salient, but…

By Krauss - Own work, CC BY-SA 4.0, Link

How should an experimenter proceed when faced with a black box?

Ashby, 1956/2015, 6/2

What properties of the Box’s contents are discoverable and what are fundamentally not discoverable?

Ashby, 1956/2015, 6/2

What methods should be used if the Box is to be investigated efficiently?

Ashby, 1956/2015, 6/2

The primary data of any investigation of a Black Box consists of a sequence of values of the vector with two components: (input state, output state).

Ashby, 1956/2015, 6/5

From this there follows the fundamental deduction that all fundamental knowledge obtainable from a Black Box (of given input and output) is such as can be obtained by re-coding the protocol [sequence of input/output measurements].

Ashby, 1956/2015, 6/5

What is being suggested now is not that Black Box behavior is somewhat like real objects, but that the real objects are in fact all Black Boxes, and that we have in fact been operating with Black Boxes all our lives.

Ashby, 1956/2015, 6/18

The theory of the Black Box is simply the study of the relations between the experimenter and his environment, when special attention is given to the flow of information.

Ashby, 1956/2015, 6/18

You say you want a revolution…

B.F. Skinner

\(S \rightarrow R\) psychology

  • \(S\) (stimulus) -> \(R\) (response)
  • \(S\) and \(R\) observable

  • \(S\) is the effect of some \(W_i\) on \(B\)
  • \(R\) is some change in \(B\) (\(\dot{B_i}\)) that is observable in \(W\)

Noam Chomsky

\(S \rightarrow C \rightarrow R\)

  • \(S\) (stimulus) -> \(C\) (cognition/computation) -> \(R\) (response)
  • \(C\) inferred from structure in \(S\)->\(R\)
  • \(C\) elaborate, probably innate
  • \(S\) alone insufficient to explain \(C\), especially language (Chomsky, 1959)

The cognitive ‘revolution’

  • \(C\) (\(M\)) viewed in algorithmic terms
  • Formal operations on information structures
  • Developed in parallel with computer science, AI

The cognitive (neuroscience) ‘revolution’

  • Developed alongside cognitive science
  • \(C (M) \leftrightarrow N\)
  • What \(N_i\) compute functions of \(C (M)\)?
  • Assume 1:1 relationship between \(N\) and \(C (M)\)

The problem with revolutions…

Linear/open-loop causal chains

  • \(S \rightarrow R\)
  • \(S \rightarrow C \rightarrow R\)

vs. closed-loop causal chains

\(R\) (actions) create effects in the World (\(W\))

\(W\) states induce sensations (\(S\))

\(S\) affect \(C\) (computations/cognitions)

\(C\) affect \(A\)


  • Properties of \(W\) downplayed, ignored
  • Why are \(S\) or \(C\) causal but not \(R\)?
  • Variability in behavior \(R\) only noise
  • Origins of \(R\), \(C\), \(A\)?
  • \(W \leftrightarrow B \leftrightarrow N\) reflect ontogenetic and phylogenetic history

Reports that say that something hasn’t happened are always interesting to me, because as we know, there are known knowns; there are things we know we know.

We also know there are known unknowns; that is to say we know there are some things we do not know.

But there are also unknown unknowns—the ones we don’t know we don’t know. And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones.

The ‘unknown-known’ of cybernetics

Alan Turing’s letter to Ross Ashby (~1946)

In working on the ACE [Automatic Computing Engine] I am more interested in the possibility of producing models of the action of the brain than in the practical applications to computing.

Turing’s letter to Ashby, 1946

The ACE will be used, as you suggest,…in an entirely undisciplined manner, similar to the action of the lower centres, although the reflexes are extremely complicated.

Turing’s letter to Ashby, 1946

The disciplined action carries with it the disagreeable feature, which you mentioned, that it will be entirely uncritical when anything goes wrong.

Turing’s letter to Ashby, 1946

It will also be necessarily be devoid of anything that could be called originality.

Turing’s letter to Ashby, 1946

It would be quite possible for the machine to try out variations in behaviour and accept or reject them in the manner you describe, and I have been hoping to make the machine do this…

Turing’s letter to Ashby, 1946

Ashby’s homeostat

Fellow travelers…

In neuroscience

Algorithms vs. Dynamics (Van Gelder, 1995)

  • How to regulate the speed of a steam engine?

Maquina vapor Watt ETSIIM.jpg
By Nicolás Pérez, CC BY-SA 3.0, Link

1. Measure the speed of the flywheel.
2. Compare the actual speed against the desired speed.
3. If there is no discrepancy, return to step 1. Otherwise,
    a. measure the current steam pressure;
    b. calculate the desired alteration in steam pressure;
    c. calculate the necessary throttle valve adjustment.
4. Make the throttle valve adjustment.
5. Return to step 1. 

Centrifugal governor.png
By R. Routledge - Image from “Discoveries & Inventions of the Nineteenth Century” by R. Routledge, 13th edition, published 1900., Public Domain, Link

David Marr & J.J. Gibson

What if we took Marr too literally?

  • Computational theories matter
  • Algorithms matter
  • Hardware matters

Biological computing

  • Constrained by space, time, energy

25 W vs. ?? MW

Biological computing

  • Computes with chemistry (when possible), and ‘wires’ (when necessary)

Biological computing

  • Engages in real-time behaviors with existential consequences (e.g., ingestion, defense, reproduction, locomotion, pursuit)
  • Interacts with and in dynamic environments

Biological computing

  • Separate neural ‘streams’ for \(S \leftrightarrow C \leftrightarrow A\)

Am I my connectome?

What we know from C. elegans and the crustacean stomatogastric ganglion, with networks involving very few neurons, is that we haven’t a clue how they work even with the essential addition of electrophysiology.

Costandi, 2012

So, while SS’s memorable (if pukeworthy) ‘I am more than my genome, I am my connectome’ may be true in a grossly superficial way, they are not going resolve connectional issues in any useful way using their current approaches.

Costandi, 2012

So, while SS’s memorable (if pukeworthy) ‘I am more than my genome, I am my connectome’ may be true in a grossly superficial way, they are not going resolve connectional issues in any useful way using their current approaches.

Costandi, 2012

Jonas & Kording, 2017

We show that [classic analytic neuroscience] approaches reveal interesting structure in the data but do not meaningfully describe the hierarchy of information processing in the microprocessor.

Jonas & Kording, 2017

This suggests current analytic approaches in neuroscience may fall short of producing meaningful understanding of neural systems, regardless of the amount of data.

Jonas & Kording, 2017

Closing the loop

Perceptual Control Theory (PCT)

William T. Powers

Powers, 1973

Powers, 1973

Tenets of PCT

  • Perception is the source of controlled variables
  • Actions control perception with respect to some referent signal
  • Nested hierarchies of perceptual control systems

Wang et al. 2012

What is being controlled?

  • Control illustrates the purposive dimension of behavior (Tolman)
  • Perceptual states (PCT) vs. actions (Ecological psychologists, forward modelers)
  • Mappings from \(S \rightarrow R\) and \(C \rightarrow C\) (cognitive control/executive functions)

What do animals control?

Internal states

  • Hunger, thirst, activity/arousal level, temperature, …

  • Mental states (mood, thought processes)?

External states

  • Direction/position/motion of sensory organs
  • Body postures, movements in space
  • Emissions (e.g., sound)
  • Secretions

Visuo-motor pursuit

Powers, 1973


  • Deep ‘theories of tasks’
  • Amenable to control variable analyses

Powers, 1973


  • But which tasks?

Krakauer et al., 2017

Criteria for task selection

  • Ethological relevance
  • Transparent control variables
  • Measurable state spaces
  • Rich ‘computational’ theory of the task

Peripheral control points

  • Vision: direction of gaze, eyes open/closed, pupil diameter, lens thickness
  • Audition: head position, ears covered/un-, acoustic/stapedius reflex, outer hair cells in cochlea


  • Foundational example of “knowing” (Turvey)
  • Systematic, often rhythmic changes in body configuration
  • Produce forces that propel organism
  • Control variables: speed, heading direction, posture

Embedded within locomotion

  • Perceiving a ‘locomotable’ trajectory
  • Perceiving stable body posture

Perceiving a ‘locomotable’ trajectory

  • From vision/audition -> surface layout
  • From kinesthesia -> body orientation relative to contact surface

Constraints on ‘locomotable’ trajectory

  • Wide enough to permit passage of body (geometric properties)
  • In general direction of desired heading (geometric properties)
  • Direction toward or away depending on valence of target(s)

Fajen & Warren 2003

Constraints on ‘locomotable’ trajectory

  • With suitable supporting surface (friction, support rigidity, size)
  • With sufficient ‘clearance’ to avoid collision or injury

Controlling speed

  • From proprioception -> energy expenditure, limb movement rate, respiration and heart rate
  • From vision -> optical flow (as a function of distance)
  • From audition -> change in heard azimuth, elevation
  • From vestibular sense -> \(\int{\ddot{x}dx}\)

Perceiving stable body posture

  • From vestibular sense -> linear and angular head accelerations (gravity +)
  • From kinesthesia -> body configuration and Center of Mass (CoM)
  • From kinesthesia -> base of support

Perceiving surface layout (visually)

  • Stabilize gaze direction (despite head/body movements)
  • Regulate binocular fixation
  • Sample visual field
  • Regulate optical focus

Stabilizing gaze direction

  • Vestibulo-ocular Response (VOR)
  • Optokinetic Nystagmus
  • Pursuit
  • Saccades & microsaccades

Controlling posture

  • Righting reflex (relative to gravity)
  • Myotatic reflexes (relative to target joint angle configuration)


  • Characterizing states
  • Characterizing state transitions
  • Is bigger better?



“Rhymes with B” -> H


Inspired by Ashby, 1956/2015


  • Dimensionality of input stream (\(S\)) state spaces
  • Energy (EM, mechanical, thermal) or material fields in \(W\)

Plenoptic function (Adelson & Bergen, 1991)


\((V_{x},V_{y},V_{z})\): locations in \(W\)

\((\theta,\phi)\): optic axis, usually aligned with \(V_{z}\)

\(\bar{\lambda}\): wavelength distribution


  • Most biologically meaningful events are multimodal
  • Handling mixtures of (within or between mode) perceptual signals?

e.g., object distance from vision (monocular, binocular, motion cues, …)

object distance from audition, olfaction, proprioception, or

body posture from vision, vestibular proprioception, muscle/joint proprioception

\(P(t)=\sum_{i=1}^n w_{i}(t)S_{i}(t)\)

\(\sum_{i=1}^n w_{i}(t) = 1\)


  • Characterizing statistics of the perceptual environment…
  • Natural scene statistics

Geisler, 2008


  • States of perceptual environment across ontogenetic development…

Gilmore et al., 2015

Gilmore et al., 2015

Visual motion statistics differ across age due to

  • Changes in body posture (crawling vs. walking)
  • Locomotor speed
  • Increasingly stable gaze

Adolph, 2013

Even earlier ontogenetic origins

  • Spontaneous activity
  • In retina (e.g., Shatz)
  • In muscles (twitching; Blumberg)


  • Behaviors sequential, multivariate
  • Mixtures of control laws…

Crawl/walk to mom + keep balance + stabilize gaze + look around + …

\(A(t)=\sum_{i=1}^n w_{i}(t)F_{i}\)

\(\sum_{i=1}^n w_{i}(t) = 1\)


  • Temporal delays
  • Info from simultaneous events propagates at speeds differing by 100x
  • Need theory of ‘psycho/neural’ relativity?
  • Comparable challenges for output streams


  • Access to sufficiently rich datasets with multiple state variables
  • Clear specification of controlled systems & control variables
  • Widespread open sharing
  • Imagine easy ‘meta-analyses’ of control variables & dynamical relationships

Krakauer et al., 2017

Despite the challenges

  • Many (most?) are ‘known-unknowns’

Take homes

Algorithms Dynamics
for x in range(0, n): \(\dot{x}=f(x,a)\)

Different ways to characterize Black Boxes


(Wang et al. 2012)

Complicate before simplifying?

And yet…

  1. An organism in some state of relative constancy (\(\dot{B}<\epsilon\)) is under the influence of one or more control systems.
  2. \(\dot{B}=f(B,N,W)\)
  3. For every \(R_i\) (\(\dot{B_i}\)) there is some \(S_i\) (\(\dot{B_i}\) or \(\dot{W_i}\)) …

Our (essential) open science future

Is there a reproducibility crisis in science?

  • Yes, a significant crisis
  • Yes, a slight crisis
  • No crisis
  • Don’t know

Have you failed to reproduce an experiment from your lab or someone else’s?

Why this matters

The advancement of detailed and diverse knowledge about the development of the world’s children is essential for improving the health and well-being of humanity…

We regard scientific integrity, transparency, and openness as essential for the conduct of research and its application to practice and policy…

Open science will accelerate discovery

  • Open sharing of materials, analysis code, & data
  • Causal/dynamic models in discoverable formats
  • Video + physio + other temporally dense time series?

An openly-shared ‘behaviorome’


## R version 3.5.2 (2018-12-20)
## Platform: x86_64-apple-darwin15.6.0 (64-bit)
## Running under: macOS Mojave 10.14.3
## Matrix products: default
## BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## other attached packages:
## [1] igraph_1.2.2       databraryapi_0.1.4
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.0       revealjs_0.9     digest_0.6.18    assertthat_0.2.0
##  [5] R6_2.4.0         magrittr_1.5     evaluate_0.13    httr_1.4.0      
##  [9] stringi_1.3.1    curl_3.3         keyring_1.1.0    rmarkdown_1.11  
## [13] tools_3.5.2      stringr_1.4.0    xfun_0.4         yaml_2.2.0      
## [17] compiler_3.5.2   pkgconfig_2.0.2  htmltools_0.3.6  knitr_1.21