2017-10-18-the-whole-elephant

2017-11-02 08:15:42

Support

Talk overview

Project status report
- The development of optic flow processing
- Play & Learning Across a Year (PLAY) project
- The Proximal Emotional Environment Project (PEEP)
Toward an open, transparent, robust, and integrative developmental science

The development of optic flow processing

Questions about optic flow

What is optic flow?
Why is optic flow important?
How does optic flow sensitivity develop?
How do brain systems for processing optic flow develop?
What shapes these patterns of development?

Approach

EEG measures of brain responses to optic flow
Psychophysical (behavioral) measures of optic flow perception
Empirical measures of experienced optic flow across development from head-mounted video cameras and computer vision analyses

What is Optic Flow?

Structured pattern of visual motion generated by observer movement

Types of Optic Flow

(Yu, Page, Gaborski, & Duffy, 2010)

Why is optic flow important?

Geometry of environment
- Surface layout, orientation
- Object motion
Visual proprioception
- Rotation, translation

Flow types specify self-motion

How Does Optic Flow Sensitivity Develop?

Sensitivity at birth, (Jouen, Lepecq, Gapenne, & Bertenthal, 2000)
4-6 mo-old infants: Larger brain responses to linear patterns, (R. Gilmore, Hou, Pettet, & Norcia, 2007)

How Does Optic Flow Sensitivity Develop?

4-6 mo-old infants: Larger brain responses to faster speeds/larger displacements, (Hou, Gilmore, Pettet, & Norcia, 2009)

How Does Optic Flow Sensitivity Develop?

Behavioral sensitivity to slow (linear) speeds develops slowly in macaque monkeys, (Kiorpes & Movshon, 2004)

What about older children?

Present: Time-varying optic flow patterns
Measure: Steady-state visual evoked potentials (SSVEPs)
- Event-related electro-encephalograms (EEGs)
- Phase-locked responses at low-order harmonics
\(n=29\) 4-8 year-olds
(R. Gilmore, Thomas, & Fesi, 2016)

2 deg/s translation

https://nyu.databrary.org/volume/75/slot/9825/-

4 deg/s rotation

https://nyu.databrary.org/volume/75/slot/9825/-

8 deg/s radial

https://nyu.databrary.org/volume/75/slot/9825/-

Complex Domain Plot of 1F1 Channels

(R. Gilmore et al., 2016)

Complex Domain Plot of 3F1 Channels

(R. Gilmore et al., 2016)

Results Summary

Anatomical & frequency separation of responses
- vs. pattern (lateral, 1F1)
- speed (medial, 3F1)
Radial & rotation \(\neq\) translation
Speed tuning (slow < med & fast)
Similar to, but different from adults

What about behavior?

Time-varying optic flow
- Radial, linear
- {2 deg/s, 8 deg/s}
- {5, 10, 15, 20%} coherence (adults)
- {15, 30, 45, 60%} and {20, 40, 60, 80%} (children)

Methods

n=30 children (4.3–8.6 yrs, M = 6.5 yrs, 19 Female)
- https://nyu.databrary.org/volume/218
n=30 adults (18.7–23.9 yrs, M = 20.8 yrs, 16 female)
- http://doi.org/10.17910/B7V88T
- (Adamiak, Thomas, Patel, & Gilmore, 2015)

Children's responses p(correct)

Adults' responses p(correct)

(Adamiak et al., 2015)

Speed effects in children

Speed effects in adults

(Adamiak et al., 2015)

Pattern effects in children

Pattern effects in adults

(Adamiak et al., 2015)

Behavioral Summary

Children's behavior: more accurate to detect fast speeds, radial patterns
Adults more accurate to detect slow speeds, radial patterns
Response speeds in children and adults (not shown) show similar patterns
But, why?

Potential factors shaping development of flow sensitivity

External
- Environment
Internal
- Posture
- Locomotion, head, eye movements

Head mounted eye tracker data from "coupled" infant/mom dyads

Adolph, K. (2015). Active vision in passive locomotion: real-world free viewing in infants and adults. Databrary. Retrieved February 18, 2017 from http://doi.org/10.17910/B7.123

Synchronized infant and mother cams

https://nyu.databrary.org/volume/123/slot/8238/-

Findings

(Raudies & Gilmore, 2014)

Findings

Infant (passengers) experience faster motion speeds than mothers
Controlling for speed of locomotion, environment

Replication

Do the findings hold in larger samples
When geometry of environment varies
Outside the lab
Across age

Experienced flow across cultures

Jayaraman, S., Smith, L.B., Raudies, F. & Gilmore, R.O. (2014). Natural Scene Statistics of Visual Experience Across Development and Culture. Databrary. Retrieved February 18, 2017 from http://doi.org/10.17910/B7988V

(R. O. Gilmore, Raudies, & Jayaraman, 2015)

Participant summary

Country	Females	Males	Age (wks)	Coded video Hrs
India	17	13	3-63	3.1 (0.5-6.0)
U.S.	15	19	4-62	4.6 (0.2-7.6)

(R. O. Gilmore et al., 2015)

Illustrative Speed Histograms - 6 weeks

(R. O. Gilmore et al., 2015)

Illustrative Speed Histograms – 34 weeks

(R. O. Gilmore et al., 2015)

Illustrative Speed Histograms – 58 weeks

(R. O. Gilmore et al., 2015)

Pattern Correlation Results

(R. O. Gilmore et al., 2015)

Conclusions: Measuring experienced flow

Fast speeds, broad speed distributions
Linear flow >> radial or rotational flow

Simulating developmental change

\(\begin{pmatrix}\dot{x} \\ \dot{y}\end{pmatrix}=\frac{1}{z} \begin{pmatrix}-f & 0 & x\\ 0 & -f & y \end{pmatrix} \begin{pmatrix}{v_x{}}\\ {v_y{}} \\{v_z{}}\end{pmatrix}+ \frac{1}{f} \begin{pmatrix} xy & -(f^2+x^2) & fy\\ f^2+y^2 & -xy & -fy \end{pmatrix} \begin{pmatrix} \omega_{x}\\ \omega_{y}\\ \omega_{z} \end{pmatrix}\)

Geometry of environment/observer: \((x, y, z)\)
Translational speed: \((v_x, v_y, v_z)\)
Rotational speed: \((\omega_{x}, \omega_{y}, \omega{z})\)
Retinal flow: \((\dot{x}, \dot{y})\)

Parameters For Simulation

Parameter	Crawling Infant	Walking Infant
Eye height	0.30 m	0.60 m
Locomotor speed	0.33 m/s	0.61 m/s
Head tilt	20 deg	9 deg

(Kretch et al., 2014)

Geometric Feature	Distance
Side wall	+/- 2 m
Side wall height	2.5 m
Distance of ground plane	32 m
Field of view width	60 deg
Field of view height	45 deg

Simulating Flow Fields

(Gilmore et al, 2015)

Simulated Flow Speeds (m/s)

Type of Locomotion	Ground Plane	Room	Side Wall	Two Walls
Crawling	14.41	14.42	14.43	14.62
Walking	9.38	8.56	7.39	9.18

Summing up

Infants commonly experience fast, linear optic flows
- Head/eye instability
- Proximity to ground
Brain and behavioral responses to optic flow develop throughout childhood
- Still immature in 5-8 year-olds
Statistics of natural experience informative

Some next steps

Individual differences in flow sensitivity
Brain responses to flow 'in the wild'
Comparing brain responses to behavioral sensitivity
Simulating the effects of development on visual experience
Automating pipelines for automated analysis of 1st person videos (using PSU's Galaxy environment)

An hour in the life…

Play & Learning Across a Year project

Play is the central context and activity of early development
But what do parents and infants actually do when they 'play'?

Play & Learning Across a Year (PLAY) project

\(n=900\) infant/mother dyads; 300 @ 12-, 18-, 24-months
30 dyads from 30 sites across the US
1 hr natural activity
- 3 min solitary toy play
- 2 min dyadic toy play
- video tour of home

Play & Learning Across a Year (PLAY) project

Videos coded for
- Emotional expression
- Object interaction
- Physical activity & locomotion
- Transcript, Communication, and Gesture
Enhancements to Datavyu for transcription, CHAT compatibility, Windows support

Play & Learning Across a Year (PLAY) project

Demographics + parent-report questionnaires about health, family, temperament
Ambient sound levels

Play & Learning Across a Year (PLAY) project

Census block group geocoding

Play & Learning Across a Year (PLAY) project

"Big data" developmental science

Play & Learning Across a Year (PLAY) project

Data shared on Databrary

Play & Learning Across a Year (PLAY) project

Video as data and documentation

(Rick O Gilmore & Adolph, 2017)

What questions would you ask about these data?

The Proximal Emotional Environment Project (PEEP)

Guess the emotion

Sample 1	Sample 2

Questions

How do children's brains respond to overheard speech with different affective prosodies?
How do children's brains respond to different speakers?
How do children perceive the affective qualities of overheard speech?
What are the acoustic properties that distinguish affectively-laden speech?
What is the natural "affective" environment

How 'angry' are the scenarios

How 'happy' are they?

'Intensity' ratings across time

What's in common?

Themes

Natural environments are rich
Perception, action, cognition & emotion inextricably linked
Need better/denser measures
- of environment
- of behavior
- of physiology

Turtles networks all the way down…

Power of

convergent evidence
data visualization
data, materials sharing
to accelerate discovery…

Limitations of

scientific culture
technology
sampling diversity, sample sizes/power, publication bias, etc.
big team science vs. small team science

Toward an open, transparent, robust, and integrative developmental science

What would a data observatory for human development look like?

Combine data from diverse domains

Link measures across people

Enable web-based data visualization, analysis

Support search, filtering by personal characteristics

Encourage self/active curation of data, materials

Provide consistent, clear sharing permissions structure

Progress

Example	Multi-measure	Indiv link/search	Visualize	Self-curate	Permissions
Databrary	✔	✔	tabular	✔	✔
Human Proj	✔	✔	?	?	✔
ICPSR	✔	?	✔	?	✔
Neurosynth	fMRI BOLD	group data	✔	public	NA
OpenNeuro	✔	?	✔	✔	public
Open Humans	✔	✔	?	?	✔
OSF	✔			✔	public
WordBank	M-CDI	group metadata	✔	?	public

Barriers

Humans are diverse

But much (lab-based) data collected are from Western, Educated Industrialized, Rich, Democratic (WEIRD) populations Henrich et al., 2010

http://www.evoanth.net/2015/01/06/evolutionary-psychology-has-problems-and-it-isnt-getting-better/

(LeWinn, Sheridan, Keyes, Hamilton, & McLaughlin, 2017)

Data sensitive, hard(er) to share

"…psychologists tend to treat other peoples’ theories like toothbrushes; no self-respecting individual wants to use anyone else’s."

Walter Mischel, 2009

"Reviewers and editors want novel, interesting results. Why would I waste my time doing careful direct replications?"

Any number of researchers I've talked with

(Munafo et al., 2017)

Let's build a Databservatory for human behavior

It should…

Store & share data & materials
Link data across studies, measures
Link across group characteristics, individuals
Enable searching & filtering by individual characteristics, tasks, settings

Support web-based data analysis, visualization; open API
Provide consistent framework for ethical data sharing
Enable data aggregation, cloning, provenance tracking
Support self/active curation
Link to publications &…

Let us embrace the whole elephant

Stack

This talk was produced on 2017-11-02 in RStudio using R Markdown and the ioslides framework. The code and materials used to generate the slides may be found at https://github.com/gilmore-lab/2017-10-18-the-whole-elephant/. Information about the R Session that produced the code is as follows:

sessionInfo()

## R version 3.4.1 (2017-06-30)
## Platform: x86_64-apple-darwin15.6.0 (64-bit)
## Running under: macOS Sierra 10.12.6
## 
## Matrix products: default
## BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## loaded via a namespace (and not attached):
##  [1] compiler_3.4.1  backports_1.1.0 magrittr_1.5    rprojroot_1.2  
##  [5] tools_3.4.1     htmltools_0.3.6 yaml_2.1.14     Rcpp_0.12.12   
##  [9] stringi_1.1.5   rmarkdown_1.6   knitr_1.17      stringr_1.2.0  
## [13] digest_0.6.12   evaluate_0.10.1

References

Adamiak, W., Thomas, A., Patel, S., & Gilmore, R. (2015). Adult Observer’s Sensitivity to Optic Flow Varies by Pattern and Speed. Journal of Vision, 15(12), 1008. https://doi.org/10.1167/15.12.1008

Gilmore, R. O., & Adolph, K. E. (2017). Video can make behavioural science more reproducible. Nature Human Behavior, 1. https://doi.org/10.1038/s41562-017-0128

Gilmore, R. O., Raudies, F., & Jayaraman, S. (2015). What accounts for developmental shifts in optic flow sensitivity? In 2015 Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob) (pp. 19–25). https://doi.org/10.1109/DEVLRN.2015.7345450

Gilmore, R., Hou, C., Pettet, M., & Norcia, A. (2007). Development of cortical responses to optic flow. Visual Neuroscience, 24(06), 845–856. https://doi.org/10.1017/S0952523807070769

Gilmore, R., Thomas, A., & Fesi, J. (2016). Children’s Brain Responses to Optic Flow Vary by Pattern Type and Motion Speed. PLOS ONE, 11(6), e0157911. https://doi.org/10.1371/journal.pone.0157911

Hou, C., Gilmore, R., Pettet, M., & Norcia, A. (2009). Spatio-temporal tuning of coherent motion evoked responses in 4–6 month old infants and adults. Vision Research, 49(20), 2509–2517. https://doi.org/10.1016/j.visres.2009.08.007

Jouen, F., Lepecq, J.-C., Gapenne, O., & Bertenthal, B. I. (2000). Optic flow sensitivity in neonates. Infant Behavior and Development, 23(3–4), 271–284. https://doi.org/10.1016/S0163-6383(01)00044-3

Kiorpes, L., & Movshon, J. A. (2004). Development of sensitivity to visual motion in macaque monkeys. Visual Neuroscience, 21(6), 851–859. https://doi.org/10.1017/S0952523804216054

LeWinn, K. Z., Sheridan, M. A., Keyes, K. M., Hamilton, A., & McLaughlin, K. A. (2017). Sample composition alters associations between age and brain structure. Nat. Commun., 8(1), 874. https://doi.org/10.1038/s41467-017-00908-7

Raudies, F., & Gilmore, R. (2014). Visual Motion Priors Differ for Infants and Mothers. Neural Computation, 26(11), 2652–2668. https://doi.org/10.1162/NECO_a_00645

Yu, C. P., Page, W. K., Gaborski, R., & Duffy, C. J. (2010). Receptive Field Dynamics Underlying MST Neuronal Optic Flow Selectivity. Journal of Neurophysiology, 103(5), 2794–2807. https://doi.org/10.1152/jn.01085.2009