Vision

How it works, how it develops, & why you should care

Rick O. Gilmore

Preliminaries

About me

  • B.A., Cognitive Science, Brown University
  • M.S. & Ph.D., Psychology (Cognitive Neuroscience), Carnegie Mellon University
  • Human brain development, perception & action, computational modeling, machine vision, big data, open science
  • Founding Director of Human Imaging, Penn State Social, Life & Engineering Sciences Imaging Center (SLEIC)
  • Co-Founder/Co-Director of Databrary.org data library
  • gilmore-lab.github.io
  • banjo player, actor, cyclist, backpacker, poet, ham (W3TM)
  • native Coloradoan, husband, dad, grandpa

Agenda

  • What’s vision for?
  • Properties of light
  • The visual eye and brain
  • Measuring vision
  • Typical and atypical development

What’s vision for?

Behavioral priorities

  • Secure sustenance (eat & drink)
  • Avoid harm
  • Reproduce

Behavioral primitives

  • Locomotion (wayfinding, steering, & balance)
  • Object interaction/manipulation
  • Communication (vocalizing & gesturing)

Vision answers…

  • Where things are; where they’re moving; where/how I’m moving
  • Who/What’s out there
  • How should I respond

Vision informs observers about…

  • Geometry and motion of objects and surfaces
  • Surface properties (color, texture, rigidity)
  • Object properties (animate/inanimate; familiar/un-; person/animal…)

Light as information source

Physics of sensation

Informal name Source
Vision Electromagnetic radiation
Audition Mechanical vibration in air/water
Touch Mechanical vibration of skin on surface
Vestibular Rotation & linear acceleration of head
Olfaction Chemical patterns in air/water
Gustation Chemical patterns in mouth
Informal name Source
Electroception Electromagnetic radiation
Magnetoreception Electromagnetic radiation patterns
Kinesthesia Position, velocity, acceleration of limbs, body

Plenoptic function

Edward H. Adelson (1991)

(Chan, 2014)

Properties of light

  • Form of electromagnetic (EM) radiation
  • Wavelength (1/frequency)
  • Intensity

EM spectrum from Wikipedia

Different surfaces/geometries -> different reflectance patterns

Source: https://www.cns.nyu.edu/~david/courses/perception/lecturenotes/color/color-slides/Slide14.jpg

Perceived color differences correspond to different patterns of light reflection.

Randeberg (2005)

Some materials refract (bend) light

Light

  • Provides fast (2.99 million m/s; 186 million mi/hr) information about surfaces at a distance
  • vs. sound (340 m/s; 767 mi/hr)
  • vs. chemical signals (min/mi)

Projects images

…that relate to object shape/orientation/geometry

Source: https://thebrain.mcgill.ca

Source: https://thebrain.mcgill.ca

and even look backward in time…

“Webb’s first deep field (NIRCam image)” (n.d.)

The visual eye and brain

The eye

is like an auto-focus, auto-exposure camera…

part of a self-stabilizing system…

(Bucalo, 2015)

Note

  • Eye + head + body movements align and point the eyes
  • Eye + head + body movements stabilize the eyes
    • When the observer moves
    • When objects move

Image formation

  • Eye’s optical components
    • Cornea (fixed refraction)
    • Iris/pupil (modifiable aperture)
    • Lens (modifiable refraction)
  • Create projection (image) on retina

The retina…

…samples spatial patterns of light intensity & wavelength patterns

via ‘wavelength-tuned’ photoreceptors

Retinal rods (green) & cones (blue)

…arranged in mosaics

Normal color vision vs. protanopic (color blind) vision; Source: https://en.wikipedia.org/wiki/Retinal_mosaic

with different concentrations in different parts of the retina

Wikipedia

Information processing

  • Separate channels for short, medium, long wavelengths (cones): chromatic (color)
  • Black/gray/white or overall illumination (rods): achromatic (dark/light)
  • ~120 M rods + ~ 5 M cones (125 M) vs. professional cameras with 100 M pixels
  • Point by point, topographic (map-like) 2D image of 3D world
  • Non-uniform resolution (center >> periphery)
  • yields focused image except…

…when eye misshapen for cornea+lens

The visual brain

(Logothetis, 1999)

Primary (feedforward, cortical) pathway

  • Retina ->

  • Thalamus ->

  • Primary visual cortex (V1) in occipital lobe

  • Subcortical pathways

Retinotopic maps via fMRI

(Charting, 2020b)

(Charting, 2020a)

https://en.wikipedia.org/wiki/Retinotopy

Visual processing dominates the primate brain

(Tootell, Tsao, & Vanduffel, 2003)

Measuring vision

Psychophysical functions

  • Map perception (psycho) to physical features
  • Among the earliest and most robust psychological methods

Psychophysical methods

  • Method of constants (fixed levels)
  • Method of adjustment (raise/lower amplitude until detectable/indetectable)
  • Method of limits (“can you see me now? now?”; often use staircases)

Psychophysiological functions

(Mirabella, Kjaer, Norcia, Good, & Madan, 2006, fig. 1)

(Mirabella et al., 2006, fig. 2)

Acuity

  • Detail/pattern vision
  • Grating acuity
  • Vernier
  • Symbol/letter (optotype) acuity

Contrast sensitivity

  • Light/dark ratio (contast)
  • vs. spatial frequency (level of detail)
  • Contrast -> edges; edges -> shape/form
  • e.g., driving in fog

Contrast sensitivity function

Pelli-Robson Contrast Sensitivity Chart

Typical development

Acuity: Similar across primates

(Kiorpes, 2016, fig. 2)

Contrast sensitivity

(Hofsten et al., 2014, fig. 1)

(Hofsten et al., 2014, fig. 4)

(Hofsten et al., 2014, fig. 3)

…similar across primate species

(Kiorpes, 2016, fig. 3)

Contrast effects on newborn vision

(Kiorpes, 2016, fig. 4)

Motion

Coherence: Signal vs. noise

(Qian, Seisler, & Gilmore, 2021)

Male/female differences

(Qian et al., 2021)

Children (4-8 year-olds)

Figure 4 from (R. O. Gilmore, Thomas, & Fesi, 2016)

Adults

Figure 21 from (R. O. Gilmore et al., 2016)

Rates of development

  • Vary across visual functions
  • Some (e.g., motion sensitivity) not adult-like until early teens

Figure 5 from (Kiorpes, 2016)

Atypical development

Autism

(Figure 1 from Koldewyn, Whitney, & Rivera, 2010)

(Figure 3 from Koldewyn et al., 2010 )

Amblyopia

  • Amblyopia: Reduced visual acuity in one or both eyes relative to the other without an obvious defect or change in the eye
  • Strabismus: Misalignment of the eyes
  • Anisometropia: Difference (between eyes) in refractive power

Wikipedia: Strabismus

(Figure 7 from Kiorpes, 2016). Characteristic amblyopic contrast sensitivity data from three macaque monkeys. Strabismic amblyopia (left) resulted from surgical alteration of the horizontal rectus muscles of one eye (data from Kiper and Kiorpes, 1994). Anisometropic amblyopia (middle) resulted from rearing with a defocusing lens over one eye (data from Kozma and Kiorpes, 2003). Deprivation amblyopia (right) resulted from rearing with one eye closed for 14 months (data from Harwerth et al., 1990). Redrawn from Kiorpes (2008).

Prematurity

Some (binocular functions) are experience-dependent

(Figure 2 from Jandó et al., 2012)

Others (pattern/contrast reversal) are not

(Figure 3 from Jandó et al., 2012)

Cataract

Wikipedia: https://en.wikipedia.org/wiki/Congenital_cataract

…rapid responses following surgical removal

(Maurer, Lewis, Brent, & Levin, 1999)

The big picture

Davida Teller

“For me, one of the major attractions of visual science is the promise it holds for empirical attacks on the mind-body problem…”

“…that is, for working out meaningful ways to explain psychophysically defined visual functions on the basis of properties of the neural substrate….

…A critical locus or critical computation for a particular perceptual function can be defined as an anatomic or computational stage at which information concerning that function is lost or importantly reorganized…”

“or more poetically, as a stage or computation that leaves its mark on that perceptual capacity.”

“Part of the appeal of visual development is its potential for extending this promise. Visual functions mature because the visual substrate matures, and the causes of functional maturation undoubtedly lie in neural maturation.”

“But the length of the big toe matures too, and we do not see it as causal in relation to the development of grating acuity. The puzzle is, which of the many immaturities of the visual substrate provide the critical immaturities that limit a particular visual capacity at a particular age?”

Other things change, too…

Posture changes visual input

(Kretch, Franchak, & Adolph, 2014)

Parameter Crawling Infant Walking Infant
Eye height 0.30 m 0.60 m
Locomotor speed 0.33 m/s 0.61 m/s
Head tilt 20 deg 9 deg

(R. O. Gilmore, Raudies, & Jayaraman, 2015)

(R. O. Gilmore et al., 2015)

Type of Locomotion Ground Plane Room Side Wall Two Walls
Crawling 14.41 14.42 14.43 14.62
Walking 9.38 8.56 7.39 9.18

But, what’s the input? The real input?

https://nyu.databrary.org/slot/7739/0,26634/asset/16747/download?inline=true

https://nyu.databrary.org/slot/7739/0,26134/asset/16749/download?inline=true

(Figure 1 from Kretch & Adolph, 2015)

Findings

(Raudies & Gilmore, 2014)

  • Infant (passengers) experience faster visual speeds than mother
  • Controlling for speed of locomotion, environment
  • Motion “priors” for infants ≠ mothers

(Raudies & Gilmore, 2014)

Summing up…

(Figure 10.1 from Atkinson & Braddick, 2013)

  • Vision develops rapidly, but approaches asymptote slowly
  • Complex interplay of brain and behavioral changes

(Figure 10.4 from Atkinson & Braddick, 2013)

Thank you!

Materials

The code and materials used to generate the slides may be found at https://github.com/gilmore-lab/csd-vision-course/.

References

Vision: How it works, how it develops, & why you should care

Atkinson, J., & Braddick, O. (2013). Visual development. In P. D. Zelazo (Ed.), The oxford handbook of developmental psychology (Vol. 1, pp. 271–309). Oxford University Press.
Bucalo, P. (2015, October). Falcon belly dance. Youtube. Retrieved from https://www.youtube.com/watch?v=JGArTWOJtXs
Chan, S. C. (2014). Plenoptic function. In K. Ikeuchi (Ed.), Computer vision: A reference guide (pp. 618–623). Boston, MA: Springer US. https://doi.org/10.1007/978-0-387-31439-6\_7
Charting, I. B. (2020a, June). Retinotopy task – Ring-Expanding run. Youtube. Retrieved from https://www.youtube.com/watch?v=DcgHJIlwQCo
Charting, I. B. (2020b, June). Retinotopy task – Wedge-Clockwise run. Youtube. Retrieved from https://www.youtube.com/watch?v=rsykP-9-moA
Edward H. Adelson, J. R. B. (1991). The plenoptic function and the elements of early vision. In Computational models of visual processing. Retrieved from http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.2.9848
Gilmore, Rick O. (n.d.). Children’s brain responses to optic flow vary by pattern type and motion speed. https://nyu.databrary.org/volume/75. https://doi.org/10.17910/B7QG6W
Gilmore, R. O., Raudies, F., & Jayaraman, S. (2015). What accounts for developmental shifts in optic flow sensitivity? In 2015 joint IEEE international conference on development and learning and epigenetic robotics (ICDL-EpiRob) (pp. 19–25). ieeexplore.ieee.org. https://doi.org/10.1109/DEVLRN.2015.7345450
Gilmore, R. O., Thomas, A. L., & Fesi, J. (2016). Children’s brain responses to optic flow vary by pattern type and motion speed. PloS One, 11(6), e0157911. https://doi.org/10.1371/journal.pone.0157911
Hofsten, O. von, Hofsten, C. von, Sulutvedt, U., Laeng, B., Brennen, T., & Magnussen, S. (2014). Simulating newborn face perception. Journal of Vision, 14(13), 16. https://doi.org/10.1167/14.13.16
Jandó, G., Mikó-Baráth, E., Markó, K., Hollódy, K., Török, B., & Kovacs, I. (2012). Early-onset binocularity in preterm infants reveals experience-dependent visual development in humans. Proc. Natl. Acad. Sci. U. S. A., 109(27), 11049–11052. https://doi.org/10.1073/pnas.1203096109
Kiorpes, L. (2016). The puzzle of visual development: Behavior and neural limits. J. Neurosci., 36(45), 11384–11393. https://doi.org/10.1523/JNEUROSCI.2937-16.2016
Koldewyn, K., Whitney, D., & Rivera, S. M. (2010). The psychophysics of visual motion and global form processing in autism. Brain: A Journal of Neurology, 133(Pt 2), 599–610. https://doi.org/10.1093/brain/awp272
Kretch, K. S., & Adolph, K. E. (2015). Active vision in passive locomotion: Real-world free viewing in infants and adults. Developmental Science, 18(5), 736–750. Retrieved from https://onlinelibrary.wiley.com/doi/abs/10.1111/desc.12251
Kretch, K. S., Franchak, J. M., & Adolph, K. E. (2014). Crawling and walking infants see the world differently. Child Development, 85(4), 1503–1518. https://doi.org/10.1111/cdev.12206
Logothetis, N. K. (1999). Vision: A window on consciousness. Scientific American, 281(5), 69–75. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/10920769
Maurer, D., Lewis, T. L., Brent, H. P., & Levin, A. V. (1999). Rapid improvement in the acuity of infants after visual input. Science, 286(5437), 108–110. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/10506555
Mirabella, G., Kjaer, P. K., Norcia, A. M., Good, W. V., & Madan, A. (2006). Visual development in very low birth weight infants. Pediatr. Res., 60(4), 435–439. https://doi.org/10.1203/01.pdr.0000238249.44088.2c
Qian, Y., Seisler, A. R., & Gilmore, R. O. (2021). Children’s perceptual sensitivity to optic flow-like visual motion differs from adults. Developmental Psychology, 57(11), 1810–1821. https://doi.org/10.1037/dev0001227
Randeberg, L. (2005). Diagnostic applications of diffuse reflectance spectroscopy. Retrieved from https://www.semanticscholar.org/paper/ec9450b79923e2e2152b54ab9241b60bc5374944
Raudies, F., & Gilmore, R. O. (2014). Visual motion priors differ for infants and mothers. Neural Computation, 26(11), 2652–2668. https://doi.org/10.1162/NECO\_a\_00645
Tootell, R. B. H., Tsao, D., & Vanduffel, W. (2003). Neuroimaging weighs in: Humans meet macaques in “primate” visual cortex. The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, 23(10), 3981–3989. https://doi.org/10.1523/JNEUROSCI.23-10-03981.2003
Webb’s first deep field (NIRCam image). (n.d.). https://webbtelescope.org/contents/media/images/2022/035/01G7DCWB7137MYJ05CSH1Q5Z1Z. Retrieved from https://webbtelescope.org/contents/media/images/2022/035/01G7DCWB7137MYJ05CSH1Q5Z1Z