One of the fundamental themes of How to Lie with Charts is that our interpretation of visual information is grounded in deep-seated notions about spatial orientation. These notions are cultural biases, even though widely held, and are not innate.
For example, we see motion from left to right as the progression of time. And motion up and down is gain or loss. Thank you, Rene Descartes for the Cartesian coordinate system that grew from these biases. The up-down is a glass-half-empty or half-full notion, and the left-right is probably rooted in the Western convention of reading from left to right on the page.
Likewise, the legacy of Johannes Mercator and his grid-plotted maps has been to link the notions of North-South to up-down and West-East to left-right, even though these choices are purely arbitrary.
Our brains make meanings automatically and involuntarily based on these assumptions. As just one example, a chart (or picture) can imply progress where there may be none simply by emphasizing rightward motion. And we can explain or rationalize such misperceptions, but the initial impression persists at some level.