Rationality Quotient: Comprehensive Assessment of Rational Thinking

Stanovich and colleagues recently developed a prototype of the first comprehensive assessment of rational thinking. The test is discussed, and presented in detail in the new book,  titled- The Rationality Quotient.

Up until publication of- The Rationality Quotient - components of rational thinking had been tested using various tasks, but a comprehensive test was not available. I first discussed the development of such a test with Stanovich, in 2013- interview here.

In the following interview (conducted in November, 2016) Stanovich provides detailed answers to important questions about the test. 

 

What are some of the initial reactions, regarding the RQ, from academics?

Uniformly positive so far, and I believe that is because we were careful in the book to be explicit about two things.  First, we were clear about what our goals were and the goals were circumscribed.  Secondly, we included an entire chapter contextualizing our test (the Comprehensive Assessment of Rational Thinking, CART) and discussing caveats regarding its use as a research instrument or otherwise. In fact, I think we have already entirely achieved our aims.  We have a prototype test that is a pretty comprehensive measure of the rational thinking construct and that is grounded in extant work in cognitive science.  Now, this is not to deny that there is still much work to be done in turning the CART into a standardized instrument that could be used for practical purposes.  But of course a finished test was not our goal in this book.  Our goal was to show a demonstration of concept, and we have done that.  We have definitively shown that a comprehensive test of rational thinking was possible given existing work in cognitive science.  This is something that I have claimed in previous books but had not empirically demonstrated with the comprehensiveness that we have here by introducing the CART.  As I said, there are more steps left in turning the CART into an “in the box” standardized measure, but that is a larger goal than we had for this book.

I think that, at least so far, most academics have understood our goals and the feedback has been good.  We wrote a summary article on the CART in a 2016 issue of the journal Educational Psychologist (51, 23-34) and the feedback from that community has been good.

Are there components of the RQ that can be expected to show a strong positive correlation with intelligence?

The CART has 20 subtests and four thinking dispositions scales (the latter are not part of the total score). Collectively they tap both instrumental rationality and epistemic rationality.  In cognitive science, instrumental rationality means behaving in the world so that you get exactly what you most want, given the resources (physical and mental) available to you.  Epistemic rationality concerns how well beliefs map onto the actual structure of the world.  The two types of rationality are related.  In order to take actions that fulfill our goals, we need to base those actions on beliefs that are properly calibrated to the world.

The CART assesses epistemic thinking errors such as: the tendency to show incoherent probability assessments; the tendency toward overconfidence in knowledge judgments; the tendency to ignore base rates; the tendency not to seek falsification of hypotheses; the tendency to try to explain chance events; the tendency to evaluate evidence with a myside bias; and the tendency to ignore the alternative hypothesis.

Additionally, CART assesses instrumental thinking errors such as:  the inability to display disjunctive reasoning in decision making; the tendency to show inconsistent preferences because of framing effects; the tendency to substitute affect for difficult evaluations; the tendency to over-weight short-term rewards at the expense of long-term well-being; the tendency to have choices affected by vivid stimuli; and the tendency for decisions to be affected by irrelevant context. 

 

Importantly, the test also taps what we call contaminated mindware.  This category of thinking problem arises because suboptimal thinking is potentially caused by two different types of mindware problems.  Missing mindware, or mindware gaps, reflect the most common type—where Type 2 processing does not have access to adequately compiled declarative knowledge from which to synthesize a normative response to use in the override of Type 1 processing.  However, in the book, we discuss how not all mindware is helpful or useful in fostering rationality.  Indeed, the presence of certain kinds of mindware is often precisely the problem.  We coined the category label contaminated mindware for the presence of declarative knowledge bases that foster irrational rather than rational thinking.  Four of the 20 subtests assess contaminated mindware.

My purpose in digressing here to describe the CART is to point out that given the number and complexity of rational thinking skills, it is likely that the subtests will have correlations with intelligence that are quite variable.  The four subtests with the highest correlations are: the Probabilistic Reasoning Subtest; the Scientific Reasoning Subtest; the Reflection Versus Intuition Subtest; and the Financial Literacy Subtest.  Correlations with these subtests tend to .50or higher.  Most of the subtests of the CART correlate with intelligence in the range of .25 to .50 (a few have even lower correlations). Some very important components of rational thinking do show considerable dissociation from intelligence.  Overconfidence (measured by the Knowledge Calibration Subtest of the CART) shows only a .38 correlation with intelligence.  This represents a substantial amount of dissociation for a key component of rational thinking.  Kahneman, for example, devoted substantial portions of his best-selling book to this component of rational thinking.  Myside bias (measured by our Argument Evaluation Subtest) likewise shows a correlation of .38, indicating a substantial dissociation.  This thinking bias is at the center of many discussions of what it means to be rational.  Some of the subtests that most directly measure the components of the axiomatic approach to utility maximization show relatively mild correlations with intelligence.  For example, the Framing Subtest shows a fairly low .28 correlation.  Framing measures a foundational aspect of rational thinking according to the axiomatic approach.

Finally, some subtests of immense practical importance show very low correlations with intelligence in the CART.  The skill of assessing numerical expected value shows a correlation of only .21, and the ability to delay for greater monetary reward shows a correlation of only .06.  The tendency to believe in conspiracies shows a modest correlation of .34.

Do you think rationality will acquire the same high level status as intelligence in the near future?

Not in the near future, no.  Our goal with the book was more modest—to simply raise awareness of the importance of rational thinking and the ability of modern cognitive psychology to measure it.  The result of our efforts will, we hope, redress the imbalance between our tendency to value intelligence versus rationality.  In our society, what gets measured gets valued.  Our aim in developing the CART was to draw attention to the skills of rational thought by measuring them systematically.  In the book, we are careful to point out that we operationalized the construct of rational thinking without making reference to any other construct in psychology, most notably intelligence.  Thus, we are not trying to make a better intelligence test.  Nor are we trying to make a test with incremental validity over and above IQ tests.  Instead, we are trying to show how one would go about measuring rational thinking as a psychological construct in its own right.  We wish to accentuate the importance of a domain of thinking that has been obscured because of the prominence of intelligence tests and their proxies.  It is long overdue that we had more systematic ways of measuring these components of cognition, that are important in their own right, but that are missing from IQ tests.  Rational thinking has a unique history grounded in philosophy and psychology, and several of its subcomponents are firmly identified with well-studied paradigms.  The story we tell in the book is of how we have turned this literature into the first comprehensive device for the assessment of rational thinking (the CART).

Why does society need a comprehensive assessment of rational thinking?

To be globally rational in our modern society you must have the behavioral tendencies and knowledge bases that are assessed on the CART to a sufficient degree.  Our society is sometimes benign, and maximal rationality is not always necessary, but sometimes, in important situations, our society is hostile.  In such hostile situations, to achieve adequate degrees of instrumental rationality in our present society the skills assessed by the CART are essential.  In Chapter 15 of The Rationality Quotient we include a table showing that rational thinking tendencies are linked to real life decision making.  In that table, for each of the paradigms and subtests of the CART, an association with a real-life outcome is indicated.  The associations are of two types.  Some studies represent investigations where a laboratory measure of a bias was used as a predictor of a real-world outcome.  Others are reports of real-world analogues of biases that were originally discovered in the lab.  Clearly more work remains to be done on tracing the exact nature of the connections—that is, whether they are causal.  The sheer number of real-world connections, however, serves to highlight the importance of the rational thinking skills in our framework.  Now that we have the CART, we could, in theory, begin to assess rationality as systematically as we do IQ.  If not for professional inertia and psychologists’ investment in the IQ concept, we could choose tomorrow to more formally assess rational thinking skills, focus more on teaching them, and redesign our environment so that irrational thinking is not so costly.  Whereas just thirty years ago we knew vastly more about intelligence than we knew about rational thinking, this imbalance has been redressed in the last few decades because of some remarkable work in behavioral decision theory, cognitive science, and related areas of psychology.  In the past two decades cognitive scientists have developed laboratory tasks and real-life performance indicators to measure rational thinking tendencies such as sensible goal prioritization, reflectivity, and the proper calibration of evidence.   People have been found to differ from each other on these indicators.  These indicators are structured differently from the items used on intelligence tests.  We have brought this work together by producing here the first comprehensive assessment measure for rational thinking, the CART.

Science and Rationality in Modern Society

Science and rationality are important in modern, technologically advanced, industrial societies. Science is a large enterprise consisting of multiple components. Science, although fallible, is the great reality detector. Rationality, in this context, refers to rationality as it is conceptualized in cognitive science. Rationality is concerned with judgment and decision making. Rationality consists of two main categories- instrumental and epistemic. Instrumental rationality reflects goal optimization, and epistemic reflects evidence based beliefs. There is overlap between the two categories of rationality. In my most recent book- In Evidence We Trust: The need for science, rationality & statistics- I provide information on various aspects of science, rationality and mathematical procedures (statistics) used in describing and making inferences in the context of scientific research. 

In Evidence We Trust

 It is often said we live in the information age, but we also live in the mis-information age.  How do we decide what constitutes knowledge and what constitutes nonsense?  Maybe there are no wrong or right answers, and just opinions?  This notion is fallacious.  There are facts and opinions, right and wrong answers.  There is a reality that extends beyond personal comforts and opinions (Mitchell & Jolley, 2010).  In the context of science  facts are tentative.  They are assertions that are supported by the preponderance of evidence.  Facts in the context of science (primary concern in this book) are based on levels of certainty, but absolute certainty is never attained.  Scientific findings are presented in terms of probabilities and data (e.g. laws, principles, theories, etc.) is revised in accordance to findings.

Testimonials, anecdotes, they-says, wishful thinking and so on do not count for evidence.  If  these types of claims and feelings are labeled  as evidence then any discussion of evidence is vacuous.  Testimonials exist for almost any claim you can imagine.  That does not mean that claims of this sort have no value.    Experiences are confounded (confused by alternative explanations). Experiences may be very important in some contexts, and they may serve as meaningful research questions.  However, a meaningful question or a possible future finding is not synonymous with evidence. Scientific evidence is drastically different than evidence as it relates to everyday discourse.  As Joy Victoria points out- it should be obvious from the book's title that the type of evidence I am referring to in the book is derived from scientific findings (paraphrased). 

 

The content in chapter one includes short-articles (old, new & revised), a science discussion roundtable (featuring individuals from various fields) and a nonsense detection kit. Some of the short articles presented in chapter one have been published on various internet sites, and some of the same or similar information may be discussed in across different articles.  There are at least two key benefits that can occur when presenting similar information across different articles (in different contexts): strengthening of memory connections, and each article can be read as a stand-alone article.  In the science discussion roundtable participants are asked two questions.  One) Do you have any tips for people that are interested in enhancing their ability to read scientific research?  Two) What is the biggest (or at least one of the biggest misconceptions) misconception about science? The Nonsense Detection Kit is presented at the end of chapter one.  The impetus for designing the Nonsense Detection Kit was similar kits devised by Sagan, Shermer, and Lilienfeld.

Chapter two features short articles on rationality.  Some of the same or similar information is contained across different articles.  There are at least a couple of advantages to presenting information in this manner (refer to previously mentioned advantages in chapter one).  Many of the articles focus on the rationality intelligence dichotomy.  Also included in this chapter are interviews with Keith Stanovich and the Stanovich Research Lab (Keith Stanovich, Richard West and Maggie Toplak).  In the interview with Stanovich, he discusses the development of an RQ Test. In the interview with the Stanovich lab, rationality and intelligence are discussed. Since the publication of the book Stanovich, West and Toplak have designed the first comprehensive test for rational thinking

Chapter three features frequently asked questions about research methods and statistics. Many of the questions are questions I have received in the past from my students.  Some of the questions address basic research and statistics problems, while other questions are more complex.  At the end of the chapter recommended sources are provided for readers that are interested in furthering their studies on research methods and statistics. 

    

The book ends with an appendices section. Practice problems, and guidelines regarding APA citations and reference lists are given.

The content in this book may be difficult for some to comprehend. However, with some effort and patience the content is learnable for most people. In the words of Albert Einstein “Things should be made as simple as possible, but not any simpler.” Science, rationality and statistics can be simplified to a degree, but relative to most other topics these topics are difficult.  This book is not written for cognitive misers (the cognitively lazy).  This book is written for individuals that are interested in separating knowledge and nonsense, and are willing to put forth at least a moderate level of cognitive effort.  This book is not written in the format often used by pop science writers.

  

I would like to thank Joy Victoria, Kitty Mervine, Jason Silvernail and Coert Visser for the review articles of IEWT they have written. 

 

RecommendedResources: In Evidence We Trust by Joy Victoria 


In Evidence We Trust (Review) by Coert Visser  

More Than Scientific Literacy

Discussions involving scientific literacy are ubiquitous. Scientific literacy is conceptualized and operationalized  in various ways (see; Norris & Phillips, 2002).  Examples used in defining scientific literacy include: understanding science and its applications, knowledge of what counts as science, general scientific knowledge, knowledge of risks and benefits of science, etc. Numerous scales are used to measure scientific literacy.   In my current research scientific literacy is synonymous with general scientific knowledge, that involves various domains.  This form of literacy is sometimes referred to as derived scientific literacy. The various forms of scientific literacy are important, but there are many other relevant science related concepts, that are as important or maybe more important.

What about scientific cognition (thinking)? Scientific cognition is not the same thing as scientific literacy; it involves multiple components and sub-components (Feist, 2006).  Deanna Kuhn asserts that the essence of scientific thinking is coordinating belief with evidence (2001).  At the very least scientific cognition involves philosophy of science, scientific methodology, quantitative reasoning, probabilistic reasoning and elements of logic. Scientific cognition requires specific cognitive abilities and cognitive style (thinking disposition). 
 

Various scales have been developed to measure scientific thinking / reasoning / cognition.  Kahan developed a scale called the Ordinary Science Intelligence Scale (OSI_2.0, Kahan, 2014).  Drummond and Fischhoff (2015) developed the Scientific Reasoning Scale.  Drummond and Fischhoff found that measures of scientific reasoning were distinct from measures of scientific literacy.  Kevin Dunbar (2000) and Zimmerman (2005) have also conducted research on scientific thinking.  Dunbar's research mostly involves examining cognitive processes underpinning thinking during the research process, while Zimmerman's research is broader, examining various scales, and development patterns of scientific thought.  Fugelsang et al. (2004) have examined strategies that scientists and non-scientists use to evaluate data that is consistent or non-consistent with expectations.

Attitudes about science, predictors of scientific eminence, association between scientific measures, rudimentary knowledge regarding meta-sciences  and group difference relating to scientific concepts are other important topics, that receive less attention than general scientific knowledge.  All of these topics are important!  A comprehensive understanding and appreciation of science and its wide range of implications is a complex task.  

 

Current Research

 

Myself and colleagues are developing and modifying instruments for the assessment of scientific cognition and scientific literacy (general scientific knowledge).  We have completed a prototype for each, and we are currently using the instruments in a study examining the relationship between scientific cognition and scientific literacy. Each instrument consists of 14 questions. The scales are derivations from previously used scales. Upon completion of the study we will probably modify the instruments accordingly.  We plan on running a statistical analysis of internal consistency once the measures are complete. 

 

I am working with a colleague on an additional paper that involves measures of general scientific knowledge, attitudes toward science, relationships between / among various science concepts and group differences regarding science outcomes. This is a relatively long paper that presents a relatively large number of statistics.   

 

These studies are part of "Project: thinking about science."  We are also in the intermediate stages of the development of a seminar that will encompass information on the vast goals and implications of "Project: thinking about science." 

References are available upon request

 

You can contact me jamie.hale1@gmail.com if interested in hosting a seminar.   

 

     

Building A Better Memory

"We are who we are because of what we learn and remember"
Eric Kandel, Nobel Laureate

"Without learning and memory processes, personality would merely be an empty, impoverished expression of our genetic constitution"

Joseph Ledoux, author of Anxious

Are learning and memory completely distinct?  No; both are experienced based.  “[M]emory is the consequence of learning from an experience- that is, the consequence of acquiring new information” , asserts James McGaugh (memory researcher, author of Memory and Emotion).  Learning is a process of memory formation.  There are 2 general categories of memory: explicit and implicit.  Explicit (declarative, conscious) – is what most people think of when they think of memory.  It involves conscious recall of people, places, objects, facts and events. As an example, direct memory testing (tests in school) reflects explicit memory.  Implicit (procedural, unconscious) – the storage of information that does not require conscious attention for recall- often in the form of habits, perceptual or motor strategies, and associative and non-associative conditioning.  Examples of implicit memory include the memory utilized for riding a bike, or throwing a ball.  IM has an automatic quality, it is recalled through performance.  The tips provided in this article are for enhancing explicit memory, but they are also applicable to implicit memory (some modification may be required). 

 

Strategies to maximize learning

Be prepared! Familiarity with class material- read all assignments – complete understanding of directions

Focused Attention! Eliminate distractions- No FB or texting – focalfilter.com

Take detailed notes! Highlight – learn highlighted material well- read aloud

Following class, review lecture (notes and reading materials)! Don’t rush- think deeply about materials,  meaning and how it is connected to information already in memory

Ask questions!  In class, out of class, e-mail

Don’t worry about if you will remember! Concentrate on understanding- understanding means strong memory formation

Foundations of Memory

Strong memory rests on some key foundations.  These foundations include: brain health, focused attention, elaborative encoding, spaced rehearsal and testing.  With the appropriate strategies most people can strengthen the foundations substantially.  When considering memory and learning some people may have some biological advantages, but in most cases the right strategies goes a long way in building strong memories. [Refer to Emotional Memories and Genes for more info on how genes may influence memory]    

All memories require the brain (explicit and implicit memory).  When a new memory is formed changes occur in the brain.  Memory reflects biological change (change in brain connections).  Short term memory does not lead to brain changes, while long term memory does.  The formation of long term memory requires protein synthesis. Due to the brain's central role in memory it is apparent that brain health is important in regards to learning / memory. The pillars of brain health are exercise, nutrition, cognitively challenging activity, positive social interaction and minimal stress. 

 

Focused attention involves being  attentive to desired sensory outputs while ignoring undesired sensory outputs. That is, attention to current goal while ignoring distraction [Refer to The Benefits of FocusedAttention to learn more about this important aspect of attention]

Another foundation of  memory is elaborative encoding or rehearsal.  It involves think deeply- about meaning and connecting the to-be-remembered information to other information already stored in memory.  When using elaborative rehearsal I often recommend that students apply the VSOC principle.  This principle involves thinking about whatever your trying to remember from the following perspectives: visual, spatial, outrageous (salient) and consequential (personal consequences in regards to yourself).  This technique helps  attach the information to a large framework of existing memories, thus leading to the possibility of many retrieval  paths. An array of variations might be used.

Spaced rehearsal (distributed practice effect) involves studying or practicing persistently over time.  Cramming is not conducive to strong memory formation.  Three 1hr sessions are more beneficial than one 3hr session.  One of the key reasons that spaced learning increases memory is that each time you study you may perceive the material from a different perspective.  [Refer to HowTo Study]   

Test yourself on the information you are trying to remember. Do not have the answers in plain view while testing.  Testing serves as a powerful mnemonic aid for future retention.Testing allows for an accurate assessment of knowledge. Individuals often over estimate their level of knowledge.  [ Refer to Does Testing Enhance Learning]

To reiterate, the foundations of memory include: brain health, focused attention, elaborative encoding, spaced rehearsal and testing.  Understanding is imperative for strong memory.  Studying should be structured: progressive, organized, spaced over multiple sessions and involve accurate evaluation.