Module_7

Module 7: Internal Validity 7.1 Overview and Objectives

Overview of Validity
In the last module, we learned how to use tools (the instruments) to collect materials (data for research) for the construction of our pyramid (research proposal). It is critical to make every effort to ensure the data we collect are reliable and valid; otherwise, our research findings are of no value and cannot be substantiated. In order to construct a solid pyramid, we must avoid using inappropriate or bad building materials, otherwise our pyramid will be a shaky and unreliable pyramid or it could end up not being a pyramid at all. Therefore, we need to know how to garner valid data in our research so that our findings are based on accurate and appropriate information that are considered trustworthy.

Objectives
In this module you will be able to understand the following concepts: · Internal vs. external validity · Sources of threats to internal validity · Controls of threats to internal validity · The //Limitations// and //Recommendations// sections as they relate to uncontrolled threats

Resources and links for Textbook
[] = =

Research Validity
· Validity of a research study refers to the //credibility// of the research findings that can be substantiated by the //controls// against potential //threats//. · There are two types of research validity: o // Internal validity // (credibility of research methodology within the internal framework of a research study); //-- Focus of this module// o // External validity // (credibility of generalizations of the current findings to other research settings).

Threats to Internal Validity
In this module you will be able to understand the following: · Threats to internal validity refer to //viable alternative explanations// for research findings that may jeopardize the credibility of research conclusions within the current research setting. · An internally valid study allows for //unambiguous// research conclusions by eliminating viable alternative explanations. · An example of threats to internal validity: // A study found that infants conceived with assisted reproductive technologies (such as test-tube method or introcytoplasmic sperm injection) have an 8.6% risk of major birth defects, compared with 4.2% in babies conceived without assisted reproductive technologies. Can the researchers conclude that infertility treatments damage babies’ genes? // To answer this question, we have to consider if there are any other reasons that could cause birth defects other than using reproductive technologies. One potential threat could be the health condition of those mothers who seek the assistance of reproductive technologies. We know that only women with abnormal health conditions will seek the technological help; therefore the higher risk of birth defects may be caused by the mothers' abnormal health condition, a threat to internal validity, rather than the reproductive technologies treatment.

Sources of Internal Threats
Internal threats mainly come from the following four sources: · // Subject // (including //Selection//, //Regression//, //Mortality//, //Maturation//, //Awareness//) · // Location // · // History // or //External Event// · // Instrumentation // or //Data Collection// (including //Instrument//, //Testing, Data collector, Carryover//) Although this module focuses primarily on internal validity, many of the threats discussed below are applicable to external validity as well. In this next section we are going to discussion these threats one by one.

Selection Threat
· Unintended subject characteristics (e.g., age, gender, SES, intelligence, motivation) may render two groups incomparable to begin with or the sample being unrepresentative of the population. // Example 1: A CNN viewer-call-in poll that surveys the attitude toward America’s military agenda on Iraq. // Discussion: For example 1, we question if the data selected could be biased because only those against the Iraq War may be more eager to express themselves. // Example 2: A researcher solicits volunteers to participate in a new treatment for cancer and compares the mortality rate of this group with that of a traditional treatment group. // Discussion: For example 2, we know that it is more likely that patients with more severe life-threatening health conditions to accept an innovative new treatment as a last resort, so the mortality rate may be higher due to the severity of their condition going into the new treatment.

Regression Threat
· Extreme pre-scores have a statistical tendency to regress toward the mean on the post-test because the factors of chance that may have contributed to the extremity are not likely to be operating again. // Example: A researcher selects the lowest 20 percent on the pre-test for a special remedial intervention and finds improved performance on the post-test //. // Discussion // : In this case, the highest 20 percent on the pre-test may have lower scores in the post-test while the lowest 20 percent on the pre-test may get higher scores because the source causing the extreme score differences does not exist in the post-test.

Mortality Threat
· During the course of study, some subjects may drop out due to //systematic// reasons, yielding an unrepresentative sample who have remained or responded in the study. // Example 1: A student satisfaction survey with low return rate, in which only unsatisfied students returned the surveys. // Discussion: The drop out group has a systematic reason for not responding, so the data collected is biased because the losing of samples makes the sample non representative of its population.

Maturation Threat
· During the course of study, subject characteristics may change due to natural aging, leading to //improvement// or //deterioration// in performance. // Example 1: A high positive correlation was found between toe length and intelligence for grade-school students. // (This example was a real research study, and it was popular for awhile that moms kept pulling their kids’ toes.) // Discussion: The older the child is, the longer the toes is; the older the child, the higher level of intelligence, if the child is normal. Therefore, the higher intelligence level is caused by maturation rather than the length of the toe. //

Awareness Threat
· // Subjects’ awareness of the study may change their performance beyond treatment effect. // o // Hawthorne Effect—Improvement in performance of treatment group due to attention or novelty of treatment. This tends to overestimate the treatment effect. // o // John Henry Effect // —Improvement in performance of control group due to sense of being excluded or left out, resulting in “//compensatory rivalry//.” This tends to underestimate the treatment effect. o // Diffusion Effect // —Subjects in the control group interact with those in the experimental group. // Example 1: Students in a technology-infusion class are told they are participating in a fun project and the whole nation is watching their performance. By the end of the study, the students perform better than a conventional class on both achievement and motivation measures. // // Discussion: The students are aware that they are being observed and are more motivated to perform well--Hawthorne Effect. //

Location Threat
· Characteristics of the physical location where research takes place may introduce systematic bias in the results, which has nothing to do with the purpose of study. // Example: A researcher is interested in examining the effectiveness of technology-enhanced instruction. The technology-enhanced instruction takes place in a classroom with better lighting, newer furniture, and more colorful wall paintings than traditional classrooms. The results show that students in the technology group perform better than those of the traditional classroom. // Discussion: The students probably are more productive in the classroom that has a better environment and working conditions.

History Threat
· Incidental or unanticipated events external to the study may affect the subjects’ performance. // Example 2: A new teaching method to promote multicultural awareness is implemented in Florida right after the 9/11/01 terrorist attacks. The study finds the method to be ineffective. // Discussion: The terrorist event may have decreased students' interest, appreciation, or created a negative attitude towards cultures outside of the United States.

Instrument Threat
· Lack of validity of the instrument or change in instrumentation may obscure the data collected. // Example 2: A merit-pay system operationalizes “teaching load” as “class size.” //

Discussion: Teaching load is not only associated with class size.

Testing Threat
· The //reactivity effect// of pre-testing alerts subjects to the purpose of the study and the practice or memory of the pretest tends to inflate their posttest performance. // Example 1: A researcher gives a pretest on the content of an instructional video, plays the video, and gives a posttest on the same content. The students show marked increase on their post-test scores. // Discussion: Students exposure to the pretest may have alerted them to specific content that would was found on the post-test.

Data Collector Threat
· The person who collects data may affect subjects’ performance. Possible sources of threat: o // Characteristics // (e.g., age, gender, race, attractiveness); o // Subjectivity // in scoring (e.g., inconsistent criteria, diffusion from parts to whole or //Halo Effect which means that when we consider a person good (or bad) in one category, we are likely to make a similar evaluation in other categories., please see the notes under this slide for definition//); o // Bias // (e.g., ask leading questions, provide nonverbal cues to desirable answers, researcher as data collector). // Example 1: The researcher scores all the essays in one class then all the essays in the other class. The same essays are scored by two scorers with the scores sitting side by side on the paper. // Discussion: The researcher may be biased and may score one class using a different standard for scoring than another class. The two scorers have knowledge of the researcher's scores which may influence how they determine scores on the same essays. // Example 2: After passing out the Course Evaluation questionnaires, the instructor strolls in the classroom to answer questions. // Discussion: The instructor's presence in the classroom during the evaluation may intimidate the students from answering questions honestly. // Example 3: An instructor designs a science mystery novel series, tries it out in her classroom, and claims tremendous successes in boosting student learning. // Discussion: Researcher bias causes the instructor to inflate the results or causes her to design a study that will produce the positive results she desires.

Carryover Effect
· // The cumulative effect of multiple treatments tends to inflate the later treatment. // // Example: A researcher tries out Method A with a group of subjects for one month, administers a test to measure the post-A effect, and follow with Method B for another month, and administers the test again to measure post-B effect. She finds post-B test performance to be higher than post-A test performance. She concludes that Method B is more effective than A. // Discussion: Students had already been exposed to Method A and may have retained knowledge from Method A rather than from Method B. If the time between methods is too short, the researcher cannot determine if Method B was the cause of the improved test performance or if it may have been influenced by Method A.

Controls for Threats
The following are some suggested ways to control for threats to internal validity · Control group (non-treatment group) for comparison (so that the non treatment effect, if any, can be determined in order to partial it out); · Randomization or matching (to make equal groups to reduce group differences from subject characteristics); · Neutralization (holding confounding variables constant, such as using the same gender in the experiment to reduce gender effects, //but// jeopardizing external validity); · Counter-balancing (order of multiple treatments is equally distributed so that carryover effect can be reduced); · Blinding (single-blind and double-blind controls prevent research outcomes from being 'influenced' by the awareness threat or observer bias. Placebo--deterioration in performance due to lack of attention can be counter-measured by some neutral treatment.) · Statistical adjustment (statistically neutralizing confounding variables) Note: 1. There are many other ways through research designs or statistical methods to control for threats to internal validity which are beyond the topic of this course. If you are interested, the following are good resources for your reference: [] 2. Reliability and validity are most often emphasized in quantitative research; however, validity should not be overlooked in qualitative research either. An excellent article on this is Anafara, V. A. & Brown, K. M. (2002). [|Qualitative analysis on stage: Making the research process more public.]. //Educational Researcher, 31(7), 28-38.//

Lack of Control over Threats to Internal Validity
An internally valid study should control for all potential threats during the planning stage. If expo-facto (after-the-fact) threats were found after the study has been completed, the researcher must discuss these threats or limitations in the //Limitations// and //Recommendations// sections. In critiquing a research paper, if not enough information is given for evaluating research validity (e.g., no words on the pre-treatment equivalency of the experimental and the control groups), this lack of information would suggest lack of control.

7.5 The Limitations and Recommendations Sections
Included in almost all research papers are the limitation and recommendation sections, in which the following points are commonly addressed. · Demonstrate awareness of potential constraints · Acknowledge uncontrolled potential threats to research findings · Controlled potential threats are described in the //Methodology// section · The //Limitations// section may appear at the end of the //Introduction// section or the //Methodology// section (the later is preferred) · Limitations lead to “//Recommendations for Future Research//” at the end of the study

7.6a Article Reviews / Critiques and Research Proposal Paper
Please continue to work on your article reviews / critiques and discussing or developing your research proposal paper. This week consider the potential threats in your proposed research and how to control them.

7.6b. Summary of Potential Threats and Control Methods
Please post a summary on the potential threats and controlling methods within the Discussion area of the course website. Your summary report should address two components: · Identify the potential threats and · Specify the methods you are going to use for controlling those threats.

7.6c. Mid Term Practice .
You can go to the textbook web site and complete the multiple choice and matching for chapters 8 and 9 as a self-assessment and in preparation for the midterm. [] []