Module_6

  Overview of Instrumentation As we discussed in the previous module, conducting a research study is like building a pyramid. A literature review is like the foundation of the pyramid, establishing a solid theoretical or conceptual base and demonstrating how the proposed research study is unique from other research studies. Sampling is like the gathering of building materials. However, we need to consider what types of tools we can use to gather good materials (data) in order to build a strong pyramid. In this module we are going to learn the types of tools we can use or create for collecting samples for our proposed research study. The information gathered from the samples is called data, and those tools used for data collection in research are called instruments.

Objectives In this module you will be able to understand the following concepts: Instrument and Instrumentation Criteria for good instrument Procedure for Data Collection Data format, measurement scale, frame of reference Instrument Search (Online Library search) Guidelines for evaluating instrumentation and procedure

Resources and links for Textbook []

Question to ponder: A study investigates the impact of SES (Socio Economic Status) on academic achievement. What instruments are likely to be used?

Types of Instrument To collect data from human subjects, there are three main types of instruments: Researcher instrument The researcher collects first-hand data through direct interaction with the subjects (e.g., observation). In this case the researcher is the instrument. Subject instrument The subject provides self-report data through a test, interview, or questionnaire. Informant instrument An informed person who is familiar with the subject provides the second-hand data to the researcher. Question to ponder: The State of Florida is interested in assessing preschool program accountability. What kind of instrument would be appropriate to use?

Three Criteria for a Good Instrument Reliability Consistency of repeated measurements under different circumstances (e.g., time and day of testing, gender of interviewer, sampled behaviors) Validity Accuracy and appropriateness of inferences based on collected data (e.g., an IQ test that taps on verbal skills only, a leadership test that predicts future job success, and SES scale that measures income only) or the extent data accurately reflects the concept that it is intended to measure Usability Practicality and utility of the instrument (e.g., time, cost, training)

 For instruments, reliability is necessary, but it is not a sufficient condition to ensure validity. Sometimes, an instrument can be reliable and produce consistent results, but it may not be valid. For example, if your bathroom scale indicates your weight as 125 pounds everyday, but your actual weight is 120 pounds--you know your bathroom scale (instrument) is adding five pounds more than your actual weight. Although it provides a consistent result which is considered reliable, and you can still use it to determine if you gain weight or lose weight; however, you can not use this result directly to provide your actual weight information to your doctor because it is not valid. The following graphs from our textbook are good explanations about the relationships between validity and reliability. See Fraenkel & Wallen (2009, p. 155)

In a research proposal or paper, after or at the end of the //Instrumentation// section, there should be a //Procedure// section detailing how the instrument is used for data collection. The Procedure section may be integrated into the Instrumentation section The Procedure section should provide enough information about //how//, //who//, //when//, and //where// the instrument was used for data collection.

Different formats of data involve different mental processes and different analytical procedures, so various instruments may be used for collecting different data. There are two data formats which are categorized according to the structures of the responses: //Selected response//—Dichotomous (e.g., T/F, Y/N, Agree/Disagree, multiple-choice, matching), polytomous (graded, ordering) //Constructed response//—Fill-in-the-blank, essay, problem-solving, story-completion, picture-drawing, anecdotal log Researchers should carefully weigh the advantages and limitations of the above two response formats. For quantitative analysis and summary purposes, constructed response data are often converted into either dichotomous or polytomous data. For example, psychologists use **//Rorschach Inkblots// (See graph 1)** instrument to examine the personality characteristics and emotional functioning of their patients by asking them what image the patient associates with the inkblots. Then the constructed data collected through the instrument is converted into different hostility scores such as "3." Graph 2 shows another constructive instrument used to collect response data from children. Children are asked to draw a person so that their mental development level can be analyzed and determined. According to the example of the drawing below and the time used, it was converted into a standard score as 71. Only when the constructed response is converted into either dichotomous or polytomous data, can quantitative data analysis be conducted.

When we use instruments to collect data, we need to decide what scales we are going to use to measure the data. The scales are called measurement scales. There are four types of measurement scales which are very important for collecting quantitative data.

Nominal scale is the type or category for differentiating identity (e.g., 1 for male and 0 for female). In other words, we use numbers to represent the identity of the data, and those assigned as nominal scale numbers do not have numerical values or degrees in magnitude. For example, if we use 1 to represent males and 0 to represents female, it does not mean males possess a higher value than females, so in this case we are only using 0 and 1 to categorize the gender differences.

If we add magnitude and direction for ranking (e.g., A > B > C, 1st < 2nd < 3rd) to the nominal scale, then it becomes an ordinal scale. An ordinal scale does not only classify the categories, the numbers used for ordinal scale also represents ranks. If we say 1st grader, 2nd grader, the 3rd grader, we know the 3rd grader is the highest level among the three. Or in a car race we could have lst place, 2nd place, or 3rd place.

If there is an equal magnitude in unit of measurement (e.g., 90ºF–89ºF = 75ºF – 74ºF), the scale is interval. For example, we know that the temperature difference between 74 degree and 75 degree is equal to the difference between 89 degrees and 90 degrees, which means that every two numbers next to each other has the equal magnitude in unit of measurement of one degree.

Ratio scale has absolute true zero for ratio comparison (e.g., income, game score). For example, if we say someone has zero income, it means that person does not have income at all. But if we say the temperature is zero, we know that the zero still represents the degree of the temperature (there is no such thing as no temperature), so temperature will be measured by interval scale rather than ratio scale. Please note, both interval scale and ratio scale are continuous variables because both can be continuously counted.

mastery of objectives in a specific content domain (e.g., 80/100=80% content mastery) For example: //**Driver’s license test in which a minimum of 35 out of 50 items correct is considered passing. **// relative performance compared to others (e.g., ranking the 99th percentile in a group) For example: **//In a scholarship competition, candidates who received the top 10 rankings are awarded the scholarship. //** cutoff point on a DR or NR scale (//Note //: Some texts use CR to refer to DR and do not make the distinction between CR and DR.) For example: **//The students in a literature class are required to read at least one book of fiction every two weeks. //**
 * //Domain-referenced //**// (DR) //
 * //Norm-referenced //**// (NR) //
 * //Criterion-referenced //** //(CR) //

To evaluate instrumentation and procedure, it is helpful to ask the following questions: Does the Instrumentation section provide enough information on the quality of the instrument and rationale for the instrumentation? Is the instrument reliable, valid, and practical? Does the Procedure (sub) section provide enough information about how, who, when, and where the instrument is used for data collection? Is the procedure for data collection appropriate?

To collect data, we can either create our own instruments or use published instruments. In the following, some useful resources are introduced to you to search for published instruments.

Educational Testing Service website (www.ets.org) provides testing items ready for researchers and teachers to use. You can use //Tests // (//Test Directory //) to find a wide variety of tests (e.g., intelligence).

You can search test reviews both to locate instruments and to review the quality of a certain instrument. The most useful tool is Mental Measurements Yearbooks (//MMY //). MMY is available in the UCF library. To get access to MMY, just go to www.library.ucf.edu and click on //Books / Catalog////__ à __// Online Resources, and enter //Mental Measurements Yearbooks //, and click on //<span style="color: black; font-family: 'Arial','sans-serif';">go " // EBSCOhost:… " enter search text (e.g.,//<span style="color: black; font-family: 'Arial','sans-serif';"> intelligence //). If you have any problems getting access to it, please ask a librarian: Call 407-823-2562 or 1-866-271-7589 (toll free)

 Please continue to work on the literature review section of your Research Proposals and choosing the articles for your two Article Reveiws/Critiques. Remember the articles selected must be **__<span style="color: black; font-family: 'Arial','sans-serif';">peer-reviewed __** articles only. The Wilsonweb database allows for you to select a search option for peer-reviewed articles. One technique to determine if you have found a true research article is to look for a Methods and/or Results heading. If you are not certain, please contact me through the course e-mail. Don't be surprised if you don't understand the results section yet. You will learn what those statistical numbers mean in this course.

 Please post a summary of your instrumentation plan for your research proposal in the Discussions area. The posting should include: A list of the instruments you plan to use to gather your research data. Briefly describe each instrument (36 multiple choice questions, 24 attitude Likert scale questions, etc.). If you have validity and reliability scores on existing instruments please provide that information. Reliability is normally determined by the Cronbach Alpha score which is designated between 0.0 (no correlation) to 1.00 (perfect correlation). Many researchers require a minimum reliability of 0.70 or higher to consider an instrument usable. If you will be designing your own instrument, provide a draft sample of the survey or sample items for survey questions with a description of instrument scales and data format. If you are going to do a purely qualitative research study, describe or identify how you will observe or interview participants in the study (i.e. observation rate scale, field notes, interview questions). Also, describe how you will ensure that your data are trustworthy. In addition, describe any type of assistive technologies or other special equipment to be used in your study (i.e. survey monkey, palm pilots, videotaping, etc.)

 You can go to the textbook web site and complete the multiple choice and matching for chapters 7 and 8 as a self-assessment and in preparation for the midterm. http://highered.mcgraw-hill.com/sites/0073525960/student_view0/chapter7/ http://highered.mcgraw-hill.com/sites/0073525960/student_view0/chapter8/**/**