Designing a Scale

Unlike a questionnaire, a scale is designed to estimate the strength of a personal characteristic or disposition held by an individual. The characteristics measured by a scale are called constructs. Scales are most often needed when measuring constructs in the affective domain. Measurement in the cognitive domain attempts to determine what people know, understand, or are able to do intellectually (e.g., recall, understand, analyze, evaluate, synthesize, problem solve, or think critically). Measurement in the psychomotor domain describes the quality of an individual’s ability to perform physical tasks (e.g., athletic or musical ability). We do not need a self-report instrument to measure these attributes; other more direct measures are available and are more appropriate. However, measurement in the affective domain targets an individual’s interests, perspective, beliefs, attitudes, or personality traits. Scales are needed to measure affect because the best (and possibly only) way to obtain data is to ask individuals to self-report. In addition, because affective constructs targeted by a scale are complex abstractions, multiple items are required to obtain an accurate measurement (each item on a scale is designed to capture a nuanced aspect of the construct). The related items on a scale are combined to provide a numerical indication of the strength of the construct for individual respondents.

Anderson and Bourke (2000) explain that affective characteristics being measured must (a) involve feeling, (b) describe a somewhat permanent trait rather than a temporary state, (c) be targeted toward something (e.g., feelings about specific people, situations, or entities), (d) have a positive or negative direction (e.g., like or dislike), and (e) vary in degree or magnitude for different individuals. For example, physical characteristics (e.g., height, hair color, or even appearance) do not involve feeling and can be measured without the need of a self-report instrument. Likewise, anything that can be simply categorized does not require measurement (e.g., age, gender, or nationality). These characteristics can be determined through simply reporting. Attitudes and perceptions, on the other hand, are personal attributes that often can only be determined through self-report. And while general feeling may be useful, specific feeling regarding specific targets are more helpful, especially when they represent typical or deeply held perceptions rather than temporary feelings (e.g., while you may have an opinion about politics in general, you would likely have contrasting and distinct opinions about various aspects of the political process). Finally, in order for affective characteristics to be measured, they must have both direction and intensity (i.e., the potential for individuals to have more or less positive or negative feelings about the topic).

The Affective Domain

Affective constructs (and dispositions) represent inherent qualities of mind and character. A person’s innate disposition can refer to one’s predominant or typical mental and emotional temperament (e.g., a positive or happy disposition); however, a disposition can also refer to a person’s typical state of mind, tendencies, and inclinations. These are attitudinal dispositions rather than innate personality traits. Attitudinal dispositions are often the target of scales used in survey research. Measuring these constructs is important because attitudes affect an individual’s behavior (see Fishbein and Ajzen’s [1975] theory of reasoned action). Attitudinal dispositions, although in the affective domain, also have a conative influence in that while our behaviors are influenced by our feelings, beliefs, and opinions, they are moderated by free will and agency (Davies, 2009). According to Ajzen’s (1991) theory of planned behavior, people typically align their behavior with their personal belief systems after taking into account how they believe others would perceive their actions. They may also experience conflicting intentions, values, and beliefs (Davies, 2009). For example, an individual may be capable of performing specific skills, understanding important concepts, recognizing the value of the behavior, or having a personal desire to behave a certain way, but they still choose to act in ways we might not expect due to other deeply held beliefs or compulsions.

Dimensionality

Some scales target unidimensional constructs, others are multidimensional. For example, when measuring students’ attitude towards school, a scale would need to be multidimensional because students' interests and attitudes are based on a variety of unidimensional aspects. A student’s attitude towards school might be influenced by the individual’s perception of school being a safe place (physically and emotionally), a welcoming or fun place (socially), or a good place to learn (academically). The attitude may also be influenced by views of individual teachers or the student's personal interest in learning (career goals). It is very possible that a student may, for example, have a positive perception of school with regards to social aspects, but no interest or a negative attitude towards the academic opportunities provided. Each dimension would be measured separately (using a variety of items), then conceivably combined to get an overall measure of the student’s attitude towards school.

Creating Scale Items

Anderson and Bourke (2000) describe two methods for writing items for a scale—the mapping-sentence and domain-reference approaches.

Mapping-Sentence Approach

When using this approach, the item developer pairs a situation with a behavior (or list of distinct actions). The respondent will either be asked to select the behavior that best describes the way they would react given the situation or indicate the likelihood they would behave in the manner specified given the situation.

Before writing items, the item developer must carefully identify the construct (defining its theoretical nature) and describe in detail how the construct might manifest itself as a behavior (operationalizing the various aspects or facets). When identifying the core of the construct, think about the kinds of beliefs and attitudes someone with this characteristic might have. When operationalizing the construct, consider what actions or behavior would be exhibited by someone who embodied this characteristic. If providing a list of possible behaviors, each potential response should represent an action likely to be taken by individuals possessing the characteristic at different intensities (e.g., high, moderate, or low). If the target is to be more specific (rather than a general measure of the construct), the situation can be contextualized.

Mapping-Sentence Example

Suppose you were attempting to measure the construct of self-confidence (in general). At its core, confidence requires a high level of self-efficacy, a belief in one’s ability or competency. In operational terms, someone with self-confidence may try new things, attempt difficult endeavors, or be willing to take risks. They might also express feelings of success. This understanding could help you develop items to place in the scale. Pairing situations with behaviors we believed would be taken by someone with self-confidence allows us to get an indication of the strength and direction of the characteristic for individuals. While we might be interested in obtaining a general measure of confidence, contextualizing the situations allows us to get targeted measure of an individual’s confidence in specific situations (i.e., ability to compete in a specific sport or succeed in a specific endeavor).

Possible situations:

  • starting a new task
  • trying a difficult maneuver
  • giving an opinion
  • learning a new skill

Anticipated behavior:

  • begin without hesitation
  • wait until encouraged
  • think carefully whether to proceed
  • let someone else try first
  • believe they will be successful

Possible items:

  • When learning a difficult maneuver for the first time, I immediately begin without hesitation.
  • When learning a new skill, I am confident I will succeed.

Domain-Reference Approach

This approach also begins with an examination of the construct’s critical features. However, particular consideration is given to the target, direction, and intensity of the characteristic. Statements are created by specifying a target, using a directional form of the “to be” verb (positive or negative), and then adding an adjective denoting intensity. Once the basic structure of the statement is determined, it can be transformed to fix grammatical issues. The statements produced should have a direct alignment between conceptual and operational definitions of the construct.

Domain-Reference Example

Suppose you were attempting use this approach to write items that could be used to measure the construct of self-confidence. Possible targets are listed. A verb is selected, followed by an adjective that describes a possible reaction. The statement should describe a precise behavior and a strong emotion or position.

Target: learning a new skill, participating in a debate, expressing my opinion

Verb: I am, is

Adjective: exciting, thrilling, eager, happy

Possible revised statements:

  • I am eager to try difficult maneuvers for the first time.
  • I am happy to express my opinion to others.
  • I usually express my opinions quite well when debating.
  • I am very good at learning new skills.

Testing and Revision

As with all survey questions, items must be carefully tested to make sure they are meaningful and clear to potential participants. Pilot testing the items using think aloud cognitive interviews can be quite useful for this purpose. Multiple revisions may be needed. Some additional guidelines include the following:

Chapter Summary

  • A precise, agreed-on theoretical definition for the construct being measured is required before writing items. This includes both conceptual and operational definitions.
  • Conceptual definitions describe the abstract meaning of the construct and the core beliefs or motivations foundational to the construct.
  • Operational definitions describe how those with the characteristic would likely behave. Manifestation of anticipated behaviors provides inferential evidence that the individual possesses the characteristic.
  • A close alignment between conceptual and theoretical definitions should exist and be evident in each item.
  • Scales are most often needed when measuring personal characteristics in the affective domain.
  • Characteristics in the affective domain must (a) involve feeling, (b) target stable traits rather than temporary states, (c) be specifically targeted (d) have direction (positive or negative, absence to increasing positive), and (e) vary in intensity by degrees.
  • Two approaches to writing items include the sentence-mapping and domain-reference approaches.
  • Multiple items for each construct should be created and tested prior to pilot testing and administering the instrument.
  • Statistical procedures can be used to validate the instrument, including exploratory and confirmatory factor analysis or structural equation modeling.
  • Constructs may be unidimensional or multidimensional.

This content is provided to you freely by EdTech Books.

Access it online or download it at https://edtechbooks.org/designing_surveys/scale_design.