Finding & Evaluating Sources
In this chapter, you will learn how to
conduct background research on a topic
select and narrow a research topic and create a research question
utilize library and electronic resources to locate relevant information
evaluate sources' reliability and usefulness
take notes and create an annotated bibliography
Note: this chapter contains a Library Research Homework Assignment as part of the practice for this section, so budget extra time to choose and narrow your topic.
I live in the high desert mountains of the West surrounded by ranchers and cowboys. Cowboys and cowgirls are tough. Finding and evaluating sources is tough. So I'm going to teach you how to find and evaluate sources western style, like a cowboy or cowgirl. So you can be tough, too.
8.1 Why Research?
If a cowgirl wants to buy a horse, she first decides what kind she wants--there's a big difference between a hard-working quarter horse and a tall, speedy thoroughbred. Then she finds out who's selling, checks out the horse's condition, learns about its heritage, tries it out, and sees how it behaves in different situations. In other words, she does her research. We, too, spend our lives doing research--whether we realize it or not. For example, between Yelp!, Rotten Tomatoes, Consumer Reports, and Amazon reviews, it seems like we're always trying to find the best products and the best deals. When we have questions, we go to Wikipedia or ask Siri or Alexa for answers. Never in the history of the world have we had so many resources literally at our fingertips.
"Google is turning 18 years old this year. I cannot believe it was just 19 years ago that I never researched anything ever."
--Comedian Kellen Erskine, "Composed" Dry Bar Comedy on VidAngel
The trick is, how do you know you're getting the best answers to your questions? How can you tell if you're using the best sources of information or if the information you find is accurate, reliable, and up-to-date? These days you even have to ask yourself if the information you find has been planted by Russian hackers trying to influence your political opinion. The problem with a glut of resources is that it's become increasingly hard to find reliable information, which is why doing good research is becoming a necessary life skill, not just a school skill. If you know how to conduct good research, you'll be more influential no matter which career you go into.
And so I give you the Cowboy and Cowgirl's Guide to finding and evaluating sources with all the steps you'll need to find solid answers to your questions.
8.2 Step One: Choose Yer Horse (Select a Topic)
My kids call their grandpa "Cowboy Phil" because he wears a big custom-made hat, likes to go on cattle drives, and taught my kids how to ride a horse. He also taught them that it matters what kind of horse you choose.
"You pick your horse based on the work at hand. But above all, you've got to really know your horse and make sure it knows you're the boss."
Choosing your topic for the journey of writing any kind of research paper should begin with similar consideration and care. You want to choose your research topic wisely; you don't want to invest a lot of time, energy, and resources into researching a topic for a long paper unless it has a lot of potential and is something you like enough to stick with for the long haul.
"A good horse costs just as much to feed as a bad one." --Cowboy Wisdom
In a typical advanced writing class, you'll probably be asked to spend a significant amount of time on this topic: you'll likely create an extensive literature review looking at what others have said about your topic, you'll perhaps write a proposal based on it and/or give an oral presentation or poster presentation explaining this topic to your peers.
So find something you love, because it won't be worth investing in unless you really love yer horse, er, topic. So what's the best way to choose a topic? Just like on a ranch, you need to do some gathering work first. Gather your string (all your ideas--or potential topics) to the corral to take a good look at the possibilities.
One of the most overlooked and underrated parts of the research process is doing background research. Students often want to just jump right in to finding sources on a topic--grabbing the first horse they see--but the problem is that no one can know how good their topic is-especially how wide or narrow it is--until they take a good look at the bigger picture.
Getting a bird's-eye view of your topic will help you understand the context of how your topic fits into your field in general and even how important it is to the bigger world. It'll also help you discover what questions are being asked, what the hot topics are in your field, and where the most promising research is going. All this will help you create a better research question and streamline your database searches down the road. Watch this quick video on starting the research process and then we'll talk about options for doing background research.
Why did you choose your major in the first place--what topics most excited you?
Can you think of problems you've heard about in your field that you want to solve?
Can you remember something you recently heard in a class that sparked your interest?
Consider a controversy or question you want to know more about but you don't know the answer to.
Think back on personal problems and questions you've had in your life and how they might relate to your discipline of study.
Mindmapping has also been shown to jumpstart your creative juices and help your brain make new connections. There's something about thinking visually rather than just linearly that allows you to explore relationships between topics in a fresh way. In fact, we'll revisit this idea later when we talk about generating ideas for paper organization.
Mindmapping can be done on a piece of paper (the old-school preferred way) or using mindmapping apps or software like Trello, FreeMind, or XMind (new school). In the middle, write a general idea for a topic in the middle and circle it. Then draw a line to branch off that idea and write something related to it, etc. Using different colors for different branches can help you visually organize your information. You might be skeptical of this method at first, but just try it. You might be surprised what kind of ideas pop into your head when you see them visually mapped rather than simply listed in a row.
As basic as it sounds, sometimes just talking out your ideas with another person can help you make connections and discover new possibilities. Find a buddy, go to one of your professor's office hours, or even ask someone out and talk about topics you're interested in. This can either be like the freewrite where you just explore ideas with little judgment or you can ask the other person to give you honest feedback. Sometimes you'll be surprised what you can come up with simply by articulating your ideas to another person and hearing feedback from them.
Wikipedia and Google
Another good place to start exploring general topics is probably already one of your go-to sources: Wikipedia. In case you've been living in a cave, Wikipedia is a summary of what's generally known about millions of topics. A Wikipedia search can give you a quick sense of the history of a certain topic, who the majors players are, and what the sub-areas of research are. It can also lead you to related ideas and areas of research you hadn't though of before. Wikipedia's structure can be helpful to you as well because it breaks down larger discussion into bite-sized sub-issues and links out to other related articles. Consider how this organization can help you choose and narrow a topic.
Because Wikipedia's not a peer-reviewed publication, you can't use it as a main source for your paper, but most information on Wikipedia includes references to the original, primary sources, so you can use their references section to help you locate reliable, peer-reviewed sources that you can use in your paper.
Doing some general Google searches will also help you see what popular sources and commercial sites appear about your topic and if it's been in the news lately. Trade journals by professionals in your field are also a good resource. They might have a "magazine" feel to them but deal with specific issues within an area of study.
Which books and articles are the most influential about a certain topic?
What's the current thinking in your field on a particular issue?
Or what are some synonyms for specific search terms that can make database searching more fruitful (like finding out you should be searching for "emerging adult" along with "young adult")?
"A good background researcher sketches out main arguments, sub-topics, and specific language that’s popping up within a wider discussion during the background research stage. This language will become important later when the researcher opens up a database"
--Elise Silva, BYU Writing Programs Librarian
As you explore ideas about your topic, don't forget to take notes about key words that come up, how many references are listed, what sub-topics emerge, whose names you see repeatedly, and which areas of research seem especially fruitful. You'll thank yourself later in the research process when you use those notes and keywords in your database searching or if you decide you want to change your focus to a different angle or even a different topic.
Here's an example of how background research helped one of my students named Justin. At first, Justin decided he wanted to study the effects of exercise on diseases or aging. But then as he did some background research, he found that there were already tons and tons of articles on that topic. So he brainstormed other ideas, talked to his peers and professors about possibilities, and even sought the help of the Economics subject librarian.
After more exploration, Justin decided to change his focus to the influence of China on the economies of African countries. There were a lot fewer articles on that specific topic, so he knew it was a more fruitful place to put his research energy. Plus, by changing his topic early on before he spent too much time gathering sources, he saved himself a lot of time in the long run.
Now it's your turn. Brainstorm some ideas and choose a general topic that interests you. (Note that for the purposes of this class, your topic should be related to your major.) Do some background research: read about it on Wikipedia, search it on Google, see if it's been in the news. Pay attention to what sub-areas emerge and who the major players are. Maybe do a freewrite or create a mind map to generate more ideas. Then write two or three possibilities for a research topic you want to explore further in this class.
Step Two: Saddle Yer Horse (Create a Research Question)
Research always starts with a question. It might not always be stated outright, but every time you look something up on your phone, you have a question--there's a gap in your knowledge that you want to fill. Once you have a general sense of a topic you're interested in, that's when you're ready to formulate a more specific research question. Trust me, you want your horse (topic) to be ready before you get on the road.
“You have to find a better reason than ‘it’s an assignment’ to devote weeks to your research and for your readers to spend time reading your article. You’ll find that better reason when you can ask a question whose answer solves a problem that you can convince readers to care about. That question and problem are what will make readers think your research is worth their time.” from The Craft of Research (Booth, Colomb, and Williams, p. 35)
Read what's been done on a topic
Figure out where the gaps of knowledge are (secondary research)
Fill one of those gaps with some kind of action that creates new knowledge--be it a survey, experiment, analysis, longitudinal study, product, etc. (primary research)
Publish the results of that action or create a prototype
Start the process over again
Your job right now is to do steps 1 and 2--the part of this process: read what's been done on a topic and figure out where the gaps of knowledge are. Then later you might even be asked to propose some primary research that could fill one of the gaps you find (although there most likely won't be time in this class to actually do the primary research, just to propose it).
So in order to find a narrow enough topic to study and to be ready to go to the next step in the research process, you need to come up with a strong research question. You might be tempted to skip this step, but trust me that if you can take the time now to devise a narrow, specific research question, the next step in the research process--database searching--will be much, much easier.
Watch this quick video about research questions:
Although, we use the term "research question," it doesn't necessarily have to be in question format; your research question can be as simple as testing a hypothesis like this one:
"I expect that smartphone usage at a young age will increase a child's likelihood of developing anxiety."
Even though this isn't in question format, it's an implied question--we assume that by making this statement, you'll next try to figure out whether your hypothesis is correct. So really, your research question turns into this:
"Does smartphone usage at a young age increase a child's likelihood of developing anxiety?"
A hypothesis means that you have a prediction to your research question that the answer will be "yes." You can create a more open ended approach where you adopt a posture of openness and curiosity like this:
"I wonder if a teacher's appearance has an effect on their students."
The "I wonder" phrase is also an implied question because presumably the way you'll learn about this topic is to find answers to questions about teacher appearance and student learning. In other words, you don't have to stick to an exact formula when developing a research question, but you do need to have a specific area of inquiry in mind.
If, on the other hand, you like to use formulas, one helpful way to create a research question is to formulate what my colleague Dr. Grant Boswell calls a "WATCO question." WATCO stands for What are the consequences of? Or to be more precise, What are the consequences of something (A) on something else (B)? This format can be very helpful because it forces you to narrow your topic--and I find students almost always need to narrow their topics more than expand them. As you will see as you begin to search databases for articles and books, the narrower or more specific your research question, the easier it'll be to find answers. Here are some examples of WATCO questions previous students have asked (they were assigned to come up with three possible questions).
WATCO (what are the consequences of) childhood obesity on:
- Self confidence?
- Adult health outcomes?
WATCO narcissism on:
- Adolescent populations?
- Adult populations?
WATCO educational funding on:
- Student Performance?
- Extracurricular Activities?
- School Resources?
WATCO meritocracy on:
- Social mobility?
- Higher education?
WATCO bilingualism on:
- Infants' differentiation abilities (Between the two languages)?
- Cognizant skills as compared with their monolingual peers?
- Infants' social skills?
WATCO a high sugar diet on:
- Metastasis and tumor growth?
- Cancer mortality?
Here are three different but related questions from one student:
- WATCO dyslexia on the encoding processes?
- WATCO early intervention with autism on ability to develop Theory of Mind?
- WATCO speech/language therapy on patients who have suffered left-lateralized strokes?
As you can see these students are starting to narrow a large topic into smaller areas of focus. The more specific the WATCO questions, the easier it'll be for them to start the next step in the process.
At this point, getting feedback can be very helpful. Seek feedback on your research question from your peers, your teacher, or other professors. It's likely that if they think something sounds interesting, it'll be interesting to other people as well. They can also give you ideas of ways you can narrow your topic or different avenues you can take.
Research Question & Feedback
Research Question & Feedback
Try to formulate three possible research questions based on the topic you've selected. If you want to use the WATCO format, you can choose at least one A term and three B terms or write three separate questions. Narrow your terms as much as possible. Post your own research question then comment on 3 of your peers' questions by clicking "Reply." Here's what to comment on for each question:
- Rate your peer's questions on a scale of 1-10 based on how much you would want to read a paper about that topic.
- Explain what interests you about that question.
- Add suggestions for improvement.
Once you've gotten feedback on your questions, choose the topic that resonates the best with others and that you believe will be the most fruitful and interesting to you. Now you're ready to move on to the next stage: finding and narrowing sources (or in cowboy terms, getting the lay of the land).
8.3 Step Three: Get the Lay of the Land (Search & Narrow)
So the first researchers who discovered cancer asked the big questions like "What is cancer?" "How does it affect the human body?" But then as they began to answer those questions, they narrowed their scope to create new knowledge. For example, cancer researchers branched their research into looking at
different types of cancer (e.g., What is lung cancer? What is skin cancer?)
or different populations (e.g., How does cancer affect people with low socioeconomic status?)
or different age groups (e.g., How does cancer affect children?).
By adding new factors to their research questions, they narrowed their scope and continued to create new information.
You'll follow a similar path when looking for sources about a topic: you'll start with a topic or research question, then as you search for sources, you'll probably find that you need to narrow your scope until you find that sweet spot--a manageable number of sources to take a closer look at.
Here's a real life example: my husband is a professor here at BYU in Mechanical Engineering and Neuroscience. He studies Traumatic Brain Injury, but if he just typed "Traumatic Brain Injury" into the Web of Science Database, he got over 38,000 results: clearly too many to handle! Then if he added another factor to narrow the scope--say, "Traumatic Brain Injury" AND "Rehabilitation"--then he got 4.800, still too many, but getting better. Now, when he added a third narrowing factor "AND Robotics," he got 32 results--the Sweet Spot! That's a manageable herd. Later you'll be finding your own Sweet Spot, so stay tuned.
So now it's your turn to get the lay of the land and start rounding up sources the way a cowboy rounds up his cattle. Where do you search? If you're a college student, you're lucky! Your university library most likely gives you free online access to specialized databases, subject guides, and hundreds of other resources, and it's all free of charge! If you're not in college, you can still check out your local library and Google Scholar for many resources. Here's where to start:
Library Research Assignment Part 1
Complete the parts of this assignment to get started navigating the library website and searching databases for your research topic.
- Write your narrowed research question or topic here (it should be related to your major or minor).
- List 3-5 possible keywords that will help you research your Research Question/Topic.
Library Research Guides
If you click on the Articles tab, you'll see that in the field of Economics, there are several databases that are used the most: EconLit for articles more under the general economics umbrella and Business Source Premier and ProQuest Business Collection for more business-related articles. These databases keep your search inside the relevant field and weed out the irrelevant junk you'd find just searching on Google or even Google Scholar. The librarians have already done the hunting for you, so take advantage!
Library Research Assignment Part 2
Go to the BYU Library home page at lib.byu.edu and click on “Research Guides.” From the “Subject Guide” list, choose which subject fits your topic. List the subject and subject librarian here (you might need to click on an extra tab like one that says Get Help).
List the top indexes or databases for journal articles that the Subject Guide recommends you use for this topic. (List 3-5)
Always go through your university library website to search in databases (or even to use Google Scholar) because once you're logged in, you should have automatic access to thousands of databases. If you're not logged in, you could be locked out of many journals and other publications or be required to pay for them. If you ever have trouble accessing databases or RefWorks, contact your librarian or teacher for help.
When you come across a perfect source that really fits your topic and you wish you had more like it, read the article's Introduction (aka Literature Review) section and check out their References page to see which articles they're citing. If they spend a lot of time talking about a certain article, you'll know it's important to their topic. You can easily search for those sources on your university library page or on Google Scholar. This is a great way to quickly find relevant sources.
In a lot of databases, you can filter for “most cited articles” and “most cited authors” The ones that are cited the most are usually the most important sources to look at because they've had the most influence. Pay the most attention to those.
Library Research Assignment Part 3
Do a thorough search of the Subject Guide. Explore several of the tabs. Look for resources you can use to get background information, bibliographies, statistics, and other research in your field. Note the resources you can use in future research projects. List three sources BESIDES indexes and databases that could help you in this project. Wise students use specialized background sources as well as articles and books during the research process.
What is a new resource that you learned about in this search?
Finally, lest you think that Subject Guides are only helpful for doing research in your field, I want you to take some time to explore a subject totally unrelated to your major or research topic so you can see what's out there. In your personal life, you'll probably need to research topics that you've don't know anything about, so take a few minutes and see how much you can learn.
Library Research Assignment Part 4
Spend a few minutes researching a subject guide in a field very different from your major. Turn up your curiosity, open up tabs, and browse. List two things you learned in your browsing.
If you still feel lost or would like more detailed guidance about finding sources or evaluating what you find, you can review Modules 3 and 4 here (scroll down). https://edtechbooks.org/-mia
8.4 Step Four: Round 'Em Up (Gather & Annotate)
Now's the time to gather your herd and decide which sources are the best. (Public Domain)
Now that you've set your sights on the sweet spot (usually between 30-50 sources), you want to start gathering like a good cowboy. Don't just randomly grab whatever you see--you need to be selective, look at many, but only gather the best sources. The first step is to make sure you have a good rope--in other words, the right tools.
Grab Yer Rope
"A Cowboy's most important tool is his rope." --Michele Morris in The Cowboy Life (p. 71)
If you don't already have a RefWorks account (or other citation software), now's the time to get one. Citation Software is an easy way to save articles and other sources you find in one electronic location. The beauty of doing this is that once you've incorporated your sources into your paper, you can quickly create a References list from the software. Although you'll still need to double-check the entries it generates (always double-check!), this will still save you a lot of time. You can also use it to organize your sources and even save .pdf versions of them in your account.
There are many options for citation software, so if you have one you already know and use, you can stick with that. Some options are EndNote, Zotero, Mendeley, EasyBib, and Citation Machine. If you're a BYU student, you have free access to RefWorks, and the library has made it very easy to save items straight to your RefWorks account, so I suggest you start there.
Here are instructions on how to set up your own RefWorks Account: https://edtechbooks.org/-oTid
Here are instructions for sending documents to RefWorks: https://edtechbooks.org/-hyrH
If you'd like more instruction on how to use RefWorks. here's a nifty playlist of tutorial videos on YouTube: https://edtechbooks.org/-QJM
Should you Read It?
Is it credible?
Is it reliable and relevant?
How to Read Online
Reading research online is very different from reading print sources; however, more of us are reading online than ever before. Before you whip out those articles, make sure you understand how to mentally prepare yourself for the online reading experience. If you're reading an e-book or an academic article in PDF format, make sure to create a helpful reading environment for yourself by mitigating distractions, and spending a pre-determined length reading/annotating before jumping to the next task. Screens distract us, but deep reading, the kind you need to do when researching, does not work when you're distracted. Turn off your phone, and concentrate.
You can also take advantage of software that allows you to (write notes directly on) articles and take notes as you go. You can do this by hand or digitally. Hypothes.is provides pretty cool software that allows you to write notes directly on online webpages and articles and also share your annotations with peers or others.
How to Read a Journal Article
Finding sources is different than understanding them—especially because academic jargon can make these texts harder to access than than what you're used to reading. Whether you're reading an academic article or an academic book, make sure you’re strategic in your approach to reading. Good researchers don't read articles cover-to-cover, so it's important to learn how to approach a source so you don't waste precious time.
First, as you're searching, look only at titles and abstracts to figure out which sources are the most relevant to your topic. Sources don't need to address all aspects of your topic to be relevant, but they should address at least one aspect that relates. When you find a source you want to take a closer look at, mark it by adding it to a RefWorks list or recording it somehow. Then don't just read straight through each article you find--that will take hours and is extremely inefficient. Instead, follow this advice from an expert:
BYU Professor Dr. E. Jeffrey Hill from the School of Family Life spoke with Wes Burr (one of the most prolific authors in the family studies field) about how to efficiently read journal articles. He offered the following suggestions (used with permission):
- "First, commit that you will spend no more that 60 minutes reading the article.
- Begin by taking 2 minutes to read the title carefully. The author(s) likely spent many hours deciding on a title. Try to get the most understanding out of it that you can.
- Next, spend 10 minutes reading the abstract. It should contain about 50% of what you need to know from the article. Have a sheet of paper ready and draw out all of the relationships mentioned in the abstract, so you get a conceptual idea of what is going on.
- Now read the first paragraph of the introduction. Much of what they couldn’t fit into the abstract will be in the first few paragraphs of the introduction.
- Next read the Conclusion/Summary. This should be the take home message of the article and will give you an idea of the main points.
- After reading the Conclusion/Summary, read each heading and first sentence of every section. You may read more if something seems particularly interesting. You may also look at figures and tables to see a succinct summary of results.
- Next read the entire discussion section. This is where the author tells what the important findings were and the meaning of those findings.
- Finally, if you have any time left, read the rest of the introduction to get a better idea of the background.
You almost never need to read the methods or results sections in their entirety unless you have a specific reason like if you're setting up an experiment using similar methodology (though it can be helpful to briefly look over any tables or figures). The idea is to be efficient and recognize which aspects of the article are relevant to your topic. Throughout the process, highlight key ideas and take notes. And always record which source a quote or idea comes from because you'll need that information later."
Remember that many of the conclusions any given author makes will be questioned by another source, so it’s fine if you don’t agree with what the author says--you simply need to understand the main points and how their article relates to your research and the other sources you've read.
One of the most important aspects of cowboy/cowgirl life is keeping good records. It's crucial to know how many cows you have and which ones are the best.
"Record-keeping is as much a part of ranch life as roping and riding." --Michele Morris, author of The Cowboy Life (1993, p. 147)
Just like there's an art to rounding up cattle, there's an art to sorting through sources. The goal of annotating and note-taking is for you to self-regulate--or make sure you’re understanding the piece while you’re reading it--but it’s also to create a helpful list that you can consult later as you’re writing your paper. One of the best strategies to make your life easier is to take notes as you go. You can take notes electronically on apps like Trello, on paper, on index cards, or even go old school and print out articles and highlight and write directly on them. No matter the method, you need to stay organized and be sure to keep track of which sources belong to which notes.
While simple highlighting might be your preferred method of annotating, it doesn’t tell you much about the content of what you're reading. Focus on summarizing and noting what will be helpful as you return to the article later in the writing process. For example, if you've recorded the article's major findings as you go, it'll be much easier to sort through all your sources later when you're trying to find a detail you remember reading but don't remember where. Or if one researcher's methods are similar to another's, make note of that so you can compare and contrast them later. If you know you'll be writing a certain kind of paper like a Research Grant Proposal or a Literature Review, then keep in mind the goals of that assignment as you go. (For example, see the section on note-taking in Chapter 11: How to Write a Literature Review).
One of the best ways to stay organized (especially if you'll be writing a Literature Review on this topic in the future), is to use a color-coding system where you assign one color to each source you've gathered. Then as you read that source, summarize the most important points in your own words. As you write your summaries, record them individually and use that source's assigned color--either by writing it on a colored card/paper/post-it note, writing with different colors of ink, or if you're using electronic software, by tagging all your notes with the assigned color. This will also help you later as you try to compare or synthesize sources. It's true that you can just copy and paste good quotes as you go, but trust me that summaries will be the most valuable notes you take. Don't forget to record which page each summary/quote came from so you can easily cite it later.
Should you accept or reject it?
Lest you think that you need to keep every source you find, watch this tutorial to see how to decide whether to accept or reject a source:
Write an Annotated Bibliography
One of the most useful forms of note-taking is writing an. Your teacher might ask you to do this in preparation for writing a longer paper. Even if it's not required, an Annotated Bibliography can be an easy way to keep track of the most important information from sources you find.
Recent research has shown that college students have major difficulty summarizing articles and books (as opposed to simply paraphrasing or quoting from them). (See Chapter 9 Talking About Sources.) Summary is an important skill, and writing an Annotated Bibliography will give you fantastic practice summarizing sources. The beauty of this is that you can also use those summaries later when you write your paper and want to refer to a source. It will also help you keep track of which sources addressed which topics. In fact, if you are assigned to write a Literature Review (which you probably will be in this class), you will use summary almost exclusively.
How to Write an Annotated Bibliography
Generally, these are the steps for writing an Annotated Bibliography:
- Write the source alphabetized and in its full reference format (follow the appropriate documentation style such as APA, Turabian, MLA, etc.). For more help on citation formats, see Chapter 9: Talking About Sources.
- Write a short paragraph (5-7 sentences)
a) summarizing the source,
b) evaluating its strengths and weaknesses, and
c) explaining how it's relevant to your specific topic.
For more specific information on how to best summarize sources, see Chapter 9: Talking About Sources
If you’d like more guidance on writing an annotated bibliography (including examples), see Purdue Online Writing Lab’s Guide to Annotated Bibliographies (scroll down):
 Several internal BYU studies as well as R.M. Howard, T. Serviss, & T. K. Rodrigue. (2010). Writing from sources, writing from sentences. Writing & Pedagogy, 2.2, 177-192.
8.5 Step Five: Corral 'Em (Analyze & Evaluate)
Corralling cattle is an exhilarating practice that requires both quick thinking and strong skills on the part of the herder. For a cowboy or cowgirl to do their job right, the cattle must be found, rounded up, inspected carefully, earmarked, and then corralled--much like your sources.
Why is this an issue?
Why do you even need to evaluate information? In the olden days (not that long ago, actually) there were texts that everyone agreed were authoritative. Maybe your parents own some old encyclopedias that they've displayed on a bookshelf. These were pretty much accepted as standard texts--you could look at them and trust the information that was printed.
Then came the internet. So much information began to be produced, it was hard to know what was credible. This has created civic debates about who believes what and what information (as people, a nation, and as communities), we should or shouldn't believe. One thing that is good about the way information is produced today is that it allows for people from all walks of life, and from all over the world to have a voice and to share it online. It's a way of democratizing access to, and the sharing of, information, points of view, and narratives that have been left out of mainstream discussions in the past.
But there is a downside: there are no longer gatekeepers of information--the editors, librarians, and experts who would fact-check information before it was produced into the encyclopedias of yester-year. This means that no matter what information you are consuming, you need to become information savvy yourself, and learn good fact-checking behaviors. This is especially true when engaging with information online, which we'll get to later in this section.
How do I evaluate scholarly material?
Scholarly material is, by its very nature, refereed before it is published. This is why academics hold it up as the "gold standard" of academic communication, and many journals engage in what is known as the. Like when you peer review in class, scholars engage in a similar practice, except for instead of having a classmate review their work, academics have their work reviewed by experts in their disciplines who recommend the work be published, revised, or not published at all. This rigorous exercise is put into place to ensure strong standards in academic communication. You can find peer-reviewed journal articles through most major databases, and if you are ever confused as to whether something is peer reviewed or not, make sure to look up the journal itself online, where you can usually tell in its description whether it is the result of peer review practices.
But just because something is peer reviewed doesn't mean it's the best material for your information need. Depending on your topic, you may need to find peer reviewed material that is published within a certain time frame. It is generally accepted that the more recent the publication, the better, but this really depends. If you are writing on a topic related to technology or media, for example, the academic conversation moves really rapidly in these areas, and you will need to find information published in the last 3-5 years for it to feel current. Other historical topics might allow for information to be older before it becomes dated. Seminal texts--or texts that are really important to the field--might have been published quite a while ago, but they may still be relevant to your conversation and are worthwhile to consult.
Relevance is also an important factor in determining whether scholarly information is good for your information need. This requires you to have a good sense of what you are writing about and why you are writing about it--so you can gather the best information out there. Some students fall into the "good enough" category where they just collect the first 20 sources that are good enough to sound somewhat educated on their topics. By so doing, they undercut the joy of the research process which is finding the best sources for their needs. But knowing what these best sources are is tricky: not every source, you see, needs to address every aspect of your exact topic to be highly relevant. Realize that all of your sources might not talk about your whole topic--but they might talk about sub-issues within a wider topic or even related issues.
Here is an example: say you are examining the way social media interactions affects teenage girls' behavior towards one another in person. You may find highly relevant sources that are on parts of this issue like an article on teenage social media usage, teenage social interactions, and how social media affects the brain. You might also find a few articles about sub points in your paper like cognitive development for females in teenage years. As you research, it is best to corral as many sources as you can at first, but as you cull the herd, be intentional about which ones you keep and which ones you cut loose. This will help you choose the best sources, and not just ones that are good enough.
Finally, consider other forms of scholarly communication like scholarly books. Many books published by academic presses (you can see what venue published the book in its first few pages) are peer reviewed and undergo a very rigorous editing process. To evaluate books, I'd suggest looking at who published it (a university press will carry the most clout in the scholarly communication world) and who wrote it, or contributed to it. Finally, the publication date will matter to those reviewing your bibliography.
Less scholarly sources are trade journals (like Psychology Today) which are meant for practitioners in a field. Though they're not peer reviewed, you may still find relevant research in these publications. Make sure to double check the information against the peer-reviewed research in your field before citing these sources, but do realize that they can provide a good starting point in timely, academic conversations.
How do I evaluate information I find online?
Here's where we get to the meat of it (sorry, cows!). The difficulty with information online is there is such a variety of it. We find a spectrum from highly reputable sources like the research done by the Pew Research Center--a non-governmental, not for profit, entity which studies issues relating to the United States--to enraged rantings on blog posts about the latest immigration scandal. The .org/.com tests no longer work to distinguish between "iffy" information types as just about anyone can get a .org nowadays. With opinion forums, open-access encyclopedias (read: Wikipedia) and cloak websites which hide behind layers of misinformation disguised as legitimate research to push political agendas, even the most educated of people feel wary when approaching information online. Many people would rather throw up their hands in disgust, and decide to make personal decisions about what to believe, rather than carefully learning how to evaluate information they find online for themselves, and become thoughtful researchers.
In the past we've relied on acronyms (like CRAAP: Currency, Relevance, Authority, Accuracy, and Purpose) to help give us rules of thumb about how to evaluate information; however, the information we find online often defies such definitions and quick tests of credibility. Instead, we need to learn two important lessons when interacting with information online:
1. Define your information need. This means that you need to think very carefully about what information you are seeking and go to the right places to find that kind of information. If you don't fully understand what information will fill your need, chances are you'll be looking willy-nilly for information and your information may end up coming from sketchy places. For instance, you wouldn't go to the same place to find information about a health concern you had as you would to find information about a TV you were thinking of buying. The same is the case for finding information you'll use in a paper. What is your paper about? Where would experts on this issue (either scholars, or others) be publishing? Defining your information need helps you decide where to search in the first place for good information--rather than sorting through bad information later.
For example, if you are writing about the psychology of self-driving vehicles and are trying to find a book on the subject, be warned: those may not have been written yet. Articles which have a quicker publication rate might be where to look for such information.
2. Act like a fact-checker. This means that you read information online very differently than you do traditional print material. Generally folks who read a book or a print newspaper read pretty traditionally: from top to bottom/left to right. When those same people approach a web page to decide if it is credible or not, they often do the same thing: read vertically--up and down. But professional fact checkers do something very different: they read laterally. This means that instead of focusing on the content of the website or publication they are verifying--they focus on the verification. They open tabs, they double check claims--they google the folks who are behind the information and where it came from. This horizontal reading means they jump off the source to check it, rather than staying on the source and trusting what it's telling them.
Fact checkers also tend to look further down on Google results lists than students would--they realize that the first few results in a Google search can be easily manipulated, so that's why they look at results further down and look at multiple source materials about a particular issue or source before believing its claims.
Fact Checking Behaviors
Find a website or article you don't know much about on a social media feed. Practice expert fact checking behaviors to decide if the information is credible or not. Record a reflection on your experience.
Many recent studies show that students really struggle to act like fact checkers. In fact, the Stanford History Education Group recently found that when students were evaluating a politically polarizing Tweet, about half of them did not click on the link provided in the tweet to corroborate the information found therein (p. 23). This is an issue because it shows that students get caught in superficial information evaluation acts: they might notice the hyperlink is there and think that the fact that it is there alone gives the source credibility without actually clicking on it and checking it out.
Self-Evaluation: Check Yourself
Perhaps the hardest thing we need to check is ourselves and our own biases. Confirmation bias plays into source evaluation behaviors for all of us. We all have confirmation bias, which means we are drawn to and tend to support/believe sources that reinforce our preexisting thoughts. Why is this an issue while evaluating information online? Well, it's an issue because we may struggle to seek out and fairly consider sources who support points of view that are not our own.
One way to deal with this is to proactively consume online material (especially news sources) that span the ideological spectrum. For example, check out this chart below that was created by a fact-checking organization. It shows the levels of bias of most news outlets today--and its results might surprise you.
As you can see, many news sources lean one way or the other--but a whole lot are neutral and might be considered "mainstream." Such mainstream sources employ vigorous fact-checking and fact-based news reporting and are generally trustworthy. No matter where your favorite news outlet falls on this chart, one of the best ways to avoid bias is to to double check information across several reports. When you get your information from only one source, you risk getting stuck in an echo chamber where what you hear only amplifies what you already believed. When you read more than one source, you are more likely to hear different points of view and get a more balanced view of a topic or situation. Similarly, it helps to realize that genre differences (like opinion vs. news feature) could change how you interact with the information you encounter. In sum, keep your fact-checking guard up when you read online--as Mad-Eye Moody says, "Constant vigilance, Harry!"
A second way you might deal with the issue of confirmation bias is by practicing intellectual humility. That is to say, being willing to reevaluate your beliefs, assumptions, and biases in the face of compelling, reliable, evidence. Practicing intellectual humility is not distrusting yourself or your gut, but instead, is realizing that you are a budding thinker and scholar, and you have a lot to learn. Intellectual humility is approaching research as an exercise in learning, exploring, and growing. It is an exercise in curiosity.
Revisiting one major principle: How did this information come to be?
End-of-Chapter Survey: How would you rate the overall quality of this chapter?
- Very Low Quality
- Low Quality
- Moderate Quality
- High Quality
- Very High Quality