Finding & Evaluating Sources
In this chapter, you will learn how to
select and narrow a research topic
utilize library and electronic resources to locate relevant information
assess sources' reliability and usefulness
take notes and create an annotated bibliography
Note: this chapter contains a Library Research Homework Assignment as part of the practice for this section, so budget extra time so you can successfully take the time needed to choose and narrow your topic.
I live in the high desert mountains of the West, and my step-dad is a cowboy. Cowboys and Cowgirls are tough. Finding and evaluating sources is tough. So I'm going to teach you how to find sources western style, like a Cowboy or Cowgirl. So you can be tough, too.
8.1 Why Research?
If a cowboy like my stepdad wants to buy a horse, he first finds out who's selling, he checks out the horse's condition, learns about its heritage, tries it out, and sees how it behaves in different situations. In other words, he does his research. We, too, spend our lives doing research--whether we realize it or not. For example, between Yelp! [https://www.yelp.com/], Rotten Tomatoes [https://edtechbooks.org/-yRr], Consumer Reports [https://edtechbooks.org/-YtQ], and Amazon reviews [https://www.amazon.com/], it seems like we're always trying to find the best products and the best deals. When we have questions, we go to Wikipedia [https://www.wikipedia.org/] or ask Siri or Alexa for answers. Never in the history of the world have we had so many resources literally at our fingertips.
"Google is turning 18 years old this year. I cannot believe it was just 19 years ago that I never researched anything ever." --Comedian Kellen Erskine ["Composed" on VidAngel [https://www.vidangel.com/] website [https://www.vidangel.com/]]
The trick is, how do you know you're getting the best answers to your questions? How can you tell if you're using the best sources of information or if the information you find is accurate, reliable, and up-to-date? These days you even have to ask yourself if the information you find has been planted by Russian hackers trying to influence your political opinion. The problem with a glut of resources is that it's become increasingly hard to find reliable information, which is why doing good research is becoming a necessary life skill, not just a school skill. It's especially true in the Social Sciences that if you know how to conduct good research, you will be more influential no matter which field you go into.
And so I give you the Cowboy and Cowgirl's Guide to finding and evaluating sources with all the steps you'll need to find solid answers to your questions.
8.2 Step One: Choose Yer Horse (Select a Topic)
My kids call their grandpa "Cowboy Phil" because he wears a big custom-made hat, likes to go on cattle drives, and taught my kids how to ride a horse. He also taught them that it matters what kind of horse you choose. "You pick your horse based on the work at hand. You don't want to take a rodeo horse on a long drive. But above all, you've gotta really love your horse."
Similarly, you want to choose your research topic wisely; you don't want to invest a lot of time, energy, and resources into researching a topic for a long paper unless you really love it. This semester you'll be spending a significant amount of time on this topic: you'll likely create an extensive literature review looking at what others have said about your topic, write a proposal based on it, and/or give an oral presentation or poster presentation explaining this topic to your peers. So find something you love, because it won't be worth investing in unless you really love yer horse, er, topic. So what's the best way to find good sources?
One of the most overlooked and underrated parts of the research process is doing background research. Students often want to just jump right in to finding sources on a topic, but the problem is that no one can know how wide or narrow their topic is until they take a good look at the bigger picture. Getting a bird's-eye view of your topic will help you understand the context of how your topic fits into your field in general and even how it fits into the bigger world. It will also help you discover what questions are being asked, what the hot topics are in your field, and where the most promising research is going. All this will help you create a better research question and streamline your database searches down the road.
But maybe you're like many of my students and don't even have an idea of what to research or don't really know where to start. Background research is also perfect for exploring topics and thinking about how your interests can morph into a great research question. For example, here are some questions you can ask yourself as part of the brainstorming process:
- Why did you choose your major in the first place--what topics most excited you?
- Can you think of problems you've heard about in your field that you want to solve?
- Can you remember something you recently heard in a class that sparked your interest?
- Consider a controversy or question you want to know more about but you don't know the answer to.
- Think back on personal problems and questions you've had in your life and how they might relate to your discipline of study.
If you want to be more creative or free in your topic exploration, consider making a mind map or doing a free write.
An easy way to generate ideas is to do a good ol' freewrite--get out a piece of paper or open a blank document on your computer and time yourself for 5 minutes. Start writing about topic ideas and don't let yourself stop writing until the timer goes off. No erasing, no judgment, just keep writing. If you run out of things to say, then write, "I can't think of anything to say" until something pops into your head. The trick is to keep writing. You'll often find that ideas start to flow when you suspend judgment like this and just let whatever comes to mind flow out of your pen/keyboard. This is also a good technique to use if you get stuck in the drafting process. Studies have shown that freewriting unlocks the creativity of your brain and helps you think of connections you wouldn't normally make. In fact, some studies have shown it can actually improve your health (see studies--see Delys], so you might as well try it.
Mindmapping has also been shown to jumpstart your creative juices and help your brain make new connections. There's something about thinking visually rather than just linearly that allows you to explore relationships between topics in a fresh way. In fact, we'll revisit this idea later when we talk about generating ideas for paper organization.
Mindmapping can be done on a piece of paper (the old-school preferred way) or using mindmapping apps or software (new school). On paper, write a general idea for a topic in the middle and circle it. Then draw a line to branch off that idea and write something related to it, etc. Using different colors for different branches can help you visually organize your information. You might be skeptical of this method at first, but just try it. You might be surprised what kind of ideas pop into your head when you see them visually mapped rather than simply listed in a row.
As basic as it sounds, sometimes just talking out your ideas with another person can help you make connections and discover new possibilities. Find a buddy, go to one of your professor's office hours, or even ask someone on a date and talk about topics you're interested in. This can be like the freewrite where you just explore ideas with little judgment or you can ask the other person to give you honest feedback. Sometimes you'll be surprised what you can come up with simply by articulating your ideas to another person.
Wikipedia and Google
Another good place to start exploring general topics is Wikipedia, which, as you probably know, is a summary of what's generally known about a topic. A Wikipedia search can give you a quick sense of the history of a topic, who the majors players are, and what the sub-areas of research are. It can also lead you to related ideas and areas of research you hadn't though of before. Wikipedia's structure can be helpful to you as well because it breaks down larger discussion into bite-sized sub-issues and links out to other related articles. Consider how this organization can help you choose and narrow a topic.
Because Wikipedia's not a peer-reviewed publication, you can't use it as a main source for your paper, but most information on Wikipedia includes references to the original, primary sources, so you can use their references section to help you locate reliable, peer-reviewed sources that you can use in your paper.
Doing some general Google searches will also help you see what popular sources and commercial sites appear about your topic and if it's been in the news lately. Trade journals by professionals in your field are also a good resource. They might have a "magazine" feel to them but deal with specific issues within an area of study. Another option is to physically walk into the library and look through the Social Science Encyclopedias--a rich resource that's often overlooked. In the BYU libarary, these are located on the northeast side of the first floor in the reference section labeled "Social Sciences Help Desk." There are student employees there who can direct you to rich background sources that are great for exploring new topics.
"A good background researcher sketches out main arguments, sub-topics, and specific language that’s popping up within a wider discussion during the background research stage. This language will become important later when the researcher opens up a database" --Elise Silva, BYU Writing Programs Librarian
As you explore ideas about your topic, don't forget to take notes about key words that come up, what sub-topics emerge, whose names you see repeatedly, and which areas of research seem especially fruitful. You will thank yourself later in the research process when you use those notes and keywords in your database searching or if you decide you want to change your focus to a different angle or even a different topic.
Student Example: One of my students Justin decided he wanted to study the effects of exercise on diseases or aging. But then as he did some background research, he found that there were already tons and tons of articles on that topic. So he brainstormed other ideas, talked to his peers and professors about possibilities, and decided to change his focus to the influence of China on the economies of African countries.
Now it's your turn. Brainstorm some ideas and choose a general topic that interests you. (Note that for the purposes of this class, your topic should be related to your major or at least fall under the Social Sciences umbrella.) Do some background research: read about it on Wikipedia, search it on Google, see if it's been in the news. Pay attention to what sub-areas emerge and who the major players are. Maybe do a freewrite or create a mind map to generate more ideas. Then write two or three possibilities for a research topic you want to explore further in this class.
Research always starts with a question. It might not always be stated outright, but every time you look something up on your phone, it's because there's a gap in your knowledge that you want to fill in. Once you have a general sense of a topic you're interested in, that's when you're ready to formulate a more specific research question.
“You have to find a better reason than ‘it’s an assignment’ to devote weeks to your research and for your readers to spend time reading your article. You’ll find that better reason when you can ask a question whose answer solves a problem that you can convince readers to care about. That question and problem are what will make readers think your research is worth their time.” (from The Craft of Research by Booth, Colomb, and Williams p. 35)
In academia and even in industry, there's a pattern to research:
- Read what's been done on a topic
- Figure out where the gaps of knowledge are (secondary research)
- Fill one of those gaps with some kind of action that creates new knowledge--be it a survey, experiment, analysis, longitudinal study, product, etc. (primary research)
- Publish the results of that action or create a prototype
- Start the process over again
Your job in this class is to do steps 1 and 2--the secondary research part of this process: read what's been done on a topic and figure out where the gaps of knowledge are. Then later you might even be asked to propose some primary research that could fill one of the gaps you find (although there won't be time in this class to actually do the primary research, just to propose it).
So in order to find a narrow enough topic to study and to be ready to go to the next step in the research process, you need to come up with a strong research question. You might be tempted to skip this step, but trust me that if you can take the time now to devise a narrow, specific research question, the next step in the research process--database searching--will be much, much easier.
Although, we use the term "research question," it doesn't necessarily have to be in question format; your research question can be as simple as testing a hypothesis like, "I expect that smartphone usage at a young age increases a child's likelihood of being exposed to pornography." Even though this isn't in question format, it's an implied question--we assume that by making this statement, you will next try to figure out whether your hypothesis is correct. So really, a your research question becomes "Does smartphone usage at a young age increase a child's likelihood of being exposed to pornography?" With a hypothesis, you just also have a prediction that the answer will be "yes." You can also create a more open ended approach where you adopt a posture of openness and curiosity: "I wonder how dismissive attachment styles play out in parenting scenarios." The "I wonder" phrase is also an implied question because presumably the way you'll learn about this topic is to find answers to questions about attachment styles and parenting. In other words, you don't have to stick to an exact formula when you start researching a topic, but you do need to have a specific area of inquiry in mind.
If, on the other hand, you like to use formulas, one helpful way to create a research question is to formulate what my colleagure Dr. Grant Boswell calls a "WATCO question." WATCO stands for What are the consequences of? Or to be more precise, What are the consequences of something (A) on something else (B)? This format can be very helpful because it forces you to narrow your topic--and I find students almost always need to narrow their topics more than expand them. As you will see as you begin to search databases for articles and books, the narrower or more specific your research question, the easier it will be to find answers. Here are some examples of WATCO questions previous students have asked (they had to come up with three possible questions.
WATCO childhood obesity on:
- Self confidence?
- Adult health outcomes?
WATCO narcissism on:
- Adolescent populations?
- Adult populations?
WATCO educational funding on:
- Student Performance?
- Extracurricular Activities?
- School Resources?
WATCO meritocracy on:
- Social mobility?
- Higher education?
WATCO bilingualism on:
- Infants' differentiation abilities (Between the two languages)?
- Cognizant skills as compared with their monolingual peers?
- Infants' social skills?
WATCO a high sugar diet on:
- Metastasis and tumor growth?
- Cancer mortality?
Here are three different but related questions from one student:
- WATCO dyslexia on the encoding processes?
- WATCO early intervention with autism on ability to develop Theory of Mind?
- WATCO speech/language therapy on patients who have suffered left-lateralized strokes?
As you can see these students are starting to narrow a large topic into smaller areas of focus. The more specific the WATCO questions, the easier it will be for them to start the next step in the process.
At this point, feedback can be very helpful. Just like you wouldn't buy a horse without getting it checked out first, it's a good idea to get feedback on your research question from your peers or your teacher. It's likely that if they think something sounds interesting, it'll be interesting to other people as well.
Research Question & Feedback
Try to formulate three possible research questions based on the topic you've selected. If you want to use the WATCO format, you can choose at least one A term and three B terms or write three separate questions. Narrow your terms as much as possible. Post your own research question then comment on 3 of your peers' questions by clicking "Reply." Here's what to comment on for each question:
- Rate your peer's questions on a scale of 1-10 based on how much you would want to read a paper about that topic.
- Explain what interests you about that question.
- Add suggestions for improvement.
Once you've gotten feedback on your questions, choose the topic that resonates the best with others and that you believe will be the most fruitful and interesting to you. Now you're ready to move on to the next stage: finding and narrowing sources (or in cowboy terms, getting the lay of the land).
If you'd like more help doing background research or choosing a topic, watch Modules 1 and 2 in the iframe below that you might remember from your first-year Writing 150 class (you can click directly in this frame like a website):
8.3 Step Two: Get the Lay of the Land (Find Sources & Narrow Scope)
A cowboy or cowgirl doesn't just jump and go on a cattle drive with no preparation. They get the lay of the land first and plan where the best areas will be to take their cattle. Not only that, but they choose the best, most hardy cattle to take with them. Once you've done enough background research to understand your topic in general and have created a narrow research question, the next step is to get the lay of the land--to figure out the state of the field on this topic. And then you can start gathering your herd/sources that you can take with you on your research journey. You know you'll need a certain number of steer (sources), but where are the best places to look? What is the landscape like? Where will obstacles be? At this point, there's no way to know if you'll have to narrow or expand your scope until you start searching for sources (although, quick tip: most students almost always have to significantly narrow their research question).
So now starts the search for a strong herd and a good path to take them on. You're looking for strong cattle of a similar breed and a route where there's enough food and water without too much treacherous terrain. In terms of research, your research question starts you in a direction, and as you look for sources about a topic, you choose the articles and books with the most potential. Do you have time to look through thousands of sources? No way. You've got to be selective. At the same time, if you only look for the minimum number of sources, some might not turn out to be as hardy as they appeared and might not make the journey. If that's the case, and you'll probably come up short. So try to find the sweet spotwhere you have enough sources to examine without too many too handle. For example, I usually say the sweet spot for a literature review is to gather about 30-50 sources that you can then sort through. That way you're guaranteed to have 20 good quality sources in the end.
How to Narrow
Areas of inquiry always get narrower the more we learn. When people started exploring the West, they started with the big questions: how big is this country? Where are the mountains and rivers? Then as people moved into the area, they narrowed their scope as their knowledge grew: Where's the most fertile soil in this valley? Where are the best trails to take my cattle?
Researchers in every field also start by asking big questions at the beginning, and as people make discoveries throughout the years, they narrow their scope to create new knowledge. So the first researchers who discovered cancer asked the big questions like "What is cancer?" "How does it affect the human body?" But then as they began to answer those questions, they narrowed their scope to create new knowledge. For example, they branched their research into looking at different types of cancer (e.g., What is lung cancer? What is skin cancer?) or different populations (e.g., How does cancer affect people people with low socioeconomic status?) or different age groups (e.g., How does cancer affect children?). By adding new factors to their research questions, they narrowed their scope and continued to create new information. You'll follow a similar path when looking for sources about a topic: you'll start with a general topic, then as you search for sources, you'll probably find that you need to narrow your scope until you find that sweet spot--a manageable number of sources to take a closer look at.
Here's a real life example: my husband is a professor here at BYU in Mechanical Engineering and Neuroscience. He studies Traumatic Brain Injury, but if you just type "Traumatic Brain Injury" into the Web of Science Database, you get 51,378 results: clearly too many to handle! But if you add another factor to narrow the scope--say, "Traumatic Brain Injury" AND "Rehabilitation," then you get 5,499--still too many, but getting better. Now, if you add a third narrowing factor "AND Robotics," you get 46 results--the Sweet Spot! That's a manageable herd. Later you'll be finding your own Sweet Spot, so stay tuned.
So now it's your turn to get the lay of the land. How do you find your herd? How do you search for sources? If you're a BYU student, you're lucky! The Harold B. Lee Library gives you free access to specialized databases, subject guides, and hundreds of other resources, and it's all free of charge! If you're not at BYU, you should still be able to do most of the tutorials listed below without an account, so never fear! Also, check out your local library and Google Scholar for resources. Here's where to start:
In your class, you'll most likely be assigned to complete the online Library Tutorials and Library Research Assignment for Advanced Writing before attending a live library session with a subject librarian. Please start by watching all six short tutorial videos below about the research process. (You may need to login in to the Library webpage in order access them. Scroll down inside the frame until you see the videos.):
The next part of the Library Research Assignment will be done in parts as you move through this chapter.
As you saw in the tutorial video, you'll need to use keywords to search for sources that fit your topic. Don't forget to use Boolean Operators (AND, OR, NOT) to get the best results. I hope you noted some good keywords as you did your background research--especially pay attention to the keywords listed after the Abstract of an article because these are the best terms for database searching. Sometimes if you can't find very many sources on a particular subject, it's not because no one's published on it but because you're using the wrong keyword.
Library Research Assignment Part 1
Complete the parts of this assignment to get started navigating the library website and searching databases for your research topic.
Write your narrowed research question or topic here (it should be related to your major or minor).
List 3-5 possible keywords that will help you research your Research Question/Topic.
Library Research Guides
Speaking of resources, I want to give extra attention to probably the best resource for preliminary research: Library Research Guides. BYU's Subject Librarians are librarians who specialize in a specific field of study and have a master's degree or PhD in that subject area plus a master's degree of library and information science. They gather resources about specific majors or fields and find the best databases and resources for locating sources in those specific fields. In other words, they do the heavy lifting for you, so all you have to do is follow their recommendations. Here's how to do it: Look on the main Library website under "Research Guides":
Click on Research Guides, and you'll find lists of the many different majors and fields of study the librarians have already found for you (scroll down to see the lists):
Now find your major or your field of study and click on it (you can do this right inside the iframe above). For example, if you're a Geography major, click on the Geography link, or if you're a Latin American Studies major, click on Latin American and Iberian Studies. If your major isn't listed or if your research topic is outside of your major, then find the closest subject to your topic. Once you click, you'll see tabs for finding the best sources like articles, books, statistics, etc. in that field. The top item in a list is the best and/or most used database in your field. There's even a picture and contact info for your particular Subject Librarian and a place where you can email them or set up an appointment in case you have further questions. They've thought of everything!
For example, let's take my student Justin as an example. Justin was a Social Science Teaching major, but even though his major falls under the category of Education, the topic he chose to research was the influence of China on African economies. Therefore, the Research Guide he clicked on was Economics. If you go to the Research Guide for Economics (see above), you'll find an amazing page written by the Subject Librarian Brian Champion with tabs listing of resources for finding articles, books, statistics, and other valuable tips--resources you wouldn't have found simply typing your topic into Google or even Google Scholar. If you click on the Article Databases tab, you'll see that in the field of Economics, there are two main databases that are used the most: Business Source Premier for more business-related articles and EconLit for articles more under the general economics umbrella. These databases keep your search inside the relevant field and weed out the irrelevant junk you'd find just searching on Google or even Google Scholar. The librarians have already done the hunting for you, so take advantage!
Library Research Assignment Part 2
Go to the BYU Library home page at lib.byu.edu and click on “Research Guides.” From the “Subject Guide” list, choose which subject fits your topic. List the subject and subject librarian here (you might need to click on an extra tab like one that says Get Help).
List the top indexes or databases for journal articles that the Subject Guide recommends you use for this topic. (List 3-5)
A Word on Database Searches
You'll learn more about finding sources when your class attends your live Library Session with a Subject Librarian, but here are some of the most important takeaways:
Always go through your university library website to search in databases (or even to use Google Scholar) because once you're logged in, you'll should have automatic access to thousands of databases. If you're not logged in, you could be locked out of many journals and other publications or be required to pay for them. If you ever have trouble accessing databases or RefWorks, email your teacher or librarian for help.
When you come across a perfect source that really fits your topic and you wish you had more like it, read the article's Introduction (aka Literature Review) section and check out their References page to see which articles they're citing. If they spend a lot of time talking about a certain article, you'll know it's important to their topic. You can easily search for those sources on your university library page or on Google Scholar. This is a great way to quickly find relevant sources.
In a lot of databases, you can filter for “most cited articles” and “most cited authors” The ones that are cited the most are usually the most important sources to look at because they've had the most influence. Pay most attention to those.
Also note that if you have questions, you can click on the Get Help tab and subject librarian Brian Champion's picture and contact information comes up so you can request extra help.
Library Research Assignment Part 3
Do a thorough search of the Subject Guide. Explore several of the tabs. Look for resources you can use to get background information, bibliographies, statistics, and other research in your field. Note the resources you can use in future research projects. List three sources BESIDES indexes and databases that could help you in this project. Wise students use specialized background sources as well as articles and books during the research process.
What is a new resource that you learned about in this search?
Finally, lest you think that Subject Guides are only helpful for doing research in your field, I want you to take some time to explore a subject totally unrelated to your major or research topic so you can see what's out there. In your personal life, you'll probably need to research topics that you've don't know anything about, so take a few minutes and see how much you can learn.
Library Research Assignment Part 4
Spend ten minutes researching a subject guide in a field very different from your major. Turn up your curiosity, open up tabs, and browse. List two things you learned in your browsing.
Now you're ready for your class meeting with the Subject Librarian. Bring your narrowed research question, your list of keywords, and the databases you plan to explore. It's helpful to try some preliminary searches with your keywords before this meeting so you know generally how many sources are available on your topic and whether you need to narrow your scope (which is most likely). If you still have too many sources appearing, you can narrow your search by adding more factors (AND) or by leaving out some results (NOT). If you use OR, then that will expand your search, but that can be helpful if there is more than one accepted term for one of your keywords. Your goal in the end will be to find that sweet spot where you have just the right sized herd (about 30-50 sources). Bring any questions or problems you've had in your preliminary searches to your session with the Subject Librarian, and you should be ready to ride.
If you still feel lost or would like more detailed guidance about finding sources or evaluating what you find, you can review Modules 3 and 4 from your first-year writing class (Writing 150) here:
8.4 Step Three: Round 'Em Up (Gather & Annotate Sources)
Now that you've set your sights on the sweet spot between 30-50 sources, you want to start gathering like a good roper. Don't just randomly grab whatever you see--you need to be selective, look at many, but only gather the best ones. First, make sure you have a good rope--in other words, the right tools.
Grab Yer Rope
"A Cowboy's most important tool is his rope." --Michele Morris in The Cowboy Life (p. 71)
If you don't already have a RefWorks account (or other citation software), now's the time to get one. Citation Software is an easy way to save articles and other sources you find in one electronic location. The beauty of doing this is that once you've incorporated your sources into your paper, you can quickly create a References list from the software. Although you'll still need to double-check the entries it generates (always double-check!), this will still save you a lot of time. You can also use it to organize your sources and even save .pdf versions of them in your account.
There are many options for citation software, so if you have one you already know and use, you can stick with that. Some options are EndNote, Zotero, Mendeley, EasyBib, and Citation Machine. Add links As a BYU student, you have free access to RefWorks, and the library has made it very easy to save items straight to your RefWorks account, so I suggest you start there.
Here are instructions from the BYU Library on how to set up your own RefWorks Account.
Here are instructions for sending documents to RefWorks:
If you'd like more instruction on how to use RefWorks. here is a 20-minute playlist on YouTube: https://edtechbooks.org/-QJM
How to Read Online
Now comes the process of reading and evaluating your sources and deciding whether they fit your paper or not. Reading research online is very different from reading print sources; however, more of us are reading online than ever before. Before you whip out those articles, make sure you understand how to mentally prepare yourself for the online reading experience. If you are reading an e-book or an academic article in PDF format, make sure to create a helpful reading environment for yourself by mitigating distractions, and spending a pre-determined length reading/annotating before jumping to the next task. Screens distract us, but deep reading, the kind you need to do when researching, does not work when you are distracted. Turn off your phone, and concentrate.
How to Read a Journal Article
Finding sources is different than understanding them—especially because academic jargon can make these texts harder to access than popular information. Whether you are reading an academic article or an academic book, make sure you’re strategic in your approach to reading. Good researchers don't read articles cover-to-cover, so it's important to learn how to approach a source so you don't waste precious time. First, as you're searching, look only at titles and abstracts to figure out which sources are the most relevant to your topic. Sources don't need to address all aspects of your topic to be relevant, but they should address at least one aspect that relates. When you find a source you want to take a closer look at, mark it by adding it to a RefWorks list or recording it somehow. Then don't just read straight through each article you find--that will take hours and is extremely inefficient. Instead, follow this advice from an expert:
BYU Professor Dr. E. Jeffrey Hill from the School of Family Life spoke with Wes Burr (one of the most prolific authors in the family studies field) about how to efficiently read journal articles. He offered the following suggestions (used with permission):
- First, commit that you will spend no more that 60 minutes reading the article.
- Begin by taking 2 minutes to read the title carefully. The author(s) likely spent many hours deciding on a title. Try to get the most understanding out of it that you can.
- Next, spend 10 minutes reading the abstract. It should contain about 50% of what you need to know from the article. Have a sheet of paper ready and draw out all of the relationships mentioned in the abstract, so you get a conceptual idea of what is going on.
- Now read the first paragraph of the introduction. Much of what they couldn’t fit into the abstract will be in the first few paragraphs of the introduction.
- Next read the Conclusion/Summary. This should be the take home message of the article and will give you an idea of the main points.
- After reading the Conclusion/Summary, read each heading and first sentence of every section. You may read more if something seems particularly interesting. You may also look at figures and tables to see a succinct summary of results.
- Next read the entire discussion section. This is where the author tells what the important findings were and the meaning of those findings.
- Finally, if you have any time left, read the rest of the introduction to get a better idea of the background.
You almost never need to read the methods or results sections in their entirety unless you have a specific reason like if you're setting up an experiment using similar methodology (though it can be helpful to briefly look over any tables or figures). The idea is to be efficient and recognize which aspects of the article are relevant to your topic. Throughout the process highlight key ideas and take notes. And always record which source a quote or idea comes from because you'll need that information later.
Remember that many of the conclusions any given author makes will be questioned by another source, so it’s fine that you don’t agree with what the author says--you simply need to understand the main points and how their article relates to your research and the other sources you've read.
Just like there's an art to rounding up cattle, there's an art to sorting through sources. The goal of annotating and note-taking is for you to self-regulate--or make sure you’re understanding the piece while you’re reading it--but it’s also to create a helpful list that you can consult later as you’re writing your paper. One of the best strategies to make your life easier is to take notes as you go. You can take notes electronically on apps like Trello [https://trello.com/], on paper, on index cards, or even go old school and print out articles and highlight and write directly on them. No matter the method, you need to stay organized and be sure to keep track of which sources belong to which notes.
While simple highlighting might be your preferred method of annotating, it doesn’t tell you much about the content of what you're reading. Focus on noting what will be helpful as you return to the article later in the writing process. For example, if you've noted the article's major findings as you go, it will be much easier to sort through all your sources later when you're trying to find a detail you remember reading but don't remember where. Or if one researcher's methods are similar to another author's, make note of that so you can compare and contrast them later. If you know you'll be writing a certain kind of paper like a Research Grant Proposal or a Literature Review, then keep in mind the goals of that assignment as you go. (For example, see the section on note-taking in Chapter 9: Literature Reviews [https://edtechbooks.org/-tmG]).
One of the best ways to stay organized (especially if you'll be writing a Literature Review on this topic in the future), is to use a color-coding system where you assign one color to each source you've gathered. Then as you read that source, summarize the most important points in your own words. As you write your summaries, record them individually and use that source's assigned color--either by writing it on a colored card/paper, writing with different colors of ink, or if you're using electronic software, by tagging all your notes with the assigned color. This will also help you later as you try to compare or synthesize sources. It's true that you can just copy and paste good quotes as you go, but trust me that summaries will be the most valuable notes you take. Don't forget to record which page each summary/quote came from so you can easily cite it later.
Lest you think that you need to keep every source you find, watch this tutorial to see how to decide whether to accept or reject a source:
Write an Annotated Bibliography
One of the most useful forms of note-taking is writing an Annotated Bibliography. Your teacher might ask you to do this in preparation for writing a longer paper. Even if it's not required, an Annotated Bibliography can be an easy way to keep track of the most important information from sources you find.
You already know the definition of a bibliography (a list of the sources you use in a paper). To “annotate” simply means to summarize, comment on, or explain more about something (especially a text). So an annotated bibliography is a bibliography that also includes a summary and/or evaluation of each source. The annotated bibliography is so helpful that it has become a legitimate genre on its own: you can find formal annotated bibliographies published in many academic journals.
An annotated bibliography may seem like busy work, but it actually is a really important part of any research writing process. The annotated bibliography teaches you how to gather and cull source material. It also helps you practice the art of synthesis. Creating an annotated bibliography, if done well, helps you gather main source material, get it cited correctly, and helps you summarize shortly what the point of the research is. This helps you begin to put all of these sources into conversation with one another before you actually start to compose.
Recent research [https://edtechbooks.org/-xhm] has shown that college students have major difficulty summarizing articles and books (as opposed to simply paraphrasing or quoting from them). (See Chapter 8 Citing Sources [https://edtechbooks.org/-TUP].) Summary is an important skill, and writing an Annotated Bibliography will give you fantastic practice summarizing sources. The beauty of this is that you can also use those summaries later when you write your paper and want to refer to a source. It will also help you keep track of which sources addressed which topics. In fact, if you are assigned to write a Literature Review(which you probably will be in this class), you will use summary almost exclusively.
You might also analyze and evaluate the sources in your Annotated Bibliography. These evaluations of the strengths and weaknesses of a source as well as any connections you make to other sources will help you see the bigger picture (the lay of the land) as you write. This will also come in handy as you write your papers because you can incorporate your analysis into your paper as well.
Generally, these are the steps for writing an Annotated Bibliography:
- Write the source alphabetized and in its full reference format (follow the appropriate documentation style such as APA, Turabian, MLA, etc.). For more help on citation formats, see Chapter 8: Citing Sources [https://edtechbooks.org/-iny].
- Write a short paragraph (5-7 sentences)
a) summarizing the source,
b) evaluating its strengths and weaknesses, and
c) explaining how it's relevant to your specific topic.
For more specific information on how to best summarize sources, see Chapter 8: Citing Sources [https://edtechbooks.org/-cVh]
If you’d like more guidance on writing an annotated bibliography (including examples), see Purdue Online Writing Lab’s Guide to Annotated Bibliographies (scroll down):
 [https://edtechbooks.org/-odPD] Several internal BYU studies as well as R.M. Howard, T. Serviss, & T. K. Rodrigue. (2010). Writing from sources, writing from sentences. Writing & Pedagogy, 2.2, 177-192.
8.5 Step Four: Corral 'Em (Evaluate & Analyze Sources)
Corralling cattle is an exhilarating practice that requires both quick thinking and strong skills on the part of the herder. For a cowgirl to do her job right, the cattle must be found, rounded up, inspected carefully, and then corralled--much like your sources!
Why is this an issue?
Why do you even need to evaluate information? In the olden days (not that long ago, actually) there were texts that everyone agreed were authoritative. Maybe your parents own some old encyclopedias that they've displayed on a bookshelf. These were pretty much accepted as standard texts--you could look at them and trust the information that was printed.
Then came the internet. So much information began to be produced, it was hard to know what was credible. This has created civic debates about who believes what and what information (as people, a nation, and as communities), we should or shouldn't believe. One thing that is good about the way information is produced today is that it allows for people from all walks of life, and from all over the world to have a voice and to share it online. It's a way of democratizing access to, and the sharing of, information, points of view, and narratives that have been left out of mainstream discussions in the past. But there is a downside: there are no longer gatekeepers of information--the editors, librarians, and experts who would fact-check information before it was produced into the encyclopedias of yester-year. This means that no matter what information you are consuming, you need to become information savvy yourself, and learn good fact-checking behaviors. This is especially true when engaging with information online, which we'll get to later in this section.
How do I evaluate scholarly material?
Scholarly material is, by its very nature, refereed before it is published. This is why academics hold it up as the "gold standard" of academic communication, and many journals engage in what is known as the peer review process. Like when you peer review in class, scholars engage in a similar practice, except for instead of having a classmate review their work, academics have their work reviewed by experts in their disciplines who recommend the work be published, revised, or not published at all. This rigorous exercise is put into place to ensure strong standards in academic communication. You can find peer-reviewed journal articles through most major databases, and if you are ever confused as to whether something is peer reviewed or not, make sure to look up the journal itself online, where you can usually tell in its description whether it is the result of peer review practices.
But just because something is peer reviewed doesn't mean it's the best material for your information need. Depending on your topic, you may need to find peer reviewed material that is published within a certain time frame. It is generally accepted that the more recent the publication, the better, but this really depends. If you are writing on a topic related to technology or media, for example, the academic conversation moves really rapidly in these areas, and you will need to find information published in the last 3-5 years for it to feel current. Other historical topics might allow for information to be older before it becomes dated. Seminal texts--or texts that are really important to the field--might have been published quite a while ago, but they may still be relevant to your conversation and are worthwhile to consult.
Relevance is also an important factor in determining whether scholarly information is good for your information need. This requires you to have a good sense of what you are writing about and why you are writing about it--so you can gather the best information out there. Some students fall into the "good enough" category where they just collect the first 20 sources that are good enough to sound somewhat educated on their topics. By so doing, they undercut the joy of the research process which is finding the best sources for their needs. But knowing what these best sources are is tricky: not every source, you see, needs to address every aspect of your exact topic to be highly relevant. Realize that all of your sources might not talk about your whole topic--but they might talk about sub-issues within a wider topic or even related issues. Here is an example: say you are examining the way social media interactions affects teenage girls' behavior towards one another in person. You may find highly relevant sources that are on parts of this issue like an article on teenage social media usage, teenage social interactions, and how social media affects the brain. You might also find a few articles about sub points in your paper like cognitive development for females in teenage years. As you research, it is best to corral as many sources as you can at first, but as you cull the herd, be intentional about which ones you keep and which ones you cut loose. This will help you choose the best sources, and not just ones that are good enough.
Finally, consider other forms of scholarly communication like scholarly books. Many books published by academic presses (you can see what venue published the book in its first few pages) are peer reviewed and undergo a very rigorous editing process. To evaluate books, I'd suggest looking at who published it (a university press will carry the most clout in the scholarly communication world) and who wrote it, or contributed to it. Finally, the publication date will matter to those reviewing your bibliography.
Less scholarly sources are trade journals (like Psychology Today [https://edtechbooks.org/-znw]) which are meant for practitioners in a field. Though they're not peer reviewed, you may still find relevant research in these publications. Make sure to double check the information against the peer-reviewed research in your field before citing these sources, but do realize that they can provide a good starting point in timely, academic conversations.
How do I evaluate information I find online?
Here's where we get to the meat of it (sorry cows!). The difficulty with information online is there is such a variety of it. We find a spectrum from highly reputable sources like the research done by the Pew Research Center [https://www.pewresearch.org/]--a non-governmental, not for profit, entity which studies issues relating to the United States--to enraged rantings on blog posts about the latest immigration scandal. The .org/.com tests no longer work to distinguish between "iffy" information types as just about anyone can get a .org nowadays. With opinion forums, open-access encyclopedias (read: Wikipedia) and cloak websites which hide behind layers of misinformation disguised as legitimate research to push political agendas, even the most educated of people feel wary when approaching information online. Many people would rather throw up their hands in disgust, and decide to make personal decisions about what to believe, rather than carefully learning how to evaluate information they find online for themselves, and become thoughtful researchers.
In the past we've relied on acronyms (like CRAAP: Currency, Relevance, Authority, Accuracy, and Purpose) to help give us rules of thumb about how to evaluate information; however, the information we find online often defies such definitions and quick tests of credibility. Instead, we need to learn two important lessons when interacting with information online:
- Define your information need. This means that you need to think very carefully about what information you are seeking and go to the right places to find that kind of information. If you don't fully understand what information will fill your need, chances are you'll be looking willy-nilly for information and your information may end up coming from sketchy places. For instance, you wouldn't go to the same place to find information about a health concern you had as you would to find information about a TV you were thinking of buying. The same is the case for finding information you'll use in a paper. What is your paper about? Where would experts on this issue (either scholars, or others) be publishing? Defining your information need helps you decide where to search in the first place for good information--rather than sorting through bad information later. For example, if you are writing about the psychology of self-driving vehicles and are trying to find a book on the subject, be warned: those may not have been written yet. Articles which have a quicker publication rate might be where to look for such information.
- Act like a fact-checker. This means that you read information online very differently than you do traditional print material. Generally folks who read a book or a print newspaper read pretty traditionally: from top to bottom/left to right. When those same people approach a web page to decide if it is credible or not, they often do the same thing: read vertically--up and down. But professional fact checkers do something very different: they read laterally. This means that instead of focusing on the content of the website or publication they are verifying--they focus on the verification. They open tabs, they double check claims--they google the folks who are behind the information and where it came from. This horizontal reading means they jump off the source to check it, rather than staying on the source and trusting what it's telling them. Fact checkers also tend to look further down on Google results lists than students would--they realize that the first few results in a Google search can be easily manipulated, so that's why they look at results further down and look at multiple source materials about a particular issue or source before believing its claims.
Fact checking behaviors
Find a website or article you don't know much about on a social media feed. Practice expert fact checking behaviors to decide if the information is credible or not. Record a reflection on your experience.
Many recent studies show that students really struggle to act like fact checkers. In fact, the Stanford History Education Group [https://edtechbooks.org/-bBN] recently found that when students were evaluating a politically polarizing Tweet, about half of them did not click on the link provided in the tweet to corroborate the information found therein (p. 23). This is an issue because it shows that students get caught in superficial information evaluation acts: they might notice the hyperlink is there and think that the fact that it is there alone gives the source credibility without actually clicking on it and checking it out.
Self-Evaluation: Check Yourself
Perhaps the hardest thing we need to check is ourselves and our own biases. Confirmation bias plays into source evaluation behaviors for all of us. We all have confirmation bias, which means we are drawn to and tend to support/believe sources that reinforce our preexisting thoughts. Why is this an issue while evaluating information online? Well, it's an issue because we may struggle to seek out and fairly consider sources who support points of view that are not our own.
One way to deal with this is to proactively consume online material (especially news sources) that span the ideological spectrum. For example, check out this chart below that was created by a fact-checking organization. It shows the levels of bias of most news outlets today--and its results might surprise you.
As you can see, many news sources lean one way or the other--but a whole lot are neutral and might be considered "mainstream." Such mainstream sources employ vigorous fact-checking and fact-based news reporting and are generally trustworthy. No matter wear your favorite news outlet falls on this chart, one of the best ways to avoid bias is to to double check information across several reports. When you get your information from only one source, you risk getting stuck in an echo chamber where what you hear only amplifies what you already believed. When you read more than one source, you are more likely to hear different points of view and get a more balanced view of a topic or situation. Similarly, it helps to realize that genre differences (like opinion vs. news feature) could change how you interact with the information you encounter. In sum, keep your fact-checking guard up when you read online--as Mad-Eye Moody says, "Constant vigilance, Harry!"
A second way you might deal with the issue of confirmation bias is by practicing intellectual humility. That is to say, being willing to reevaluate your beliefs, assumptions, and biases in the face of compelling, reliable, evidence. Practicing intellectual humility is not distrusting yourself or your gut, but instead, is realizing that you are a budding thinker and scholar, and you have a lot to learn. Intellectual humility is approaching research as an exercise in learning, exploring, and growing. It is an exercise in curiosity.
Revisiting one major principle: How did this information come to be?
At this point you might be a little frustrated and thinking, "Well what can I trust? I guess I can't trust anything or anyone!" Distrust of the media is a bit of a scary phenomenon since it seems to give license for individuals to believe whatever they want rather than becoming informed on salient issues. That is why I advocate an informed approach to information consumption rather than an approach which is self-defeating. If there is one idea I want you to walk away from this section with it is this: in order to really evaluate information, you need to figure out how that information was created; in other words, figure out how it came to be.
Such a question: "How was this information created?" hits on many points: who is behind this information? Who wrote it? Who published it? What kind of a publication process did it go through? Was it vetted? Was it reviewed and edited? By answering these questions you will be able to construct a picture of the process this information went through in order to come to you. You might even be able to surmise if someone had an agenda or motivation that seemed hidden at first. As with everything, you will need to make some value judgments along the way. For example, some kinds of information will help you establish your credibility in a paper, whereas other kinds of source material will lend humor or pathos to an argument. As you engage with such sources consider your audience and what kind of information they would expect you to use for your writing. It all comes back to context when you make the decision to trust a source or to use it purposefully in your writing.
Now that you've evaluated, analyzed, and started to synthesize your sources, you're ready to start figuring out how to cite your sources once you've found them. In other words, now that you've corralled the herd, it's time to do some branding.
Charles, C. C. & Silva, E. (2019). Finding & Evaluating Sources: A Cowboy and Cowgirl's Guide. In C. C. Charles, Writing in the Social Sciences. EdTech Books. Retrieved from https://edtechbooks.org/writing/resources
End-of-Chapter Survey: How would you rate the overall quality of this chapter?
- Very Low Quality
- Low Quality
- Moderate Quality
- High Quality
- Very High Quality