At the conclusion of this chapter, you will be able to:
Searching for information is often nonlinear and iterative, requiring the evaluation of a range of information sources and the mental flexibility to pursue alternate avenues as new understanding develops (Association of College & Research Libraries, 2016 [https://edtechbooks.org/-cWK]).
You developed a viable research question, compiled a list of subject headings and keywords and spent a great deal of time searching the literature of your discipline or topic for sources. It’s now time to evaluate all of the information you found. Not only do you want to be sure of the source and the quality of the information, but you also want to determine whether each item is appropriate fit for your own review. This is also the point at which you make sure that you have searched out publications for all areas of your research question and go back into the literature for another search, if necessary.
In general, when we discuss evaluation of sources we are talking about looking at quality, accuracy, relevance, bias, reputation, currency, and credibility factors in a specific work, whether it’s a book, ebook, article, website, or blog posting. Before you include a source in your literature review, you should clearly understand what it is and why you are including it. According to Bennard et al., (2014 [https://edtechbooks.org/-cWK]), “Using inaccurate, irrelevant, or poorly researched sources can affect the quality of your own work.” (para. 4).
When evaluating a work for inclusion in, or exclusion from, your literature review, ask yourself a series of questions about each source.
For primary and secondary sources you located in your search, use the ASAP mnemonic to evaluate inclusion in your literature review:
Is it outdated? The answer to this question depends on your topic. If you are comparing historical classroom management techniques or building on educational theory, something from 1965 might be appropriate.
If relying on the book to provide empirical evidence, however, a general rule of thumb would be 5-10 years for education or technology.
Check reference or bibliography sources as well as those listed in footnotes or endnotes. Skim the list to see what kinds of sources the author used. When were the sources published? If the author is primarily citing works from 10 or 15 years ago, the book may not be what you need.
Does the author have the credentials to write on the topic? Does the author have an academic degree or research grant funding? What else has the author published on the topic?
Look for academic presses, including university presses. Books published under popular press imprints (such as Random House or Macmillan, in the U.S.) will not present scholarly research in the same way as Sage, Oxford, Harvard, or the University of Washington Press.
Other questions to ask about the book you may want to include in your literature review:
In your research, it is likely you will discover information on the web that you will want to include in your literature review. For example, if your review is related to the current policy issues in public education in the United States, a potentially relevant information source may be a document located on the National Center for Education Statistics (NCES) website titled The Condition of Education 2017 [https://edtechbooks.org/-sgX]. Likewise, for nursing, an article titled Discussing Vaccination with Concerned Patients: An Evidence-Based Resource for Healthcare Providers [https://edtechbooks.org/-zC] is available through the nursingcenter.com website. How do you evaluate these resources, and others like them?
Use the RADAR mnemonic (Mandalios, 2013 [https://edtechbooks.org/-cWK]) to evaluate internet sources:
How did you find the website and how is it relevant to your topic?
Look for the About page to find information about the purpose of the website . You may make a determination of its credibility based on what you find there. Does the page exhibit a particular point of view or bias? For example, a heart association or charter school may be promoting a particular perspective – how might that impact the objectivity of the information located on their site? Is there advertising or is there a product information attached to the content?
What is the web address or URL? This can give you a clue about the purpose of the website, which may be to debate, advocate, advertise or sell, campaign, or present information. Here are some common domains and their origins:
Mike Caulfield (2017 [https://edtechbooks.org/-cWK]), the author of Web Literacy for Student Fact-Checkers [https://edtechbooks.org/-vE], recommends a few simple strategies to evaluate a website (as well as social media):
It is likely that most of the resources you locate for your review will be from the scholarly literature of your discipline or in your topic area. As we have already seen, peer-reviewed articles are written by and for experts in a field. They generally describe formal research studies or experiments with the purpose of providing insight on a topic. You may have located these articles through Google, Google Scholar, a subscription or open access database, or citation searching. You now may want to know how to evaluate the usefulness for your research. As with the other resources, you are again looking for authority, accuracy, reliability, relevance, currency, and scope. Looking at each article as a separate and unique artifact, consider these elements in your evaluation:
ASK: Who is the author? Is this person considered an expert in their field?
Citation analysis is the study of the impact and assumed quality of an article, an author, or an institution, based on the number of times works and/or authors have been cited by others. Google Scholar is a good way to get at this information.Google Scholar
Check the facts. ASK:
ASK: Is there an obvious bias? That doesn’t mean that you can’t use the information, it just means you need to take the bias into account.
ASK: The hard questions:
To determine and evaluate in this category, ASK:
Although social media (for example, Twitter or Facebook) is generally treated as an object under study rather than a source of information on a topic, the prevalence of social media as communication and sharing platforms must be acknowledged. It’s important to be skeptical of these sources, especially for inclusion in a literature review. However, as with any other web resource,you can evaluate a social media posting for authenticity by asking the following questions:
Another way to think about evaluation of sources is to ask the 5W questions:
Locating sources for your literature review by using discovery layers, library catalogs, databases, search engines, and other search platforms may take a great deal of time and effort. Does everything you found and retrieved have value or worth to you as you write your own literature review? If the resource has not met the criteria above and you can’t justify its place in your literature review, it doesn’t deserve to be mentioned in your work. Include high-quality materials that are current, accurate, credible, and most importantly relevant to your research question, hypothesis, or topic.
Using a search engine like Google, do a quick search for a topic that interests you. Select a website from your list of results and evaluate it using the elements of website evaluation listed earlier in this chapter.
Select a subject specific book or ebook that you can access quickly and evaluate it based on the ASAP criteria.
You can practice evaluation using the attached articles. You don’t need to spend a lot of time with the article, but see if you can identify each of the elements of evaluation. Remember the elements of evaluation for articles are:
Your topic is music therapy in kindergarten classrooms in the United States. Which of the two resources would you include in your literature review?
Frederiksen, L., Phelps, S. F., & Kimmons, R. (2018). Evaluating Sources. In R. Kimmons, Rapid Academic Writing. EdTech Books. Retrieved from https://edtechbooks.org/rapidwriting/lit_rev_evaluating_sources