| Research Methods Week 3: Reading and Reviewing |
Course OverviewSpecial Needs Assistance
| Week Number and/or Date |
Lecture/Seminar/ Field Trip/Other |
Topic/Theme/Title | Assessment |
| Week 1 | Lecture 1 | The Research Process Pt1 | |
| Week 2 | Lecture 2 | The Research Process Pt1 | |
| Week 3 | Lecture 3 | Reading and Reviewing | |
| Week 4 | Lecture 4 | Writing and Presenting Information |
|
| Week 5 | Lecture 5 | Experimental Methods: Overview and Research Design |
|
| Week 6 | Lecture 6 | Experimental Methods: Sample Design, Measurement and Scaling |
|
| Week 7 | Midterm |
| Week Number and/or Date |
Lecture/Seminar/ Field Trip/Other |
Topic/Theme/Title | Pre-reading |
| Week 1 | Lecture 1 | The Research Process Pt1 | |
| Week 2 | Lecture 2 | The Research Process Pt1 | |
| Week 3 | Lecture 3 | Reading, Writing and Reviewing | |
| Week 4 | Lecture 4 | Presenting | |
| Week 5 | Lecture 5 | Experimental Methods: Overview and Research Design |
|
| Week 6 | Lecture 6 | Experimental Methods: Sample Design, Measurement and Scaling |
|
| Week 7 | Midterm |
Course Overview
| Week Number and/or Date |
Lecture/Seminar/ Field Trip/Other |
Topic/Theme/Title | Assessment |
| Week 7 | Midterm | 13 Nov Assessment 2 Deadline |
|
| Week 8 | Lecture 7 | Experimental Methods: Data Collection |
|
| Week 9 | Lecture 8 | Experimental Methods: Data Analysis |
|
| Week 10 | Lecture 9 | Review of legal, ethical, social and professional issues |
|
| Week 11 | Lecture 10 | Assignment presentations | |
| Week 12 | Lecture 11 | Commercialisation | 11 Dec Assessment 3 Deadline |
| Week 13 | Lecture 12 | Revision/ Assignment support |
| Overview |
| • What is Critical Reading? • -Critical Reading and Critical Thinking. • -Critical Thinking • How to Read Critically • Process of Critical Reading …. |
| Overview |
| • What to read? – Front-line literature – Discerning front-line literature – The Internet • Reading Critically – Finding and evaluating the argument – Five Critical Synopsis Questions <cont> |
| Critical Reading |
| • What is Critical Reading? • Critical Reading and Critical Thinking • Critical Thinking • How to Read critically • The Review Process |
| What is Critical Reading? |
Facts vs. Interpretation
• Non critical reading
• Text = facts
• What is written
• Memorize facts
• Critical reading
• Text = portrayal of facts
• What’s written and how
it’s written
• Appreciate how a
perspective can lead to
an understanding.
Reaching for an Interpretation
• Step 1: What the text says.
– i.e. non critical reading.
• Step 2: What the text does.
– Examples? Argument? Sympathy? Contrast?
• Step 3: What it means.
– Based on earlier analysis.
Discussion
• Step 1: What the text says.
– > Restatement
• Step 2: What the text does.
– > Discussion – discusses aspects of the discussion.
• Step 3: What it means.
– > Analysis – analysis the text and asserts a
meaning for the text as a whole.
Goals
•to recognize an author’s purpose
–involves inferring a basis for choices of content and
language
•to understand tone and persuasive elements
–involves classifying the nature of language choices
•to recognize bias
–involves classifying the nature of patterns of choice
of content and language
Recognize and analyse
Critical reading is not simply close and careful
reading.
To read critically, one must actively recognize
and analyze evidence upon the page.
Analysis and Inference
• Critical reading requires analysis and inference
– Analysis
• what we look for
• involves recognizing those aspects of a discussion that
control the meaning.
– Inference
• How to think about what you find
• interpretation of data from within the text
Example
‘The Senator admitted owning the gun that
killed his wife.’
Example
‘The Senator admitted owning the gun that
killed his wife.’
– There is a Senator.
– He owns a gun.
– He is married.
– His wife is dead.
– That gun caused her death.
– The Senator admitted owning that gun.
Inferences
• The gun (or at least a bullet) has probably
been recovered and identified as the murder
weapon.
– Or the admission of the senator would not make
sense.
Unwarranted inferences
• We do not necessarily know if
– The Senator’s admission is true
– the Senator is responsible for his wife’s death
– The wife died of gunshot wounds (she might have
been hit over the head with the gun)
– It was murder (it may have been suicide)
Exercise
A man and his son are driving in a car. The car
crashes into a tree, killing the father and
seriously injuring his son. At the hospital, the
boy needs to have surgery. Upon looking at the
boy, the doctor says, “ I cannot operate on him.
He is my son.”
How?
“I cannot operate on him. He is my son.”
Implications For Reading
• All reading is
– Active
– Reflective
– A problem-solving process
• Readers can use the notion of inference
equations to test whether or not the
ingredients for a given inferences are indeed
present. …
Implications For Writing
• Writers must assure that the ingredients of
the equation are
– present
– Clear
– and that the desired relationships are signaled in a
clear and effective way.
• As writers, we must be aware that our readers
will interpret our thoughts.
| Critical Reading and Critical Thinking |
| • Not critical-reading • Critical reading • Critical thinking |
Non-critical (or pre-critical) reading
• Recognizing what a text says about the topic.
Information.
• Make sense of the text
– Information
– Ideas
– Opinions
• Linear activity
Critical reading
• Analytic non-linear activity
• The reader rereads a text to identify
patterns of elements –
– information,
– values,
– assumptions,
– and language usage
— throughout the discussion. These elements
are tied together in an interpretation, an
assertion of an underlying meaning of the text
as a whole.
Critical thinking
• Involves taking into account
– outside knowledge,
– biases,
– and values
to evaluate the presentation and decide what
ultimately to accept as true.
| Critical Reading and Critical Thinking |
Critical Reading v. Critical Thinking
• Critical reading is a technique
for discovering information and ideas within a
text.
• Critical thinking is a technique
for evaluating information and ideas, for
deciding what to accept and believe.
Example
• “Parents are buying expensive cars for their
kids to destroy them.”
Example
• “Parents are buying expensive cars for their
kids to destroy them.”
– Critical reading
– Critical thinking
Example
• “Parents are buying expensive cars for their
kids to destroy them.”
– Critical reading
• Analysis – What does it mean?
– Critical thinking
• Evaluation – Do we believe it
Example
• “Parents are buying expensive cars
for their kids to destroy them.”
– Critical reading
• Analysis – What does it mean?
• Parents are buying expensive cars for
their kids to destroy the expensive cars.
– Critical thinking
• Evaluation – Do we believe it
• No.
Example – Alternative Interpretation
• “Parents are buying expensive cars
for their kids to destroy them.”
– Critical reading
• Analysis – What does it mean?
• Parents are buying the kids expensive
cars and don’t care that they do and if
they destroy them.
– Critical thinking
• Evaluation – Do we believe it
• Yes, the parents are rich.
The Two Together in Harmony
Critical Reading
Critical Thinking
What does it mean
Is it reasonable
The Usefulness of the Distinction
• Critical reading
– Is objective.
• We can read each text on its own merits, not imposing
our prior knowledge or views on it.
• Important to be able to learn something new.
Reading Critically: How Well Does The
Text Do What It Does
• Writer has a job to do:
–a specific topic must be addressed
–terms must be clearly defined
–evidence must be presented
–common knowledge must be accounted for
–exceptions must be explained
–causes must be shown to precede effects and to be
capable of the effect
–conclusions must be shown to follow logically from
earlier arguments and evidence
Reading Critically: How Well Does The
Text Do What It Does
• Tasks must be realized:
– Completely
– Comprehensively
– Consistently
Thinking Critically:
Evaluating The Evidence
• Critical thinking
– Useful for real world tasks
• Building roads, writing code, writing legislation etc.
• We want to accept as fact only that which is actually
true using our best judgment.
• Truth is based on prior understanding and evidence.
| Critical Thinking |
| Critical Thinking |
| • Rationality • Self-awareness • Honesty • Open-mindedness • Discipline • Judgment |
Rationality
• Rely on reason rather than
emotion,
• Require evidence, ignore no known
evidence, and follow evidence
where it leads, and
• Be concerned more with finding the
best explanation rather than being
right, analyzing apparent confusion
and asking questions.
Self-awareness
• Know yourself.
• Weigh the influences of motives and bias.
• Recognize our own
– Assumptions
– Prejudices
– Biases
– point of view
Honesty
• Recognize
– emotional impulses
– selfish motives
– nefarious purposes
– or other modes of self-deception
Open-mindedness
• Evaluate all reasonable inferences.
• Consider a variety of possible viewpoints or
perspectives.
• Remain open to alternative interpretations.
• Accept a new explanation, model, or paradigm
because it explains the evidence better, is
simpler, or has fewer inconsistencies or covers
more data.
Open-mindedness
• Accept new priorities in response to a
reevaluation of the evidence or reassessment
of our real interests
• Do not reject unpopular views out of hand.
Discipline
• Precise, meticulous, comprehensive, and
exhaustive.
• Resist manipulation and irrational appeals.
• Avoid snap judgments.
Judgment
• Recognize the relevance and/or merit of
alternative assumptions and perspectives
• Recognize the extent and weight of evidence
Critical thinkers
• Are skeptical. They approach texts with the same
skepticism and suspicion as they approach
spoken remarks.
• Are active, not passive. They ask questions and
analyze. They consciously apply tactics and
strategies to uncover meaning or assure their
understanding.
• Do not take an egotistical view of the world. They
are open to new ideas and perspectives. They
are willing to challenge their beliefs and
investigate competing evidence.
Exercise
What do you think the characteristics of a noncritical thinker might be?
Non-critical thinkers
•They see things in black and white, as either-or,
rather than recognizing a variety of possible
understanding.
•They see questions as yes or no with no subtleties.
•They fail to see linkages and complexities.
•They fail to recognize related elements.
•They take an egotistical view of the world
–their facts are the only relevant ones.
–their perspective is the only sensible one.
–their goal as the only valid one.
| How to Read Critically |
Exercise
WHY DO IT THE HARD WAY when you can be rich NOW!!!
“It took me five years to make my first million. I made my second million in six weeks.
Now I just can’t stop making money. I own four luxury villas on three continents, five
top-of-the-range sports cars and my own helicopter. Most important of all, the
financial security of my family is ensured.
Now I want to share my good fortune with you. By following my simple instructions
you too can be a millionaire within just a few months. There is no risk and it just can’t
fail. I have already helped hundreds of people attain their dream of a new life. They
are so grateful to me – no longer do they worry about domestic bills, healthcare or
their children’s education. Their future is certain. And yours can be too.
Just call me on the number below, and I will send you my introductory pack free
of charge. It will explain to you how my failsafe method can bring you guaranteed
wealth and happiness. Call now, and let your life change forever for the better.”
18662178474
Exercise
What’s wrong with this?
Should we change our career?
Critical reading in Academia
In general academics aren’t charlatans, so why
do we need critical reading?
Reasonable skepticism
• The authors mean to be honest, but may have been
misled by the evidence into saying something that I
consider untrue.
• The authors mean to be logical, but may have
developed a line of reasoning that contains a flaw.
• The authors mean to be impartial, but may have
incorporated into the account some assumptions that I
don’t share.
• The authors mean to tell me something new, but may
not have taken into account other information that I
possess.
Exercise
‘In the reading test, the five children who were
taught to read using phonics performed better
overall than the five children taught using the
whole word method. This shows that the
phonics method is a better choice for schools.’
Exercise
1. What questions would you ask?
2. Where would you expect to find the
answers?
Exercise
• Is a study of just ten children sufficient to draw such a
strong conclusion?
• What does ‘performed better overall’ signify? Did some
children taught using the whole word method perform
better than some children taught using phonics? If so, what
does this mean for the results?
• Were the differences between the two groups sufficiently
great for us to be satisfied that they would occur in a re-run
of the experiment with different subjects?
• How were the two teaching programmes administered, and
might there have been ‘leakage’ of whole word teaching
into the phonics teaching and vice versa?
Exercise
• What was the reading test actually testing, and might it
have been unintentionally biased to favor the children
taught using phonics?
• What care was taken to check how parental involvement at
home might have influenced what and how the children
learned?
• Were the two sets of five children matched for intelligence,
age, gender or other factors?
• Is it reasonable to infer that what works well in a small
experimental study will work well in school environments?
• How does Browning envisage phonics being used in
schools? Would there still be a place for the whole word
method?
Review
Browning (2005) found that children taught to read using
phonics did better in a reading test than children taught
using the whole word method. However, the study was
small, the test rather limited, and the subjects were not
tightly matched either for age or gender. An examination
of Browning’s test scores reveals that, although the mean
score of the phonics group was higher, two of the highest
scorers in the test were whole word learners. Since this
indicates that the whole word method is effective for
some learners at least, Browning is perhaps too quick to
propose that ‘the phonics method is a better choice for
schools’ (p. 89).
Critical Reading and Writing
• Critical reading normally supports writing
• Critical reading
– assessing the extent to which authors have provided
adequate justification for the claims they make.
– Depends on
• what the authors have communicated
• other relevant knowledge, experience and inference that
you are able to bring into the frame.
• Self-critical writing
• convincing your readers to accept your claims.
• Effective communication of adequate reasons and evidence
for these claims.
Self Critical Writing
• Demand of yourself evidence to back up your
claims;
• Be alert to the possibility of making an illogical
jump in your reasoning;
• Become sensitive to your own assumptions and
how they might affect your claims;
• Realize the importance of checking the literature
thoroughly to ensure that your understanding is
sufficiently deep.
To be continued….
Being Critical as a Requirement of
Academic Study
Critical thinking and creativity: managing
creative processes in self and others; Organizing
thoughts, analysis, synthesis, critical appraisal.
This includes the capability to identify
assumptions, evaluate statements in terms of
evidence, detect false logic or reasoning,
identify implicit values, define terms adequately
and generalize appropriately.
Being Critical as a Requirement of
Academic Study
• Be able to
– Evaluating what you read
– Relating what you read to other information
• About assessing the quality of the case made
for evaluating and interpreting the facts in
some way rather than the ‘facts’ themselves.
• Assessing the evidence and the validity of the
interpretation and generalisations.
Objects of Scrutiny
• The evidence provided in the account;
• Whether the reasoning of the author’s argument
follows logically to the conclusion that has been
drawn;
• Explicit or implicit indications of the author’s
values and assumptions;
• The match between the author’s claims and
those of other authors;
• The match between the author’s claims or
predictions and the reader’s own research
evidence or knowledge.
| Process of Critical Reading |
| Process of Critical Reading |
| • Task Based Reading • Deciding What to Read • Reading Critically |
Task-driven approach
• Two components
– Tasks
• What you need to do
– Questions
• What you need to ask
?
Tasks
• You have been told to read it in preparation for a class.
• You are doing background reading on your subject, just
to get your bearings.
• It reports a particular approach or technique that you
want to see in action.
• It addresses a particular question that you want to
know the answer to.
• You are looking for evidence to counter-balance
something else that you have read.
• You have a particular story to tell, and you need some
supporting evidence for it.
Questions
• Too vague
– ‘What did you do, and what did you find out?’ For
example
• Better
– Is this author’s method of investigation the best
one for me to emulate in my own work?
– How does this author’s position compare with
that of another author whose work I’ve read?
– Would this author challenge the claims that I am
making in my own work?
Questions
• Helps to
– Guide you to what you need to read.
– Determine how thoroughly you read it.
– Stay focused and maintain discipline
• May mean reading the same paper twice
• Saves time overall
– There is simply too much literature to read
normally and be led by the text after the initial
background reading stage.
| What to read? |
| What to read |
| • Deciding what to read • Types of literature • The internet |
Why read?
• Review a paper
• Write a paper
• Applying for funding
• Research project
• MSc/PhD
• Developing a course
• Etc.
Why read?
• Review a paper
• Write a paper
• Research project
• MSc/PhD
• Develop a course
• Etc.
You are doing research
the reading process is more or less the same
Deciding what to read
• Two Stage strategy
– Draw up a list of documents that look important
– Create a shortlist of those that appear most useful
Long list
• Use any recommended reading list for your module or subject area,
including those from past years.
• Search the Internet for reading lists posted up for similar modules
at other universities, and identify texts that are repeatedly
recommended.
• Look up one or two important texts in the library catalogue. Then
do a search using their subject code to see what else has been
classified as covering the same topic.
• Go to the library shelves and see what is physically stored under the
same class mark as the key recommended texts.
• Note how many copies the library has of a particular text. If there
are plenty, it has evidently been a recommended text at some
point.
• As you begin to read, note texts that are often cited by others, and
whether positively or negatively (both may be useful).
Long list
• Make a list of the three or four journals most often carrying
papers that have been recommended or frequently cited,
then check the back and current issues of those journals for
similar papers.
• Use abstracts databases to search for papers via keywords
and author names that you associate with the topic.
• Look through the catalogues (on paper or on-line) of the
leading academic publishers to see what has come out
recently.
• Check what books have been reviewed in recent academic
journals.
• Improvise, find out for yourself which methods help you to
find the most relevant material. …
From long-list to short-list
Short list should include some or all of;
a) An overview of what the key issues in the field are and
why they are important.
b) An overview of what has been done and found out, and a
summary of where the field of enquiry currently stands.
c) Papers that pose the question that you need to answer
with your research.
d) Some specific examples of the sorts of methodology,
results and analysis reported by individual researchers.
e) Answers to one or more specific questions that you have
been required, or have chosen, to address.
etc..
From long-list to short-list
• Not just one paper..
– A key component of your research will be to
combine and contrast information from different
sources.
Example reading lists?
• Look at any high quality* academic research
paper and check the references.
– The list of references is the short-list
– The position of the citation tells us what the
reference refers to.
| Types of literature |
| • Support literature – Textbooks – Readers, handbooks and encyclopedias • Front line literature – Journal and conference papers, theses, tech reports etc. – Discerning front line literature |
discern
verb (used with object)
1.to perceive by the sight or some other sense or by the
intellect; see, recognize, or apprehend:
“They discerned a sail on the horizon.”
2.to distinguish mentally; recognize as distinct or different;
discriminate
discerning
adjective
1.showing good or outstanding judgment and understanding
“a discerning critic of French poetry.”
Textbooks
• Skills textbooks
– to help you learn such things as how to design a
robust investigation or analyze data statistically.
• Subject textbooks
• introduce readers to a field of academic
enquiry, and are explicitly designed to support
students’ learning.
Textbooks
• Relatively cheap compared with research materials.
• Words like ‘introduction’, ‘guide’ or ‘study’ appear in
the title or the series title.
• Available in softback, and have an eye-catching cover.
• The title evidently encompasses a field or sub-field
rather than a particular research agenda (e.g., A Short
History of the English Language) or else it covers a
particular skill (e.g., Statistics in the Social Sciences).
• The cover blurb indicates a student target readership.
• There are multiple copies in academic bookshops and
libraries. Also, popular textbooks often run to more
than one edition.
Textbooks
• Good place to start
– Overview of subject area
– Identify front line texts
• Limited for research
– Basic
– One commentators account
– Prescriptive, authoritative
– Often include subjective interpretations of original work,
you need to go to the source.
– In the interest of offering a clear story they make strong
claims without proper evidence or over simplify
complicated issues.
Textbooks
Textbooks often give us the Hollywood version…
…you need to read the book
(or rather the original published papers).
| Readers, Handbooks and Encyclopedias |
| • Readers • Handbooks • Enclopeaedias |
Readers
• Collections of classic
papers in particular
research field.
– Personal selection of editors,
may not be considered
representative by other
authors.
– Normally full papers are
included, but sometimes
necessary to check the
original.
Handbooks and Specialist
Encyclopedias
• Like readers but articles are commissioned.
– Overviews of research, theory or methodology.
– Useful for gaining an understanding of the stateof-the-art.
– Like papers, written from the authors perspective.
– Sometimes in the reading process, it’s better to
form your own perspective by selectively reading
front line material.
Over-reliance on support literature
• Referring to ideas and evidence without referring
to the original source.
• Giving references to works without having read
them yourself.
• Referring just to works mentioned in the support
text.
• Using secondary referencing, e.g. ‘Jones (cited in
Smith, 2009) found …’.
• Plagiarizing by presenting an identical or slightly
rewritten version of the support text, as if you
had done the reading and thinking.
| Front-line literature |
| Front-line literature |
| • Types of front-line literature • Discerning front-line literature |
| Types of front-line literature |
| • Theoretical descriptions and explanations • Reports of original research • Accounts of current practice • Policy statements |
Theoretical Descriptions and
Explanations
• Models the way things are (or should be), by
using evidence …
– Experiments
– Observations
– Experience
– Ideas
– (not necessarily from the authors themselves)
• … to identify patterns.
– Which can be formalized into a model..
– .. to make future predictions (i.e. hypotheses)
Theoretical Descriptions and
Explanations
• How theory and evidence interact through
modelling
Theoretical Descriptions and
Explanations
• Model can
– Help deepen understanding
– Anticipate future observations
– Strengthen case for making changes
Theoretical Descriptions and
Explanations
• Example
– A paper presents a predictive model for the
consumption of earths natural resources
– The model predicts that, at current rates of
consumption, some resources will be used up within
fifty years.
– This study could be used to
• Criticize policy
• Recommend change
• Influence thought
• Etc.
Reports of original research
• AKA data-driven literature.
• Observations about the real world, often
related to a prediction or hypothesis derived
from a model.
Reports of original research
• Data
– Observational
– Experimental
• Situation manipulated by the researcher
– Observational-experimental (between the two)
• Observing two contrasting groups or situations that occur
naturally (a natural experiment).
• Observation in which the researcher participates in the
observed activity or situation (participant observation).
• Detailed observation of one or more individuals or groups
with the same, or contrasting, profiles (case studies).
Reports of original research
• Results can
– deepen understanding
– Help explain problems
– Demonstrate why a method works better than
another
– Influence changes in policy or practice
– etc.
Practice literature
• Accounts of how things are done,
– Written by experience practitioners
– Features strongly in applied fields of enquiry
focusing on practical activity e.g. nursing
– To recommend practices that are most effective,
or identify shortcomings.
– For example,
• A personal illustration of how a nurse working for a
relief agency has learned to cope with the extreme
demands of over-crowded refugee camps.
Policy literature
• Emphasizes changes that improve practice,
– Written by policy makers
– Features strongly in applied fields of enquiry
focusing on practical activity (as with practice
literature)
– For example,
• The government publish a report describing a fault in
practice, proposing an alterative policy as a remedy
• A pressure group with different values publish a critical
report suggesting an alternative policy
| Discerning front-line literature |
Discerning front-line literature
• A skill that is learnt over time
• Useful indicators
– Top Journals; Science, Nature .. Information
Visualization, BMC bioinformatics (in my field) etc.
– Journal papers with Thomas Routers impact factor
– Book publishers Springer, Elsevier, Academic Press etc.
– Indexed papers IEEE, ACM, SPIE etc.
– Author reputation (Citation indexes)
– Position of big author (1st, 2nd, 3rd or last)
– Number and quality of citations to a paper
– Position in web searches (?)
Discerning front-line literature
• Content:
– Reading these three sections (in this order) can
normally give you a good idea of the content and
quality of a paper.
• Abstract
• Conclusion
• Introduction
– If the paper is poorly written, confused* or vague
then stop reading. Life is too short.
Discerning front-line literature
• *Paper is confused
– Internal inconsistencies
– Lack of clarity or unnecessary complexity
– Unsubstantiated assertions
– Stop reading
• You are confused
– Concepts are inherently complex
– You lack the background knowledge to understand the
assertions made
– Take your time and refer to support literature or
references. Keep reading.
| The Internet |
Opportunities
• Fantastic source of material
– Especially with links to e-resources at XJTLU
• Convenient
• Optimized for research
– Academic roots
– Hyperlinks are like references
– Sources linked to citations
– Export to endnote etc.
Dangers
• Copy and Paste
– Passing work found on the internet as your own is
plagiarism and carries harsh penalties.
• Reliability
– The internet is a huge, amoral, uncoordinated
dissemination forum.
– Anyone can say anything.
Solution
• Use the internet as a catalogue rather than as
a repository for information.
• Always refer to the original resources (which
are often linked to).
• Don’t copy and paste.
Internet resources for research
• Google Scholar
• Microsoft Academic Search
• Citeseer
• DBLP Bibliography
• Online bibliographies
• Author homepages
• University Course Pages
Internet Material
• Likely to be reliable
– Peer-reviewed journal articles that are also published
in an academic journal.
– Peer-reviewed journal articles published in genuine
electronic journals.
– Already published journal articles and book chapters
that have been posted on an academic’s home page.
– Electronically readable books written by subject
experts
– Official materials published on a recognized
institutional website
Internet Material
• Likely to be fairly reliable
– Pre-peer-reviewed material, as described in (3)
above – but track down the published version if
possible.
– 2 Lecture or research notes on the site of an
academic working at a recognized institution.
Internet Material
• Likely to be unreliable:
– Material on the home pages of individuals.
– Material on organization websites that is written
by enthusiasts rather than experts.
– 3 Free-for-all post-your-views sites (unless
restricted to a recognized set of academic
contributors).
– Web-logs (blogs), chatroom, pontifications, etc.
Referencing Internet Sources
1. Use a, more permanent, print version where possible.
2. Attribute the material to a person if possible.
3. If (and only if) no author is named, give the
institutional details instead.
4. Give the date when it was posted or last updated.
5. Indicate the date on which you last accessed it.
6. Double check the URL.
Wray, A. (2010) ‘What is formulaic language?’,
www.cardiff.ac.uk/encap/research/networks/flarn/whatis/i
ndex.html (accessed 5 January 2010).
| Reading Critically |
Reading Strategies
• Scanning – looking through a text to find
specific sections or key words and phrases
indicating where the information you are
seeking is located.
• Skimming – reading quickly through those
parts of a text that can give you an overview
of the content.
• Intensive reading – carefully reading every
word of a text from beginning to end.
Balanced Strategy
• Scan and Skim first to find what might be
useful.
• Read the more important texts in more detail.
Summary
• Critical Reading
– Active
– Objective
– Analysis
– Inference
– Task driven
Summary
• Critical Reading
– Active
– Objective
– Analysis
– Inference
– Task driven
– Focus on what to look for in a text
– Identify and Evaluate the Argument
Central and Review Question
• Central Question
– Broad, covers research task
• Review Questions
– More specific
– Derived from a broader central question
– Can;
• Directly contribute to central question
• Help with theoretical questions
• Justify methodology
Central and Review Question
• Central Question
– ‘Can Information Visualization help with IR’
• Review Questions
– ‘Can IV be used with text’
• Directly contribute to central question
– ‘How can I design IV software to handle text’
• Help with theoretical questions
– ‘What can I learn from other IV software with text’
• Justify methodology
Evaluating Usefulness
• Not everything is equally useful
• Articles should
– Contribute directly to your task
– Be reliable
– Be plausible
– Extreme views should be treated with caution
Evaluating Usefulness
• Not everything is equally useful
• Articles should
– Contribute directly to your task
– Be reliable
– Be plausible
– Extreme views should be treated with caution
• We need to identify and evaluate the
argument.
Argument
• Argument = Conclusion + Warranting
– Conclusion:
• One or more claims that something is, or should be, the
case.
– Warranting:
• The justification for why the claims should be accepted.
• Normally based on evidence from research or personal
experience. Sometimes based on evidence from the
literature.
• A robust conclusion is one that is sufficiently
warranted by appropriate evidence.
Argument
Argument-warranting=Opinion
“Correlation between CO2 emission and rising
temperature does not by itself warrant the
conclusion of a causal relationship.”
Argument
• Authors job: Provide the best possible
warranting for their conclusion.
• Your job: Judge whether the warranting is
enough to make the conclusion convincing
and so whether to accept or reject the
conclusion.
Exercise
“For example, one practitioner researcher commented
that ‘I think that my TLRP [Teaching and Learning
Research Programme] experience was very, very positive.
It caused me to reflect back on where I was and to accept
that I am really happy in FE [further education], that I
don’t want to be a lecturer in HE [higher education].’
Building research capacity is not just about building the
next cohort of professors and senior academics, it can
also relate to the building of one’s own personal capacity
to engage with research and practice.”
Claim? Warranting?
Exercise
“For example, one practitioner researcher commented
that ‘I think that my TLRP [Teaching and Learning
Research Programme] experience was very, very positive.
It caused me to reflect back on where I was and to accept
that I am really happy in FE [further education], that I
don’t want to be a lecturer in HE [higher education].’
Building research capacity is not just about building the
next cohort of professors and senior academics, it can
also relate to the building of one’s own personal capacity
to engage with research and practice.”
Claim Warranting
Exercise
“For example, one practitioner researcher commented
that ‘I think that my TLRP [Teaching and Learning
Research Programme] experience was very, very positive.
It caused me to reflect back on where I was and to accept
that I am really happy in FE [further education], that I
don’t want to be a lecturer in HE [higher education].’
Building research capacity is not just about building the
next cohort of professors and senior academics, it can
also relate to the building of one’s own personal capacity
to engage with research and practice.”
Claim Warranting Convincing?
What makes an argument convincing
• Quote?
• Number?
• Percentage?
• Statistic?
What makes an argument convincing
• Quote?
• Number?
• Percentage?
• Statistic?
• Depends on the claim
Example
In the reading test, the five children who were
taught to read using phonics performed better
overall than the five children taught using the
whole word method. This shows that the
phonics method is a better choice for schools.
Example
In the reading test, the five children who were
taught to read using phonics performed better
overall than the five children taught using the
whole word method. This shows that the
phonics method is a better choice for schools.
Browning (2005) found that children taught to read using
phonics did better in a reading test than children taught
using the whole word method. However, the study was
small, the test rather limited, and the subjects were not
tightly matched either for age or gender. An examination
of Browning’s test scores reveals that, although the mean
score of the phonics group was higher, two of the highest
scorers in the test were whole word learners. Since this
indicates that the whole word method is effective for
some learners at least, Browning is perhaps too quick to
propose that ‘the phonics method is a better choice for
schools’ (p. 89).
Browning (2005) found that children taught to read using
phonics did better in a reading test than children taught
using the whole word method. However, the study was
small, the test rather limited, and the subjects were not
tightly matched either for age or gender. An examination
of Browning’s test scores reveals that, although the mean
score of the phonics group was higher, two of the highest
scorers in the test were whole word learners. Since this
indicates that the whole word method is effective for
some learners at least, Browning is perhaps too quick to
propose that ‘the phonics method is a better choice for
schools’ (p. 89).
Convincing Argument
Convincing Argument
=
Conclusion
(containing claims)
+
Adequate Warranting
(based on sufficient appropriate evidence)
Identifying the conclusion and
warranting of arguments
• Key words;
– therefore, because, since, so, it follows that, it can be
concluded that
• Examples
– Since research shows that girls mature faster than boys,
studies should take age and gender into account when
exploring child development.
– Child development studies should take age and gender
into account because research shows that girls mature
faster than boys.
– Research shows that girls mature faster than boys.
Therefore, studies of child development should take age
and gender into account.
Identifying the conclusion and
warranting of arguments
• Examples with weight on warranting implying
that conclusion is not universally true
– In so far as girls are believed to mature faster than
boys, studies of child development should take age
and gender into account.
– In conditions where girls mature faster than boys,
studies of child development should take age and
gender into account.
– Where it is relevant to the investigation that girls
mature faster than boys, studies should take age and
gender into account.
| Incomplete or flawed arguments |
| • Conclusion without warranting • Potential warranting without conclusion • Warranting leading to an illogical conclusion • Conclusion not explicitly linked to warranting • Conclusion with inadequate warranting |
Conclusion without warranting
“The best musicians make the worst teachers”
“Why do you think that? How do you know?”
“The eye for fine detail possessed by the best
musicians tends to make them over-critical and
discouraging with pupils (Goodman, 2009).”
Potential warranting without
conclusion
“Johnson’s research shows that people often
sign legal agreements without reading them.
Legal documents can be difficult to read”
“So what? What do these different pieces of
evidence, together, imply?”
“People may fail to read legal documents
because they are difficult to read.”
Warranting leading to an illogical
conclusion
“People in English-speaking countries tend not
to know another language. This indicates that
they are poor language learners”
“does this reasoning add up? Aren’t there more
plausible conclusions”
“This may suggest that English speakers do not
see the need to know other languages.”
Conclusion not explicitly linked to
warranting (missing steps)
“Statistics show that students are drinking far to
much to be good for their health. Alcoholic drinks
should be increased in price.”
“What causal relationship between the factors are
you meaning to suggest?”
“Since students have only limited money, raising the
price of alcohol might result in their drinking less.”
Conclusion with inadequate
warranting
“Trainee managers learn more effectively when they are
praised than when their efforts are criticized. In a survey
of female trainee managers in a retail company, 77% said
they liked to be praised.”
“Is the evidence adequate to justify the extent of the
claim? Is the evidence appropriately interpreted?”
“However, males and females may respond differently to
praise. Furthermore, there is no evidence of a link
between ‘liking to be praised’ and learning more
effectively.”
| Logical fallacies |
| • Flaws in reasoning that lead to illogical statements. • Masquerade as reasonable statements. • Attempts to manipulate readers by appealing to their emotions instead of their intellect. |
Hasty Generalization
• Conclusions from inadequate evidence
– E.g. “My hometown is the best place to live” with
two examples to support the statement.
• That’s not enough evidence!
• Others might not feel the same way.
– Other examples are stereotyping and sexism.
False analogy
• Differences outweigh the similarities or the
similarities are irrelevant.
– E.g. ““Old Joe Smith would never make a good
president because an old dog can’t learn new
tricks”
• Joe Smith isn’t a dog.
• Learning the role of being a president is in no way
similar to that of being a president.
– These analogies sound clever but fall apart under
examination.
Begging the question
• Aka Circular reasoning
• Tries to offer proof simply by stating another
version of the argument.
– E.g. “Wrestling is a dangerous sport because it is
unsafe” or “Wrestling is a dangerous sport
because wrestlers get injured.”
– dangerous unsafe
Irrelevant argument
• An irrelevant argument reaches a conclusion
that doesn’t follow from the premises.
– E.g. “Jane Jones is a forceful speaker, so she’ll
make a good mayor.”
– Why? What exactly does speaking ability have to
do with being a good mayor?
False cause
• A false cause assumes that because two
events are related in time, the first caused the
second.
– E.g. ‘The new weather satellite caused the rain
storms’
– How do you know that?
– Superstition.
Self-contradiction
• Two premises that can’t both be true at the
same time.
– E.g. “Only when nuclear weapons have finally
destroyed us will we be convinced of the need to
control them”
– Not possible for both to happen
– Contradiction
Red herring
• Tries to distract attention from one issue by
introducing a second that’s unrelated to the
first.
– E.g. “Why worry about pandas becoming extinct
when we haven’t solved the plight of the
homeless?”
– What does the second have to do with the first
Argument to the person
• Attacking the person making the argument
rather than the argument itself.
– E.g. “We’d take her position on employment
issues seriously if she were not so lazy.”
Guilt by association
• A person’s arguments, ideas, or opinions lack
merit because of that person’s activities,
interests, or companions.
– E.g. “Jack belongs to the International Hill
Climbers Association, which declared bankruptcy
last month. This makes him unfit to be mayor of
our city.”
Jumping on the bandwagon
• Something is right or permissible because
“everyone does it.”
– E.g. “How could bungee jumping be unhealthy if
thousands of people have done it?”
False or irrelevant authority
• Citing the opinion of someone who has no
expertise in the subject at hand.
• Transfer prestige from one area to another
– E.g. “Lionel Messi eats Bimbo bread”
Card-stacking
• Ignores evidence on the other side of a
question.
• Aka Cherry Picking or Special pleading.
– E.g. “the three studies we looked at all had
positive results”
– How did you choose these tests? What about any
other tests?
The either-or fallacy
• Offers only two alternatives when more exist
– E.g. “Either go to college or forget about getting a
job.”
– Rigid two-sided statement that ignores the truth.
Taking something out of context
• Deliberately distorts an idea or a fact by
removing it from its previously surrounding
material.
– e.g Die Hard 4.0
– Blurb New York Daily News “Hysterically …
entertaining.”
– written line: “The action in this fast-paced,
hysterically overproduced and surprisingly
entertaining film is as realistic as a Road Runner
cartoon.”
Appeal to ignorance
• Make an incorrect argument based on
something never having been shown to be
false or true.
• Often exist in the fuzzy language of doublenegatives.
• Absence of evidence isn’t evidence of absence
– ‘I haven’t seen it so it mustn’t be there’
Ambiguity and equivocation
• Statements open to more than one
interpretation, thus concealing the truth.
– E.g. “The software lived up to the users
expectations”
– Ambiguous. We don’t know if this is good or bad.
| Five Critical Synopsis Questions |
| A. Why am I reading this? B. What are the authors trying to do in writing this? C. What are the authors saying that is relevant to what I want to find out? D. How convincing is what the authors are saying? E. In conclusion, what use can I make of this? |
A: Why am I reading this?
• To stay focused
• For example;
– Background
– Methodology
– Contrasting approach
– etc.
B: What are the authors trying to do in
writing this?
• For example;
– Report the findings of their own research.
– Review others’ work.
– Develop theory.
– Express particular values or opinions.
– Criticize what is currently done.
– Advise on what should be done in the future.
• Also, who is it written for
C: What are the authors saying that is
relevant to what I want to find out?
• What the text is actually about – what it
reports, how any empirical work was carried
out, what was discovered and what the
authors conclude about it.
• Where any overlap lies between the authors’
concerns and your own interests – the authors
are unlikely to have been asking the same
questions as you are.
D: How convincing is what the authors
are saying?
• Quality of data and argument.
– Strength of warranting for claims
– False arguments
– Underlying assumptions
E: In conclusion, what use can I make
of this?
• How important is the article?
– Mention briefly
– Discuss in detail
• Do I agree with the claims made?
• How well does if answer my review question?
A Critical Synopsis of a text
A. Why am I reading this?
B. What are the authors trying to do in writing
this?
C. What are the authors saying that is relevant to
what I want to find out?
D. How convincing is what the authors are saying?
E. In conclusion, what use can I make of this?
Code (1) = Return to this for detailed analysis; (2) = An
important general text; (3) = Of minor importance; (4) =
Not relevant.
Tutorial Exercise
Choose a text in your area of expertise and write
a critical synopsis of the text for the purposes of
one of your own review questions.
A. Why am I reading this?
Part of reading to answer the review question
‘….
| The Review Process |
External Review (revision)
• Part of the Scientific Method
• Evaluation by experts.
• Peer review
– Unbiased anonymous criticism.
– Scientific journals, conferences.
– Better journal or conference ->
work perceived as being of a higher standard or
better quality.
Peer Review
• Quality control to check if a work should be
funded or published
• Peer
= people of equal standing
= other experts in my field
= other computer scientists
• Not all journals or conferences have peerreview. Just the ones that matter.
Why publish
• Support future research
• Personal reasons
– Announce formally the results
of work
– Associate your name with an
important discovery, thereby
creating a permanent record
– Promote your research
interests and attract additional
funds. Publish or perish.
Numbers
International Journal of Service Industry Management
(Emerald)
Step 1: Journal Editors
• Paper is relevant
• Paper matches editorial platform, e.g.
– Relevant
– Innovative
– Groundbreaking
• Only a few papers survive until peer review
Step 2: Reviewers
• Comment on the validity of the science,
identifying scientific errors and evaluating the
design and methodology used
• Judge the significance by evaluating the
importance of the findings
• Determine the originality of the work based on
how much it advances the field. Reviewers also
identify missing or inaccurate references.
• Recommend that the paper be published or
rejected. Editors don’t have to heed this
recommendation, but most do.
Process
• Single-blind review
– Reviewers know the author’s identity, but not vice
versa
• Double-blind review
– Identities of the author and referees are both
hidden
• Open peer review
– Author’s and reviewers’ identities are known to
each other
Exercise
• What are the relative advantages of each type
of peer review.
– Single-blind review
– Double-blind review
– Open peer review
Advantages
• Single-blind review
– Reviewers can be honesty without fear of
retribution
• Double-blind review
– Reviewers aren’t influenced by authors’ identity
and the work needs to speak or itself
• Open peer review
– Reviewers are more accountable and forced to be
more thoughtful
Single-blind vs Double blind
• True double-blind review is difficult since the
identity of an established author is often easy
to guess.
Advantages of Peer Review
• For Authors
– Validates work, respectability, prospect of future
funding.
• Editors
– Informs decision making, improves journal reputation
(when it works).
• Scientists
– Helps us prioritize what we read and judge the quality
of published research.
• Non-scientists
– Helps us make sense of scientific claims.
Limitations of Peer Review
• Doesn’t necessarily find errors.
– Reviewers can concentrate on style rather than
content.
• Can’t detect fraud (i.e. falsified results)
– Referees don’t have access to actual data
– Better at detecting plagiarism than fraud
• Abuse of inside information by reviewers
Limitations of Peer Review
• Slows down advances in knowledge
– Publication of an article can take over a year.
• Suppression of Valuable Results
– Reviewer might suppress innovative results that
challenge his own standpoint.
– Research undertaken in developing nations is
often overlooked in top-end journals.
– Language issues.
PC era peer-review
• Peer review software
– Streamlined process
– Blogs? Open-data? Post-publication review?
Conclusion
• Know how to read and think critically
• Recognize and evaluate an argument
• Write a critical review
• Understand the peer-review process
Resources
• Critical Reading and Writing for Postgraduates
– Mike Wallace and Alison Wray
– SAGE Study Skills
• TED talks, www.ted.com/talks
– Naomi Oreskes
Research Proposal Abstract
The objective of this exercise is for you to prepare a short (one page or
half page) abstract for a project proposal with an original research
question and some outline plan for answering that research question.
You are welcome to choose your own project ideas as long as they fit
the project specifications. These can be related to things such as:
• The evaluation of an aspect of your current research or software
that you have developed in another part of the course,
• a hobby or side project that you are interested in, or
• a replication of an existing study, where you extend or vary some
aspect of it in an interesting way.
Research Proposal Abstract
• To keep the work relevant to the CSSE masters course, the
problem must be grounded in computer science or
software-engineering.
• In practice, the best problem statements are ones that
identify a small but interesting problem.
• There should be some value to solving the problem
(Justification).
• It should be reasonable to expect that you can undertake
all the required work within the time frame of the module.
• You should be careful to avoid overly-ambitious problems
or anything that might end up taking too much time.
The problem could be related to either one of the
following:
• The work you plan to do for your Masters
dissertation,
• a replication of a published study,
• an on-going research project (or related sideproject) in your research lab
• something suggested by the course leader or
another faculty member.
Due to the time-constraints of the project I would
advise against the development of new software
from scratch
• Use software that already exists
• Software that can be implemented quickly using a
software toolkit.
• Use code from other sources (give proper credit
to the original authors).
• Lo-fi prototyping is another feasible option.
• So are surveys or questionnaires
Examples of projects that could work well are:
• An empirical comparison of different information visualizations for
hierarchy navigation
– using the perfuse infovis toolkit to implement different visualizations of a
familiar network drive hierarchy
• An investigation into the problems with mobile maps
– this could be an exploratory study to investigate sources of frustration for
mobile map users
• An Evaluation of Quantile Distortion Plots for the Effective Visualization
of Bivariate Correlation Patterns
– comparing quantile curve distorted plots with unscaled plots using data
describing various economic and social indicators for countries around the
world
• A user centered redesign of the Suzhou bus map
– investigating how a user centered design methodology can be employed
to improve the user experience for a public-transport map of the city
• You should have your first abstract proposal
finished at least by 25th October 2019.
• If you finish the abstract earlier I can give you
early feedback that allows you to improve
your proposal.
• You should not start your main project
proposal before your abstract is approved by
the module leader.
You are encouraged to approach academic staff
in the department to help develop your
proposal idea. The aim of this exercise is to
develop your idea of what a suitable research
project might be. You will have some feedback
from the module leader who can highlight any
potential problems with your proposal.
2015 Projects
• AI Bear
• Deed learning for Object
Recognition
• Mental health app
• Effect of SNS on student
performance
• IR for library collection
• Evaluation of Optimization
Strategies for Hadoop
• Mobile browsing survey
• Security watch
• Word Cloud
• Mobile Language game (with
sound)
• Web English Learning Game
• Security Problems in ecommerce
• Bitcoin survey
• IV for running watches
• A Survey of Wireless Sensor
Networks
• Car pooling app
• Word segment analysis
• Quantum communication
• Smart Car Control
• Evaluation of Decision Tree
Algorithms