CSCM21 Research Report Assignment
Dr. Matt Roach
By submitting this work, electronically and/or hard copy, you state that you fully under- stand and are
complying with the university’s policy on Academic Integrity and Academic Misconduct. The policy can be
found at https://www.swansea.ac.uk/academic-services/academic-guide/assessment-issues/academicintegrity-academic-misconduct
Task Introduction
This report is worth 50% of the CSCM21 module. The core challenge in this course work is to
You will conduct a personal research study and present the findings in a report (written in
the style of a paper). Your study can be undertaken in any of the topics and concepts
covered in CSCM21, broadly summarised by:
• Bias in Algorithms / Models
o Reducing, Detecting, Measuring, Mitigating.
• Explaining Blackbox Algorithms
o Algorithms, use case, limitations, insights.
• Fair ML
o Concepts, metrics, principles, frameworks.
The Assignment is split into two parts, the first part “Definition & Scope” (worth 5% of the
module) acts a gateway to the second part “Full Research Report” (worth 45% of the
module).
Part 1: Definition & Scope (5%)
For the first part you will need to provide a presentation “Research Report: Definition &
Scope”. This section acts as a gateway to the second part to ensure that your plan is
suitable and within scope of the assignment. You should not proceed to the second part of
the assignment until you have approval to do so from the first part. This part of the
assignment provides an opportunity for feedback on your proposed research report. Please
seek me out the seminar sessions in office hours if you would like to discuss ideas and
proposals for your research report before the deadline.
Your “Research Report Definition & Scope” should include the following information at a
minimum.
• Title & Study Type
• Short statement (Hypothesis or Conjecture) on what is the subject of the report.
• Main section
o In this section try to highlight the precise focus of the study – be specific
rather than general and abstract with your language.
o Name and briefly outline the experimental method(s) that will be used to
carry out the study. Include:
§ any data to be used – emphasises on scope size of data, number of
data sets etc.
§ Concepts to be discussed – emphasises on scope number of concepts.
Deliverable
Use the CHI Proceedings template available at https://chi2020.acm.org/authors/chiproceedings-format/ to format your report. You can remove the “Author Keywords” and
“CCS Concepts” sections of the template, but your work should otherwise be formatted as a
written report, including figures/tables/references as applicable.
Upload your “Definition & Scope” to Canvas as a single pdf file (max 1 page, excluding
references1).
Part 2: Full Report (45%)
The second part of the coursework is a report (paper) of no more than six pages (not
including the references) written in the style of a conference paper. Although the exact
heading and number of sections and subsections will be up to you to decide all reports must
cover five core areas:
• Purpose and importance of the study relevant to the Hypothesis or Conjecture.
• Background & Literature Review
• Study Design / Experimental Method
• Results & Analysis
• Conclusions or Discussion
Deliverable
Use the CHI Proceedings template available at https://chi2020.acm.org/authors/chiproceedings-format/ to format your report. You can remove or ignore the “Author
Keywords” and “CCS Concepts” sections of the template, but your work should otherwise be
formatted as a written report, including figures/tables/references as applicable.
Upload your “Full Research Report” to Canvas as a single pdf file (max 6 pages excluding
references, no appendices or annexes are allowed).
If you have python code/notebooks and experimental data this is to be uploaded separately
from the pdf file in a zip file using the link on Canvas).
1 Refences will not be considered in the marking of Part 1 “Definition and Scope”).
Appendix – Further Explanation of Study
There are three types of study you can choose from for your assignment (only choose one
study type). The following section describes what a good research study could look like for
this assignment through some example descriptions and papers. The papers are example of
the “type” of study however, since they are peer reviewed publications they are much larger
and extensive in scope than the assignment so use them as inspiration for your ideas only.
Critical Concepts Discussion
This type of study is characterised as a study that critically apprises current state-of-the-art
approaches to: fair ML, Bias or Explainability through a multidisciplinary perspective(s).
This type of study would be motivated by the concepts and style covered in the debate
seminars of the module. Foreach of the four core areas of the research report the emphasis
for a critical concepts study is highlighted below:
• Purpose of the Study
• Hypothesis / Conjecture
• Background & Literature Review
• Clarity of concepts: definitions and their origin (social science, legal studies)
• Study Design
• Structured Reviews
• Thematic Analysis
• Discussion & Further Work
• Critical discussion on current state-of-the-art philosophical and empirical use
of the concepts in formalized ML / data driven approaches
• Further Work
Inspiration for a Critical Concepts Discussion Report
• An example just one of multiple frameworks and ethical principles drawn up by
many organisations see Google’s white paper [1]. There are many others NHS EU [2]
etc.
• A peer review paper, and therefore more rigorous in its method of generation (or a
least in the explanation thereof) is Microsoft researchers’ publication on Human-AI
interaction [3]
• A very nice position piece on fair ML is presented in [4] it is a meta analysis of a
topic and provide a very clear well-presented conceptual piece
• An excellent critical conceptual discussion is presented by Binns in [5] where he
mixes a thorough review and critical appraisal of related fairness concepts from
multidisciplinary domains; and positions their use in ML with thought experiments
and illustrative examples.
• And Overview of explaining Black box algorithms by Guidotti [6] provides an insight
into topics and a structured review style paper.
• Rudin advocates a provocative position to stop explaining black box machine
learning [7].
Empirical Experimental Study
This type of study is characterised by coding (or using libraries) method(s) applying them to
data, presenting empirical results and includes discussions on uses and limitations of the
methods. This type of study would be motived by the labs and could be an extension and
write-up of a lab(s) or a practical exploration of method(s) covered in the course but not
explicitly covered in the labs.
• Purpose of the Study
• What would you like to investigate?
• Empirical exploration of a method or methods to address a challenge.
• Motivated by the lab(s) – Extension and write-up
• Background & Literature Review
• Origins of methods used and explored in the study.
• Applications and limitations.
• Study Design
• Explanation of experimental design and approach to explore the purpose of
the study.
• Empirical Results Presented
• Discussion & Further Work
• Critical analysis of the results
• Conclusions & limitations
• Further study proposals
Inspiration for an Empirical Experimental Study Report
• Measuring detecting bias in natural langue models could be very important given the
potential for wide adoption of very large langue models such as word2vec, BERT and
GPT2 in in software applications. An interesting paper is presented on how to debias
a word2vec model is presented in [8].
• Original papers on methods explored in the labs would help think of ides to extend
the analysis of the labs into these methods with a write-up.
o Interacting with predictions [9]
o Salience maps [10]
o Face Recognition [11] [12]
o Etc. see lab workbook for specific references.
• Flip test is a great empirical study on using GANs to transform models to remove bias
[13], a reminder this type of work is extensive, but a small study analogous ideas
could be an interesting report.
• Another example of transforming model spaces using Adversarial Learning, this time
on word embeddings [14].
Large Scale Research Design
This type of study is characterised by requiring a large well-resourced research study in
order to carry it out. Since executing a study like this well outside the scope of this
assignment this type of report requires that you present a robust experimental design for
such a study. This would be motivated by a hypothesis that, AI is being applied unfairly, for
example using mobile phone audio (without permission – or explicit denial) to drive
personalised advertising.
• Purpose of the Study
• Hypothesis
• Motivated by real world problem that has not be investigated in the
literature to date.
• Background & Literature Review
• Contextual Setting Explanation
• Importance, prevalence or significance of study
• Survey of similar studies
• Study Design
• Explanation of experimental design & method
• Technical architecture
• Participant journey, expectation, engagement,
• Discussion & Further Work
• Most Critical risks difficulties with study
• Limitations of conclusions
• Potential challenges to study
Inspiration for a Large Scale Research Design Report
• An example of a real-world study into algorithmic equity is presented by Katell et al
[15].
• A large scale empirical study using web scraping and crowed workers is presented by
Datta [16]
• Use of AI in healthcare setting provokes an interesting question into weather we
need algorithms to be explainable in this context [17].
• Explaining model failure in retail forecasting [18].
• Using real world data for counter factual risk assessment [19]
• Finally Bhat et al present Explainable Machine Learning in Deployment [20]
References
Please note this is not an exhaustive list please do your own research and reading in
preparation for this assignment.

Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 2020,
pp. 514–524.

Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 2020,
pp. 45–55.