1
CSE402 Research Methods
Assessment task #3Assessment Brief
Pilot Study and Project Write-up
Overview
This objective of this assessment task is for you to conduct and write-up a pilot study. This can be
based on the project proposal from assessment #2 or a new project proposal adhering to the
requirements specified in the documentation for assessment task #2 (any new project proposal should
be cleared by the module leader before advancing to the pilot study). The pilot study will include a
small scale experiment to evaluate the feasibility and predict the costs of a full-scale study. This will
give you experience of planning, running, debugging, analysing and interpreting the results of a smallscale evaluation.
Milestones
Milestones for this assessment are outlined below.
1. Pilot Study
Experiments, like computer programs with bugs, can be flawed easily. A pilot study often allows early
testing which would help identify problems. Typical problems include flaws and inefficiencies in
methodology, and data analysis. You should first test a computer program by yourself, then perhaps
with classmates or colleagues. You should also analyse the data produced from your pilot studies or
experiments, try to make a sense of what final results of your project would really be like. In rare
occasions, you may have to consider changes to be made to your Problem Statement and
Methodological Approach.
Run your study with a small number of people (e.g., classmates or friends). The goal here is to both
debug your study, to see how amenable your data is to analysis, to see what kind of results you are
getting, and ultimately to revise your study as needed. You will probably have to:
| |
Book rooms if needed. Prepare any necessary paperwork, equipment and/or software (including consent forms). Solicit and schedule subjects. Run subjects and collect data/observations. Analyse the data and interpret the results. |
Depending on results you might want to repeat this stage one or more times with a modified
procedure to ensure that you have sufficient information for the final report.
2. Project Report
This will be a 4-8 page paper in ACM CHI Archive Format prepared in LaTeX using the document How
to Structure Reports on Experiments in Human Computer Interaction (Appendix) as a guide for
structure and content. The paper will combine your developed project proposal, the materials and your
ethics application, along with the following sections:
Title, author(s)1, abstract, etc.
1
The project should be your own work but if a lecturer or a fellow student provides support or advice that
contributes in a meaningful and substantive way to the proposal’s intellectual content you can acknowledge them
2
| |
Introduction (motivation, background, and detailed problem statement). Detailed methodology used for pilot study. Data and results observed. Initial analysis of results. An analysis and interpretation of the pilot results, including expected trends. A discussion of any legal, ethical, social and professional issues. A critical appraisal of the limitations of the study and any problems in the analysis. Results and conclusions. |
The introduction and methodology sections can be carried across and summarised from assignment
two but you should paraphrase to avoid repeating text wherever possible. The discussion section
should include an outline of any ethical and legal issues related to the methodology and an outline of
the social and professional implications of the research.
3. Project interview
The final milestone of the project will be to attend a short interview with the module leader where you
will be expected to answer questions about your project, show the data generated during the project
and demonstrate any software or hardware developed for the project. This interview will be scheduled
to take place during the normal class time of your tutorial in the week following the assignment handin. During the interview you will be required to have your project material available (including the
project data plus any hardware or software developed for the project) and demonstrate how your
experimental data was generated and analyzed. You will not receive a mark for this assignment if you
are not available for your interview, cannot provide the appropriate project material, or do not answer
the questions related to your project to satisfactorily demonstrate that the project is your own work.
Dates
The deadline for submission is 11nd December 2020 23:00.
Electronic copies of the Initial Write-up and Project Report should be uploaded to ICE in pdf format.
The electronic copy of the final report should include additional electronic appendices including all the
data generated by the experiments and any other documents generated throughout the project. An
additional hard-copy of the report (without appendices) should be left in the lecturer’s mailbox.
Learning Outcomes
After completing this assessment you should be able to:
A – Critically evaluate scholarly work including your own in terms of its contribution to the field
of computer science.
B – Conduct and critically analyse experimental work in computer science.
C – Demonstrate critical awareness of legal, ethical, social and professional issues in computer
science.
D – Demonstrate effectively the skills to structure, write and deliver formal presentations such
as technical and research reports, adhering to relevant codes of practice.
by adding their name as a co-author. If someone helps you out in a way that does not impact on the proposal’s
intellectual content, you should acknowledge their contribution in a separate Acknowledgements section at the
end of the paper.
3
Marking Scheme
A total of 100 points are available for this assessment. Your mark will be your score as a percentage
of the total points available. This will be given a weighting of 55% towards the final mark for the
module. So, for example, if your total mark for the assignment is 70 points this will contribute 38.5%
(70/100*55%=38.5%) toward your final grade for the module. The list below describes the total
points available for each assignment component.
A) Analysis of empirical data (20 points)
The project should include an experiment that tests your hypothesis from assignments two or generates
data to answer your research question.
| i. | Empirical data is generated |
| ii. iii. iv. |
Analysis is attempted Analysis is adequate (i.e. the results can help with the research question) Analysis is good (i.e. gives a good idea of what needs to be changed for a successful full scale study) |
v. Analysis is excellent (full numeric analysis describes the extent of the result and how all
variables, including the number of test subjects, need to change for a successful full scale
study)
B) Conclusions and findings (20 points)
a. General
i. Findings are presented
| ii. | Adequate presentation of findings |
| iii. | Some conclusions are drawn |
| iv. | Full conclusions are evaluated |
| b. | Critical reflection |
i. Use of anecdotal evidence linking the results to the conclusions
| ii. | Some argument linking the results to the conclusions |
| iii. | Critical argument discussing the conclusion from at least one perspective |
| iv. v. |
Sound argument for conclusions accounting for different perspectives Impressive critical ability, demonstrated by an extended critical discussion of most of the issues |
C) Discussion (20 points)
| i. | Discussion of methodology |
| ii. | Evidence of creativity in methodology |
| iii. | Awareness of ethical and legal issues |
| iv. | Discussion of social and professional implications of the research |
v. Demonstrates the contribution of the research to the wider literature
D) Quality of writing, expression of ideas, logical and critical arguments, and conformity to
conventions of referencing (20 points)
i. Writing has a logical and coherent structure
| ii. | Conforms to the conventions of academic writing |
| iii. | Writing is fluent |
| iv. | Arguments are well formed |
| v. | The quality of the work is consistent with that required for publication in an academic or professional journal |
E) Level of the project (20 points)
| i. ii. |
The project fits the minimum requirements to be considered an empirical study. You have overcome some reasonable challenges to realize this project. The project includes a challenging literature review, some development work or overcomes some technical difficulties in the implementation or analysis of experiments. You have overcome significant challenges in one of the following areas: developing an |
| iii. |
understanding of challenging background literature, building software, implementing
technical solutions, running experiments, or analyzing experimental data.
4
iv. You have overcome significant challenges in more than one of the following areas:
developing an understanding of challenging background literature, building software,
implementing technical solutions, running experiments, or analyzing experimental data.
v. The project demonstrates a significant original contribution to knowledge and the work could
be considered for publication in a top conference or high quality scientific journal.
The proposal must be formatted using LaTeX. This is a compulsory requirement.
The report must be written by you using your own words and all ideas that are not your own should be
properly attributed to the original author or source material (please see the University guidance on
plagiarism and collusion). Quotes are allowed if appropriate, limited in their extent, and clearly
indicated as such. None of the work presented as original work in this report should be the same as
work presented for another module. If work presented for another module is referred to in this report
then it should be clearly indicated where this work is not original to this module.