COMP5329 – Deep Learning Assignment-2 Due: 19-May-2022 (Week 12) Assignment-2 has two tracks: competition track and reseach track. Students should attend one of the two tracks. 2 or 3 students are suggested to form a group to attend one of these two tracks. 1. Competition track description [100 Marks]: In this assignment, you are to solve the multi-label classification task. Each sample in this dataset includes: • an image, • one or more labels, • a short caption that summarizes the image. Your goal is to implement an image classifier that predicts the labels of image data sample. You may optionally include the caption in the input of your classifier — it’s up to you! Please submit your submission file via Kaggle https://www.kaggle.com/competitions/multi-label-classification-competition-2023 Remember: the ranking contributes to 20% of your assignment mark. Please make sure you name your team in the following format. {unikey1}_{unikey2}_{unikey3} The evaluation metric for this assignment is Mean F1-Score. The F1 score, commonly used in information retrieval, measures accuracy using the statistics precision p and recall r. Precision is the ratio of true positives (tp) to all predicted positives (tp + fp). Recall is the ratio of true positives to all actual positives (tp + fn). The F1 score is given by: The F1 metric weights recall and precision equally, and a good retrieval algorithm will maximize both precision and recall simultaneously. Thus, moderately good performance on both will be favored over extremely good performance on one and poor performance on the other. Submission Format For every image in the dataset, submission files should contain two columns: image id and labels. Labels should be a space-delimited list. For example ImageID,Labels 1.jpg, 1 8.jpg, 8 9.jpg, 9 10 10.jpg, 10 9 etc. You can use any methods in deep learning to accomplish the classification task. You must guarantee that the submitted codes are self-complete, and can be successfully run in common python3 and PyTorch environment. Instructions to hand in the assignment • Go to Canvas and upload the report. The report should include each member’s details (student ID and name). If you work as a group, only one student needs to submit the report which must be named as student ID numbers of all group members separated by underscores. E.g. “xxxxxxxx_xxxxxxxx_xxxxxxxx.pdf” • The report must include a link of your code, model and data (e.g., a shared Google cloud folder, so we can easily run it on Colab). Clearly provide instructions on how to run your code in the appendix of the report or include a readme.txt in your shared folder. In your shared folder, please include a file “Predicted_labels.txt”, which contains the predicted labels of test exampels. You may want to submit the prediction that achieves the best performance on kaggle. Don’t update the code/data any more after the submission. If the latest modified time of the shared folder is signiftanly late after the submisison deadline, the whole submission will be taken as a late submission. • The report must clearly show (i) details of your modules, (ii) the predicted results from your classifier on test examples, (iii) run-time, and (iv) hardware and software specifications of the computer that you used for performance evaluations. • There is no special format to follow for the report but please make it as clear as possible and similar to a research paper. • The use of ChatGPT or other AI tools is prohibited in the assignments. A plagiarism checker will be used. Late submission: Suppose you hand in work after the deadline: If you have not been granted special consideration or arrangements – A penalty of 5% of the maximum marks will be taken per day (or part) late. After ten days, you will be awarded a mark of zero. – e.g. If an assignment is worth 40% of the final mark and you are one hour late submitting, then the maximum marks possible would be 38%. – e.g. If an assignment is worth 40% of the final mark and you are 28 hours late submitting, then the maximum marks possible marks would be 36%. – Warning: submission sites get very slow near deadlines. – Submit early; you can resubmit if there is time before the deadline. Marking scheme Category Criterion Report [70] Introduction [10] – What’s the aim of the study? – Why is the study important? – The general introduction of your used method in the assignment and your motivation for such a solution. Related works [10] – Existing related methods in the literature. Techniques [20] – The principle of your method used in this assignment. – Justify the reasonability of the method. – Any advantage or novelty of the proposed method. Experiments and results [20] – Accuracy/efficiency (Figures or Tables) – Extensive analysis (ablation studies, comparison methods, hyper parameter analysis) Conclusions and Discussion [5] – Meaningful conclusion and discussion. Other [5] – At the discretion of the marker: for impressing the marker, excelling expectation, etc. Examples include fast code, using LATEX, etc. Code [10] Code runs within a feasible time [5] Size of resulting deep models for the prediction < 100MB [5] Classification performance [20] Groups in top 10% [20] Groups in top 10%-30% [15] Groups in top 30%-50% [10] Groups in top 50%-80% [5] Other valid submissons [3] Badly written code: [-20] Penalties [-] Not including instructions on how to run your code: [-30] Late submission 2. Research track description [100 Marks]: This research track calls for brave new ideas on deep learning. In this track, you are encouraged to propose and investigate new algorithms or problems in deep learning. You must not use the project (e.g. your honours, capstone or SSP project) that you have aleady done or are currently doing in other units to participate in this track. You are encouraged to contact our teaching team, if you want to have some discussion on your proposed research problems. Submit the report (including a link to your source codes, models and data) on Canvas before the due date. TA will further contact you for the preseantion slides by Week 13. Marking scheme Category Criterion Report [90] Introduction [10] – What’s the aim of the study? – Why is the study important? – The general introduction of your used method in the assignment and your motivation for such a solution. Related works [10] – Existing related methods in the literature. Techniques [30] – The principle of your method used in this assignment. – Justify the reasonability of the method. – Any advantage or novelty of the proposed method. Experiments and results [30] – Accuracy/efficiency (Figures or Tables) – Extensive analysis (ablation studies, comparison methods, hyper parameter analysis) Conclusions and Discussion [5] – Meaningful conclusion and discussion. Other [5] – At the discretion of the marker: for impressing the marker, excelling expectation, etc. Examples include fast code, using LATEX, etc. Presentation [10] Penalties [-] Presentation in week 13 Late submission