MLSP: Machine Learning Systems Project

INTRODUCTION TO THE COURSE

The machine learning systems project is a research project that CDT ML Systems students will engage in over the course of the first year. The project itself will form a part of your PhD, but at the same time should be sufficiently well constrained to be able to report on it in the first year. The general process for the project will be to agree a preliminary exploration goal with the supervisor, followed by a project proposal, that should be agreed between student and supervisor by December 1 at the latest. Then the student undertakes the project and produces the "project report" for making a complete submission two weeks before the deadline. The deadline each year is at 4pm on May 1st (or 4pm the immediate Monday after if May 1st is on the weekend), so we expect a complete submission on April 23rd. Further edits can be made up to the deadline, but we do expect students to work and submit things early enough to cover eventualities - for example if you are ill near the deadline, that does not count as grounds for any special consideration.

STAGE 1: EXPLORATION

Every student should agree with their supervisory team a target problem and goal for their project, and an exploratory process to help firm up that project. Note this can be a team effort, and indeed will be better if it is! An explicit machine learning goal, and an explicit systems goal should both be articulated. This exploratory process should not just be a literature review (though that will be part), but it should also be some initial test coding, analysis and baselines. It should also identify the different contributions that different members of the team can make. The point of the exploratory process is to be able, by 1 December, to have a full project proposal for implementing the project, and confidence in the feasibility and likely success.

STAGE 2: PROJECT PROPOSAL

A project proposal should be written following the pro forma:

Title
Principal goal
Project description
Project evaluation mechanisms
Model paper: a paper that might form a good template for what is required for the work - i.e. the work extends this paper, or the work can follow a similar evaluatory framework
Completion criteria, and definition of success
If all goes well, the research contribution claim of the paper from this work
The minimal work to have some reasonable project writeup; the ease and expected time for achieving that
Risks and mitigations
Skills required, or training required and how it will be obtained in sufficient time
Resources required, and how they will be obtained
At least four key references

Note this is not required to be extensive; it should be focused. In reality students will need to establish more than just this pro-forma to be sure of the feasibility and likely success of the project, but this template provides a good guide of what should be considered. Where there is a Team of multiple students involved, it can be the same for each student but should identify the individualised contributions. There is every expectation that the overall goal of the project proposal will be defined by the supervisor; it is then the student's job to fill in the details above.

What makes a good project proposal?

A project proposal should target a well-defined problem. It should separate out the problem definition completely from the solution approach, it should have contingencies in place in case things don't work. There should be potential research novelty and contribution. There should be a vision for a publishable paper from the project.

Note that projects that don't require things to work are safer than projects where if things don't perform well, the student is left with nothing. A project of "Compare N things to identify what is best and why" doesn't depend on your `new method' delivering the goods. That said, it is perfectly acceptable to submit a project with negative results. It is just much harder to complete and to write up such a project. There is no requirement that "things work". Just a requirement that your approach and write-up is scholarly.

STAGE 3: DOING THE RESEARCH

The project involves significant work alongside the courses and other potential distractions. You will need to learn to balance time effectively. You should meet your supervisor regularly, either individually or as a Team, typically weekly or at least fortnightly (barring absences etc). Your supervisor should make time for you if you need a 1 to 1 with them - if you find the balance of a team is proving problematic for you, bring that up sooner rather than later. Your work is your own, your supervisor is your advisor, but in general you ignore supervisory advice at your peril.

One key target of a research project is to get to the critical point (the point where you have the results you need to write something good up) sooner rather than later. This then gives you time to improve the work, consolidate it, or engage in most risky further developments.

STAGE 4: WRITEUP

The main project write up will be written up in your preferred format. Preferably this would be the format of a conference or journal you might choose to submit to, if it is a decent length (e.g. 9 page) conference format. The write up will likely involve additional supplementary material to describe the work in further detail. Second there is a reflection component (typically 5-8 double spaced pages), which will satisfy the additional learning requirements of the project (more is required here for the longer 80 credit and 100 credit project. See the degree programme tables for more information and the SUBMISSION section below). In all cases this reflection will state:

A delimitation of the general field of the project and the longer term goals of the project.
A broader literature review covering the wider relevant contributions to this field than those covered in the project write up, along with why they are relevant
A summary of how the specific project fits within this broader scope
A detailed breakdown and description of the work done by different individuals and their contributions (when the work was done as a team or with collaboration with others)
A precise statement of both the ML and the Systems contributions of the project
A summary of the future goals and plan
IMPORTANT: A reflection on issues of responsible research and innovation
CRITICAL: For larger projects (80 or 100 credits), a reflection on the experience of collaboration and/or how cross ml-systems stack integration can be achieved with future collaboration. For an 100 credit course, please also propose on how one might manage any future ML-Systems collaboration effectively? Note these are course requirements for the longer projects, and so will be a marking criterion.

Note that the main project paper can be identical across different individuals if this was team work. The reflection however must be an individual document, written by the student and submitted as all their own work. Please be aware of the University plagiarism regulations. If in any doubt please discuss with the supervisor and/or the CDT directorship.

I recommend all students at least take a look at my Rules for Writing, and ideally follow them. I recommend buying and reading the recommended book as early as possible in the PhD.

STAGE 5: SUBMISSION

The student must submit one pdf document which is a merge of two (or three) documents into 1 pdf:

A description of the work in approximately 9 pages of conference format not including references or appendices, along with whatever appendices you wish, all in a single pdf. If your preferred conference format has a separate supplementary material, please just join the two pdfs into one
A reflection document, as above

For the purpose of the project, these documents will be marked on the basis of the scholarship, not on the basis of the research. Have you been rigorous? have you been comprehensive? Is your project and writing well organised? Have you done a good amount of work? Is your writing clear and unambiguous? Have you justified design decisions? Have you critiqued your work and the work of others? Have you considered the responsible research elements of the work? Is your work properly contextualised in the backgroun d literature, and in the future plan?

Separate from this, the writeup will also form a basis for the annual PhD review, along with an annual review meeting. The criteria there will be different. In that we will be looking for evidence for:

An ability to undertake research, and an understanding of what research means
An understanding of the point of the work, the direction of future work, and the more immediate plan
That you have done sufficient work on the project
An ability to write up work with sufficient rigour
An ability to do experimental and computational work of sufficient level for the project
An ability to do more theoretical or mathematical work of sufficient level for the project
An understanding and knowledge of the background work in the field
A sufficiently organised, systematic and joined up approach to research
Progress in the project, the research and the understanding, with some novel research contribution
That advice or collaboration from others is sought and considered
An ability to present your work to an audience.

Where an element of this is missing, this will be discussed in the annual review with a plan made for its achievement. These annual review criteria are not explicitly considered in the project marking. The project marking focuses on scholarship. The research content is the subject of the PhD research project.

License