IRR: Machine Learning

Machine Learning

Introduction

Machine Learning (ML) is about developing algorithms that adapt their behaviour to data, to provide useful representations or make predictions. Some ML research is driven by fundamental questions in artificial intelligence, some to solve practical problems in other research areas, while some attempts to make machine learning have a positive impact on society.

Some research in machine learning is deeply mathematical. For example, the field of computational learning theorystudies what learning is, when it is possible, and what formal guarantees we can attach to systems built with machine learning.

However, current theory is not sufficient to guide most choices made in building machine learning systems. New architectures and methods are usually tested empirically. Understanding whether an experimental comparison is fair, or really tests the questions asked can be challenging.

Unless your review is deeply mathematical, or on core methodology for well-established tasks with a clear route for evaluation, you are likely to have to consider other research areas. For example, if your methods are primarily motivated by an application area (e.g., computer vision, natural language processing, speech technology, forecasting, recommendations), some consideration of what matters in those areas would be part of any responsible review.

Are there go-to sources to find reviews and perspectives?

There isn’t one central place to find review papers. Foundations and Trends in Machine Learning contains some high-quality in-depth treatments, but it’s selection of topics is far from comprehensive. Once you know what sub-area you are interested in, you are more likely to find a relevant review article through a general web search, or seeing it cited in other papers.

One way to find some sub-areas that are interesting to review, and their current key questions, is to browse workshops. Workshops are usually smaller meetings than full conferences, on a narrower sub-field of research. The workshop meeting will often have short talks and posters, but there are often also short papers posted informally on the workshop’s website, and occasionally in more formal proceedings. These papers are often more preliminary than those published in conferences and journals.

The main machine learning conferences (see below) usually have satellite workshops associated with them. More people want to run workshops than there is space, so there is a competitive process. A list of workshops at one of these meetings gives some of the topics that some leaders in the field think are currently interesting and important.

Take the name of a conference (e.g., NeurIPS) and a particular year (e.g., 2020) then do a web search (e.g., “NeurIPS 2020 workshops”). You might also search for “topic-name workshop” (possibly also adding “machine learning” to your search). On a workshop’s website you might find a description of the field and/or a list of talks or papers that give you an indication of what was important that year. This strategy is to complement a traditional literature search, not to replace it.

What are the key journals or conferences in the field?

The top two general machine learning conferences are often said to be:

• NeurIPS, Neural Information Processing Systems, (called NIPS before 2018)
• ICML, International Conference on Machine Learning

Other top international conferences include:

• AAAI (“triple-A I”), Association for the Advancement of Artificial Intelligence. A more general AI conference, including good ML work.
• ICLR (“I-clear”), International Conference on Learning Representations. Narrower scope than NeurIPS and ICML, more-or-less just for “deep learning”.
• AISTATS, AI and Statistics. For research on the border with statistics.
• UAI, Uncertainty in Artificial Intelligence. Leans towards methods that capture and use uncertainty. More “computer sciencey” than AISTATS.
• COLT, Conference on Learning Theory. As in the name, more theoretical.

There is a lot of overlap between these meetings. Which conference a paper was submitted to was often driven by which deadline was closest! There is also a lot of randomness in acceptance (cf https://arxiv.org/abs/2109.09774), do not take venue as a reliable source of quality.

Some longer papers, some with earlier conference versions appear in The Journal of Machine Learning Research (JMLR). While there are other good journals, this one is being highlighted because it is run by the community, generally has good papers, and is free for all authors and readers.

A sister publication site Proceedings of Machine Learning Research (PMLR) hosts proceedings from some (but not all) machine learning conferences and workshops.

Are there any particular hi-profile or rapidly growing research areas?

If you find recent workshops you will find many ideas, that are often quite narrow. Do look at those, and see if there are any that you find interesting. Perhaps an application area you care about like healthcare or climate. The list here gives some broader areas, but is by no means exhaustive.

• As machine learning systems now impact society, research on these impacts has grown rapidly. There is theoretical and empirical work on fairness, ethics, and privacy in machine learning. Supporting this work is research on explainability and transparency – which can be helpful for deciding whether a system is fair and making it accountable.
• There is growing interest in making ML systems more robust when they are deployed in the real world. Predictions based on data gathered in a particular way, and in the past, may not work well in future, when a system is exposed to a greater variety of situations, and as the world changes. The language that’s used, and the focus of the research, can depend on the application area, which determines the sort of data and learning signals available. Some research will talk about robustness, mis-specification and distribution shift. The whole area of reinforcement learning also cares about being able to respond well in future when data was gathered in the past. There is also growing interest in causal reasoning, learning the underlying mechanisms that lead to an outcome, rather than just predicting what will happen. Causal reasoning promises to let us better predict the effect of our proposed actions or policy changes.
• Computer vision is currently seeing an increase in use of “unsupervised pre-training” or “self-supervised training”: fitting models based on tasks like image completion that don’t require expert labels. These models are then adapted, using a little supervision for particular tasks, such as object classification or segmentation. Much of the last decade in ML for computer vision has focussed on adapting models from “supervised” tasks like ImageNet classification. There are also new datasets and tasks emerging that reveal the limitations of just doing research on ImageNet.
• In contrast, Natural Language Processing (NLP) has been pre-training models based on unlabelled text for longer. There is still an explosion of interest in how to best use these large models for a range of real-world tasks, and mitigating the potential harms of doing so. Particularly challenging are low-resource tasks (languages or situations for which there is little data), which can be a useful test-bed for general ML research on transfer or multi-task learning.
• Computer vision and NLP are two of the application areas of machine learning methods that are most represented in machine learning meetings. They have also had different trajectories in the use of deep learning methods. Computer vision became a heavy user of deep learning after the ImageNet 2012 results. However, it took longer for deep learning to show large gains in natural language processing tasks – partly because words (with no representation learning) are already more useful features than raw pixels. Machine learning research for these two areas share many of the same methods, but often not found to be useful at the same time, and researchers often face different difficulties. It could be useful to review how a method has been used differently in these two (or other) areas, or whether there are methods that have become popular in one area that might be useful in another.
• Large-scale generative models that produce images (e.g. of faces) are often popular with students and the public, because we can immediately understand the task and what has been achieved. For several years there was an explosion of papers on GANs (Generative Adversarial Networks). In the last year, there have been really nice results on Score-Based Diffusion Models, which connect with other large-scale generative models (Continuous Time, Normalizing Flows). Other probabilistic models that could generate (and potentially explain) data include variational autoencoders (VAEs). All these models make different trade-offs. Some of them might be more appropriate in different applications.

Any other remarks or advice?

Periodically a machine learning paper will look back at many proposed variants of a popular machine learning architecture, and fail to reproduce much of an advantage from the variants. There’s also some folk wisdom that applied machine learning researchers should look at the latest papers, and then use the baselines that they compare to, rather than the latest method that the paper proposes. As another depressing example, there are manypapers on new optimization methods for deep learning, yet almost everyone seems to use SGD with momentum or ADAM; perhaps the optimization papers claiming their method is better doesn’t reproduce meaningfully for other people?

Don’t lurch towards being too cynical and dismissing everything you read; there has been a lot of real progress in machine learning! We really can do things now that would have seemed impossible a decade ago. However, results in a single paper may not convincingly prove a method. Look at whether the ideas make sense, whether the experiments seem to have made an effort to make a good comparison, and whether you can relate those results to those in other papers. Also see whether an idea has actually been adopted by anyone else.

Contributor: I Murray

Last update 13 October 2021