Notes

Last updated Jun 23, 2022

• Type note
• Status good
• Source article
• On quantum computing machine learning

Metadata

• Source:: Why measuring performance is our biggest blind spot in quantum machine learning (pennylane.ai)
• Author:: Maria Schuld
• Publish Date:: 2022-03-07
• Review Date:: 2022-03-09

• A major portion of QML research is driven by whether quantum hardware will be useful for ML when it is available at scale.
• There is a lack of a suitable framework to study what makes a quantum algorithm “good” for ML.
• Further detail on this topic can be found in 2022 Is quantum advantage the right goal for quantum machine learning.

• Two most wide-spread measures in QML
  • Running Benchmarks
    • Show with simulations that model learns well.
    • Adopted from ML.
  • Proving asymptotic runtime advantage
    • Show that models from ansatz are classically intractable.
    • Adopted from QC.
• Other emerging measures
  • Expressivity or generality of a computation.
  • Quantum model emulates a popular Machine Learning concept.
• Looking closely at these, one can easily see some major flaws with these approaches.

• Adopted from Machine Learning, benchmarks
  • typically measure the error a trained model makes on a test dataset.
  • indicate generalization power of the model.
• But even in classical ML, they pose some challenges (see the panel discussion The Role of Benchmarks in the Scientific Progress of Machine Learning at NeurIPS 2021) such as
  • what is a fair comparison
  • how to control the impact of the choices for hyperparameters
  • how to avoid overfitting a single dataset over the years
• In case of QML, unlike classical ML, we currently don’t have any hardware or simulation capability to run benchmarks on a relevant scale.
• We fix this problem by
  • choosing tiny datasets and few-qubit experiments
  • proposing hybrid classical-quantum setups
  • downscaling the data
  • focusing on data that can be classically emulated
• In most cases, we don’t know the effects of these limitations on the results.
• So, when using benchmarks
  • think very carefully about the scope of the imposed limitations
  • put more effort into drawing the right conclusions from the results
• Some good things to think about:
  • Does the conclusion really lie in the scope of the benchmark?
  • What was the quantum model compared to, a classical neural net with similar parameters, or similar expressive power, or a similar training time?
  • Was the data down-scaled by another algorithm, and how does that influence the result?
  • How robust is the outcome when we change the ansatz or hyperparameters in training?
  • What settings break the model?
• We should focus more on the conscious use of benchmarks instead of trying to make QML look good as compared to ML.

• We adopted the notion of asymptotic runtime complexity as a figure of merit from the quantum computing research.
• Some arguments as to why this isn’t a right figure of merit for ML
  • Machine Learning problems are messy.
    • This makes it genuinely difficult for complexity analysis as we often do not know a lot about the general mathematical structure of the data.
    • Often, a problem has to be reduced to an absurd minimum to make it accessible to complexity considerations.
  • Lack of comprehensive theoretical proofs for the effective runtime of classical algorithms.
    • Training neural networks means solving a non-convex optimization problem that is in principle Class NP-Hard.
  • Most machine learning algorithms already take only linear time in the number of data in practice.
    • There is not much room for speedups.
    • Sublinear algorithms require assumptions that circumvent the linear-time process of loading the data.
  • Only considering what is provable can be severely limiting.
    • This hinders our ability to explore new and exciting domains/applications.
• There may be cases where runtime complexity may be the right figure of merit, but those applications are very very far in the future.

• Expressivity is often used when considering a family of quantum models defined by a parameterized quantum circuit (whose internal design is fixed by an ansatz).
• In this setting, Expressivity can refer to two very different concepts
  • How large is the class of unitaries that an ansatz can express?
  • How large is the class of models that the quantum circuit can express?
• More expressivity is not always better, because:
  • A very expressive parametrized ansatz is hard to train.
    • This is an important result in the literature on “barren plateaus”.
  • Expressive model classes can overfit.
  • Theoretical statements about large classes of models tend to be loose.
  • Theoretical statements about large classes of models can even be misleading.
    • For example, for Quantum Kernels, researchers warned that this method cannot scale.
      • For large qubit numbers, two quantum states have a high probability of having a zero overlap, and the distance measure will become useless.
• Expressivity is an interesting measure, but it is not something we should blindly maximize.

• What works in classical ML does not have to work when combined with quantum circuits.
  • Overparametrized deep quantum circuits trained by stochastic gradient descent could be useless on large scales, unlike in Classical ML.
  • Neural tangent kernels help us to understand deep learning, but they may not help us to understand quantum machine learning.
  • Perceptrons are the building blocks of neural networks, but they may not be a suitable mathematical mechanism to build models from quantum theory.
• The right figure of merit here should not be the emulation of classical methods, but targeting the crucial principles that lead to the success of classical methods.
• So, we should not fall prey to spurious beliefs that adding “quantum” to fancy classical terms is a valid argument for good performance.

• When reading a paper look for arguments used by the authors when advertising their methods
  • Don’t just accept them blindly.
  • Ask which culture of science they come from
    • if they make sense in the study
  • Find out if the authors discuss their limitations convincingly.
• When reviewing a paper
  • acknowledge how hard it is to design a study that is aware of methodological issues.
  • reward those who make an effort to ask uncomfortable questions about their own work.
• When writing a paper
  • be courageous to focus on small but well-posed questions that help our understanding.
  • resist the pressure to sell quantum machine learning.
  • consider investigation designs that try to break or test our methods.
  • be clear about your personal goal
    • the papers that will be relevant for quantum machine learning in 5 years’ time may be very different from the ones that make it into prestigious journals today.
• Thinking about how to define good QML is crucial for innovation in the field.