Bamboozeled by Gaussian Processes

For the past 2 years or so I've been trying to get comfortable with Gaussian processes (GPs). I thought I understood GPs, especially after working through a basic example. The problem is in that example I used an external Julia package called GaussianProcesses.jl and recently I've been trying to implement GPs from scratch for multivariate cases¹.

Usually, the application of GPs is straightforward because it's just the construction of the covariance matrix via the chosen kernel (which does have some hyperparameters to tune). What's nice is that the kernel trick handles the scenario for any dimension feature input, $k:{\mathbb{R}}^d\times {\mathbb{R}}^d\to \mathbb{R}$. But for some reason, this confuses me in the programming of it, I just need to stop second-guessing myself. Also, the tuning of the kernel parameters seems not to be that trivial.

To alleviate my frustrations in feeling like I don't understand GPs as well as I would like to, I'm going to be spending the weekend reading through:

• R. Garnett, Bayesian Optimization, Cambridge University Press, 2023.
• D. Packwood, Bayesian Optimization for Materials Science, Springer, Singapore, 2017. https://doi.org/10.1007/978-981-10-6781-5.

Both are excellent texts. The one by R. Garnett is particularly complete and ideal for self-learning. I do wish it had some code (from scratch) to go along with the book. I would say the book is as good as Gaussian Processes for Machine Learning, but as the title suggest, it is focused on Bayesian optimization.

The book by D. Packwood addresses the lack of hands-on examples in Garnett's book because Ch. 2 goes through every detail to implement the GPs prior used in the Bayesian optimization and along with the companion R code. A while back I reworked the R code in Julia; I'll revisit this for sure.

I actually really love the book by D. Packwood because it's concise and directly applicable to my domain area. It's a bit expensive for how brief it is (42 pages), but worth it. The best part is you can read it over and over again because it is so short, making a great reference. It does assume a good foundation probability and at times the symbol choice and nomenclature are confusing or unclear.

Credit Kat Bailey

There are probably also tons of good talks on GPs. I found this one by Neil D. Lawrence which seems very detailed and in-depth. Not to mention very clean slide deck. There is also this quick and simple implementation from scratch in Python by Katherine Bailey. I particularly like her meme

Footnotes

---

1. I always get this confused, but when I say multivariate I think of multiple inputs single output, where multivariable, is multiple inputs and outputs. ↩

2. You can actually access Ch. 2 for free from the publisher in the provided product sample file here. ↩

Reuse and Attribution

Bringuier, S., Bamboozeled by Gaussian Processes, Dirac's Student, (2023). Retrieved from https://www.diracs-student.blog/2023/07/bamboozeled-by-gaussian-processes.html.

  @misc{Bringuier_20JUL2023,
  title        = {Bamboozeled by Gaussian Processes},
  author       = {Bringuier, Stefan},
  year         = 2023,
  month        = jul,
  url          = {https://www.diracs-student.blog/2023/07/}# 
                 {bamboozeled-by-gaussian-processes.html},
  note         = {Accessed: 2025-04-15},
  howpublished = {Dirac's Student [Blog]},
  }