Feedback on 3.2 Compression via Denoising

[DongliLiu]

This part is so important that I have to raise my feedback.

I would like to propose a major revision to Section 3.2 (“Seeking Low-Dimensional Distributions through Denoising”) to improve its clarity and pedagogical flow.
The current version contains 98 numbered equations (from 3.2.1 to 3.2.98), which makes the core ideas hard to follow, especially for beginners. The numbering is also confusing—there are multiple “3.2” headings at different levels, and the logical thread is obscured.

My suggestion is to restructure the section as follows:

1. Move many of the detailed derivations and technical lemmas to an appendix. The sheer number of equations (98) is overwhelming; only the most essential ones should remain in the main text.

2. Fix the numbering scheme. Currently, there are several instances of duplicate “3.2” headings (e.g., both the section title and some subsections are numbered 3.2). A clearer hierarchical numbering (e.g., 3.2.1, 3.2.2, …) should be used consistently.

3. Make the core logical flow explicit. The essential chain of ideas is:

   step1: Core diffusion equation (e.g., Eq. (3.2.1) / (3.2.85))

step2: Tweedie’s formula (Eq. (3.2.23) / (3.2.86))

step3: DDIM / iterative denoising update (Eq. (3.2.82) / (3.2.87))

step4: Training objective (Eq. (3.2.90))

This sequence could be highlighted in a box or a diagram, and the surrounding text should explicitly guide the reader along this path.

4. Include a complete, worked-out example that runs through the entire process. For instance, a simple Gaussian mixture or a toy image dataset could be used to illustrate the forward diffusion, the learned denoiser, and the reverse sampling steps. This would greatly help readers connect the abstract equations to a concrete implementation.

I believe these changes would make Section 3.2 much more accessible and would better serve the book’s goal of providing a principled yet practical introduction to representation learning.

[DruvPai]

Thanks for the feedback!

Move many of the detailed derivations and technical lemmas to an appendix. The sheer number of equations (98) is overwhelming; only the most essential ones should remain in the main text.

I think we will not be doing this: our goal is to give an introduction to the mathematical foundations of the material, and we will only postpone to the appendices content which is either (a) preliminary, (b) generally beyond the ken of the intended audience (cf covering number arguments which are indeed postponed to Appendix B). The material in Ch3 is neither, we took a lot of trouble to make sure it was accessible to people with basic linear algebra, calculus, and probability knowledge.

Fix the numbering scheme. Currently, there are several instances of duplicate “3.2” headings (e.g., both the section title and some subsections are numbered 3.2). A clearer hierarchical numbering (e.g., 3.2.1, 3.2.2, …) should be used consistently.

I do not see this duplicate numbering on either the PDF or the website, can you point it out? Thanks! Indeed this would be a serious issue that we would fix.

Make the core logical flow explicit. The essential chain of ideas is: ...

I don't see exactly what you mean. Do you mean that we should highlight the dependency graph? The idea of our book/chapter is not necessarily to give a minimal presentation of diffusion models but rather show how it connects to the rest of the ideas throughout the book (for which much of this exposition is required).

Include a complete, worked-out example that runs through the entire process. For instance, a simple Gaussian mixture or a toy image dataset could be used to illustrate the forward diffusion, the learned denoiser, and the reverse sampling steps. This would greatly help readers connect the abstract equations to a concrete implementation.

We introduce Gaussian/Gaussian mixture diffusion and include it several times as a running example (see Example 3.3, Example 3.5, everything below eq (3.2.69).) We believe this ends up better than combining all of these instances into one big example and leaving readers bereft of them for large parts of the chapter :)

[DongliLiu]

OK, I got your point, anyway how to write a book is your decision :)

For the duplicate numbering issue, yes after I raised this I found it happened on the Chinese version on Website only, e.g. :

• Theorem 3.1 (Simplified version of Theorem [B.2] -> 3.1 ( [B.2]的简化版)
• Example 3.2 (Adding noise to a mixture of Gaussians) -> 3.2 (向高斯混合添加噪声)

[DruvPai]

Oh thanks for pointing out the numbering issue! The fix turns out to not be trivial, I will play around with it.