Rabiul Awal

Denoising Embedding Models Are Self-Supervised Learners

Rabiul Awal

Mila, Université de Montréal

Work in progress

A grid of 32 small generated images of animals, objects and scenes.
Uncurated images generated without labels, by one network trained only by denoising.

For the past two years, I have worked on generative world models for control. In my DRIFT project, I found that the features of a denoising model are less suited to control. One explanation is that reconstruction does not produce invariant, abstract features. This is also part of why Yann LeCun favors joint-embedding methods such as JEPA over generative models.

I was mostly fascinated and consumed by diffusion models, so I went looking for a diffusion-native solution to both representation and generation. In this project, a single denoising objective trains one network to do both. The network generates representations and pixels, so representation learning and generation share one model and one objective. In this view, self-supervised learning becomes a form of generative modeling.

Abstract

This paper shows that a generative model trained only by denoising can learn visual representations competitive with self-supervised learning and generate images without labels. Our approach is simple. In joint-embedding self-supervised learning, a network predicts the embedding of one view of an image from another view. We observe that this prediction is denoising at full noise, where the noisy embedding carries no information. We train the network on this view-conditioned denoising at all noise levels and call the result a denoising embedding model (DEM). A DEM is a diffusion model in representation space, with self-supervised prediction at its full-noise end. Without the view, the same network generates embeddings, and as a pixel denoiser it turns them into images. On ImageNet-100, DEM's linear probe exceeds that of DINOv2 at the same training length (76.1% against 72.8%). With a single network, it generates without labels at FID 26.1, against 22.9 for RCG, which trains three separate models. We hope this simple model will motivate rethinking self-supervised learning as a form of generative modeling.

These are current results on ImageNet-100 at 64x64 resolution. They may change as the work progresses.

Method

In joint-embedding learning, a network predicts the embedding of one view of an image from another view. Suppose the network also receives a noisy copy of the embedding it predicts. If the copy is pure noise, it carries no information, and the task reduces exactly to self-supervised prediction.

One network with three uses. A view of the image goes in and the network outputs its embedding, the representation. Given a noisy embedding, the network outputs the denoised embedding, and without the view it generates embeddings from noise. Given an embedding and noisy pixels, it outputs an image. A noise-level axis below runs from no noise to full noise, where denoising the embedding is self-supervised prediction. View x′ zτ Noisy embedding Noisy pixels, given ẑ One network Transformer Embedding, the representation ẑ Denoised embedding Image no noise full noise full noise = self-supervised prediction noise level of zτ
Denoising in embedding space turns joint-embedding prediction into a generative model.

We train the network on this view-conditioned denoising at all noise levels. Without the view, it samples embeddings as a representation generator, and as a pixel denoiser it turns them into images. Its features also serve as the image representation.

Results

Linear probe accuracy on ImageNet-100 against FID without labels. DEM has the highest probe and the second lowest FID. Single-task methods sit on the N/A lanes. N/A706050403020N/A4050607080 FID without labels (lower is better, right) Linear probe (%) DINOv2: probe 72.8%, FID N/ADINOv2BYOL: probe 70.1%, FID N/ABYOLI-JEPA: probe 39.7%, FID N/AI-JEPAMAE: probe 43.1%, FID N/AMAEJiT (uncond.): probe N/A, FID 64.0JiT (uncond.)RCG (DINOv2 enc.): probe N/A, FID 22.9RCG (DINOv2 enc.)MAGE: probe 35.4%, FID 67.9MAGEDEM: probe 76.1%, FID 26.1DEM
One generative model learns representations competitive with self-supervised methods and generates images without labels.
DEMBaseline
Linear probe76.1%72.8% (DINOv2)
FID-50k without labels26.122.9 (RCG, three models)

ImageNet-100, 64x64, current results.

One embedding, many images

The embedding is learned from views of the image. It can therefore stay invariant to color and framing, while the pixel denoiser reproduces these variations.

Four rows of eight generated images. Images in a row share content and differ in detail.
Each row decodes one sampled embedding with different pixel noise.

Completing a crop

Below full noise, samples spread more as the view shows less of the image. A small crop fits many images, so the embeddings sampled from it decode to different images.

Four rows. Each starts with an image where only a small boxed crop is visible, followed by four generated images that contain the crop's content.
The first column shows the crop in a blue box, and the others show images decoded from sampled embeddings.

In-painting

If we keep the visible pixels while sampling, DEM fills in a hidden region without being trained for this task.

Eight examples in two blocks. Each shows an image with its centre hidden, then two filled-in versions.
Each triple shows the input with its centre hidden and two samples. The examples are uncurated.

BibTeX

@misc{awal2026dem,
  title  = {Denoising Embedding Models Are Self-Supervised Learners},
  author = {Awal, Rabiul},
  year   = {2026}
}