Super Resolution using Convolutional Neural Networks

This module contains several learning-based algorithms for upscaling an image.

Usage

Run the following command to build this module:

cmake -DOPENCV_EXTRA_MODULES_PATH=<opencv_contrib>/modules -Dopencv_dnn_superres=ON <opencv_source_dir>

Refer to the tutorials to understand how to use this module.

Models

There are four models which are trained.

EDSR

Trained models can be downloaded from here.

• Size of the model: ~38.5MB. This is a quantized version, so that it can be uploaded to GitHub. (Original was 150MB.)
• This model was trained for 3 days with a batch size of 16
• Link to implementation code: https://github.com/Saafke/EDSR_Tensorflow
• x2, x3, x4 trained models available
• Advantage: Highly accurate
• Disadvantage: Slow and large filesize
• Speed: < 3 sec for every scaling factor on 256x256 images on an Intel i7-9700K CPU.
• Original paper: Enhanced Deep Residual Networks for Single Image Super-Resolution [1]

ESPCN

Trained models can be downloaded from here.

• Size of the model: ~100kb
• This model was trained for ~100 iterations with a batch size of 32
• Link to implementation code: https://github.com/fannymonori/TF-ESPCN
• x2, x3, x4 trained models available
• Advantage: It is tiny and fast, and still performs well.
• Disadvantage: Perform worse visually than newer, more robust models.
• Speed: < 0.01 sec for every scaling factor on 256x256 images on an Intel i7-9700K CPU.
• Original paper: Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network [2]

FSRCNN

Trained models can be downloaded from here.

• Size of the model: ~40KB (~9kb for FSRCNN-small)
• This model was trained for ~30 iterations with a batch size of 1
• Link to implementation code: https://github.com/Saafke/FSRCNN_Tensorflow
• Advantage: Fast, small and accurate
• Disadvantage: Not state-of-the-art accuracy
• Speed: < 0.01 sec for every scaling factor on 256x256 images on an Intel i7-9700K CPU.
• Notes: FSRCNN-small has fewer parameters, thus less accurate but faster.
• Original paper: Accelerating the Super-Resolution Convolutional Neural Network [3]

LapSRN

Trained models can be downloaded from here.

• Size of the model: between 1-5Mb
• This model was trained for ~50 iterations with a batch size of 32
• Link to implementation code: https://github.com/fannymonori/TF-LAPSRN
• x2, x4, x8 trained models available
• Advantage: The model can do multi-scale super-resolution with one forward pass. It can now support 2x, 4x, 8x, and [2x, 4x] and [2x, 4x, 8x] super-resolution.
• Disadvantage: It is slower than ESPCN and FSRCNN, and the accuracy is worse than EDSR.
• Speed: < 0.1 sec for every scaling factor on 256x256 images on an Intel i7-9700K CPU.
• Original paper: Deep laplacian pyramid networks for fast and accurate super-resolution [4]

Benchmarks

Comparing different algorithms. Scale x4 on monarch.png (768x512 image).

Refer to the benchmarks located in the tutorials for more detailed benchmarking.

References

[1] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee, "Enhanced Deep Residual Networks for Single Image Super-Resolution", <i> 2nd NTIRE: New Trends in Image Restoration and Enhancement workshop and challenge on image super-resolution in conjunction with CVPR 2017. </i> [PDF] [arXiv] [Slide]

[2] Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A., Bishop, R., Rueckert, D. and Wang, Z., "Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network", <i>Proceedings of the IEEE conference on computer vision and pattern recognition</i> CVPR 2016. [PDF] [arXiv]

[3] Chao Dong, Chen Change Loy, Xiaoou Tang. "Accelerating the Super-Resolution Convolutional Neural Network", <i> in Proceedings of European Conference on Computer Vision </i>ECCV 2016. [PDF] [arXiv] [Project Page]

[4] Lai, W. S., Huang, J. B., Ahuja, N., and Yang, M. H., "Deep laplacian pyramid networks for fast and accurate super-resolution", <i> In Proceedings of the IEEE conference on computer vision and pattern recognition </i>CVPR 2017. [PDF] [arXiv] [Project Page]