Video Codec with Deep Learning

High Efficiency Video Codec (HEVC)
  Electronics devices, such as cameras in cell phones or video camcorder, are evolving these days. As new digital devices come out, video compression technology in those devices are evolving day by day and also the resolution of the videos.  Therefore, video compression performance is improving accordingly.

  Currently, HEVC is the latest standard video codec. It has been improved 40% of performance compare to the before standard video, H.264. It uses the various size of blocks for coding efficiency which H.264 didn’t, and in-loop filters for removing compression artifacts. 


 

Video codec with CNN
  As CNN shows great performance in Super-Resolution problem, people started to apply it at removing compression artifacts like artifacts caused by JPEG or HEVC.

  There are many ways to insert CNN in HEVC with in-loop filtering or CU division and so on. Among those methods, we are trying to replace CNN to in-loop filter to improve coding efficiency. There are many researches that modified in-loop filter already and it shows the possibility to improve coding efficiency with CNN.

 

Current researches
  At first, IFCNN[1] shows some improvment in All-Intra(AI) mode with only three convolution layers. But the result was restricted with small resolution. Then, VRCNN[2] came up with the idea that HEVC uses various size of Coding Unit(CU). It showed better performance, but the experiment only held on AI mode. To get better result on inter-compression mode, other researchers used inter-compressed frames.

  These days, people want to make new network which works well with all of compression modes (like AI, LDP, and RA and so on) and high resolutions. Also, they are trying to train end-to-end network for video codec.

 

Reference
[1] Park, Woon-Sung, and Munchurl Kim. “CNN-based in-loop filtering for coding efficiency improvement.” Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), 2016 IEEE 12th. IEEE, 2016.
[2] Dai, Y., Liu, D., and Wu, F. A convolutional neural network approach for post-processing in hevc intra coding. In: International Conference on Multimedia Modeling. Springer, Cham, 2017. p. 28-39.

 

[이소윤, 홍진형, 오병태, “CNN 기반 HEVC 루프 필터의 성능 비교,” 한국방송미디어공학회 추계학술대회, 2017.]

MPLAB