Implicit neural representations (INR) has found successful applications across diverse
domains. To employ INR in real-life, it is important to speed up training. In the field of
INR for video applications, the state-of-the-art approach employs grid-type parametric
encoding and successfully achieves a faster encoding speed in comparison to its
predecessors. However, the grid usage, which does not consider the video’s dynamic nature,
leads to redundant use of trainable parameters. As a result, it has significantly lower
parameter efficiency and higher bitrate compared to NeRV-style methods that do not use a
parametric encoding. To address the problem, we propose Neural Video representation with
Temporally coherent Modulation (NVTM), a novel framework that can capture dynamic
characteristics of video. By decomposing the spatio-temporal 3D video data into a set of 2D
grids with flow information, NVTM enables learning video representation rapidly and uses
parameter efficiently. Our framework enables to process temporally corresponding pixels at
once, resulting in the fastest encoding speed for a reasonable video quality, especially
when compared to the NeRV-style method, with a speed increase of over 3 times. Also, it
remarks an average of 1.54dB/0.019 improvements in PSNR/LPIPS on UVG (Dynamic) (even with
10% fewer parameters) and an average of 1.84dB/0.013 improvements in PSNR/LPIPS on MCL-JCV
(Dynamic), compared to previous grid-type works. By expanding this to compression tasks, we
demonstrate comparable performance to video compression standards (H.264, HEVC) and recent
INR approaches for video compression. Additionally, we perform extensive experiments
demonstrating the superior performance of our algorithm across diverse tasks, encompassing
super resolution, frame interpolation and video inpainting.
