|
|
|
|
|
|
|
|
|
|
Arias |
.png)
.png)
.png)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Arias |
|
|
|
|
|
|
Recently, MIMO (Multiple Input, Multiple Output) architectures has been proposed for the video restoration task. They have a better performance/running time trade-off than still single output architectures (MISO: Multiple Input, Multiple Output). In this paper, we focus on the three main state-of-the-art architectures for denoising in the low-latency setting (limited number of frames in the output stack, typically 5 or 7 frames). We show that the PSNR of MIMO networks is not uniform within the output stack and propose to use recurrence across stacks (RAS) and output stack overlap (OSO) to smooth
this non uniformity. See Figure (a). We also show that MIMO networks are temporally consistent within output stacks. However, they have very strong and unwanted changes at stack transitions. See Figure (b). The proposed contributions (shortened to ROSO when applied together) significantly reduce these changes. |
|
Project developed at the ENS Paris-Saclay, Centre Borelli and accepted at BMVC 2024. |
| MIMO (multiple input, multiple output) approaches are a recent trend in neural network architectures for video restoration problems, where each network evaluation produces multiple output frames. The video is split into non-overlapping stacks of frames that are processed independently, resulting in a very appealing trade-off between output quality and computational cost. In this work we focus on the low-latency setting by limiting the number of available future frames. We find that MIMO architectures suffer from problems that have received little attention so far, namely (1) the performance drops significantly due to the reduced temporal receptive field, particularly for frames at the boundaries of the stack, (2) there are strong temporal discontinuities at stack transitions which induce a step-wise motion artifact. We propose two simple solutions to alleviate these problems: recurrence across MIMO stacks to boost the output quality by implicitly increasing the temporal receptive field, and overlapping of the output stacks to smooth the temporal discontinuity at stack transitions. These modifications can be applied to any MIMO architecture. We test them on three state-of-the-art video denoising networks with different computational cost. The proposed contributions result in a new state-of-the-art for low-latency networks, both in terms of reconstruction error and temporal consistency. As an additional contribution, we introduce a new benchmark consisting of drone footage that highlights temporal consistency issues that are not apparent in the standard benchmarks. |
|
Our proposed ROSO mechanism. We propose two contributions to drastically alleviate the problems mentionned above: Output Stack Overlapping (OSO) and Recurrence Across Stack (RAS). In the three figures, the red dashed lines indicates the RAS. (a) Baseline MIMO network applied to two non-overlapping frame stacks (b) OSO: the output in the overlapped frames is computed as a weighted averafe of the two denoised versions. (c) OSO with RAS with a specific overlapped recurrence using the output overlapped frames from the previous stack. |
|
Visual results at stack transition. We display here the results of the baseline network and with our proposed RAS & OSO (abbreviated to ROSO). The network architecture is M2Mnet. We show the frames at stack transition t . We show the two last frames of one stack and the first frame of the next stack. Between them, we display the warping error. The contrast has been enhanced for better visualization. The strong transition is clearly visible with the baseline network. Using RAS slightly alleviate the transition, while ROSO does not show larger transition inside a stack (see here at time t-2 to t-1 ) than acros two stacks (transition t-1 to t ). |
| Video results of M2Mnet, BasicVSR++ and ReMoNet. In this video, we present in a split view: the clean ground-truth, the noisy sequence (AWGN σ = 40), the results of the baseline networks and the results of ours baseline+ROSO. The video is played twice, the second time with enhanced contrast to highlight the differences between the methods. The sequence is a crop from one of the videos of the proposed drone benchmark containing an urban scene. The step-wise motion effect induced by the stack transitions can be noticed in low contrast textures (e.g. building facades), and also in the motion of the vehicles. With the extensions proposed in the paper (+ROSO), the textures are much more stable and the cars on the road follow a more fluid motion. |
|
(left) PSNR vs running time landscape. The proposed ROSO improves over the current Pareto frontier by a significant increase in the PSNR
(right> inter-TC vs running time landscape. We used a metric based on the warping error designed to measure the temporal consistency at output stack transitions. This metric is called the inter-tc . Our contribution largely improves the inter-tc among low-latency networks. |
| Video drone dataset. Temporal consistency issues are masked in existing benchmark datasets. Consequently, we introduce a new evaluation dataset of 14 stabilized videos taken with drone-mounted cameras. This dataset features smooth motions to highlight temporal consistency issues. We hope that this will encourage research towards better restoration on stabilized video, which is a very relevant use case. For that, we make this validation dataset publicly available. |
 |
V. Dewil, Z. Zheng, A. Barral, L. Raad, N. Nicolas, I. Cassagne J-M. Morel, G. Facciolo B. Galerne and P. Arias Adapting MIMO video restoration networks to low latency constraints. In BMVC 2024. (hosted on ArXiv) |
Acknowledgements |