OOM Google Colab

[USERGAS]

MP4 files to be processed:
./input/video/023_klingai_reedit.mp4
Number of MP4 files: 1
Number of lines in the text file: 1
Processing video file: ./input/video/023_klingai_reedit.mp4 with prompt: A serene scene of a panda bear playing a guitar at sunset unfolds by a tranquil lake. The panda, with its black-and-white fur, strums the guitar while seated on a rock. Behind, a breathtaking mountain range glows under the orange and pink hues of the setting sun, contrasting beautifully with the lake's deep blue. The composition highlights the panda's relaxed interaction with the guitar, set against the stunning natural landscape, creating depth and peaceful harmony.
/usr/local/lib/python3.11/dist-packages/timm/models/layers/init.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {name} is deprecated, please import via timm.layers", FutureWarning)
2025-01-21 04:48:06.518097: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-01-21 04:48:06.547366: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-01-21 04:48:06.556982: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-01-21 04:48:06.586304: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-01-21 04:48:08.876819: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/usr/local/lib/python3.11/dist-packages/torchvision/datapoints/init.py:12: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: pytorch/vision#6753, and you can also check out pytorch/vision#7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(_BETA_TRANSFORMS_WARNING)
/usr/local/lib/python3.11/dist-packages/torchvision/transforms/v2/init.py:54: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: pytorch/vision#6753, and you can also check out pytorch/vision#7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(_BETA_TRANSFORMS_WARNING)
2025-01-21 04:48:11,270 - video_to_video - INFO - checkpoint_path: ./pretrained_weight/model.pt
2025-01-21 04:48:49,764 - video_to_video - INFO - Build encoder with FrozenOpenCLIPEmbedder
2025-01-21 04:49:56,160 - video_to_video - INFO - Load model path ./pretrained_weight/model.pt, with local status
2025-01-21 04:49:56,410 - video_to_video - INFO - Build diffusion with GaussianDiffusion
vae/config.json: 100% 609/609 [00:00<00:00, 3.88MB/s]
diffusion_pytorch_model.fp16.safetensors: 100% 196M/196M [00:08<00:00, 23.9MB/s]
2025-01-21 04:50:07,523 - video_to_video - INFO - Build Temporal VAE
2025-01-21 04:50:12,252 - video_to_video - INFO - input video path: ./input/video/023_klingai_reedit.mp4
2025-01-21 04:50:12,263 - video_to_video - INFO - text: A serene scene of a panda bear playing a guitar at sunset unfolds by a tranquil lake. The panda, with its black-and-white fur, strums the guitar while seated on a rock. Behind, a breathtaking mountain range glows under the orange and pink hues of the setting sun, contrasting beautifully with the lake's deep blue. The composition highlights the panda's relaxed interaction with the guitar, set against the stunning natural landscape, creating depth and peaceful harmony.
2025-01-21 04:50:12,655 - video_to_video - INFO - input fps: 15.0
2025-01-21 04:50:13,168 - video_to_video - INFO - input resolution: (240, 426)
2025-01-21 04:50:13,168 - video_to_video - INFO - target resolution: (960, 1704)
2025-01-21 04:50:13,287 - video_to_video - INFO - video_data shape: torch.Size([78, 3, 960, 1704])
2025-01-21 04:52:54,039 - video_to_video - INFO - step: 0
Traceback (most recent call last):
File "/content/STAR/./video_super_resolution/scripts/inference_sr.py", line 137, in
main()
File "/content/STAR/./video_super_resolution/scripts/inference_sr.py", line 133, in main
star.enhance_a_video(input_path, prompt)
File "/content/STAR/./video_super_resolution/scripts/inference_sr.py", line 73, in enhance_a_video
output = self.model.test(data_tensor, total_noise_levels, steps=self.steps,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/content/STAR/video_to_video/video_to_video_model.py", line 110, in test
gen_vid = self.diffusion.sample_sr(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/content/STAR/video_to_video/diffusion/diffusion_sdedit.py", line 410, in sample_sr
x0 = solver_fn(
^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/content/STAR/video_to_video/diffusion/solvers_sdedit.py", line 172, in sample_dpmpp_2m_sde
denoised = model(x * c_in, sigmas[i], variant_info=variant_info)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/content/STAR/video_to_video/diffusion/diffusion_sdedit.py", line 343, in model_chunk_fn
x0_chunk = self.denoise(xt_chunk, t, None, model, model_kwargs, guide_scale,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/content/STAR/video_to_video/diffusion/diffusion_sdedit.py", line 81, in denoise
y_out = model(xt, t=t, **model_kwargs[0], **model_kwargs[2], variant_info=variant_info)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/content/STAR/video_to_video/modules/unet_v2v.py", line 1746, in forward
control = self.VideoControlNet(x, t, y, hint=hint, t_hint=t_hint,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/content/STAR/video_to_video/modules/unet_v2v.py", line 2189, in forward
x = self._forward_single(block, x, e, context, time_rel_pos_bias,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/content/STAR/video_to_video/modules/unet_v2v.py", line 2238, in _forward_single
x = module(x, context)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/fairscale/nn/checkpoint/checkpoint_activations.py", line 171, in _checkpointed_forward
return original_forward(module, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/content/STAR/video_to_video/modules/unet_v2v.py", line 1058, in forward
x = block(x, h=h, w=w)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/content/STAR/video_to_video/modules/unet_v2v.py", line 483, in forward
x = self.attn1(
^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/content/STAR/video_to_video/modules/unet_v2v.py", line 161, in forward
k = self.to_k(context)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 824.00 MiB (GPU 0; 14.75 GiB total capacity; 12.68 GiB already allocated; 793.06 MiB free; 13.78 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
All videos processed successfully.

[kirikorneev]

I have the same issue, the free T4 GPU gets out of memory :(
I have also tried checking out clarin-ebtio800090's fork and upscaling only to the factor of 2

[yhliu04]

Hi @kirikorneev, @USERGAS,

The free T4 GPU provided by Colab may not meet the VRAM requirements. We recommend using a paid GPU (preferably A100) to ensure proper execution.