Summary: National Day tourist attractions are too crowded, the photos taken are all people, cars, cars, what should I do? May wish to try this black technology to make your travel vlog a sci-fi blockbuster in seconds.
This article is shared from the HUAWEI CLOUD community " National Day Travel Artifact, Magic Black Technology is a new creation, let vlog become a science fiction blockbuster in seconds! ", author: technical torchbearer.
When traveling on National Day, whether it is shooting people, sceneries or other things, the "sky" is a key element. For example, a picture of an unremarkable scene with the afterglow of the setting sun will give you a sense of atmosphere.
Of course, the natural landscape of the sky is not the coolest. Today, I will introduce to you a AI processing method based on native video , which can not only replace the sky background with one click, but also create any "city in the sky".
For example, changing to the vast starry sky and spaceships in "Star Trek", turning the mediocre vlog seconds that I shot easily into a sci-fi blockbuster, there is no sense of contradiction in the picture.
This method is derived from the open source project SkyAR on Github, which can automatically recognize the sky, then cut the sky out of the picture, and then replace the sky with the target sky, thereby realizing the magic sky.
Below, we will start from scratch based on SkyAR and ModelArts' JupyterLab As long as the brain is big enough, using this AI technology, you can create an infinite variety of gameplay.
This case can run under both CPU and GPU. It is estimated that it will take 9 minutes to run in the CPU environment and 2 minutes to run in the GPU environment.
Experiment goal
Learning through this case:
Understand the basic applications of image segmentation;
Understand the basic applications of motion estimation;
Understand the basic applications of image mixing.
Precautions
- If you are using JupyterLab for the first time, please refer to " ModelArts JupyterLab User Guide " to understand how to use it;
- If you encounter an error while using JupyterLab, please refer to " ModelArts JupyterLab Common Problem Solutions " to try to solve the problem.
Experimental steps
1. Install and import dependent packages
import os
import moxing as mox
file_name = 'SkyAR'
if not os.path.exists(file_name):
mox.file.copy('obs://modelarts-labs-bj4-v2/case_zoo/SkyAR/SkyAR.zip', 'SkyAR.zip')
os.system('unzip SkyAR.zip')
os.system('rm SkyAR.zip')
mox.file.copy_parallel('obs://modelarts-labs-bj4-v2/case_zoo/SkyAR/resnet50-19c8e357.pth', '/home/ma-user/.cache/torch/checkpoints/resnet50-19c8e357.pth')
INFO:root:Using MoXing-v1.17.3-43fbf97f
INFO:root:Using OBS-Python-SDK-3.20.7
!pip uninstall opencv-python -y
!pip uninstall opencv-contrib-python -y
Found existing installation: opencv-python 4.1.2.30
Uninstalling opencv-python-4.1.2.30:
Successfully uninstalled opencv-python-4.1.2.30
WARNING: Skipping opencv-contrib-python as it is not installed.
!pip install opencv-contrib-python==4.5.3.56
Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple
Collecting opencv-contrib-python==4.5.3.56
Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/3f/ce/36772cc6d9061b423b080e86919fd62cdef0837263f29ba6ff92e07f72d7/opencv_contrib_python-4.5.3.56-cp37-cp37m-manylinux2014_x86_64.whl (56.1 MB)
|████████████████████████████████| 56.1 MB 166 kB/s eta 0:00:01|█████▋ | 9.8 MB 9.4 MB/s eta 0:00:05 MB 9.4 MB/s eta 0:00:05███▏ | 26.6 MB 9.4 MB/s eta 0:00:04/s eta 0:00:03��██▍ | 35.8 MB 9.4 MB/s eta 0:00:03�███████████▌ | 42.9 MB 9.4 MB/s eta 0:00:02��██████████████▎ | 49.6 MB 166 kB/s eta 0:00:40
Requirement already satisfied: numpy>=1.14.5 in /home/ma-user/anaconda3/envs/PyTorch-1.4/lib/python3.7/site-packages (from opencv-contrib-python==4.5.3.56) (1.20.3)
Installing collected packages: opencv-contrib-python
Successfully installed opencv-contrib-python-4.5.3.56
WARNING: You are using pip version 20.3.3; however, version 21.1.3 is available.
You should consider upgrading via the '/home/ma-user/anaconda3/envs/PyTorch-1.4/bin/python -m pip install --upgrade pip' command.
cd SkyAR/
/home/ma-user/work/Untitled Folder/SkyAR
import time
import json
import base64
import numpy as np
import matplotlib.pyplot as plt
import cv2
import argparse
from networks import *
from skyboxengine import *
import utils
import torch
from IPython.display import clear_output, Image, display, HTML
%matplotlib inline
# 如果存在GPU则在GPU上面运行
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
INFO:matplotlib.font_manager:generated new fontManager2. Preview the original video
video_name = "test_videos/sky.mp4"
def arrayShow(img):
img = cv2.resize(img, (0, 0), fx=0.25, fy=0.25, interpolation=cv2.INTER_NEAREST)
_,ret = cv2.imencode('.jpg', img)
return Image(data=ret)
# 打开一个视频流
cap = cv2.VideoCapture(video_name)
frame_id = 0
while True:
try:
clear_output(wait=True) # 清除之前的显示
ret, frame = cap.read() # 读取一帧图片
if ret:
frame_id += 1
if frame_id > 200:
break
cv2.putText(frame, str(frame_id), (5, 15), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 1) # 画frame_id
tmp = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) # 转换色彩模式
img = arrayShow(frame)
display(img) # 显示图片
time.sleep(0.05) # 线程睡眠一段时间再处理下一帧图片
else:
break
except KeyboardInterrupt:
cap.release()
cap.release()
3. Preview the sky picture to be replaced
img= cv2.imread('skybox/sky.jpg')
img2 = img[:,:,::-1]
plt.imshow(img2)
<matplotlib.image.AxesImage at 0x7fbea986c590>
4. Customize training parameters
You can modify the following parameters according to your needs
skybox_center_crop: center offset of skybox
auto_light_matching: automatic brightness matching
relighting_factor: relighting
recoloring_factor: recoloring
halo_effect: halo effect
parameter = {
"net_G": "coord_resnet50",
"ckptdir": "./checkpoints_G_coord_resnet50",
"input_mode": "video",
"datadir": "./test_videos/sky.mp4",
"skybox": "sky.jpg",
"in_size_w": 384,
"in_size_h": 384,
"out_size_w": 845,
"out_size_h": 480,
"skybox_center_crop": 0.5,
"auto_light_matching": False,
"relighting_factor": 0.8,
"recoloring_factor": 0.5,
"halo_effect": True,
"output_dir": "./jpg_output",
"save_jpgs": False
}
str_json = json.dumps(parameter)
class Struct:
def __init__(self, **entries):
self.__dict__.update(entries)
def parse_config():
data = json.loads(str_json)
args = Struct(**data)
return args
args = parse_config()
class SkyFilter():
def __init__(self, args):
self.ckptdir = args.ckptdir
self.datadir = args.datadir
self.input_mode = args.input_mode
self.in_size_w, self.in_size_h = args.in_size_w, args.in_size_h
self.out_size_w, self.out_size_h = args.out_size_w, args.out_size_h
self.skyboxengine = SkyBox(args)
self.net_G = define_G(input_nc=3, output_nc=1, ngf=64, netG=args.net_G).to(device)
self.load_model()
self.video_writer = cv2.VideoWriter('out.avi',
cv2.VideoWriter_fourcc(*'MJPG'),
20.0,
(args.out_size_w, args.out_size_h))
self.video_writer_cat = cv2.VideoWriter('compare.avi',
cv2.VideoWriter_fourcc(*'MJPG'),
20.0,
(2*args.out_size_w, args.out_size_h))
if os.path.exists(args.output_dir) is False:
os.mkdir(args.output_dir)
self.output_img_list = []
self.save_jpgs = args.save_jpgs
def load_model(self):
# 加载预训练的天空抠图模型
print('loading the best checkpoint...')
checkpoint = torch.load(os.path.join(self.ckptdir, 'best_ckpt.pt'),
map_location=device)
self.net_G.load_state_dict(checkpoint['model_G_state_dict'])
self.net_G.to(device)
self.net_G.eval()
def write_video(self, img_HD, syneth):
frame = np.array(255.0 * syneth[:, :, ::-1], dtype=np.uint8)
self.video_writer.write(frame)
frame_cat = np.concatenate([img_HD, syneth], axis=1)
frame_cat = np.array(255.0 * frame_cat[:, :, ::-1], dtype=np.uint8)
self.video_writer_cat.write(frame_cat)
# 定义结果缓冲区
self.output_img_list.append(frame_cat)
def synthesize(self, img_HD, img_HD_prev):
h, w, c = img_HD.shape
img = cv2.resize(img_HD, (self.in_size_w, self.in_size_h))
img = np.array(img, dtype=np.float32)
img = torch.tensor(img).permute([2, 0, 1]).unsqueeze(0)
with torch.no_grad():
G_pred = self.net_G(img.to(device))
G_pred = torch.nn.functional.interpolate(G_pred,
(h, w),
mode='bicubic',
align_corners=False)
G_pred = G_pred[0, :].permute([1, 2, 0])
G_pred = torch.cat([G_pred, G_pred, G_pred], dim=-1)
G_pred = np.array(G_pred.detach().cpu())
G_pred = np.clip(G_pred, a_max=1.0, a_min=0.0)
skymask = self.skyboxengine.skymask_refinement(G_pred, img_HD)
syneth = self.skyboxengine.skyblend(img_HD, img_HD_prev, skymask)
return syneth, G_pred, skymask
def cvtcolor_and_resize(self, img_HD):
img_HD = cv2.cvtColor(img_HD, cv2.COLOR_BGR2RGB)
img_HD = np.array(img_HD / 255., dtype=np.float32)
img_HD = cv2.resize(img_HD, (self.out_size_w, self.out_size_h))
return img_HD
def process_video(self):
# 逐帧处理视频
cap = cv2.VideoCapture(self.datadir)
m_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
img_HD_prev = None
for idx in range(m_frames):
ret, frame = cap.read()
if ret:
img_HD = self.cvtcolor_and_resize(frame)
if img_HD_prev is None:
img_HD_prev = img_HD
syneth, G_pred, skymask = self.synthesize(img_HD, img_HD_prev)
self.write_video(img_HD, syneth)
img_HD_prev = img_HD
if (idx + 1) % 50 == 0:
print(f'processing video, frame {idx + 1} / {m_frames} ... ')
else: # 如果到达最后一帧
break5. Replace the sky
The output video after replacement is out.avi, and the before and after comparison video is compare.avi
sf = SkyFilter(args)
sf.process_video()
initialize skybox...
initialize network with normal
loading the best checkpoint...
processing video, frame 50 / 360 ...
processing video, frame 100 / 360 ...
no good point matched
processing video, frame 150 / 360 ...
processing video, frame 200 / 360 ...
processing video, frame 250 / 360 ...
processing video, frame 300 / 360 ...
processing video, frame 350 / 360 ... 6. Compare the original video and the replaced video
video_name = "compare.avi"
def arrayShow(img):
_,ret = cv2.imencode('.jpg', img)
return Image(data=ret)
# 打开一个视频流
cap = cv2.VideoCapture(video_name)
frame_id = 0
while True:
try:
clear_output(wait=True) # 清除之前的显示
ret, frame = cap.read() # 读取一帧图片
if ret:
frame_id += 1
cv2.putText(frame, str(frame_id), (5, 15), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 1) # 画frame_id
tmp = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) # 转换色彩模式
img = arrayShow(frame)
display(img) # 显示图片
time.sleep(0.05) # 线程睡眠一段时间再处理下一帧图片
else:
break
except KeyboardInterrupt:
cap.release()
cap.release()
If you want to generate your own video, just replace the sky.mp4 video in test_videos and the sky.jpg picture in skybox with your own video and picture, and then run it again with one click. Hurry up and give it a try to make your National Day blockbuster even more brilliant!
HUAWEI CLOUD community wishes everyone a happy National Day and a happy holiday!
appendix
This case comes from the HUAWEI CLOUD AI Gallery: 1615e62ba54733 magic black technology, can change the sky and
Click to follow and learn about Huawei Cloud's fresh technology for the first time~
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用。你还可以使用@来通知其他用户。