Want to know what your baby will look like in the future?

Abstract: This case can generate a photo of a child based on a photo of the front face of a father and mother, and can adjust the parameters to see the looks of children of different genders and ages.

This article is shared from the HUAWEI CLOUD community " BabyGAN: Generate Child Photos Based on Parent Photos", author: Shanhaizhiguang.

In this case, a photo of the front face of a father and mother can be used to generate a photo of the child, and the parameters can be adjusted to see the looks of children of different genders and ages.

In order to ensure the effectiveness of the photos, when uploading photos of parents, try to upload photos that show the facial features and have a light background.

This case is only used for learning and communication, please do not use it for other purposes.

In addition, due to imperfect technology, the generated child photos may be distorted or distorted. You can replace different parent photos and regenerate the child photos until a satisfactory result is achieved.

Let's start running this case step by step.

1. Install the required modules

This step takes about 4 minutes

!pip install imutils moviepy dlib

2. Download code and model files

import os
import moxing as mox

root_dir = '/home/ma-user/work/ma_share/'
code_dir = os.path.join(root_dir, 'BabyGAN')
if not os.path.exists(os.path.join(root_dir, 'BabyGAN.zip')):
    mox.file.copy('obs://arthur-1/BabyGAN/BabyGAN.zip', os.path.join(root_dir, 'BabyGAN.zip'))
    os.system('cd %s; unzip BabyGAN.zip' % root_dir)

os.chdir(code_dir)

3. Load related modules and models

import cv2
import math
import pickle
import imageio
import warnings
import PIL.Image
import numpy as np
from glob import glob
from PIL import Image
import tensorflow as tf
from random import randrange
import moviepy.editor as mpy
import matplotlib.pyplot as plt
from IPython.display import clear_output
from moviepy.video.io.ffmpeg_writer import FFMPEG_VideoWriter

import config
import dnnlib
import dnnlib.tflib as tflib
from encoder.generator_model import Generator

%matplotlib inline
warnings.filterwarnings("ignore")

Load the model file. This code block can only be executed once. If an error occurs, please restart the kernel and re-run all the code

tflib.init_tf()
URL_FFHQ = "./karras2019stylegan-ffhq-1024x1024.pkl"
with dnnlib.util.open_url(URL_FFHQ, cache_dir=config.cache_dir) as f:
    generator_network, discriminator_network, Gs_network = pickle.load(f)
generator = Generator(Gs_network, batch_size=1, randomize_noise=False)
model_scale = int(2 * (math.log(1024, 2) - 1))
age_direction = np.load('./ffhq_dataset/latent_directions/age.npy')
horizontal_direction = np.load('./ffhq_dataset/latent_directions/angle_horizontal.npy')
vertical_direction = np.load('./ffhq_dataset/latent_directions/angle_vertical.npy')
eyes_open_direction = np.load('./ffhq_dataset/latent_directions/eyes_open.npy')
gender_direction = np.load('./ffhq_dataset/latent_directions/gender.npy')
smile_direction = np.load('./ffhq_dataset/latent_directions/smile.npy')
def get_watermarked(pil_image: Image) -> Image:
    try:
        image = cv2.cvtColor(np.array(pil_image), cv2.COLOR_RGB2BGR)
        (h, w) = image.shape[:2]
        image = np.dstack([image, np.ones((h, w), dtype="uint8") * 255])
        pct = 0.08
        full_watermark = cv2.imread('./media/logo.png', cv2.IMREAD_UNCHANGED)
        (fwH, fwW) = full_watermark.shape[:2]
        wH = int(pct * h * 2)
        wW = int((wH * fwW) / fwH * 0.1)
        watermark = cv2.resize(full_watermark, (wH, wW), interpolation=cv2.INTER_AREA)
        overlay = np.zeros((h, w, 4), dtype="uint8")
        (wH, wW) = watermark.shape[:2]
        overlay[h - wH - 10: h - 10, 10: 10 + wW] = watermark
        output = image.copy()
        cv2.addWeighted(overlay, 0.5, output, 1.0, 0, output)
        rgb_image = cv2.cvtColor(output, cv2.COLOR_BGR2RGB)
        return Image.fromarray(rgb_image)
    except:
        return pil_image
def generate_final_images(latent_vector, direction, coeffs, i):
    new_latent_vector = latent_vector.copy()
    new_latent_vector[:8] = (latent_vector + coeffs * direction)[:8]
    new_latent_vector = new_latent_vector.reshape((1, 18, 512))
    generator.set_dlatents(new_latent_vector)
    img_array = generator.generate_images()[0]
    img = PIL.Image.fromarray(img_array, 'RGB')
    if size[0] >= 512: img = get_watermarked(img)
    img_path = "./for_animation/" + str(i) + ".png"
    img.thumbnail(animation_size, PIL.Image.ANTIALIAS)
    img.save(img_path)
    face_img.append(imageio.imread(img_path))
    clear_output()
    return img
def generate_final_image(latent_vector, direction, coeffs):
    new_latent_vector = latent_vector.copy()
    new_latent_vector[:8] = (latent_vector + coeffs * direction)[:8]
    new_latent_vector = new_latent_vector.reshape((1, 18, 512))
    generator.set_dlatents(new_latent_vector)
    img_array = generator.generate_images()[0]
    img = PIL.Image.fromarray(img_array, 'RGB')
    if size[0] >= 512: img = get_watermarked(img)
    img.thumbnail(size, PIL.Image.ANTIALIAS)
    img.save("face.png")
    if download_image == True: files.download("face.png")
    return img
def plot_three_images(imgB, fs=10):
    f, axarr = plt.subplots(1, 3, figsize=(fs, fs))
    axarr[0].imshow(Image.open('./aligned_images/father_01.png'))
    axarr[0].title.set_text("Father's photo")
    axarr[1].imshow(imgB)
    axarr[1].title.set_text("Child's photo")
    axarr[2].imshow(Image.open('./aligned_images/mother_01.png'))
    axarr[2].title.set_text("Mother's photo")
    plt.setp(plt.gcf().get_axes(), xticks=[], yticks=[])
    plt.show()

4. Prepare photos of father and mother

In this case, a default parent photo has been prepared. In the file resource management window on the left sidebar, enter the ma_share/BabyGAN directory, and then enter the father_image or mother_image directory to see the provided parent photo ,As shown below:

If you need to change the photos of your parents, please check section 11 "Replace the photos of father and mother" in this article

if len(glob(os.path.join('./father_image', '*.jpg'))) != 1 or (not os.path.exists('./father_image/father.jpg')):
    raise Exception('请在 ma_share/BabyGAN/father_image 目录下准备一张父亲的照片，且命名为father.jpg')

if len(glob(os.path.join('./mother_image', '*.jpg'))) != 1 or (not os.path.exists('./mother_image/mother.jpg')):
    raise Exception('请在 ma_share/BabyGAN/father_image 目录下准备一张母亲的照片，且命名为mother.jpg')

5. Get the father's face area and align the face

!python align_images.py ./father_image ./aligned_images

Look at the father's face

if os.path.isfile('./aligned_images/father_01.png'):
    pil_father = Image.open('./aligned_images/father_01.png')
    (fat_width, fat_height) = pil_father.size
    resize_fat = max(fat_width, fat_height) / 256
    display(pil_father.resize((int(fat_width / resize_fat), int(fat_height / resize_fat))))
else:
    raise ValueError('No face was found or there is more than one in the photo.')

6. Obtain the mother's face area and align the face

!python align_images.py ./mother_image ./aligned_images

View mother's face

if os.path.isfile('./aligned_images/mother_01.png'):
    pil_mother = Image.open('./aligned_images/mother_01.png')
    (mot_width, mot_height) = pil_mother.size
    resize_mot = max(mot_width, mot_height) / 256
    display(pil_mother.resize((int(mot_width / resize_mot), int(mot_height / resize_mot))))
else:
    raise ValueError('No face was found or there is more than one in the photo.')

7. Extract facial features

This step takes about 3 minutes

!python encode_images.py \
    --early_stopping False \
    --lr=0.25 \
    --batch_size=2 \
    --iterations=100 \
    --output_video=False \
    ./aligned_images \
    ./generated_images \
    ./latent_representations

if len(glob(os.path.join('./generated_images', '*.png'))) == 2:
    first_face = np.load('./latent_representations/father_01.npy')
    second_face = np.load('./latent_representations/mother_01.npy')
    print("Generation of latent representation is complete! Now comes the fun part.")
else:
    raise ValueError('Something wrong. It may be impossible to read the face in the photos. Upload other photos and try again.')

8. Generate photos of a family of three

Please modify the gender_influence and person_age parameters in the code below,

gender_influence: gender influence factor, the value range is [0.01, 0.99], the closer the value is to 0, the greater the influence of the father's appearance, and vice versa, the greater the influence of the mother;

person_age: Age influence factor, value range [10, 50], after setting this value, the face of the child of the corresponding age will be generated.

Every time you modify the parameter value, re-run the following code block to generate a new photo of the child

genes_influence = 0.8 # Gender influence factor, the value range is [0.01, 0.99], the closer the value is to 0, the greater the influence of the father’s appearance, and vice versa the greater the influence of the mother
person_age = 10 # Age influence factor, the value range is [10, 50], after setting this value, the appearance of the child of the corresponding age will be generated

style = "Default"
if style == "Father's photo":
    lr = ((np.arange(1, model_scale + 1) / model_scale) ** genes_influence).reshape((model_scale, 1))
    rl = 1 - lr
    hybrid_face = (lr * first_face) + (rl * second_face)
elif style == "Mother's photo":
    lr = ((np.arange(1, model_scale + 1) / model_scale) ** (1 - genes_influence)).reshape((model_scale, 1))
    rl = 1 - lr
    hybrid_face = (rl * first_face) + (lr * second_face)
else:
    hybrid_face = ((1 - genes_influence) * first_face) + (genes_influence * second_face)

intensity = -((person_age / 5) - 6)
resolution = "512"
size = int(resolution), int(resolution)

download_image = False
face = generate_final_image(hybrid_face, age_direction, intensity)
plot_three_images(face, fs=15)

9. Check the appearance of children at all ages

Please modify the gender_influence parameter in the code below. This parameter is a gender influence factor. The value range is [0.01, 0.99]. The closer the value is to 0, the greater the influence of the father's appearance, and vice versa.

Every time you modify the parameter value, you must re-run the following code block

gender_influence = 0.8  # 性别影响因子，取值范围[0.01, 0.99]，取值越接近0，父亲的容貌影响越大，反之母亲影响越大

!rm -rf ./for_animation
!mkdir ./for_animation
face_img = []
hybrid_face = ((1 - gender_influence) * first_face) + (gender_influence * second_face)
animation_resolution = "512"
animation_size = int(animation_resolution), int(animation_resolution)
frames_number = 50
download_image = False
for i in range(0, frames_number, 1):
    intensity = (8 * (i / (frames_number - 1))) - 4
    generate_final_images(hybrid_face, age_direction, intensity, i)
    clear_output()
    print(str(i) + " of {} photo generated".format(str(frames_number)))

for j in reversed(face_img):
    face_img.append(j)

automatic_download = False

if gender_influence <= 0.3:
    animation_name = "boy.mp4"
elif gender_influence >= 0.7:
    animation_name = "girl.mp4"
else:
    animation_name = "animation.mp4"

imageio.mimsave('./for_animation/' + animation_name, face_img)
clear_output()
display(mpy.ipython_display('./for_animation/' + animation_name, height=400, autoplay=1, loop=1))

10. Check the faces of children of different genders

Please modify the person_age parameter in the code below. This parameter is an age influence factor, with a value range of [10, 50]. After setting this value, the appearance of a child of the corresponding age will be generated.

Every time you modify the parameter value, you must re-run the following code block

person_age = 10  # 小孩的年龄，取值范围[10, 50]，设置该值后，将生成对应年龄的小孩的容貌

!rm -rf ./for_animation
!mkdir ./for_animation
face_img = []
intensity = -((person_age / 5) - 6)
animation_resolution = "512"
animation_size = int(animation_resolution), int(animation_resolution)
frames_number = 50  # 容貌变化的图像数，取值范围[10, 50]
download_image = False

for i in range(1, frames_number):
    gender_influence = i / frames_number
    hybrid_face = ((1 - gender_influence) * first_face) + (gender_influence * second_face)
    face = generate_final_images(hybrid_face, age_direction, intensity, i)
    clear_output()
    print(str(i) + " of {} photo generated".format(str(frames_number)))

for j in reversed(face_img):
    face_img.append(j)

animation_name = str(person_age) + "_years.mp4"
imageio.mimsave('./for_animation/' + animation_name, face_img)
clear_output()
display(mpy.ipython_display('./for_animation/' + animation_name, height=400, autoplay=1, loop=1))

11. Change photos of father and mother

Next, you can upload photos of your parents that you are interested in to the father_image and mother_image directories, and re-run the code to generate new photos of your children.

You need to follow the following rules and steps:

1. Refer to the operation in the figure below and enter the ma_share/BabyGAN directory;

2. Prepare a father of photos, uploaded to father_image directory name must be father.jpg; (if you do not know the method to upload files to JupyterLab, please check the documentation )

3. Prepare a picture of your mother and upload it to the mother_image directory. The name must be mother.jpg;

4. Only one photo is allowed in both father_image and mother_image directories;

5. Re-run the code in steps 4~10.

Click to follow, and learn about Huawei Cloud's fresh technology for the first time~