Easily build web applications with Flutter and Firebase

Author/ Very Good Ventures Team

We (Very Good Ventures team) cooperated with Google to launch photo booth interactive experience (I/O Photo Booth) at this year's Google I/O conference. You can take photos with the beloved Google mascots: Flutter’s Dash , Android Jetpack, Chrome’s Dino and Firebase’s Sparky, and decorate the photos with various stickers, including party hats, pizza, fashionable glasses, etc. Of course, you can also download and share via social media, or use it as your personal avatar!

△ Flutter's Dash, Firebase's Sparky, Android Jetpack and Chrome's Dino

We used Flutter web and Firebase build an I/O photo booth. Because Flutter now supports the creation of a web application , we think this will be a good way to allow participants from all over the world to easily access this application at this year's online Google I/O conference. Flutter web eliminates the barriers that must be installed through the app store, and users can also flexibly choose the device on which the app runs: mobile device, desktop device or tablet. Therefore, as long as the browser can be used, users can directly use the I/O photo booth without downloading.

Although the I/O photo booth is designed to provide a web experience, all code is written in a platform-independent architecture. When the support for native functions such as camera plug-ins is ready on each platform, this set of codes can be universal on all platforms (desktop, web, and mobile devices).

Use Flutter to build a virtual photo booth

Build a Web version of the Flutter camera plug-in

The first challenge is to build a camera plug-in for Flutter on the web. Initially, we contacted the Baseflow team because they are responsible for maintaining the existing open source Flutter camera plug-in . Baseflow is committed to building first-class camera plug-in support for iOS and Android, and we are also happy to cooperate with it, using the joint plug-in method to provide web support for the plug-in. We conform to the official plugin interface as much as possible so that we can merge it back into the official plugin when we are ready.

We have identified two APIs that are essential for building the I/O photo booth camera experience in Flutter.

• initialize camera: application first needs to access your device camera. For desktop devices, the webcam may be accessed, while for mobile devices, we chose to access the front camera. We also provide the expected resolution of 1080p to fully improve the shooting quality according to the user's device type.
• Take a picture: We used the built-in HtmlElementView , which uses the platform view to render native web elements as Flutter widgets. In this project, we VideoElement as native HTML elements, which is what you will see on the screen before taking a picture. We also used a CanvasElement to capture images from the media stream when you click the camera button.

Future<CameraImage> takePicture() async {
 final videoWidth = videoElement.videoWidth;
 final videoHeight = videoElement.videoHeight;
 final canvas = html.CanvasElement(
   width: videoWidth,
   height: videoHeight,
 );
 canvas.context2D
   ..translate(videoWidth, 0)
   ..scale(-1, 1)
   ..drawImageScaled(videoElement, 0, 0, videoWidth, videoHeight);
 final blob = await canvas.toBlob();
 return CameraImage(
   data: html.Url.createObjectUrl(blob),
   width: videoWidth,
   height: videoHeight,
 );
}

Camera permission

After completing the Flutter camera plug-in on the web, we created an abstract layout to display different interfaces based on camera permissions. For example, while waiting for your permission or denial to use the browser camera, or if there is no camera available for access, we can display an explanatory message.

Camera(
 controller: _controller,
 placeholder: (_) => const SizedBox(),
 preview: (context, preview) => PhotoboothPreview(
   preview: preview,
   onSnapPressed: _onSnapPressed,
 ),
 error: (context, error) => PhotoboothError(error: error),
)

In the abstract layout above, the placeholder will return to the initial interface while the app is waiting for you to grant the camera permission. Preview will return to the real interface after you grant permission and display the real-time video stream of the camera. The Error construct statement at the end can catch the error and display the corresponding message when the error occurs.

Generate mirrored photos

Our next challenge is to generate mirror images. If we use the photo taken by the camera as it is, what you see will be different from what you see when you look in the mirror. Some devices will provide a setting option specifically to deal with this problem, so if you take a photo with the front camera, what you see is actually a mirrored version of the photo.

In our first method, we try to capture the default camera view and then flip it 180 degrees around the y axis. This method seems to work, but then we ran into a problem , namely Flutter occasionally override this flip, causing the video to return to non-mirrored version.

With the help of the Flutter team, we put the VideoElement in DivElement , and updated the VideoElement to fill the width and height of the DivElement to solve this problem. In this way, we can apply mirroring to the video element, and because the parent element is a div, the flip effect will not be overwritten by Flutter. In this way, we have the required mirror camera view!

△ unmirrored view

△ Mirror view

keep the aspect ratio

Keeping the 4:3 aspect ratio on the big screen and the 3:4 aspect ratio on the small screen is more difficult than it seems! Maintaining the aspect ratio is very important, not only to conform to the overall design of the web application, but also to ensure that when sharing photos on social media, the pixels in the photos show a clear natural color effect. This is a challenging task because the aspect ratios of the built-in cameras on different devices vary greatly.

To force the aspect ratio to be maintained, the application first uses JavaScript getUserMedia API request the maximum possible resolution from the device camera. Then, we pass this API to the VideoElement stream, which is what you see in the camera view (of course the mirrored version). We also applied the object-fit CSS property to ensure that the video element can cover its parent container. We use the AspectRatio widget that comes with Flutter to set the aspect ratio. Therefore, the camera does not make any assumptions about the aspect ratio of the display; it always returns the maximum supported resolution and then obeys the constraints provided by Flutter (in this case 4:3 or 3:4).

final orientation = MediaQuery.of(context).orientation;
final aspectRatio = orientation == Orientation.portrait
   ? PhotoboothAspectRatio.portrait
   : PhotoboothAspectRatio.landscape;
return Scaffold(
 body: _PhotoboothBackground(
   aspectRatio: aspectRatio,
   child: Camera(
     controller: _controller,
     placeholder: (_) => const SizedBox(),
     preview: (context, preview) => PhotoboothPreview(
       preview: preview,
       onSnapPressed: () => _onSnapPressed(
         aspectRatio: aspectRatio,
       ),
     ),
     error: (context, error) => PhotoboothError(error: error),
   ),
 ),
);

add stickers by drag and drop

One of the important experiences of the I/O photo booth is to take photos with your favorite Google mascot and add props. You can drag and drop the mascot and props in the photo, as well as adjust the size and rotation, until you get the image you like. You will also find that when adding mascots to the screen, you can drag and resize them. The mascots still have animation effects-this effect is achieved by sprite sheets.

for (final character in state.characters)
 DraggableResizable(   
   canTransform: character.id == state.selectedAssetId,
   onUpdate: (update) {
     context.read<PhotoboothBloc>().add(
       PhotoCharacterDragged(
         character: character, 
         update: update,
       ),
     );
   },
   child: _AnimatedCharacter(name: character.asset.name),
 ),

To adjust the size of the object, we created a draggable, resizable widget that can accommodate other Flutter widgets. In this case, they are mascots and props. The widget will use LayoutBuilder to handle the scaling of the widget according to the constraints of the window. Internally, we use GestureDetector to hook into onScaleStart, onScaleUpdate and onScaleEnd events. These callbacks provide detailed information about the necessary gestures to reflect the user's actions on the mascot and props.

by multiple GestureDetectors, the 160c2e6da908e9 Transform widget and the 4D matrix transformation can handle zooming, as well as rotating the mascot and props according to various gestures made by the user.

Transform(
 alignment: Alignment.center,
 transform: Matrix4.identity()
   ..scale(scale)
   ..rotateZ(angle),
 child: _DraggablePoint(...),
)

Finally, we created a separate package to determine whether your device supports touch input. The draggable and resizable widget will adjust accordingly according to the touch function. On devices with touch input capabilities, you cannot see the resized anchor point and rotation icons, because you can directly manipulate the image by pinching and panning with two fingers; while on devices that do not support touch input (such as your Desktop devices), we have added anchor points and rotation icons to accommodate click and drag operations.

Optimize Flutter for the Web

Use Flutter to develop for the Web

This is one of the first pure web projects we built with Flutter, which has different characteristics from mobile apps.

We need to ensure that the application has responsiveness and adaptability for any browser on any device. In other words, we must ensure that the I/O photo booth can be scaled according to the size of the browser and can handle input from mobile devices and the web. We did this in several ways:

• Responsive resizing: users can adjust the size of the browser at will, and the interface can be responsive. If your browser window is portrait, the camera will flip from 4:3 landscape view to 3:4 portrait view.
• Responsive Design: for desktop browsers. We design to display Dash, Android Jetpack, Dino and Sparky on the right side, and for mobile devices, these elements will be displayed on the top. For desktop devices, we designed a navigation drawer on the right side of the camera, and for mobile devices, we used the BottomSheet class.
• Adaptive input: If you use a desktop device to access the I/O photo booth, mouse clicks will be regarded as input, if you are using a tablet or mobile phone, use touch input. This is especially important when you adjust the size of the sticker and place it in the photo. Mobile devices support two-finger pinch and pan gestures, and desktop devices support click and drag operations.

Scalable Architecture

We have also built a scalable mobile application for this application. Our I/O photo booth has a solid foundation at the beginning of its creation, including good air security, internationalization, and 100% unit and widget test coverage from the first submission. We use flutter_bloc for state management, because it allows us to easily test business logic and observe all state changes in the application. This is particularly useful for generating developer logs and ensuring traceability, because we can accurately observe changes from one state to another and isolate problems faster.

We have also implemented a single code base structure driven by functions. For example, stickers, sharing, and real-time camera previews are all implemented in their own folders, where each folder contains its own interface components and business logic. These functions also use external dependencies, such as the camera plug-in located in the package subdirectory. Using this architecture, our team can process multiple functions in parallel without interfering with each other, minimize merge conflicts, and effectively reuse code. For example, the interface component library is photobooth_ui , and the camera plug-in is also separate.

By dividing components into independent packages, we can extract individual components that are not tied to this particular project and open source them. Similar to Material and Cupertino component libraries, we can even open-source the interface component library package for use by the Flutter community.

Firebase + Flutter = perfect combination

Firebase Auth, storage, hosting, etc.

The photo booth utilizes the Firebase ecosystem for various back-end integrations. firebase_auth package allows users to log in anonymously immediately after the application is launched. Each session uses Firebase Auth to create anonymous users with unique IDs.

This setting will come into play when you come to the sharing page. You can download the photo to save as a personal avatar, or you can directly share it to social media. If you download a photo, it will be stored on your local device. If you share a photo, we will use firebase_storage package to store the photo in Firebase for later retrieval and post generation through social media.

Firebase Security Rule on the Firebase storage partition to ensure that the photos are immutable after they are created. This prevents other users from modifying or deleting photos in the storage partition. In addition, we use the object lifecycle management provided by Google Cloud to define a rule to delete all objects 30 days ago, but you can follow the instructions listed in the app to request that your photos be deleted as soon as possible.

This application also uses Firebase Hosting fast and secure hosting. We can use action-hosting-deploy GitHub Action to automatically deploy the application to Firebase Hosting according to the target branch. When we merge the changes to the master branch, this action triggers a workflow for building a specific development version of the application and deploying it to Firebase Hosting. Similarly, when we merge the changes into the release branch, this action will also trigger the deployment of the production version. By combining GitHub Action and Firebase Hosting, our team can iterate quickly and always get a preview of the latest version.

Finally, we use Firebase Performance Monitoring to monitor the main web performance indicators.

Use Cloud Functions for social networking

Before generating your social posts, we will first make sure that the photo content is pixel-perfect. The final image contains beautiful borders to present the I/O photo booth characteristics, and is cropped in 4:3 or 3:4 aspect ratios for excellent results on social posts.

We use OffscreenCanvas API or CanvasElement to synthesize layers of original photos, mascots and props, and generate a single image that you can download. This processing step is executed image_compositor

Then, we use Firebase's powerful Cloud Functions to share the photos to social media. When you click the share button, the system will take you to the new tab page and automatically generate a post to be published on the selected social platform. The post also contains a link to the Cloud Functions we wrote. When the browser analyzes the URL, it will detect the dynamic metadata generated by Cloud Functions, and accordingly display a beautiful preview of the photo in your social posts, as well as a link to the sharing page, where your fans can view Photos, and navigate back to the I/O photo booth app to get their own photos.

function renderSharePage(imageFileName: string, baseUrl: string): string {
 const context = Object.assign({}, BaseHTMLContext, {
   appUrl: baseUrl,
   shareUrl: `${baseUrl}/share/${imageFileName}`,
   shareImageUrl: bucketPathForFile(`${UPLOAD_PATH}/${imageFileName}`),
 });
 return renderTemplate(shareTmpl, context);
}

The finished product is as follows:

For more information on how to use Firebase in a Flutter project, please check this Codelab .

final result

This project demonstrates in detail how to build applications for the Web. To our surprise, compared to the experience of building a mobile application with Flutter, the workflow for building this web application is very similar. We must consider elements such as window size, adaptation, touch and mouse input, image loading time, browser compatibility, and all other factors that must be considered when building a Web application. However, we can still use the same pattern, architecture, and coding standards to write Flutter code, which makes us feel very comfortable when building web applications. The tools provided by the Flutter package and the evolving ecosystem, including the Firebase tool suite, helped us realize the I/O photo booth.

△ The Very Good Ventures team who built the I/O photo booth

We have opened up all the source code. Welcome to GitHub to view the photo_booth project, and don’t forget to take pictures and show it!