How to put the app on the car

Authors of this article: Lu Kang, Chen Chijing, Nie Shuai

At present, new forces in car-making are getting more and more popular, and car intelligence has become the outlet. Many mobile phone applications hope to expand the car-machine scene. Cloud Music and its Look live broadcast have also made some explorations in the car-machine scene. The following is a summary and sharing of some of the process. Experience

Types and characteristics of current in-vehicle development

There are currently three main ways of in-vehicle access. The first is to extend access to the mobile app represented by Huawei Hicar. The second is to provide external OpenApi. Car companies develop their own applications for access. The last is the most common. Independent app access for car and machine.

1. Extend access to mobile app, take Huawei HiCar as an example

This method does not require the provision of a separate car version apk for the car, but the application of the mobile phone is connected to the Hicar sdk, which is directly developed on the original project.

At present, many mobile phone manufacturers adopt the car-linked solution based on the Android system's own MediaSession framework for template development. The mobile phone application only needs to prepare data according to the template provided by the manufacturer. The specific UI display is completed by the car machine equipment, and the developer There is no need to care about screen adaptation and UI style uniformity. The specific synchronization of broadcast control commands is also done through the MediaSession framework.

This access method needs to formulate the structure of the Media Data Tree by itself. Because the display of ViewTree is handed over to the outside for rendering, we can often only get the media information clicked and played on the car through the mediaId and extras in the onPlayFromMediaId callback. The mediaId can be constructed such as tab -> page -> listId -> songId Hierarchical relationship, we can know which song from which page and which song in which playlist is played. This is also the implementation method adopted in the official Universal Android Music Player Sample of Android.

This in-vehicle access method has the following characteristics

Convenient access, developed directly on the basis of the original project, basic capabilities are readily available, and the delivery form is a mobile phone apk, which is consistent with the original
Convenient adaptation. For example, Hicar directly provides template-based development capabilities for different types of applications. Audio applications only need to focus on the preparation of audio data and the realization of playback services. Other tedious tasks, such as drawing car interface and HiCar ensures compatibility of various resolutions, manages audio desktop cards and realizes audio task connection, etc.
Easy to update, you only need to update the application on the mobile phone to update the display logic of the car, compared to updating the car application, the cost of guiding users is much lower
The limitation of the scope of application is that it is only bound to a specific platform. For example, Hicar only supports Huawei mobile phones and requires that the car machine is connected to the Huawei Hicar system. At present, several domestic mainstream mobile phone manufacturers are trying to promote similar ecosystems. In the momentum of building cars on the Internet, auto manufacturers have also accelerated the introduction of these systems, but in terms of total volume, they are still a small part of the cars.

2. OpenAPI access

This way of implementation is that the server provides the corresponding OpenAPI interface according to our service content. Manufacturers can design their own demand plan and visual plan, and call different interfaces according to different demand ranges to obtain data and display, but in the end, it generally needs to pass our review before publishing. The development resources in this access method are also provided by the manufacturers themselves, and the bearer platforms include multiple system environments such as Linux and Android. Since this method is not the focus of this article, I won't repeat it here.

This access method has the following characteristics

Our labor cost is small, and the main development cost is concentrated on the manufacturer’s side
Can be adapted to various environments, not limited to a certain type of car-machine system
The controllability is small. Part of the data acquisition depends on the manufacturer's provision. We can only get the number of interface calls, which is prone to disagreements on issues related to settlement.
Iteration is difficult and depends on the manufacturer’s own development resources

3. Independent app access

It can be seen that there are still some key problems in the implementation of the above OpenAPI, so in general, we will give priority to the independent app access method, which is currently the more common method and the access method mainly described in this article. This method is similar to mobile application development, but it also has some characteristics

The fragmentation of the car-machine system is more serious than today's more mature mobile phone ecology (most of the shares are in the head manufacturers). Many manufacturers develop their own car-machine systems based on Android, relying on equipment such as square control, desktop widgets, and instrument displays. Ability, manufacturers often provide their own set of access SDK, so channel subcontracting is imperative
The interaction requirements of the car-machine application are concise, highlight the key points, and the application supporting voice operation will be a great attraction for users
Lack of equipment for testing vehicles
The system version span is large, and the devices currently in contact can be covered from Android 4.3 to Android 10.
Performance is generally weak, so pay special attention to performance bottlenecks during development

Design

In view of some of the characteristics of the development of independent in-vehicle apps mentioned above, we have conducted a series of explorations in channel subcontracting, decoupling vehicle and machine dependence, voice operation access, resolution adaptation, etc. Here are a few related solutions design

1. Multi-channel access capability abstraction

As mentioned above, the car-machine system is relatively fragmented. There are generally two situations to realize the control of the car-machine, desktop widgets, instrument display, etc.

The manufacturer’s relevant control implements the Android native MediaSession specification. In this case, we have to respond to the relevant KeyEvent and call the MediaSession api to update the status at various playback-related occasions.
The manufacturer provides SDK access for related control. In this case, we have to follow the manufacturer’s custom specifications.

Considering that the upper-level business code should not perceive platform differences, it was decided to encapsulate and isolate the channel access capabilities.

As shown in the figure above, the ability to rely on channels is abstracted as the EnvironmentDependency interface. Different channels rely on their respective vehicle SDKs to implement this interface, and the Mediasession specification implements a common class separately. What the business layer sees is the channel-independent DependcyWrapper proxy instance. It only needs to call the corresponding method of the proxy at each business processing time, avoiding the business layer writing channel-related code. The response capability of the party control is abstracted as the EventCallback interface. After the business is implemented, the corresponding dependcy instance is injected and triggered by it at the right time.
Aiming at the problem of sub-channel packaging, the productFlavors solution that comes with AGP is adopted. Different channels contain different source folders to isolate SDK dependencies.

flavorDimensions "channel"
    productFlavors {
        //小鹏
        xp {
            dimension "channel"
            buildConfigField("String", "channel", "\"xp\"")
        }
        //比亚迪
        byd {
            dimension "channel"
            buildConfigField("String", "channel", "\"byd\"")
        }
       ......
    }

2. Design and implementation of voice control

To do voice control, you first need to think about the following questions

Should the application implement it by itself or use the vehicle and machine capabilities?
From the perspective of docking experience, it is not common for manufacturers to provide open voice capabilities for car machines. Even if individual manufacturers provide them, their access and customization procedures are more complicated and require a long period of time. Therefore, it is necessary to integrate three-party SDKs by themselves. A more reasonable choice, but we also need to provide corresponding solutions for some manufacturers who need to support their own voice assistants.
How to evoke voice control? (In addition to the page click, can you provide other shortcuts)
If you want to realize a specific short sentence to awaken the voice assistant, the voice recognition sdk is required to receive long-term radio during the application life cycle, and it has been occupying the focus of the mic, causing the voice assistant of the vehicle system to fail to work (some vehicles have implemented multi-microphone arrays) , That is, the system uses a separate mic channel for radio, but this kind of car machine is very rare), therefore, the short sentence arousal scheme is not feasible. So, can we use Fang control? Fang control can generally provide a response to the confirmation key. If the application service itself does not require the confirmation key (for example, the application is a live broadcast service, it does not need to be paused or resumed), the confirmation key can be used directly to call up the voice assistant. If necessary, a certain point can also be designed Arouse by way (such as long press or double tap, this can be done by judging the time interval of key events at the business layer), of course, the corresponding guidance also needs to keep up, such as showing the floating layer plus voice guidance when the user enters for the first time
How to map the text recognized by the voice to the corresponding operation? The most convenient method is definitely for the client to directly judge the text matching, such as switching to the next live broadcast when it recognizes the "next song", but this approach is less fault-tolerant, and the user's statement will be invalid if the user adjusts it slightly, which is more reasonable The method is to add a semantic recognition link after the speech-to-text link. The process is as follows

After solving these basic problems, consider the complete interaction process of the next more complete voice assistant. After the assistant is awakened, it will first enter the inquiry mode and prompt the operation type of voice support, and then the user will input. If the input is overtime, it will prompt the assistant to close , After normal input, the request will be parsed. After the result is obtained, some operations will directly close the panel, and some operations will directly display the results on the panel and return to the query state. If it cannot be resolved, it will directly prompt and return to the query state. It can be seen that the entire process on the client is more suitable for abstraction as a state machine

If you need to connect the voice assistants that come with different vehicles, the callbacks involving control-related instructions and playback information need to be separated from the more common interfaces to implement them. For common instructions, such as play, pause, previous song, The next song, favorites, search on-demand, etc. need to be packaged into independent methods. Different car and machine apps are registered with different servers, and the realization of the client is processed by the same client, and the results processed by the client can be returned to the server. For display, the advantage of this is that the part of the docking with the car is completely handed over to the server for processing, and the client only needs to perform corresponding operations according to the issued instructions. The first half is decoupled, and the second half is reused.

3. Multi-resolution adaptation

In the front visual interaction design, taking into account the driving scene, the commonly used operation area should be placed as close to the driving side as possible, and the interaction process should be as simple as possible, and the page jump level should not be too many. In addition to the mainstream horizontal screen layout, the screens of BYD and Xiaopeng will also have vertical screens.
Common screen adaptation schemes include smallestWidth adaptation, headline modification DisplayMetrics#density scheme, using percentage layout, etc. Combining the actual situation of the project, we recommend that most of the layouts adopt a stream layout. You only need to change the direction of the recyclerView in the layout to adapt to the switching of horizontal and vertical screens. At the same time, the card layout should be as flat as possible, such as Guideline, layout_constraintHeight_percent, etc. in ConstraintLayout Attributes can help us to achieve percentage layout easily. If you encounter a screen with a very strange ratio and the page cannot use the flow layout, you can consider the scheme of combining the sw qualifier, and let the visual students give the layout adjustment strategy. A small number of special screens are adapted.
When developing visual adaptations, our first reaction was of course to let the manufacturer provide all the equipment that may be involved in the car, but this is unrealistic. From our docking experience, the test car is quite in short supply. Manufacturers are not even able to provide even cars and machines for the time being. They only provide documents so that we can adapt and test them internally. In this case, we can only simulate different resolution devices. The adb shell wm size command is the solution. It accepts parameters in the format of total length pixel value x total width pixel value, and can be adjusted to the corresponding aspect ratio after running. The test process only needs to run commands with different parameters on the same device. Can realize the simulation of different resolutions.

Performance optimization

As mentioned above, compared with mobile phones, car machines lag far behind in overall performance. At the beginning, due to historical burdens, component reuse and other factors, on the other hand, performance-related issues were often ignored when writing code, which made the experience of app running on the car quite bad, slow installation, slow startup, and freezes. Performance problems such as frame loss were obviously exposed, so we made a series of targeted optimizations

1. Reduce the size of the package

Reducing the package size includes both code and resources. The usual practice is as follows:

Image Compression
Resource confusion
Reduce the number of Dex

2. Reduce the number of processes

Multi-process operation requires more system resources. On devices with weaker performance, the single-app multi-process operation mode will put more pressure on the device's CPU, memory, etc.

3. Reduce the number of threads

Similar to processes, too many threads and frequent switching during startup bring a lot of overhead costs, and the execution time of the main thread will be reduced.

4. IO optimization

Too many file IOs during the startup process will slow down the startup speed and minimize unnecessary file reading and writing

5. Reduce the number of jumps in Activity

In order to display the interface faster or perform a specific function, it is best to reduce the jump level of the Activity in the startup process. Each additional Activity will increase the time-consuming by several hundred milliseconds; when requesting some interfaces, the request should also be considered Timing, whether parallel requests can be forwarded, or requests can be merged to reduce the RT of the interface

6. Optimize the layout level to reduce over-drawing

Let's share an example of performance optimization. In the process of cooperation with a certain car manufacturer, the manufacturer reported that the voice arousal stage was particularly slow from cold start to start, which was nearly 8 seconds. We tested it on the mobile phone and there was no problem at all, but due to the performance of the car and the machine, after repeated rounds of communication and joint adjustments, we mainly made the following optimizations

Significantly reduce the package size, delete a large number of useless business codes, and reduce the package size by about 80%. Because the package size is greatly reduced, the number of dex that needs to be decompressed during the startup process is also reduced, the loaded classes are reduced, and the speed is improved by a few seconds.
The playback process is merged into the main process, multi-process is changed to single process, aidl is removed, and several file reads and writes before and after the aidl communication are removed, reducing the time consumption by about 2s
Combine LoadingActivity and homepage Activity into one, reduce the number of activities in the start link process, and reduce the time-consuming by about hundreds of milliseconds
Combining multiple interfaces into one reduces network requests and reduces time-consuming by about 200 milliseconds

In the end, the time was compressed to within 3s. Our optimization process focused on optimizing the obvious time-consuming part from the early stage, and the effect was obvious. After analyzing the startup log, it was a little bit of detail, and finally passed the manufacturer's acceptance. Before starting to optimize, you need to quantify the specific indicators, clarify the goals and then proceed. Using data to measure the optimization effect can make the optimization process smoother

Pit guide

During the development of the car, I also encountered some uncommon problems in the development of mobile phone applications. I am deeply impressed. Share it here.

1. After the pre-installation, the RN page is not working

In the car scene, the frequency of users actively downloading and updating apps is much lower than that of mobile phones, so pre-installation is a very important means of shopping, but when we finally negotiated pre-installation with a certain channel, we found a strange problem , All the pages implemented with RN enter or preload will cause the application to crash. The direct cause of the crash stack prompt is that libjsexcutor.so, the JS parsing library that RN depends on, failed to load, so I took a preliminary look at the source code of the crash location of RN , I found that RN's so library is loaded through the Facebook tool of SoLoader (the official document says that it is mainly used to be compatible with the so loading dependency of versions below 4.3), and other business so in the application are working normally, so I guess SoLoader will have problems in application pre-installation scenarios, so reproduce and focus on the logs related to Soloader

The picture above is the RN loading log on the problem channel, and the picture below is the RN loading log in a normal scenario

It can be seen that the difference between the two lies in the problem channel. The so search path marked in red is not added (this path is actually the soft link of the so path after the application is installed), while the normal channel is found on this path RN-related so was loaded, and following this idea, I checked the source code of SoLoader and found the following logic

That is, after judging that the current application is a system application, the app default so path is not added to the search path, causing RN-related libraries to be loaded with Soloader will fail. After locating the cause, carefully go over the SoLoader to load the so related source code and find that it provides With the setting interface of setSystemLoadLibraryWrapper, the upper layer can define how to load dependent so for system application scenarios, so we only need to set the scene to use the original so loading method of the application to solve the problem, as shown in the following code

SoLoader.setSystemLoadLibraryWrapper {
    ReLinker.loadLibrary(context, it)
}

2. Strange problems on vehicle test equipment

After the test car of a certain channel was connected to the company’s wifi, it was still unable to access the network and communicated with the manufacturer. They informed that it was the first time that the test car was provided to the outside, and internal use was fine, so they could only locate it by themselves. Considering that the probability is related to the network environment, the iptables tool is used to view the car and machine network rules (iptables is an application software running in the user space. By controlling the Linux kernel netfilter module, it manages the processing and forwarding of network data packets, and the details of the data packets. The flow of circulation is shown in the figure below, and rules can be added in each link to intercept)

After reviewing the results, I found that some of the rules are quite special. The guess is that the test car machine was originally only used by the manufacturer. In order to prevent problems after the outflow, the network environment was identified. Once the non-manufacturer company’s intranet was found, the data packet was discarded, so Use the following command to clean up the rules and the problem is solved
```
iptables -F
iptables -X
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT
```
In the development process of a certain channel's car and machine, it was found that some interfaces reported an error. After a closer look, I found that the error interface is all https protocol (the development stage is still in the test environment, most of the interfaces are http protocol), the error content seen in the adb log is roughly as follows

javax.net.ssl.SSLHandshakeException: com.android.org.bouncycastle.jce.exception.ExtCertPathValidatorException: 
Could not validate certificate: Certificate not valid until Wed Dec 16 
09:00:05 GMT+08:00 2015 (compared to Sun Oct 12 16:20:03 GMT+08:00 1980)

It seems that the time and the validity period of the certificate are not correct. Checking the system time and finding that it is indeed wrong, it turns out that the system time will be reset every time the car is started, and the verification process of the SSL client includes the verification of the validity period of the certificate. Adjust the system time The problem can be solved later
It can be seen from the above that the test car opportunity brings some strange development problems due to some special settings, but the better point is that these test car machines are often rooted, so the command authority is large enough to conduct in-depth analysis.

Experience beyond technology

Participate in the whole process of in-vehicle applications from launch to official launch. In addition to technology, there are some other experiences

There is a big difference between car factory project management and Internet products. Its style is more rigorous and meticulous, seeking stability but not fast. Without the concept of rapid Internet iteration, it is often difficult to accept the practice of bringing some problems online and subsequent iteration fixes, so its test cycle Usually longer, there are more rounds of feedback, and the angle of feedback is also more diverse (product design, content operation, technical points), and the application side needs to be psychologically prepared and deal with it patiently.
In the initial communication with the car factory, it is necessary to align the delivery standards, such as the scope of adaptation requirements, application performance indicators, etc., to avoid back and forth communication and rework due to the inconsistent delivery standards, and also according to our own project conditions. Develop your own standard baseline and pass Monkey and automated performance tests to ensure the stability of the app
At present, the overall process of each car factory connecting to the app cannot be said to be perfect. There are problems such as lack of documentation, lack of test vehicles, unstable simulators, and slow response to feedback problems. Sort out as soon as possible, communicate with vendors in time, and predict risks. Later, as application access becomes more and more common, the construction of vendors should be improved.

summary

This article introduces some of the current status of in-vehicle development, shares some design ideas and typical problems encountered during the development process, and hopes to be helpful to everyone's application on the car!

This article was published from big front-end team of NetEase Cloud Music . Any form of reprinting of the article is prohibited without authorization. We recruit front-end, iOS, and Android all year round. If you are ready to change jobs and you happen to like cloud music, then join us at grp.music-fe(at)corp.netease.com!