1. Multimedia Application Architecture
1.1 Traditional application architecture of audio and video
Typically, a traditional multimedia application that plays audio or video consists of two parts:
- Player: for ingesting digital media and rendering it as video and/or audio;
- Interface: with transport controls to run the player and display the player status (optional);
In Android app development, building your own player from scratch also consider the following options:
- MediaPlayer : Provides the basic functionality of a barebones player, supporting the most common audio/video formats and data sources.
- ExoPlayer : An open source library that provides low-level Android audio APIs. ExoPlayer supports high performance features like DASH and HLS streaming which are not available in
MediaPlayer
.
As we all know, if you want to continue playing audio in the background of the application, the most common way is to place the Player in the Service, and the Service provides a Binder to realize the communication between the interface and the player . However, if you encounter a lock screen, if you want to communicate with the Service, you have to use the AIDL interface/broadcast/ContentProvider to complete the communication with other applications, and these communication methods not only increase the communication between application developers. Communication costs also increase the coupling between applications. In order to solve the above problems, Android officially provides the MediaSession framework starting from Android5.0.
1.2 MediaSession Framework
The MediaSession framework standardizes the communication interface between the interface and the player in audio and video applications, and realizes the complete decoupling between the interface and the player. The MediaSession framework defines two important classes of media session and media controller, which provide a complete technical framework for building multimedia player applications.
The media session and media controller communicate with each other by using predefined callbacks corresponding to standard player actions (play, pause, stop, etc.), as well as extensible custom calls that define special behavior unique to the application.
media session
The media session is responsible for all communication with the player. It hides the player's API from the rest of the app. The system can only invoke the player from the media session that controls the player.
A session maintains a representation of the player state (playing/pausing) and information about what is playing. A session can receive callbacks from one or more media controllers. This way, the app's interface and a companion device running Wear OS and Android Auto can control your player. The logic for responding to callbacks must be consistent. The response to the MediaSession
callback is the same regardless of which client application initiated the callback.
media controller
The role of the media controller is to isolate the interface, the code for the interface only communicates with the media controller (not the player itself), and the media controller converts transfer control operations into callbacks to the media session. It also receives callbacks from the media session whenever the session state changes, which provides a mechanism for automatically updating the associated interface, and the media controller can only be connected to one media session at a time.
When you use media controllers and media sessions, you can deploy different interfaces and/or players at runtime. This allows you to individually change the app's appearance and/or performance based on the capabilities of the device it's running on.
2. MediaSession
2.1 Overview
The MediaSession framework is mainly used to solve the communication problem between the music interface and the service. It belongs to a typical C/S architecture. There are four commonly used member classes, namely MediaBrowser, MediaBrowserService, MediaController and MediaSession, which are controlled by the entire MediaSession framework process. core.
- MediaBrowser: The media browser is used to connect the media service MediaBrowserService and subscription data. In the registered callback interface, you can obtain the connection status of the Service and obtain music data, which are generally created in the client.
- MediaBrowserService: media service, it has two key callback functions, onGetRoot (controls the connection request of the client media browser, and determines whether the connection is allowed in the return value), onLoadChildren (will be called when the media browser sends a data subscription request to the server , generally perform the operation of asynchronously fetching data here, and then send the data back to the interface registered by the media browser).
- MediaController: The media controller, which works in the client, sends instructions to the media server through the controller, and then sets the callback function through MediaControllerCompat.Callback to accept the status of the server. When the MediaController is created, the pairing token of the controlled end is required, so the MediaController needs to be created after the browser is connected successfully.
- MediaSession: The media session, the controlled end, receives the command sent by the MediaController by setting the MediaSessionCompat.Callback callback. After receiving the command, the callback method in the Callback will be triggered, such as playback pause. Session is generally created in the Service.onCreate method. Finally, the setSessionToken method needs to be called to set the token for pairing with the controller and notify the browser that the connection to the service is successful.
Among them, MediaBrowser and MediaController are used by the client, and MediaBrowserService and MediaSession are used by the server. Since the client and the server communicate asynchronously, a large number of callbacks are used, so there are a large number of callback classes. The schematic diagram of the framework is as follows.
2.2 MediaBrowser
MediaBrowser is a media browser used to connect to MediaBrowserService and subscribe to data. Through its callback interface, we can obtain the connection status with the Service and obtain the music library data in the Service .
Created in the client (that is, the aforementioned interface , or the control side ). The MediaBrowser is not thread-safe, all calls should be made on the thread where the MediaBrowser is constructed.
@RequiresApi(Build.VERSION_CODES.M)
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
val component = ComponentName(this, MediaService::class.java)
mMediaBrowser = MediaBrowser(this, component, connectionCallback, null);
mMediaBrowser.connect()
}
2.2.1 MediaBrowser.ConnectionCallback
Connection status callback. When MediaBrowser initiates a connection request to the service, the request result will be returned in this callback. The obtained meidaId corresponds to the mediaId set by the server in the onGetRoot function. If the connection is successful, you can create a media controller and so on. operation.
@RequiresApi(Build.VERSION_CODES.M)
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
val component = ComponentName(this, MediaService::class.java)
mMediaBrowser = MediaBrowser(this, component, connectionCallback, null);
mMediaBrowser.connect()
}
private val connectionCallback = object : MediaBrowser.ConnectionCallback() {
override fun onConnected() {
super.onConnected()
... //连接成功后我们才可以创建媒体控制器
}
override fun onConnectionFailed() {
super.onConnectionFailed()
}
override fun onConnectionSuspended() {
super.onConnectionSuspended()
}
}
2.2.2 MediaBrowser.ItemCallback
The media controller is responsible for sending commands such as playback pause to the service, and the execution results of these commands will be returned in this callback. There are many overridable functions, such as changing the playback state, changing the music information, and so on.
private val connectionCallback = object : MediaBrowser.ConnectionCallback() {
override fun onConnected() {
super.onConnected()
... //返回执行结果
if(mMediaBrowser.isConnected) {
val mediaId = mMediaBrowser.root
mMediaBrowser.getItem(mediaId, itemCallback)
}
}
}
@RequiresApi(Build.VERSION_CODES.M)
private val itemCallback = object : MediaBrowser.ItemCallback(){
override fun onItemLoaded(item: MediaBrowser.MediaItem?) {
super.onItemLoaded(item)
}
override fun onError(mediaId: String) {
super.onError(mediaId)
}
}
2.2.3 MediaBrowser.MediaItem
Contains information about a single media item for browsing/searching media. MediaItem
depends on the server, so the framework itself cannot guarantee that the values it contains are correct.
2.2.4 MediaBrowser.SubscriptionCallback
After the connection is successful, the first thing you need is the subscription service, and you also need to register the subscription callback. If the subscription is successful, the server can return a sequence of music information, and the obtained music list data can be displayed on the client. For example, here's a callback that subscribes to changes to the MediaBrowser.MediaItem list in MediaBrowserService.
private val connectionCallback = object : MediaBrowser.ConnectionCallback() {
override fun onConnected() {
super.onConnected()
// ...
if(mMediaBrowser.isConnected) {
val mediaId = mMediaBrowser.root
//需要先取消订阅
mMediaBrowser.unsubscribe(mediaId)
//服务端会调用 onLoadChildren
mMediaBrowser.subscribe(mediaId, subscribeCallback)
}
}
}
private val subscribeCallback = object : MediaBrowser.SubscriptionCallback(){
override fun onChildrenLoaded(
parentId: String,
children: MutableList<MediaBrowser.MediaItem>
) {
super.onChildrenLoaded(parentId, children)
}
override fun onChildrenLoaded(
parentId: String,
children: MutableList<MediaBrowser.MediaItem>,
options: Bundle
) {
super.onChildrenLoaded(parentId, children, options)
}
override fun onError(parentId: String) {
super.onError(parentId)
}
override fun onError(parentId: String, options: Bundle) {
super.onError(parentId, options)
}
}
2.3 MediaController
The media controller is used to send control commands to the server, such as: play, pause, etc., created in the client. The media controller is thread-safe, and the MediaController also has an associated permission android.permission.MEDIA_CONTENT_CONTROL (not a mandatory permission) that must be obtained by system-level applications. Fortunately, in-vehicle applications are generally system-level applications.
Meanwhile, MediaController
can only be created after the MediaBrowser is connected successfully. So, the code to create the MediaController is as follows:
private val connectionCallback = object : MediaBrowser.ConnectionCallback() {
override fun onConnected() {
super.onConnected()
// ...
if(mMediaBrowser.isConnected) {
val sessionToken = mMediaBrowser.sessionToken
mMediaController = MediaController(applicationContext,sessionToken)
}
}
}
2.3.1 MediaController.Callback
It is used to receive callbacks from MediaSession, so when using it, you need to register MediaController.Callback with MediaSession, as follows:
private val connectionCallback = object : MediaBrowser.ConnectionCallback() {
override fun onConnected() {
super.onConnected()
// ...
if(mMediaBrowser.isConnected) {
val sessionToken = mMediaBrowser.sessionToken
mMediaController = MediaController(applicationContext,sessionToken)
mMediaController.registerCallback(controllerCallback)
}
}
}
private val controllerCallback = object : MediaController.Callback() {
override fun onAudioInfoChanged(info: MediaController.PlaybackInfo?) {
super.onAudioInfoChanged(info)
... //回调方法接收受控端的状态,从而根据相应的状态刷新界面 UI
}
override fun onExtrasChanged(extras: Bundle?) {
super.onExtrasChanged(extras)
}
// ...
}
2.3.2 MediaController.PlaybackInfo
Get the currently playing audio information, including the playback progress, duration, etc.
2.3.3 MediaController.TransportControls
Interface for controlling media playback in a session. The client can send media control commands through the session, using the following methods:
private val connectionCallback = object : MediaBrowser.ConnectionCallback() {
override fun onConnected() {
super.onConnected()
// ...
if(mMediaBrowser.isConnected) {
val sessionToken = mMediaBrowser.sessionToken
mMediaController = MediaController(applicationContext,sessionToken)
// 播放媒体
mMediaController.transportControls.play()
// 暂停媒体
mMediaController.transportControls.pause()
}
}
}
2.4 MediaBrowserService
Media Browser Service, inherited from Service, MediaBrowserService belongs to the server, and is also the container that carries players (such as MediaPlayer, ExoPlayer, etc.) and MediaSession. After inheriting MediaBrowserService, we need to overwrite onGetRoot
and onLoadChildren
two methods. The return value passed by onGetRoot determines whether the client's MediaBrowser is allowed to connect to the MediaBrowserService. The onLoadChildren method is triggered when the client calls MediaBrowser.subscribe
. The following is a use case:
const val FOLDERS_ID = "__FOLDERS__"
const val ARTISTS_ID = "__ARTISTS__"
const val ALBUMS_ID = "__ALBUMS__"
const val GENRES_ID = "__GENRES__"
const val ROOT_ID = "__ROOT__"
class MediaService : MediaBrowserService() {
override fun onGetRoot(
clientPackageName: String,
clientUid: Int,
rootHints: Bundle?
): BrowserRoot? {
// 由 MediaBrowser.connect 触发,可以通过返回 null 拒绝客户端的连接。
return BrowserRoot(ROOT_ID, null)
}
override fun onLoadChildren(
parentId: String,
result: Result<MutableList<MediaBrowser.MediaItem>>
) {
//由 MediaBrowser.subscribe 触发
when (parentId) {
ROOT_ID -> {
// 查询本地媒体库
result.detach()
result.sendResult()
}
FOLDERS_ID -> {
}
ALBUMS_ID -> {
}
ARTISTS_ID -> {
}
GENRES_ID -> {
}
else -> {
}
}
}
}
Finally, you also need to register the Service in the manifest.
<service
android:name=".MediaService"
android:label="@string/service_name">
<intent-filter>
<action android:name="android.media.browse.MediaBrowserService" />
</intent-filter>
</service>
2.4.1 MediaBrowserService.BrowserRoot
Returns containing information that the browser service needs to send to the client when it first connects. The constructor is as follows:
MediaBrowserService.BrowserRoot(String rootId, Bundle extras)
In addition, there are two methods:
- getExtras(): Get additional information about browser services
getRootId(): Get the root ID for browsing
2.4.2 MediaBrowserService.Result<T>
Contains the result set returned by the browser service to the client. By calling sendResult()
the result is returned to the caller, but before that requires calling detach()
.
- detach(): detach this message from the current thread and allow a later call to sendResult(T)
sendResult(): Send the result back to the caller.
2.5 MediaSession
The media session , the controlled end. By setting the MediaSession.Callback
callback to receive the instructions sent by the media controller MediaController
, such as controlling the [previous song], [next song] and so on.
After creating MediaSession
, you need to call the setSessionToken()
method to set the token for pairing with the ** controller. The usage is as follows:
const val FOLDERS_ID = "__FOLDERS__"
const val ARTISTS_ID = "__ARTISTS__"
const val ALBUMS_ID = "__ALBUMS__"
const val GENRES_ID = "__GENRES__"
const val ROOT_ID = "__ROOT__"
class MediaService : MediaBrowserService() {
private lateinit var mediaSession: MediaSession;
override fun onCreate() {
super.onCreate()
mediaSession = MediaSession(this, "TAG")
mediaSession.setCallback(callback)
sessionToken = mediaSession.sessionToken
}
// 与 MediaController.transportControls 中的大部分方法都是一一对应的
// 在该方法中实现对 播放器 的控制,
private val callback = object : MediaSession.Callback() {
override fun onPlay() {
super.onPlay()
// 处理 播放器 的播放逻辑。
// 车载应用的话,别忘了处理音频焦点
}
override fun onPause() {
super.onPause()
}
}
override fun onGetRoot(
clientPackageName: String,
clientUid: Int,
rootHints: Bundle?
): BrowserRoot? {
Log.e("TAG", "onGetRoot: $rootHints")
return BrowserRoot(ROOT_ID, null)
}
override fun onLoadChildren(
parentId: String,
result: Result<MutableList<MediaBrowser.MediaItem>>
) {
result.detach()
when (parentId) {
ROOT_ID -> {
result.sendResult(null)
}
FOLDERS_ID -> {
}
ALBUMS_ID -> {
}
ARTISTS_ID -> {
}
GENRES_ID -> {
}
else -> {
}
}
}
override fun onLoadItem(itemId: String?, result: Result<MediaBrowser.MediaItem>?) {
super.onLoadItem(itemId, result)
Log.e("TAG", "onLoadItem: $itemId")
}
}
2.5.1 MediaSession.Callback
Receive media buttons, transmission controls and commands from the client or system, enter [Previous Song] and [Next Song]. There is a one-to-one correspondence with most of the methods in MediaController.transportControls.
private val callback = object : MediaSession.Callback() {
override fun onPlay() {
super.onPlay()
if (!mediaSession.isActive) {
mediaSession.isActive = true
}
//更新播放状态.
val state = PlaybackState.Builder()
.setState(
PlaybackState.STATE_PLAYING,1,1f
)
.build()
mediaSession.setPlaybackState(state)
}
override fun onPause() {
super.onPause()
}
override fun onStop() {
super.onStop()
}
}
2.5.2 MediaSession.QueueItem
A single item that is part of the playback queue has an additional ID attribute compared to MediaMetadata. Commonly used methods are:
- getDescription(): Returns the description of the medium, including the basic information of the medium, such as title, cover, etc.
getQueueId(): Get the queue ID of this item.
2.5.3 MediaSession.Token
Represents an ongoing session and can be passed to the client by the session owner to allow communication between the client and the server.
2.6 PlaybackState
Class used to carry playback state. Such as the current playback position and the current control function. After MediaSession.Callback
change the state, you need to call MediaSession.setPlaybackState
to synchronize the state to the client.
private val callback = object : MediaSession.Callback() {
override fun onPlay() {
super.onPlay()
// 更新状态
val state = PlaybackState.Builder()
.setState(
PlaybackState.STATE_PLAYING,1,1f
)
.build()
mediaSession.setPlaybackState(state)
}
}
2.6.1 PlaybackState.Builder
PlaybackState.Builder is mainly used to create a PlaybackState object, which is created using the builder mode, as follows.
PlaybackState state = new PlaybackState.Builder()
.setState(PlaybackState.STATE_PLAYING,
mMediaPlayer.getCurrentPosition(), PLAYBACK_SPEED)
.setActions(PLAYING_ACTIONS)
.addCustomAction(mShuffle)
.setActiveQueueItemId(mQueue.get(mCurrentQueueIdx).getQueueId())
.build();
2.6.2 PlaybackState.CustomAction
CustomActions
can be used to extend the functionality of standard transport controls by sending application-specific actions to MediaControllers
.
CustomAction action = new CustomAction
.Builder("android.car.media.localmediaplayer.shuffle",
mContext.getString(R.string.shuffle),
R.drawable.shuffle)
.build();
PlaybackState state = new PlaybackState.Builder()
.setState(PlaybackState.STATE_PLAYING,
mMediaPlayer.getCurrentPosition(), PLAYBACK_SPEED)
.setActions(PLAYING_ACTIONS)
.addCustomAction(action)
.setActiveQueueItemId(mQueue.get(mCurrentQueueIdx).getQueueId())
.build();
Common APIs are as follows:
- getAction(): returns the action of CustomAction
- getExtras(): Returns extras that provide additional application-specific information about the operation
- getIcon(): Returns the resource ID of the icon in the package
getName(): Returns the display name for this operation
2.7 MediaMetadata
Contains basic data about the item, such as title, artist, etc. Generally, the server needs to query the original data from the local database or the remote end, encapsulate it into MediaMetadata, and then return it to the client 's MediaController.Callback.onMetadataChanged
MediaSession.setMetadata(metadata)
Common APIs are as follows:
- containsKey(): Returns true if the given key is contained in the metadata.
- describeContents(): Describes the kinds of special objects contained in the marshaled representation of this packable instance.
- getBitmap(): Returns the Bitmap for the given key, or null if there is no bitmap for the given key.
- getBitmapDimensionLimit(): Get the width/height limit of the bitmap when this metadata was created
- getDescription(): Get a simple description of this metadata for display.
keySet(): Returns a Set containing the strings used as keys in this metadata.
3. Examples
The following figure shows the communication process of the core class of the MediaSession framework.
It can be seen that in the MediaSession framework, the client first connects to the MediaBrowserService through the MediaBrowserService, and the MediaBrowserService processes the relevant requests after receiving the request.
Client sample code:
class MainActivity : AppCompatActivity() {
private lateinit var mMediaBrowser: MediaBrowser
private lateinit var mMediaController: MediaController
@RequiresApi(Build.VERSION_CODES.M)
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
val component = ComponentName(this, MediaService::class.java)
mMediaBrowser = MediaBrowser(this, component, connectionCallback, null);
// 连接到 MediaBrowserService,会触发 MediaBrowserService 的 onGetRoot 方法。
mMediaBrowser.connect()
findViewById<Button>(R.id.btn_play).setOnClickListener {
mMediaController.transportControls.play()
}
}
private val connectionCallback = object : MediaBrowser.ConnectionCallback() {
override fun onConnected() {
super.onConnected()
if (mMediaBrowser.isConnected) {
val sessionToken = mMediaBrowser.sessionToken
mMediaController = MediaController(applicationContext, sessionToken)
mMediaController.registerCallback(controllerCallback)
// 获取根 mediaId
val rootMediaId = mMediaBrowser.root
// 获取根 mediaId 的 item 列表,会触发 MediaBrowserService.onLoadItem 方法
mMediaBrowser.getItem(rootMediaId,itemCallback)
mMediaBrowser.unsubscribe(rootMediaId)
// 订阅服务端 media item 的改变,会触发 MediaBrowserService.onLoadChildren 方法
mMediaBrowser.subscribe(rootMediaId, subscribeCallback)
}
}
}
private val controllerCallback = object : MediaController.Callback() {
override fun onPlaybackStateChanged(state: PlaybackState?) {
super.onPlaybackStateChanged(state)
Log.d("TAG", "onPlaybackStateChanged: $state")
when(state?.state){
PlaybackState.STATE_PLAYING ->{
// 处理 UI
}
PlaybackState.STATE_PAUSED ->{
// 处理 UI
}
// 还有其它状态需要处理
}
}
//音频信息,音量
override fun onAudioInfoChanged(info: MediaController.PlaybackInfo?) {
super.onAudioInfoChanged(info)
val currentVolume = info?.currentVolume
// 显示在 UI 上
}
override fun onMetadataChanged(metadata: MediaMetadata?) {
super.onMetadataChanged(metadata)
val artUri = metadata?.getString(MediaMetadata.METADATA_KEY_ALBUM_ART_URI)
// 显示 UI 上
}
override fun onSessionEvent(event: String, extras: Bundle?) {
super.onSessionEvent(event, extras)
Log.d("TAG", "onSessionEvent: $event")
}
// ...
}
private val subscribeCallback = object : MediaBrowser.SubscriptionCallback() {
override fun onChildrenLoaded(
parentId: String,
children: MutableList<MediaBrowser.MediaItem>
) {
super.onChildrenLoaded(parentId, children)
}
override fun onChildrenLoaded(
parentId: String,
children: MutableList<MediaBrowser.MediaItem>,
options: Bundle
) {
super.onChildrenLoaded(parentId, children, options)
}
override fun onError(parentId: String) {
super.onError(parentId)
}
}
private val itemCallback = object : MediaBrowser.ItemCallback() {
override fun onItemLoaded(item: MediaBrowser.MediaItem?) {
super.onItemLoaded(item)
}
override fun onError(mediaId: String) {
super.onError(mediaId)
}
}
}
The following is the sample source code of the server, which is mainly used to process the client's request and return the result to the client.
const val FOLDERS_ID = "__FOLDERS__"
const val ARTISTS_ID = "__ARTISTS__"
const val ALBUMS_ID = "__ALBUMS__"
const val GENRES_ID = "__GENRES__"
const val ROOT_ID = "__ROOT__"
class MediaService : MediaBrowserService() {
// 控制是否允许客户端连接,并返回 root media id 给客户端
override fun onGetRoot(
clientPackageName: String,
clientUid: Int,
rootHints: Bundle?
): BrowserRoot? {
Log.e("TAG", "onGetRoot: $rootHints")
return BrowserRoot(ROOT_ID, null)
}
// 处理客户端的订阅信息
override fun onLoadChildren(
parentId: String,
result: Result<MutableList<MediaBrowser.MediaItem>>
) {
Log.e("TAG", "onLoadChildren: $parentId")
result.detach()
when (parentId) {
ROOT_ID -> {
result.sendResult(null)
}
FOLDERS_ID -> {
}
ALBUMS_ID -> {
}
ARTISTS_ID -> {
}
GENRES_ID -> {
}
else -> {
}
}
}
override fun onLoadItem(itemId: String?, result: Result<MediaBrowser.MediaItem>?) {
super.onLoadItem(itemId, result)
Log.e("TAG", "onLoadItem: $itemId")
// 根据 itemId,返回对用 MediaItem
result?.detach()
result?.sendResult(null)
}
private lateinit var mediaSession: MediaSession;
override fun onCreate() {
super.onCreate()
mediaSession = MediaSession(this, "TAG")
mediaSession.setCallback(callback)
// 设置 token
sessionToken = mediaSession.sessionToken
}
// 与 MediaController.transportControls 中的方法是一一对应的。
// 在该方法中实现对 播放器 的控制,
private val callback = object : MediaSession.Callback() {
override fun onPlay() {
super.onPlay()
// 处理 播放器 的播放逻辑。
// 车载应用的话,别忘了处理音频焦点
Log.e("TAG", "onPlay:")
if (!mediaSession.isActive) {
mediaSession.isActive = true
}
// 更新状态
val state = PlaybackState.Builder()
.setState(
PlaybackState.STATE_PLAYING, 1, 1f
)
.build()
mediaSession.setPlaybackState(state)
}
override fun onPause() {
super.onPause()
}
override fun onStop() {
super.onStop()
}
//省略其他方法
}
}
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用
。你还可以使用@
来通知其他用户。