Audio / Video

The BlackBerry 10 OS lets you create visually astonishing UIs for your mobile apps with fully functioning multimedia support for both audio and video. You can design apps that record multimedia content and allow your users to play, seek, and rewind their audio and video files. The BlackBerry 10 OS also supports streaming audio and video. Users can interact with streamed content in the same way as local audio and video files.

Cascades APIs

The Cascades multimedia suite supports most of the popular audio and video file formats, as well as streaming protocols, and provides support for accessible apps.

For information about BlackBerry media support, see BlackBerry media support at a glance.

The Cascades multimedia suite provides four components:

Audio recorder
This component can be used to record audio files in your app.
Multimedia player
This component allows your apps to play audio and video files including both local and streamed content. It can be used to play a sequence of files (audio and video), such as a playlist. The Multimedia player can be used to monitor and report its current state and it can also be used to build accessible multimedia apps.
System sound
This component allows your apps to use system sounds that come preloaded on the device.
Notifications
This component provides media notifications that can be used to monitor the media content that's currently playing. Your app can respond to these media notifications as they become available.

You cannot use the Cascades multimedia suite to record video. To record video you must use the Cascades Camera APIs that are available in the BlackBerry 10 Native SDK. For more information about recording video, see Camera.

There are more features available in Cascades than can be mentioned here in this introduction. One important feature of Cascades is that you can rapidly develop fully functional multimedia apps in a relatively short amount of time without sacrificing quality.

Flow and states

There are some flow and states that must be discussed before you begin to work with the AudioRecorder. The architectural flow diagram shown below illustrates the flow and various states of the AudioRecorder. The AudioRecorder must proceed through these states before reaching the Started state where it can record content.

Unprepared state

The AudioRecorder begins in an Unprepared state. In the Unprepared state, the AudioRecorder cannot record content because it doesn't have the necessary resources, such as exclusive access to the recorder or free space allocated in the file system, to do so.

Prepared and Started states

When the AudioRecorder acquires the resources it needs to record content, it's in a Prepared state. The Prepared state usually moves quickly into the Started state where it can record media content. Your app can go from the Unprepared state directly to the Started state and begin recording content. In this case, it may appear that the Prepared state was skipped, but it's not skipped, it's just occurring briefly.

AudioRecorder architectural flow diagram

Accessing media notification areas

The MediaPlayer class provides the functionality to play media content. However, the NowPlayingConnection class notifies the system and your app of external events, such as skip, play and pause, or revoked. The NowPlayingConnection class also gives your app system access to the volume overlay and the media notification area. The volume overlay is where the media that is currently playing on the device is described to your user.

Screen showing the volume overlay.

Volume overlay and media notification area

The media notification area is visible in the volume overlay when the overlayStyle property is set to OverlayStyle::Fancy. The media notification area is where you'll find information on media metadata and also any artwork that represents the media content that's currently playing. The media notification area is where you'll also find playback controls.

Notification area signals

The NowPlayingConnection class gives you access to the volume overlay. The volume overlay can be used to display information about the media that's currently playing.

The notification area emits signals that can be used by your app to keep track of when the Play, Pause, Prev or Next media player control buttons have been pressed.

The volume overlay on top of a stock media player app

Working with the notification area

Playing media that is located in the device's shared data areas requires file system access, access permissions, and some familiarity with the device file system. For more information about the shared data areas of the device, see File system access.

Your app must acquire the necessary resources before it can attempt to play any media content. When your app has acquired the resources needed to play media content, it can begin to play the content and use the volume overlay to display media information to users. Your app can acquire exclusive access to the notification area, as you'll see in the following code samples:

The acquired() signal is emitted when an app has successfully acquired exclusive access to the notification area. Using an onAcquired signal handler from within your QML code, your app can handle the acquired() signal by sending media data to the NowPlayingConnection object. The data is then used to set some important properties, such as duration and position. The following QML code sample demonstrates how to use the NowPlayingConnection object to set some media properties, and move the NowPlayingConnection object into the Started state. These NowPlayingConnection properties can only be set after the acquired() signal has been successfully received.

import bb.multimedia 1.4

// ...

attachedObjects: [
    NowPlayingConnection {
        id: nowPlaying

        connectionName: "myConnection"
        icon: "/tmp/artwork.png"

        onAcquired: {
            var metadata = { "track": "MyTrack", "artist" : "MyArtist" };

            // Set a duration of 2 minutes in milliseconds
            nowPlaying.duration = 120000;
            nowPlaying.position = 0;

            // You can go from an Unprepared state directly to
            // the Started state. The Prepared state is implied
            nowPlaying.mediaState = MediaState.Started;

            nowPlaying.setMetaData(metadata);
        }

        // A Pause command has been received from the UI
        onPause: {
            mymedia.pause();
        }


        // A Play command has been received from the UI
        onPlay: {
            mymedia.play();
        }

        // A Stop command has been received from the UI
        onRevoked: {
            mymedia.stop();
        }
    }
]

// ...

onMyMediaWantsToPlay: {
    nowPlaying.acquire();
}

onMyMediaPaused: {
    nowPlaying.mediaState = NowPlayingConnection.MediaStatePaused;
}

onMyMediaPositionChanged: {
    nowPlaying.position = ms;
}

onMyMediaNoLongerWantsToPlay: {
  nowPlaying.revoke();
}

onMyMediaTrackHasChanged: {
  nowPlaying.trackChange();
}

// ...

Not applicable

Not applicable

While it's true that typically the acquired() signal is used to set media data, artwork, and initial position, it can also be used to make function calls to next(), previous(), play(), pause() or stop(). When the acquired() signal has been received, your app can call the most appropriate function to work with your media type.

You have the flexibility to set or update any of the properties of the NowPlayingConnection object until the revoked() signal is received. No other updates are possible after the revoked() signal has been received by the NowPlayingConnection object. It's important to note that signals that are triggered by the notification areas are only emitted while the service is acquired and not currently preempted by a connection with a higher priority, such as a phone connection.

Here's a code sample that shows you how to use the notification area in your app:

Not applicable

The first step is to define a NowPlayingConnection object. In the following C++ code sample, the NowPlayingConnection object is named np. This object is used to acquire exclusive access to the notification area, and set some of the properties of the NowPlayingConnection. The following C++ code sample shows you how to create a pointer to a NowPlayingConnection object. In this code sample, the NowPlayingConnection() constructor, which takes two parameters, is used to create the instance of the NowPlayingConnection object. The first parameter is the name of the service that you want to connect to and the second parameter is the parent object. In this case, the parent is the current object.

#include <bb/multimedia/NowPlayingConnection>

// ...

// Define a NowPlayingConnection object
NowPlayingConnection np("myConnection", this);

// ...

The notification area is used for displaying media metadata in the volume overlay for content that is currently playing. It's also responsible for receiving signal notifications when the media playback controls displayed in it are used.

The next step is to define a QVariantMap to store some metadata that describes your media content. This metadata is used with your np object and the volume overlay to display information to your users. The following C++ code sample shows you how to define some metadata for your NowPlayingConnection.

// ...

// Define a QVariantMap object and
// set some metadata properties
QVariantMap metadata;

metadata[MetaData::Title] = "Sample Track";
metadata[MetaData::Artist] = "Sample Artist";
metadata[MetaData::Album] = "Sample Album";

np->setMetaData(metadata);

// ...

Next, you can define the path to your artwork icon. This step is not necessary to use the NowPlayingConnection but it has been included here to show you how to do it. This icon can be any piece of artwork that represents your media content, such as CD cover art. The following C++ code sample shows you how to use the setIconUrl() function to set the path to your content artwork. The path is defined in the QUrl parameter.

// ...

// Set the path to the artwork icon
// for your media content
np->setIconUrl(QUrl("file://%1/app/native/assets/artwork.png")
    .arg(QDir::currentPath()));

// ...

Finally, your app must make a call to the acquire() function to acquire exclusive access to the notification area. If the NowPlayingConnection is able to acquire exclusive access to the notification area, then it can continue to set some media content properties such as the duration of the track, and the initial starting position. It can also move the NowPlayingConnection object into the Started state.

// ...

// Acquire exclusive access to the notification
// area which is used for displaying now
// playing media metadata 
np->acquire();

// If exclusive access to the notification area
// has been successfully acquired, we can set
// some properties of NowPlayingConnection

// Represents 2 minute duration in milliseconds
np->setDuration(120000);

// Represents the starting position of the track
np->setPosition(0);

// Sets the media state of the NowPlayingConnection
np->setMediaState(MediaState::Started);

// ...

Here are all of the above pieces of the code sample put together.

#include <bb/multimedia/NowPlayingConnection>

// Define a NowPlayingConnection object
NowPlayingConnection np(this);

// Define a QVariantMap object and
// set some metadata properties
QVariantMap metadata;

metadata[MetaData::Title] = "Sample Track";
metadata[MetaData::Artist] = "Sample Artist";
metadata[MetaData::Album] = "Sample Album";

np->setMetaData(metadata);

// Set the path to the artwork icon
// for your media content
np->setIconUrl(QUrl("file://%1/app/native/assets/artwork.png")
        .arg(QDir::currentPath()));

// Acquire exclusive access to the notification
// area which is used for displaying now
// playing media metadata 
np->acquire();

// If exclusive access to the notification area
// has been successfully acquired, we can set
// some properties of NowPlayingConnection

// Represents 2 minute duration in milliseconds
np->setDuration(120000);

// Represents the starting position of the track
np->setPosition(0);

// Sets the media state of the NowPlayingConnection
np->setMediaState(MediaState::Started);

Not applicable

Connection priority

A NowPlayingConnection can only serve one user of its connection service at a time. The NowPlayingConnection uses two connection priorities to manage its service availability. The connection priorities are either low (which is preemptible) or high (which is non-preemptible). Preemptible connections can be automatically restarted, as long as their audio routing has not changed to a more public device, such as from a headset to the handset. Non-preemptible connections cannot be restarted, and can be revoked by a connection with a higher priority.

Before the connection service can be used to record or play media content, it must be acquired by a connection service user, such as the media player. When a connection is acquired, it's considered active. If the connection service is already playing a media file, the service may revoke any previously connected users. This process of revocation is not automatic, it's decided by connection priority, which controls whether the connection is considered by the system to be preemptible or non-preemptible. Connection revocation is done to give exclusive use of the connection service to the acquiring connection user, which becomes the current user of the NowPlayingConnection.

A connection is considered to be preemptible if it can be automatically paused and subsequently restarted by the connection service. Preemptible connections remain active while they're paused, which allows them to be automatically restarted by the service.

There are several use cases that help demonstrate connection priority.

Low priority connection acquired by another low priority connection
If a low priority connection is currently acquired and another low priority connection tries to acquire the service, the first connection is revoked. When a connection has been revoked, it's no longer considered to be acquired, and cannot be automatically restarted. In this use case, the first low priority connection is considered to be non-preemptible because it cannot be automatically restarted.
Low priority connection acquired by a high priority connection
If a low priority connection is currently acquired and a high priority connection tries to acquire the service, the first connection is preempted. This means that the first connection's use of the service is temporarily paused (made non-active), but the connection still remains acquired. When the second connection (with higher priority) has finished using the service and is revoked, then the first connection automatically becomes active again. In this use case, the low priority connection is considered to be preemptible, which means that it can be automatically paused and subsequently restarted.
High priority connection acquired by another low or high priority connection
If a high priority connection is currently acquired, any attempt by any other connection, either with low or high priority, to acquire the service will fail with a "resource unavailable" error. In this use case, all subsequent connections are considered to be non-preemptible because they're all automatically revoked as they attempt to acquire the use of the connection service.

The controllerActive property

You can use the controllerActive property to determine if one or more controllers are actively listening for updates on the connection that is currently playing. You should update this property when your app has been started, even if your app remains inactive. You can use the controllerActive property to determine many useful things (for example, if the volume overlay is being displayed, if Bluetooth is connected, or if any other controller that you're looking for is currently active). You can use the isControllerActive() function to check the value of the controllerActive property. The isControllerActive() function returns true if one or more controllers are active on the connection that is currently playing, and false otherwise.

Multimedia renderer APIs

As an alternative to using the Cascades multimedia suite, you can use the C APIs of the multimedia renderer (mm-renderer) service. The C APIs in the mm-renderer service give you more control over the functionality of your multimedia apps, which can be used to increase your app's performance.

You cannot use the mm-renderer service to record video. To record video, you must use the Camera APIs that are available in the BlackBerry 10 Native SDK. For more information about recording video, see Camera.

The mm-renderer service provides functions that allow you to:

  • Record audio files in your apps
  • Play audio and video files in your apps including local and streamed content
  • Play a sequence of files (audio and video), such as a playlist
  • Monitor and report its current state and report on various state properties, such as its position and buffer level
  • Build accessible multimedia apps
  • Monitor a specific context, which represents the multimedia object that's currently playing

There are more features available in the mm-renderer service than can be mentioned here in this introduction. The trade-off in using the C APIs of the mm-renderer service instead of Cascades is that you must handle the complexities that are abstracted away when using Cascades. Handling these complexities may extend the amount of time required to develop your app.

Working with mm-renderer

The multimedia rendering service (mm-renderer) allows your multimedia apps to request and control the playback of audio and video from files and devices.

The mm-renderer service has APIs that allow your apps to:

  • Specify a set of media files to play
  • Issue playback control commands
  • Retrieve the current playback status
  • Request notifications when the status changes
  • Provide dynamic metadata (such as position in a track or playlist) for media content

The BlackBerry 10 Device Simulator does not include any codecs other than those codecs that support .wav files. To play media files with other formats, such as .mp3 or .aac files, your app must use real hardware that supports those formats.

The multimedia renderer service allows your app to control the playback and recording of multimedia content. It also allows your app to receive events that can be used to monitor various media operations.

When performing permission-sensitive operations, such as opening files, the multimedia renderer service uses the client app's user ID and group IDs while performing those operations.

Supported media categories

The mm-renderer service supports playback of tracks and playlists. It can play content read either from the local file system or through HTTP streams.

A track is an audio or video file such as an MP3 or MP4 file.

A playlist is a list of track URLs. The following types of playlists are supported:

M3U files
The input URL must be a full path (with or without a file: or http: prefix) of a file with an M3U extension. In its simplest form, an M3U file is a plain-text file containing the path names or URLs of the tracks to play (one per line). Playlists for HTTP Live Streaming (HLS) are supported, and these may have an M3U or M3U8 file extension.
SQL queries
The input URL must be of the form sql:database?query=querystring , where database is a full path to a database file and querystring is an SQL query that returns a column of track names. If the query contains any reserved characters, they must be URL-encoded (for example, %26 instead of &).

For a current list of supported media formats, see list of supported media for BlackBerry devices.

Media sources

You can play audio and video tracks from the following sources:

Files
To play media content from files, specify the path (in POSIX format) of a local file in the input URL (see mmr_input_attach() for information on setting the input URL).
HTTP streams
To play media content from HTTP streams, specify an HLS source or another type of HTTP stream as the input URL. The mm-renderer service supports cookies, SSL, and authentication, which enables secure playback of HTTP streams.

Abstraction layers

The multimedia renderer uses a layered architecture to process playback commands and manage the flow of media content from input to output.

The mm-renderer architecture has three abstraction layers:

The messaging layer
This layer decodes client messages and delivers them to contexts. A context is an object capable of playing one piece of input media at a time on a set of output devices. The input can contain both audio and video signals (for example, a movie), and the set of outputs can consist of both audio and video devices (for example, speakers and a screen).
The context management layer
This layer:
  • Keeps track of the outputs attached to each context
  • Maps each output to the engine plugins that can support that output type
  • Selects the correct engine plugins and attaches them to the context
  • Preserves the context state between detaching and reattaching inputs
  • Delivers client requests (such as play) to engine plugins
The plugin management layer
This layer keeps track of all available plugins.

Contexts

The mm-renderer service can create contexts, which can be used to play a media stream concurrently with, and independently of, other contexts. Each context can move output to different hardware devices or files, which creates independent zones of operation.

The operations that are available for a context at a particular point in time depend on the type of input and outputs attached to that context. For example, changing playlists doesn't apply unless your input type is a playlist. Seeking to a new track position doesn't apply to some input streams, such as radio stations, or to devices, such as microphones.

You can set the output of a context to perform operations other than playing. For example, you can record rather than play an audio stream by setting the context's output to a file and its input to an audio recording device, such as a microphone.

Your app must connect to the mm-renderer service and use it to create a context. When the context is created, it has a unique name but no other properties are set. To perform subsequent operations, your app must access the context through the context handle that was assigned to it by mm-renderer service when it created the context.

You can control the properties of the context's operation, such as audio volume, by attaching parameters to the context, to the input, or to each of the outputs.

For more information about parameters, see Parameters for mm-renderer.

Outputs

Each context has to have one or more outputs attached to it before it can play anything. The outputs attached to a context allows it to determine where to send the content. The output for a context can be a file, an audio device, or a video device.

Some engine plugins determine whether to play an input by the type of outputs that are attached to it. In some cases, the engine plugin may not support detaching and reattaching outputs after an input is connected. Therefore, you must attach the outputs before attaching an input.

Inputs

Each context must have at least one input attached to it, which allows the context to determine what to play. When an input is attached, the context management layer selects the correct engine plugin and attaches it to the context. Your app has to identify the input type for mm-renderer because the service cannot automatically detect the type of the attached input.

The input type determines how mm-renderer responds to certain playback requests, such as seeking to a track position or changing playlists. Although an input may be attached to more than one context, mm-renderer doesn't detect or manage conflicting playback operations. Your app must manage potential playback conflicts.

Plugins

Engine plugins are the components used by mm-renderer to process the flow of media data from an input to one or many outputs. The implementation of the engine plugins is invisible to mm-renderer (and its client apps). The context management layer selects the best engine plugin for the task based on the types of inputs and outputs attached to the context, and on the rating that each engine gives itself for the specified inputs and outputs.

QNX Sound Architecture APIs

The QNX Sound Architecture (QSA) provides you with C libraries that give you the lowest level of access to the device's audio hardware components. Working in the QSA involves working directly with PCM devices, memory buffers, and sound chips with very little abstraction, which allows you to work as close to the device hardware as possible. Having direct access to the device's hardware components inherently provides you with limitless flexibility to develop the audio capabilities of your app. This level of access means that your app must handle all of the complexities that are abstracted away when using one of the other choices shown above.

To record or play audio, your app must have access to the following items:

  • A sound card or sound chip on a device
  • A device driver for the sound card or sound chip
  • An API to communicate with the driver

The QSA is based on version 0.5.2 of the Advanced Linux Sound Architecture (ALSA).

Cards and devices

The basic piece of hardware needed to play or record sound is an audio chip or sound card, referred to as a card. All of the sound devices are attached to a card. To reach a device, you must know what card it's attached to. The following diagram shows cards and devices:

Diagram showing cards and devices.

The devices include:

  • Control device
  • Mixer device
  • Pulse Code Modulation (PCM) devices

The resulting list includes one control device for every sound card, starting from card 0, as well as the PCM and mixer devices for each card.

Control device

There's one control device for each sound card in a system. This control device is special because it doesn't directly manage any real hardware. It's a concentration point for information about its card and the other devices that are attached to its card. The control device includes the type and number of additional devices that are attached to the card.

Mixer devices

Mixer devices are responsible for combining or mixing the various analog signals on the sound card. A mixer may also provide a series of controls for selecting which signals are mixed and how they're mixed together, adjusting the gain or attenuation of signals, or muting signals.

For more information, see Working with audio mixers.

Data formats

The QSA supports various data formats and the asound.h header file defines two sets of constants for these data formats. The two sets are related (and easily converted between) but serve different purposes:

SND_PCM_SFMT_*
A single selection from the set of data formats. For a list of the supported formats, see snd_pcm_get_format_name().
SND_PCM_FMT_*
A group of (one or more) formats within a single variable. This constant is useful for specifying the format capabilities of a device, for example.

Generally, the SND_PCM_FMT_* constants are used to convey information about raw potential, and the SND_PCM_SFMT_* constants are used to select and report a specific configuration.

You can build a format from its width and other attributes, by calling snd_pcm_build_linear_format().

You can use the following functions to check the characteristics of a format:

Pulse Code Modulation (PCM) devices

PCM devices are responsible for converting digital sound sequences to analog waveforms, or analog waveforms to digital sound sequences. Each device works only in one mode or the other. When it converts digital to analog, it's a playback channel device; when it converts analog to digital, it's a recording channel device.

The attributes of PCM devices include:

  • The data formats that the device supports (16-bit signed little-endian, 32-bit unsigned big-endian, and so on). For more information, see Data formats.
  • The data rates that the device can run at (48KHz, 44.1kHz, and so on)
  • The number of streams that the device can support (2-channel stereo, mono, and 4-channel surround, and so on)
  • The number of simultaneous client apps that the device can support, referred to as the number of subchannels the device has. Most sound cards support only one subchannel, but some cards can support more. For example, Sound Blaster Live! supports 32 subchannels.

    The maximum supported number of subchannels is a hardware limitation. On single-subchannel cards, this limitation is artificially surpassed using a software solution: the software subchannel mixer. The software subchannel mixer allows eight software subchannels to exist on top of the single hardware subchannel.

    The number of subchannels that a device advertises as being supported is defined using the best-case scenario; in fact, the device may support fewer subchannels. For example, a device might support 32 simultaneous client apps if they all run at 48 kHz, but it may only support eight client apps when the rate is 44.1 kHz. In this case, the device still advertises 32 subchannels.

PCM state machine

A PCM device is a data buffer that's converted, one sample at a time, by either a Digital Analog Converter (DAC) or an Analog Digital Converter (ADC), depending on direction. This idea becomes a little more complicated in QSA because of the concept that the PCM subchannel is in a state at any given moment.

These states are defined as follows:

SND_PCM_STATUS_NOTREADY
The device is in its initial state.
SND_PCM_STATUS_READY
The device has its parameters set for the data it operates on.
SND_PCM_STATUS_PREPARED
The device has been prepared for operation and is able to run.
SND_PCM_STATUS_RUNNING
The device is running, transferring data to or from the buffer.
SND_PCM_STATUS_UNDERRUN
This state happens only to a playback device and is entered when the buffer has no more data to be played.
SND_PCM_STATUS_OVERRUN
This state happens only to a recording device and is entered when the buffer has no room for data.
SND_PCM_STATUS_PAUSED
The device has been paused.

It's possible to transition from the SND_PCM_STATUS_PAUSED state to the SND_PCM_STATUS_PREPARED state when the PCM start mode (snd_pcm_channel_params_t.start_mode) is set to SND_PCM_START_DATA or SND_PCM_START_FULL (using snd_pcm_plugin_params()). The state change occurs after calling one of the snd_pcm_*_resume() functions.

SND_PCM_STATUS_UNSECURE
The app marked the stream as protected, the hardware level supports a secure transport (such as HDCP for HDMI), and authentication was lost.
SND_PCM_STATUS_ERROR
A hardware error has occurred, and the stream must be prepared again.
SND_PCM_STATUS_CHANGE
The stream has changed and must be prepared again.

In some cases, audio is redirected transparently between different audio devices, with different capabilities. You can enable this behavior by calling:

snd_pcm_plugin_set_enable(handle, PLUGIN_ROUTING);

If routing is enabled and the preferred device changes to an external device, such as HDMI, audio is redirected automatically to that device. When routing changes, the app receives a status of SND_PCM_STATUS_CHANGE.

When the app receives this status, it may call snd_pcm_channel_prepare() to enter the SND_PCM_STATUS_PREPARED state and be ready to continue playback, but the capabilities may have changed. For example, HDMI may support surround sound output but a local speaker does not. In this case, the app should call snd_pcm_channel_info() to check the capabilities and change the audio characteristics in response before preparing.

SND_PCM_STATUS_PREEMPTED
Audio is blocked because another libasound session has initiated playback, and the audio driver has determined that the other session has higher priority. So, the lower priority session is terminated with a state of SND_PCM_STATUS_PREEMPTED. When it receives this state, the app should stop playback and not try to resume until either the sound it wants to produce has increased in priority or a user initiates a retry.

The following diagram shows the transition between various PCM states.

Diagram showing the various PCM states.

The transition between states is the result of making a function call, or the result of conditions that occur in the hardware.

Software PCM mixing

In the case where the sound card has a playback PCM device with only one subchannel, the device driver writer can choose to include a PCM software mixing device. This device appears as a new PCM playback device that supports many subchannels, but it has a few differences from a true hardware device:

  • The mixing of the PCM streams is done in software using the CPU. Even with only one stream, the CPU is used more than if the hardware device was used.
  • When the PCM software mixer is started, it opens a connection to the real hardware device. If the real hardware device is already in use, the PCM software mixer can't run. Likewise, if the PCM software mixer is running, the real hardware device is in use and is unavailable.

The PCM software mixer is attached to a single hardware PCM device. This one-to-one mapping allows a function call to identify the PCM software-mixing device that's associated with its hardware device.

PCM plugin converters

In some cases, an app creates data in one format but the PCM device that it's trying to use can only accept data in a different format. In this case the app, such as an MPG decoder, can reformat its data to change it into a format that the PCM device can accept. Another option is for the app to use the QSA to do the conversion for it.

The conversion is accomplished by the QSA by using a series of plugin converters, each capable of doing a specific conversion. For example, the rate plugin converter converts a stream from one sampling frequency to another. There are plugin converters for bit conversions (8 to 16-bit, and so on), endian conversion (little-endian to big-endian or the opposite way), and voice conversions (stereo to mono, and so on).

To minimize CPU usage, the minimum number of plugin converters are used to translate the input format to the output format. Your app can access the plugin converters by using the PCM plugin API functions. These API functions all have the word "plugin" in their names. For more information, see Audio Library.

The ability to convert audio to match hardware capabilities (such as voice conversion, rate conversion, type conversion, and so on) is enabled by default. This ability impacts the functions snd_pcm_channel_params(), snd_pcm_channel_setup(), and snd_pcm_channel_status(). These functions behave as snd_pcm_plugin_params(), snd_pcm_plugin_setup(), and snd_pcm_plugin_status(), unless you've disabled the conversion by calling:

snd_pcm_plugin_set_disable(handle, PLUGIN_CONVERSION);

To avoid undefined or unpredictable behavior, don't mix the plugin API functions with the nonplugin functions.

Optimizing audio

Here are the best practices to reduce audio latency:

Match sample rates and data formats Ensure that sample rates and data formats are matched between the libasound app and the audio hardware. This eliminates the need for a Sample Rate Conversion (SRC) or any other data conversion in libasound.

Disable the libasound sub-buffering plugin Make sure that the read and write tasks for your app are in the exact audio fragments or block size, and disable the libasound sub-buffering plugin by calling snd_pcm_plugin_set_disable():

snd_pcm_plugin_set_disable(pcm_handle,
                           PLUGIN_DISABLE_BUFFER_PARTIAL_BLOCKS);

Use a small audio fragment or block size In your app, configure the audio interface to use a smaller audio fragment or block size. When playing to the software mixer, the fragment size is locked to the software mixer's fragment size, which is 4 KB by default.

Make sure to look at the fragment size that's returned by the snd_pcm_plugin_setup() function, because the audio interface may not be able to satisfy your fragment size request. This situation depends on various factors such as Direct Memory Access (DMA) alignment requirements, potentially required data conversions, and so on.

Use the SND_PCM_START_GO playback start mode In your app, set the playback start mode to be SND_PCM_START_GO and then issue the "go" command by calling snd_pcm_playback_go() after you've written two audio fragments or blocks into the interface. To record, use SND_PCM_START_DATA, which enables recording in the app as soon as one fragment of data is available.

Avoid buffer underrun condition When you're using the software mixer, you must wait until three audio fragments or blocks are written into the audio interface before issuing the "go" command, otherwise a buffer underrun may occur.

Related resources

Web-based training

Design guidelines

Last modified: 2015-07-24



Got questions about leaving a comment? Get answers from our Disqus FAQ.

comments powered by Disqus