Audio

Overview

The audio API will allow the user to make use of all the functionalities of the audio system and / or audio codecs that may be installed in the GMV equipments and are making use of the Deeepsy Platform. The aim of this API is to supply functions and give an easy access to the different audio capabilities such as play or record in a efficient way.

A general outline of the Audio API can be observed in the following figure:

audioApi.svg
Audio API Diagram

Main functionalities

The Audio API provides the following functionalities:

  • Audio Clients and ports
    • Get output and input port list
    • Create port
    • Delete port
    • Assign/release a GPO to port
  • Audio connections
    • Create connection between ports
    • Delete connection
    • List existing connections
    • Set connection volume
    • Set connection gain
  • Play and record sound tracks
    • Play audio track
    • Stop audio track
    • Start recording
    • Stop recording
    • Get task status (track or recording)
    • Get all tasks list
  • Audio events
    • Port events (created / deleted)
    • Connection list events (created / deleted)
    • Connection status events (enabled / disabled / updated)
    • Track events
    • Track status events
    • Recording events
    • Recording status events
    • Port Gpio assignment/release
  • Check if the service is available by the use of request method isAlive() or subscription method .

What should you know before starting your application development?

Audio API guide

In this guide we will explain you what is happening behind the scenes whenever you use either the Audio API or the Audio tool.

  • Getting started with Audio API

    The Audio API allows the user to know all the system Audio Output and Input ports available in the system or created by the user by retrieving the port list. Each port has a unique client and a name. This port list will show the user:

    • Input ports, which work as receivers of the different sound streams (sound sinks).

      System input ports may look like this:

      audio_system_input_ports_example.png
      Example of system input ports



    • Output ports, which work as senders of sound streams (sound source).

      System output ports may look like this:

      audio_system_output_port_example.png
      Example of system output port



  • Creating user ports

    In order to work with the system ports the user will need to create one or more output and input ports for his/her application. The client of user ports must start with "user" and is unique. Within the "Create Port" functionality the user will be able to do this easily but, what type of ports do we need ?
    Well, if our purpose is to play a specific sound track or music track and send it to a specific speaker or headphones, we need an output port. This port will be responsible of sending the data stream directly from our client application to the system port. Once created, a user port (client = user , port = output, type = Output) may look like this:

    audio_user_output_port_example.png
    Example of user output port



    If the main purpose of our application is to is record or save a sound that is being received via a microphone or played via a speaker we would create an input port.

  • Using and connecting ports

    Once we have created at least a user port to work with, in order to use it and transmit/receive audio, we need to connect it to a specific port ("Create Connection" functionality). Connections must always have a output port as an origin port and a input port as destination port. Taking this information into account, connecting a user with a system port may look something like this:

    audio_user_port_one_connection.png
    Example of user connection: user port connected to system port



    Right now we are ready to send an audio via the user port "user:output" to the output "external_speaker_r:playback_1".
    It might be possible to connect one output port to multiple input ports at the same time. This depends on the OBU and priority limitations as will be seen later. As an example this would lead us to something like this:

    audio_user_port_two_connections.png
    Example of user connection: user port connected to two system ports



  • Playing a sound

    Given we have a user port and we have connected it to the desired system port, playing a sound is straight forward. We just need to specify the output port we want to use and the audio file path . Following the previous example, if we decide to play a sound using the same user port as before (client = user , port = output, type = Output), the result would be:

    audio_play_sound_stream.png
    Example of play sound



    Note: if the user port is connected to two different inputs, the specified sound will be sent, and therefore heard, in both of them.


Manage connection priorities

In this section we will explain the priority concept for a connection and how they are enabled or disabled according to it.

  • Priority motivation

    There are limited audio resources in a device. It can happen for example that two applications access the same physical audio resource at the same time (for example the speaker system). One could be panel information and the other an alert warning to the passengers. It is necessary to establish which will take control of the resource for avoiding collision between applications. In the previous example, the passenger alert information will have a higher priority than others, and thus, the audio resource will be available booked for this application.

  • Getting started with connection priority

    The priority is a parameter associated with a connection. When a connection is created using the API, the priority value can be set. If a connection has a priority higher than another, the audio service will book the necessary audio resources for that connection, blocking or denying that resource to other connections with lower priority. The priority value can be from 0 (HIGHEST) to 100 (LOWEST). The priority value MUST be unique between the connections except the value 0. That is explained in the section Priority Zero

    audio_system_priority_example.png
    Example connections with priority set

  • Connection priority management

    The audio service reads the connection priority list from highest to lowest priority. When a connection uses two audio resources that are free the audio service books both and the connection is marked as ENABLED state. A connection that requires a resource that is already booked is marked as the DISABLED state. Tasks over disabled connections won't be played or recorded because the connection is not active.

  • Connection life-cycle

    When a connection is created, is created with the DISABLED state. The audio service then checks all the connections (sorted by priority) and according to the resources available internally creates a list of connections that need to change their status to ENABLED or DISABLED. Firstly, it manages those connections then changes their status to disabled and then manages connections that change to enabled status. When a connection is disabled, the audio service disposes the link between the origin and destination port. When a connection is enabled, the audio service makes all the necessary actions, (physical or software) for linking the origin and destination port. After a connection changes its status successfully, the service publishes an event indicating the new state.

    When a connection is deleted, the audio service repeats the process for establishing the status of the remaining connection list.

    audio_system_enabled_example.png
    Example of enabled and disabled connections

  • Blocking list

    Apart from the status ENABLED or DISABLED, the connection has another property related to this topic: the blocking list. The blocking list shows the enumeration of connections that prevent this connection to be ENABLED. It is because they have the same audio resources (one or both) and have a higher priority than his connection. Connection with status ENABLED has this list empty because no other connection blocks this to be ENABLED.

    audio_system_blocking_list_example.png
    Example of a connection blocked by other

  • Priority and hardware limitations

    For considering a connection available, in addition to the priority, other hardware limitations are considered. Different devices have different hardware limitations on left / right channels, speakers, and microphones. It can happen that a DISABLED connection apparently should be ENABLED, if this happens it is due to the internal hardware connection it is already booked by another connection.

  • Effect of priorities on audio playbacks

    Given the audio track volume and speaker control volume are configured correctly, if we are using connections with priorities (priority greater than 0) when playing an audio track through an user origin port the result will depend on the status of the connections that use that origin port:

    • If the origin port has an enabled connection while playing audio track, sound will be heard though the speaker linked to the input port of that connection.
    • If the origin port has no enabled connection while playing audio track, sound is not heard, but playback starts and continues normally until the end.
    • If a disabled connection is enabled while playing audio track, sound will begin to be heard in the middle of the playback.
    • If an enabled connection is disabled while playing audio track, sound will stop being heard through the input port of that connection.

    Note: When using connections with priorities an audio task will be only be queued if there is another audio task for the same port in progress. Disabling or enabling a connection will not affect the task status.

  • Effect of priorities on voice calls

    Normally, when using VoiceCall service two type of connections will be involved when making calls:

    • Connection between call output port (e.g. baresip:output_port) and system input port. If this connection is enabled and volume is configured correctly, sound from the call should be sent through speaker.
    • Connection between system output port and call input port (e.g. baresip-01:input_port). If this connection is enabled and capture is configured correctly, audio should be captured from microphone and sent to call.

  • Priority Zero

    When a connection has the priority 0 (highest) will be ENABLED always. It will be abled to use any audio resource if it is available. If there are two or more connections with priority 0 they will use the audio resources regardless that other connections with priority 0 may be using the same resources at the same time. That may lead to unexpected errors during playing or recording if the resource is already in use. So creating a connection with priority 0 is not recommended and should be used only in very specific scenarios.

    audio_system_priority_zero_example.png
    Example of connection list with priority zero


Manage connection volume and gain

Every connections have two parameters linked to each of them: volume and gain. However, depending on the type of connection this parameter may or may not affect.

  • Connection between user/call output port and system input port. If this connection is enabled, the volume percentage of this connection is applied to the control of the speaker linked to the system input port. The gain parameter does not affect.
  • Connection between system output port and user/call input port. If this connection is enabled, the gain percentage of this connection is applied to the control of the microphone linked to the system output port. The volume parameter does not affect.
  • Connection between system output port and system input port. If this connection is enabled, the volume percentage of this connection is applied to the control of the speaker linked to the system input port and the gain percentage is applied to the control of the microphone linked to the system output port.

Remark: Connection volume and volume of audio configuration are not the same. Connection volume modifies the controls of speakers both in playbacks and voice calls, while configuration volume modifies the volume of audio tracks that are played.

When creating a connection, user may not want to set any value of volume and/or gain. In that case, they are initialized with a value of -1 indicating that this connection is not modifying any control of volume or gain.

Manage connections for voice calls

When making calls using VoiceCall service new ports will be created depending on the call type and the call state. User will then be able to connect these ports to system ports.

VoIP

For an incoming call, when call state changes to Ringing the following port is created.

  • baresip:output_1 : Output port that can be connected to a system input port (speaker). Ring tone is played through this port.

Then, when call state changes to Conversation the following ports are created:

  • baresip-01:output_1 : Output port that can be connected to a system input port (speaker). Sound coming from the call is sent through this port.
  • baresip-02:input_1 : Input port that can be connected to a system output port (microphone). Sound coming from the output port is sent to the call through this port.

For an outgoing call, when call state changes to Conversation the following port is created.

  • baresip:output_1 : Output port that can be connected to a system input port (speaker). Sound coming from the call is sent through this port.
  • baresip-01:input_1 : Input port that can be connected to a system output port (microphone). Sound coming from the output port is sent to the call through this port.

GSM

Both for incoming and outgoing calls, when call state changes to Conversation the following ports are created:

  • builtin_modem_in:capture_1 : Output port that can be connected to a system input port (speaker). Sound coming from the call is sent through this port.
  • builtin_modem_out:playback_1 : Input port that can be connected to a system output port (microphone). Sound coming from the output port is sent to the call through this port.

Port assigned GPIO management

Any user can assign a GPIO to an audio port to be activated when desired port is present in an active connection. To release a GPIO from port assignment, an empty string must be passed as GPIO alias parameter. A port can be assigned an unique GPIO, if another one is assigned, previously selected GPIO will be replaced. On the other hand, a single GPIO can be assigned to multiple ports.

GPIO activation/deactivation policy

Assigning GPIO to port

When a port is assigned to a GPIO, if there is an active connection using that port, GPIO will be activated inmediately. Otherwise, GPIO will be deactivated.

Releasing GPIO to port

When a GPIO is released from a port assignment, if there is no active connection using that port or any other port which that GPIO is assigned to, GPIO will be deactivated inmediately. Otherwise, GPIO will remain at its previous state.

Activating a connection

When a connection using a GPIO assigned port becomes active, GPIO will be activated inmediately. Otherwise, GPIO will remain at its previous state.

Deactivating a connection

When a connection using a GPIO assigned port becomes inactive, if there is no active connection using that port or any other port which that GPIO is assigned to, GPIO will be deactivated. Otherwise, GPIO will remain at its previous state

Sound system for the different OBUs

RMDT100 + DTD100 / UC100

By default, the Audio service is launched and will show the system and user input and output ports available. To be able to use the Audio service / API for RMDT100 + DTD100 / UC100 properly, some compatibility considerations should be taken into account:

  • The RMDT100 / UC100 board provides 4 system inputs which, at the moment, are named like this:
    • internal_speaker_l:playback_1 : Amplified internal speaker
    • external_speaker_r:playback_1 : Amplified external speaker
    • headphone_jack_r:playback_1 : Headphone jack
    • headphone_jack_l:playback_1 : Headphone jack

    Remark: Due to audio codec channels limitation it is just possible to play sounds trough a maximum of 2 RMDT100 / UC100 system inputs at the same time.


  • The RMDT100 / UC100 board provides 2 system outputs which, at the moment, are named like this:

    • external_microphone:capture_1 : External microphone
    • internal_microphone:capture_1 : Internal microphone

  • If the DTD100 is connected and available, the system will also list the ports that exist in this board, in this case one additional input port and one output port:
    • hmi100:to_slave_1 : Default Input port, used for playing (sink)
    • hmi100:from_slave_1 : Default Output port, used as a recording source (unavailable for the moment)

    Remark: DTD100 input port is independent and is not limited by any other RMDT100 / UC100 system input port.

TV10

By default, the Audio service is launched and will show the system and user input and output ports available. To be able to use the Audio service / API for TV10 properly, some compatibility considerations should be taken into account:

  • The TV10 board provides 1 system input :
    • default_speaker:playback_1 : Default TV10 speaker for playback

EP100

The EP100 does not launch the Audio service by default as the only available audio source is the EP100 modem.

EP200

By default, the Audio service is not launched. We can add it to the system using the following commands:

dpyplatformmanagertool - -add Audio audio_service /usr/bin/ 5 true false

dpyplatformmanagertool - -start Audio

To be able to use the Audio service / API for EP200 properly, some compatibility considerations should be taken into account:

  • The EP200 provides 3 system inputs which, at the moment, are named like this:
    • driver_speaker:playback_1 : amplified driver speaker
    • PIS_internal:playback_1 : passenger information system internal speaker
    • PIS_external:playback_1 : passenger information system external speaker

    Remark: There are two versions of EP200, one with single codec and one with dual codec. EP200 with single codec has the limitation that it is only possible to play sounds trough a maximum of 2 system inputs at the same time. In contrast, EP200 with dual codec does not have this limitation, so it allows to play sounds through up the 3 system inputs at the same time.


  • The EP200 provides 2 system outputs which, at the moment, are named like this:
    • ambient_microphone:capture_1 : Ambient microphone
    • driver_microphone:capture_1 : Driver microphone

TV100

By default, the Audio service is not launched. We can add it to the system using the following commands:

dpyplatformmanagertool - -add Audio audio_service /usr/bin/ 5 true false

dpyplatformmanagertool - -start Audio

To be able to use the Audio service / API for TV100 properly, some compatibility considerations should be taken into account:

  • The TV100 provides 1 system input which, at the moment, is named like this:
    • passenger_speaker:playback_1 : passenger speaker

DTD200

By default, the Audio service is launched and will show the system and user input and output ports available. To be able to use the Audio service / API for DTD200 properly, some compatibility considerations should be taken into account:

  • The DTD200 provides 5 system inputs which, at the moment, are named like this:
    • internal_speaker:playback_1 : internal speaker
    • passenger_speaker:playback_1 : passenger speaker
    • driver_speaker:playback_1 : amplified driver speaker
    • PIS_internal:playback_1 : passenger information system internal speaker
    • PIS_external:playback_1 : passenger information system external speaker

    Remark: DTD200 uses two codecs. One codec is shared by internal_speaker, driver_speaker and PIS_internal and due to channels limitation it is only possible to play sounds trough a maximum of 2 of those 3 system inputs at the same time. The other codec is shared by passenger_speaker and PIS_external, which are not limited by the other system inputs. This means it could be possible to play through up to 4 ports at the same time


  • The DTD200 provides 3 system outputs which, at the moment, are named like this:
    • ambient_microphone:capture_1 : ambient microphone
    • driver_microphone:capture_1 : driver microphone
    • internal_microphone:capture_1 :internal microphone

REC30

By default, the Audio service is launched and will show the system and user input and output ports available. To be able to use the Audio service / API for REC30 properly, some compatibility considerations should be taken into account:

  • The REC30 provides 5 system inputs which, at the moment, are named like this:
    • radio_output:playback_1 : Radio speaker
    • modem_output:playback_1 : Modem speaker
    • external_speaker:playback_1 : External speaker
    • passenger_speaker:playback_1 : Passenger speaker
    • driver_speaker:playback_1 : Driver speaker

    Remark: At the moment it is just possible to play sounds trough a maximum of 2 system inputs at the same time.


  • The REC30 provides 4 system outputs which, at the moment, are named like this:
    • passenger_microphone:capture_1 : Passenger microphone
    • driver_microphone:capture_1 : Driver microphone
    • radio_input:capture_1 : Radio microphone
    • modem_input:capture_1 : Modem microphone

    Remark: At the moment it is just possible to capture sounds through 1 system output at the same time.

Generated by   GMV