Android RTMP协议直播之音视频采集

上文Android RTMP协议直播之H264标准(二)中对H264标准进行了总结,本文将介绍Android中的音视频结构并介绍如何利用Android现有API进行音视频数据采集。

Android音频架构

在Android系统中按实现功能分为多个层,如下图。



  • 应用框架:
    应用框架中包含多个Android系统定义的音频API,API内部通过JNI以访问与音频硬件互动的原生代码,这些API都在android.media包下,如本篇文章将要介绍的AudioRecord。
  • JNI:
    与 android.media 关联的 JNI 代码可调用较低级别的原生代码,以访问音频硬件。JNI 位于 frameworks/base/core/jni/ 和 frameworks/base/media/jni 中。
  • 原生框架:
    原生框架可提供相当于 android.media 软件包的原生软件包,从而调用 Binder IPC 代理以访问媒体服务器的特定于音频的服务。原生框架代码位于 frameworks/av/media/libmedia 中。
  • Binder IPC:
    Binder IPC 代理用于促进跨越进程边界的通信。代理位于 frameworks/av/media/libmedia 中,并以字母“I”开头。
  • 媒体服务器:
    媒体服务器包含音频服务,这些音频服务是与您的 HAL 实现进行互动的实际代码。媒体服务器位于 frameworks/av/services/audioflinger 中。
  • HAL:
    HAL 定义了由音频服务调用且您必须实现以确保音频硬件功能正常运行的标准接口。音频 HAL 接口位于 hardware/libhardware/include/hardware 中。如需了解详情,请参阅 audio.h。
  • 内核驱动程序:
    音频驱动程序可与您的硬件和 HAL 实现进行互动。您可以使用高级 Linux 声音体系 (ALSA)、开放声音系统 (OSS) 或自定义驱动程序(HAL 与驱动程序无关)。

Android媒体架构

Android系统在Android.media包中定义了许多支持媒体操作的API,例如硬件编解码MediaCodec,通过下图我们来了解Android媒体结构。



因本次视频采集使用Camera采集数据,本片文章不对相关媒体进行介绍。后续文章将对MediaCodec进行介绍。

音视频采集

本次项目采用软编码方式实现,音视频编码和推流全都在Native下实现。框架结构如下;



使用Camera采集直播数据

使用SurfaceView预览Camera,大致流程SurfaceHolder和Camera进行绑定,Camera将缓冲区的数据渲染到SurfaceView上达到预览效果,具体使用流程本文不做详细介绍,只针对直播过程的注意点作说明。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
public class VideoPusher extends APusher implements SurfaceHolder.Callback,VideoPushInterface, Camera.PreviewCallback {
private Builder builder;
private Camera mCamera;
private byte[] callbackBuffer;

public Builder getBuilder() {
return builder;
}

public VideoPusher(Builder builder) {
this.builder = builder;

}

@Override
public void prepare() {
builder.surfaceHolder.addCallback(this);
}

@Override
public void startPush() {
isPushing = true;
builder.getNativePush().setNativeVideoOptions(builder.getWidth(),builder.getHeight(),builder.getBitrate(),builder.getFps());
builder.getNativePush().startPush();
}

@Override
public void pausePush() {
isPushing = false;
builder.getNativePush().pausePush();
}

@Override
public void stopPush() {
isPushing = false;
builder.getNativePush().stopPush();

}

@Override
public void free() {
stopPush();
stopPreview();
builder.getNativePush().free();

}


@Override
public void surfaceCreated(SurfaceHolder holder) {
startPreview();
}

@Override
public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {

}

@Override
public void surfaceDestroyed(SurfaceHolder holder) {

}

@Override
public void stopPreview() {
if (mCamera != null) {
mCamera.stopPreview();
mCamera.release();
mCamera = null;
}
}

@Override
public void startPreview() {
//当surfaceView被创建的时候启动摄像头预览
try {
mCamera = Camera.open(builder.getCameraId());
Camera.Parameters parameters = mCamera.getParameters();
List<Camera.Size> sizeList = parameters.getSupportedPreviewSizes();

parameters.setPreviewFormat(ImageFormat.NV21);
//parameters.setPreviewFpsRange(24,25);
parameters.setPictureSize(builder.getWidth(),builder.getHeight());
parameters.setPreviewSize(builder.getWidth(),builder.getHeight());
mCamera.setParameters(parameters);
mCamera.setPreviewDisplay(builder.getSurfaceHolder());
mCamera.setDisplayOrientation(90);
//ARGB 8888 1pix 4字节
callbackBuffer = new byte[builder.getWidth() * builder.getHeight() * 4];
mCamera.addCallbackBuffer(callbackBuffer);
mCamera.setPreviewCallbackWithBuffer(this);

mCamera.startPreview();
} catch (Exception e) {
e.printStackTrace();
}
}


public void switchCamera() {
stopPreview();
startPreview();
}

@Override
public void onPreviewFrame(byte[] data, Camera camera) {
if (mCamera != null) {
mCamera.addCallbackBuffer(callbackBuffer);
}

if (isPushing) {
builder.getNativePush().sendVideo(data);
}


}

public static class Builder {
private int width;
private int height;
private int cameraId;
private int bitrate = 480000;
private int fps = 25;
private NativePush nativePush;
private SurfaceHolder surfaceHolder;

public int getWidth() {
return width;
}

public Builder width(int width) {
this.width = width;
return this;
}

public int getHeight() {
return height;
}

public Builder height(int height) {
this.height = height;
return this;
}

public SurfaceHolder getSurfaceHolder() {
return surfaceHolder;
}

public Builder surfaceView(SurfaceHolder surfaceHolder) {
this.surfaceHolder = surfaceHolder;
return this;
}

public int getCameraId() {
return cameraId;
}

public Builder cameraId(int cameraId) {
this.cameraId = cameraId;
return this;
}

public int getBitrate() {
return bitrate;
}

public Builder bitrate(int bitrate) {
this.bitrate = bitrate;
return this;
}

public int getFps() {
return fps;
}

public Builder fps(int fps) {
this.fps = fps;
return this;
}

public NativePush getNativePush() {
return nativePush;
}

public Builder nativePush(NativePush nativePush) {
this.nativePush = nativePush;
return this;
}
public VideoPusher build() {
return new VideoPusher(this);
}

}

}

注: 当设置Camera的PictureSize和PreviewSize时候,应该使用当前设备支持的分辨率,否则将会出现异常。
当启动推送的时候会调用startPush方法,在该方法中有以下代码

1
2
//该代码用于在推送之前初始化在Native层视频编码前的准备工作
builder.getNativePush().setNativeVideoOptions(builder.getWidth(),builder.getHeight(),builder.getBitrate(),builder.getFps());

setNativeVideoOptions最终调用native方法,这里贴出native代码后续文章在对其作解释。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
JNIEXPORT void JNICALL
Java_com_ben_android_live_NativePush_setNativeVideoOptions(JNIEnv *env, jobject instance,
jint width, jint height, jint bitrate,
jint fps) {
LOGI("%s", "setNativeVideoOptions...");

//0延迟
x264_param_default_preset(&param, "ultrafast", "zerolatency");
param.i_csp = X264_CSP_I420;
param.i_width = width;
param.i_height = height;

//设置yuv长度
y_len = width * height;
u_len = y_len / 4;
v_len = u_len;

//码率控制,CQP(恒定质量),CRF(恒定码率),ABR(平均码率)
param.rc.i_rc_method = X264_RC_CRF;
//码率 单位(Kbps)
param.rc.i_bitrate = bitrate / 1000;
//瞬时最大码率
param.rc.i_vbv_max_bitrate = bitrate / 1000 * 1.2;
//通过fps控制码率,
param.b_vfr_input = 0;
//帧率分子
param.i_fps_num = fps;
//帧率分母
param.i_fps_den = 1;
param.i_timebase_den = param.i_fps_num;
param.i_timebase_num = param.i_fps_den;
//是否把SPS PPS放入每个关键帧,提高纠错能力
param.b_repeat_headers = 1;
//设置level级别,5.1
param.i_level_idc = 51;

//设置档次
x264_param_apply_profile(&param, "baseline");

x264_picture_alloc(&pic, param.i_csp, param.i_width, param.i_height);
x264_encoder = x264_encoder_open(&param);
if (x264_encoder) {
LOGI("initVideoOptions:%s", "success");
} else {
LOGE("initVideoOptions:%s", "failed");
}

}

通过设置PreviewCallbackWithBuffer回调函数,当程序启动直播时就会不断的回调函数,将Camera预览的数据发送到Native进行编码处理。

NativePush

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
public class NativePush {

static{
System.loadLibrary("live");
}

public native void setNativeVideoOptions(int width,int height,int bitrate,int fps);
public native void setNativeAudioOptions(int sampleRateInHz,int channel);
public native void sendVideo(byte[] data);
public native void sendAudio(byte[] audioData, int offsetInBytes, int sizeInBytes);
public native void prepare();
public native void startPush();
public native void pausePush();
public native void stopPush();
public native void free();

}

使用AudioRecord采集音频数据

当直播开始时,启动一个线程并不断的从AudioRecord中读取数据发送到Native编码。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
public class AudioPusher extends APusher {
private Builder builder;
private int bufferSize;
private AudioRecord audioRecord;
private Thread mAudioThread;

public AudioPusher(Builder builder) {
this.builder = builder;
}

@Override
public void prepare() {
bufferSize = AudioRecord.getMinBufferSize(builder.getSampleRateInHz(), builder.getChannelConfig(), builder.getAudioFormat());
audioRecord = new AudioRecord(MediaRecorder.AudioSource.MIC, builder.getSampleRateInHz(), builder.getChannelConfig(), builder.getAudioFormat(), bufferSize);

}

@Override
public void startPush() {
isPushing = true;
builder.getNativePush().setNativeAudioOptions(builder.getSampleRateInHz(), builder.getChannelConfig());
mAudioThread = new Thread(new AudioPushTask());
mAudioThread.start();
}

@Override
public void pausePush() {
isPushing = false;
}

@Override
public void stopPush() {
isPushing = false;
builder.getNativePush().stopPush();
}

@Override
public void free() {
if (audioRecord != null) {
audioRecord.stop();
audioRecord.release();
audioRecord = null;
}
builder.getNativePush().free();
}


public static class Builder {
private int audioSource;
private int sampleRateInHz;
private int channelConfig;
private int audioFormat;
private int bufferSizeInByte;

private NativePush nativePush;


public int getAudioSource() {
return audioSource;
}


public int getSampleRateInHz() {
return sampleRateInHz;
}

public Builder sampleRateInHz(int sampleRateInHz) {
this.sampleRateInHz = sampleRateInHz;
return this;
}

public int getChannelConfig() {
return channelConfig;
}

public Builder channelConfig(int channelConfig) {
this.channelConfig = channelConfig;
return this;
}

public int getAudioFormat() {
return audioFormat;
}

public Builder audioFormat(int audioFormat) {
this.audioFormat = audioFormat;
return this;
}

public int getBufferSizeInByte() {
return bufferSizeInByte;
}

public NativePush getNativePush() {
return nativePush;
}

public Builder nativePush(NativePush nativePush) {
this.nativePush = nativePush;
return this;
}

public AudioPusher build() {
return new AudioPusher(this);
}
}

private class AudioPushTask implements Runnable {

@Override
public void run() {
audioRecord.startRecording();

while (isPushing && audioRecord != null) {
byte[] buffer = new byte[bufferSize];
int len = audioRecord.read(buffer, 0, bufferSize);
if (len > 0) {
// builder.getNativePush().sendAudio(buffer, 0, len);
}
}
}
}
}

和视频编码一样在程序开始直播推送前需要初始化相关音频编码设置,如采样率、声道等。

1
builder.getNativePush().setNativeAudioOptions(builder.getSampleRateInHz(), builder.getChannelConfig());

在Native中初始化FAAC,这里不对其作解释,后续文章将会单独讲解AAC编码。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
/**
* Faac初始化
* Call faacEncOpen() for every encoder instance you need.
*To set encoder options, call faacEncGetCurrentConfiguration(), change the parameters in the structure accessible by the returned pointer and then call faacEncSetConfiguration().
*As long as there are still samples left to encode, call faacEncEncode() to encode the data. The encoder returns the bitstream data in a client-supplied buffer.
*Once you call faacEncEncode() with zero samples of input the flushing process is initiated; afterwards you may call faacEncEncode() with zero samples input only.
*faacEncEncode() will continue to write out data until all audio samples have been encoded.
*Once faacEncEncode() has returned with zero bytes written, call faacEncClose() to destroy this encoder instance.
* @param env
* @param instance
* @param sampleRateInHz
* @param channel
*/
JNIEXPORT void JNICALL
Java_com_ben_android_live_NativePush_setNativeAudioOptions(JNIEnv *env, jobject instance,
jint sampleRateInHz, jint channel) {
faacEncodeHandle = faacEncOpen(sampleRateInHz, channel, &inputSamples, &maxOutputBytes);
if (!faacEncodeHandle) {
LOGE("%s", "FAAC encode open failed!");
return;
}
faacEncConfigurationPtr faacEncodeConfigurationPtr = faacEncGetCurrentConfiguration(
faacEncodeHandle);
//指定MPEG版本
faacEncodeConfigurationPtr->mpegVersion = MPEG4;
faacEncodeConfigurationPtr->allowMidside = 1;
faacEncodeConfigurationPtr->aacObjectType = LOW;
faacEncodeConfigurationPtr->outputFormat = 0; //输出是否包含ADTS头
faacEncodeConfigurationPtr->useTns = 1; //时域噪音控制,大概就是消爆音
faacEncodeConfigurationPtr->useLfe = 0;
faacEncodeConfigurationPtr->quantqual = 100;
faacEncodeConfigurationPtr->bandWidth = 0; //频宽
faacEncodeConfigurationPtr->shortctl = SHORTCTL_NORMAL;

//call faacEncSetConfiguration
if (!faacEncSetConfiguration(faacEncodeHandle, faacEncodeConfigurationPtr)) {
LOGE("%s", "faacEncSetConfiguration failed!");
return;
}

LOGI("%s", "faac initialization successful");
}

总结

本文对Android音视频以及Android媒体架构做了简单的描述,RTMP直播项目结构大致分为应用层以及Native层。应用层主要负责与用户交互相关的逻辑以及音视频数据采集。使用AudioRecord进行音频数据采集并发送至Native层,由Native层进行AAC编码。视频采集使用Camera采集数据并将预览数据并发送Native层,使用x264进行编码。后续文章将进入Native层讲解如何使用x264以及faac进行音视频编码。ok,本次文章到此就要结束了,另附上直播项目开源地址AndroidRTMPLive,喜欢本文的同学动动金手指点个star吧(* ̄︶ ̄)

随意分享,您的支持将鼓励我继续创作!