我用google的MLKit检测人脸,获取到人脸的信息,包含人脸的133关键点(google mlkit 文档)。获取到坐标后,进行了归一化处理,把坐标范围处理成[-1, 1]。
渲染顺序:检测图和贴纸先后渲染到fbo,再显示到屏幕。
添加到视椎体如下:
public void onInputSizeChanged(int width, int height) {
super.onInputSizeChanged(width, height);
mRatio = (float) width / height;
// 设置透视投影
Matrix.frustumM(mProjectionMatrix, 0, -mRatio, mRatio, -1.0f, 1.0f, 3.0f, 9.0f);
Matrix.setLookAtM(mViewMatrix, 0, 0, 0, 6.0f, 0.0f, 0.0f, 0.0f, 0.0f, 1.0f, 0.0f);
}
计算贴纸中心点坐标相关代码如下:
oneFace.vertexPoints是float数组,133个点坐标,连续两个是一个点的xy;
stickerData.centerIndexList是中心坐标索引列表,有可能是多个关键点计算中心点;
mImageHeight和mImageWidth是检测图的宽高
private void calculateStickerVertices(DynamicStickerNormalData stickerData, OneFace oneFace) {
if (oneFace == null || oneFace.vertexPoints == null) {
return;
}
System.out.println("calculateStickerVertices");
// 步骤一、计算贴纸的中心点和顶点坐标
// 备注:由于frustumM设置的bottom 和top 为 -1.0 和 1.0,这里为了方便计算,直接用高度作为基准值来计算
// 1.1、计算贴纸相对于人脸的宽高
System.out.println("mImageWidth, mImageHeight: " + mImageWidth + ", " + mImageHeight);
float stickerWidth = (float) FacePointsUtils.getDistance(
(oneFace.vertexPoints[stickerData.startIndex * 2] * 0.5f + 0.5f) * mImageWidth,
(oneFace.vertexPoints[stickerData.startIndex * 2 + 1] * 0.5f + 0.5f) * mImageHeight,
(oneFace.vertexPoints[stickerData.endIndex * 2] * 0.5f + 0.5f) * mImageWidth,
(oneFace.vertexPoints[stickerData.endIndex * 2 + 1] * 0.5f + 0.5f) * mImageHeight) * stickerData.baseScale;
float stickerHeight = stickerWidth * (float) stickerData.height / (float) stickerData.width;
System.out.println("stickerWidth, stickerHeight: " + stickerWidth + ", " + stickerHeight);
// 1.2、根据贴纸的参数计算出中心点的坐标
float centerX = 0.0f;
float centerY = 0.0f;
for (int i = 0; i < stickerData.centerIndexList.length; i++) {
centerX += (oneFace.vertexPoints[stickerData.centerIndexList[i] * 2] * 0.5f + 0.5f) * mImageWidth;
centerY += (oneFace.vertexPoints[stickerData.centerIndexList[i] * 2 + 1] * 0.5f + 0.5f) * mImageHeight;
}
System.out.println("centerX1, centerY1: " + centerX + ", " + centerY);
centerX /= (float) stickerData.centerIndexList.length;
centerY /= (float) stickerData.centerIndexList.length;
centerX = centerX / mImageHeight * ProjectionScale;
centerY = centerY / mImageHeight * ProjectionScale;
System.out.println("centerX2, centerY2: " + centerX + ", " + centerY);
// 1.3、求出真正的中心点顶点坐标,这里由于frustumM设置了长宽比,因此ndc坐标计算时需要变成mRatio:1,这里需要转换一下
float ndcCenterX = (centerX - mRatio) * ProjectionScale;
float ndcCenterY = (centerY - 1.0f) * ProjectionScale;
System.out.println("ndcCenterX, ndcCenterY" + ndcCenterX + ", " + ndcCenterY);
//
// // 1.4、贴纸的宽高在ndc坐标系中的长度
float ndcStickerWidth = stickerWidth / mImageHeight * ProjectionScale;
float ndcStickerHeight = ndcStickerWidth * (float) stickerData.height / (float) stickerData.width;
System.out.println("ndcStickerWidth, ndcStickerHeight" + ndcStickerWidth + ", " + ndcStickerHeight);
//
// // 1.5、根据贴纸参数求偏移的ndc坐标
float offsetX = (stickerWidth * stickerData.offsetX) / mImageHeight * ProjectionScale;
float offsetY = (stickerHeight * stickerData.offsetY) / mImageHeight * ProjectionScale;
System.out.println("offsetX, offsetY: " + offsetX + ", " + offsetY);
//
// // 1.6、贴纸带偏移量的锚点的ndc坐标,即实际贴纸的中心点在OpenGL的顶点坐标系中的位置
float anchorX = ndcCenterX + offsetX * ProjectionScale;
float anchorY = ndcCenterY + offsetY * ProjectionScale;
System.out.println("anchorX, anchorY" + anchorX + ", " + anchorY);
//
// // 1.7、根据前面的锚点,计算出贴纸实际的顶点坐标
mStickerVertices[0] = anchorX - ndcStickerWidth; mStickerVertices[1] = anchorY - ndcStickerHeight;
mStickerVertices[2] = anchorX + ndcStickerWidth; mStickerVertices[3] = anchorY - ndcStickerHeight;
mStickerVertices[4] = anchorX - ndcStickerWidth; mStickerVertices[5] = anchorY + ndcStickerHeight;
mStickerVertices[6] = anchorX + ndcStickerWidth; mStickerVertices[7] = anchorY + ndcStickerHeight;
mVertexBuffer.clear();
mVertexBuffer.position(0);
mVertexBuffer.put(mStickerVertices);
System.out.println("顶点坐标: " + mStickerVertices[0] + ", " + mStickerVertices[1]);
System.out.println("顶点坐标: " + mStickerVertices[2] + ", " + mStickerVertices[3]);
System.out.println("顶点坐标: " + mStickerVertices[4] + ", " + mStickerVertices[5]);
System.out.println("顶点坐标: " + mStickerVertices[6] + ", " + mStickerVertices[7]);
// 步骤二、根据人脸姿态角计算透视变换的总变换矩阵
// 2.1、将Z轴平移到贴纸中心点,因为贴纸模型矩阵需要做姿态角变换
// 平移主要是防止贴纸变形
Matrix.setIdentityM(mModelMatrix, 0);
Matrix.translateM(mModelMatrix, 0, ndcCenterX, ndcCenterY, 0);
// 2.2、贴纸姿态角旋转
// TODO 人脸关键点给回来的pitch角度似乎不太对??SDK给过来的pitch角度值太小了,比如抬头低头pitch的实际角度30度了,SDK返回的结果才十几度,后续再看看如何优化
System.out.println("pitch, yaw, roll: " + oneFace.pitch + ", " + oneFace.yaw + ", " + oneFace.roll);
float pitchAngle = -(float) (oneFace.pitch * 180f / Math.PI);
float yawAngle = (float) (oneFace.yaw * 180f / Math.PI);
float rollAngle = (float) (oneFace.roll * 180f / Math.PI);
System.out.println("pitchAngle, yawAngle, rollAngle: " + pitchAngle + ", " + yawAngle + ", " + rollAngle);
// 限定左右扭头幅度不超过50°,销毁人脸关键点SDK带来的偏差
if (Math.abs(yawAngle) > 50) {
yawAngle = (yawAngle / Math.abs(yawAngle)) * 50;
}
// 限定抬头低头最大角度,消除人脸关键点SDK带来的偏差
if (Math.abs(pitchAngle) > 30) {
pitchAngle = (pitchAngle / Math.abs(pitchAngle)) * 30;
}
// 贴纸姿态角变换,优先z轴变换,消除手机旋转的角度影响,否则会导致扭头、抬头、低头时贴纸变形的情况
// Matrix.rotateM(mModelMatrix, 0, rollAngle, 0, 0, 1);
// Matrix.rotateM(mModelMatrix, 0, yawAngle, 0, 1, 0);
// Matrix.rotateM(mModelMatrix, 0, pitchAngle, 1, 0, 0);
Matrix.rotateM(mModelMatrix, 0, -oneFace.roll, 0, 0, 1);
Matrix.rotateM(mModelMatrix, 0, oneFace.yaw, 0, 1, 0);
Matrix.rotateM(mModelMatrix, 0, oneFace.pitch, 1, 0, 0);
// 2.4、将Z轴平移回到原来构建的视椎体的位置,即需要将坐标z轴平移回到屏幕中心,此时才是贴纸的实际模型矩阵
Matrix.translateM(mModelMatrix, 0, -ndcCenterX, -ndcCenterY, 0);
// 2.5、计算总变换矩阵。MVPMatrix 的矩阵计算是 MVPMatrix = ProjectionMatrix * ViewMatrix * ModelMatrix
// 备注:矩阵相乘的顺序不同得到的结果是不一样的,不同的顺序会导致前面计算过程不一致,这点希望大家要注意
Matrix.setIdentityM(mMVPMatrix, 0);
Matrix.multiplyMM(mMVPMatrix, 0, mProjectionMatrix, 0, mViewMatrix, 0);
Matrix.multiplyMM(mMVPMatrix, 0, mMVPMatrix, 0, mModelMatrix, 0);
}
最终,实际显示在屏幕上的位置是,x坐标对,但是y坐标的位置是原图上下垂直翻转的位置
然后我把带检测的图片先上下翻转之后再获取到相关的人脸关键点坐标,再进行贴图,y的坐标是对了,x反了
Bitmap bitmap = BitmapFactory.decodeResource(getResources(), R.drawable.grapefruit);
Matrix matrix = new Matrix();
matrix.postScale(1, -1);
Bitmap bitmap180 = Bitmap.createBitmap(bitmap, 0, 0, bitmap.getWidth(), bitmap.getHeight(), matrix, false);
有大佬能帮我看看这个问题吗?
感谢您的帮助。