目录

夜雨飘零

记录精彩的程序人生

X

在Android手机上使用腾讯的ncnn实现图像分类

   
    0 0 yeyupiaoling

前言

在之前笔者有介绍过《在Android设备上使用PaddleMobile实现图像分类》，使用的框架是百度开源的PaddleMobile。在本章中，笔者将会介绍使用腾讯的开源手机深度学习框架ncnn来实现在Android手机实现图像分类，这个框架开源时间比较长，相对稳定很多。

ncnn的GitHub地址：https://github.com/Tencent/ncnn

使用Ubuntu编译ncnn库

1、首先要下载和解压NDK。

wget https://dl.google.com/android/repository/android-ndk-r17b-linux-x86_64.zip
unzip android-ndk-r17b-linux-x86_64.zip

2、设置NDK环境变量，目录是NDK的解压目录。

export NDK_ROOT="/home/test/paddlepaddle/android-ndk-r17b"

设置好之后，可以使用以下的命令查看配置情况。

root@test:/home/test/paddlepaddle# echo $NDK_ROOT
/home/test/paddlepaddle/android-ndk-r17b

3、安装cmake，需要安装较高版本的，笔者的cmake版本是3.11.2。

下载cmake源码

wget https://cmake.org/files/v3.11/cmake-3.11.2.tar.gz

解压cmake源码

tar -zxvf cmake-3.11.2.tar.gz

进入到cmake源码根目录，并执行bootstrap。

cd cmake-3.11.2
./bootstrap

最后执行以下两条命令开始安装cmake。

make
make install

安装完成之后，可以使用cmake --version是否安装成功。

root@test:/home/test/paddlepaddle# cmake --version
cmake version 3.11.2

CMake suite maintained and supported by Kitware (kitware.com/cmake).

4、克隆ncnn源码。

git clone https://github.com/Tencent/ncnn.git

5、编译源码。

# 进入到ncnn源码根目录下
cd ncnn
# 创建一个新的文件夹
mkdir -p build-android-armv7
# 进入到该文件夹中
cd build-android-armv7
# 执行编译命令
cmake -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \
    -DANDROID_ABI="armeabi-v7a" -DANDROID_ARM_NEON=ON \
    -DANDROID_PLATFORM=android-14 ..
# 这里笔者使用4个行程并行编译
make -j4
make install

6、编译完成，会在build-android-armv7目录下生成一个install目录，我们编译得到的文件都在该文件夹下：

  • include 调用ncnn所需的头文件，该文件夹会存放在Android项目的src/main/cpp目录下；
  • lib 编译得到的ncnn库libncnn.a，之后会存放在Android项目的src/main/jniLibs/armeabi-v7a/libncnn.a

转换预测模型

1、克隆Caffe源码。

git clone https://github.com/BVLC/caffe.git

2、编译Caffe源码。

# 切换到Caffe目录
cd caffe
# 在当前目录执行cmake
cmake .
# 使用4个线程编译
make -j4
make install

3、升级Caffe模型。

# 把需要转换的模型复制到caffe/tools，并切入到该目录
cd tools
# 升级Caffe模型
./upgrade_net_proto_text mobilenet_v2_deploy.prototxt mobilenet_v2_deploy_new.prototxt
./upgrade_net_proto_binary mobilenet_v2.caffemodel mobilenet_v2_new.caffemodel

4、检查模型配置文件，因为只能一张一张图片预测，所以输入要设置为dim: 1。

name: "MOBILENET_V2"
layer {
  name: "input"
  type: "Input"
  top: "data"
  input_param {
    shape {
      dim: 1
      dim: 3
      dim: 224
      dim: 224
    }
  }
}

5、切换到ncnn的根目录，就是我们上一部分克隆的ncnn源码。

cd ncnn/

6、在根目录下编译ncnn源码。

mkdir -p build
cd build
cmake ..
make -j4
make install

7、把新的Caffe模型转换成NCNN模型。

# 经过上一步，会生产一个tools，我们进入到以下目录
cd tools/caffe/
# 把已经升级的网络定义文件和权重文件复制到当目录，并执行以下命令
./caffe2ncnn mobilenet_v2_deploy_new.prototxt mobilenet_v2_new.caffemodel mobilenet_v2.param mobilenet_v2.bin

8、对象模型参数进行加密，这样就算别人反编译我们的apk也用不了我们的模型文件。把上一步获得的mobilenet_v2.param、mobilenet_v2.bin复制到该目录的上一个目录，也就是tools目录。

# 切换到上一个目录
cd ../
# 执行命令之后会生成mobilenet_v2.param、mobilenet_v2.id.h、mobilenet_v2.mem.h
./ncnn2mem mobilenet_v2.param mobilenet_v2.bin mobilenet_v2.id.h mobilenet_v2.mem.h

经过上面的步骤，得到的文件中，以下文件时需要的：

  • mobilenet_v2.param.bin 网络的模型参数；
  • mobilenet_v2.bin 网络的权重；
  • mobilenet_v2.id.h 在预测图片的时候使用到。

开发Android项目

  • 我们在Android Studio上创建一个NCNN1的项目，别忘了选择C++支持。
    这里写图片描述

其他的可以直接默认就可以了，在这里要注意选择C++11支持。
这里写图片描述

  • 在main目录下创建assets目录，并复制以下目录到该目录：
  • mobilenet_v2.param.bin 上一步获取网络的模型参数；
  • mobilenet_v2.bin 上一步获取网络的权重；
  • synset.txt label对应的名称，下载地址：https://github.com/shicai/MobileNet-Caffe/blob/master/synset.txt。
  • 在cpp目录下复制在使用Ubuntu编译NCNN库部分编译得到的include文件夹，包括里面的C++头文件。
  • 把mobilenet_v2.id.h复制到cpp目录下。
  • 在main目录下创建jniLibs/armeabi-v7a/目录，并把使用Ubuntu编译NCNN库部分编译得到的libncnn.a复制到该目录。
  • 在cpp目录下创建一个C++文件，并编写以下代码，这段代码是用于加载模型和预测图片的：
#include <android/bitmap.h>
#include <android/log.h>
#include <jni.h>
#include <string>
#include <vector>
// ncnn
#include "include/net.h"
#include "mobilenet_v2.id.h"
#include <sys/time.h>
#include <unistd.h>

static ncnn::UnlockedPoolAllocator g_blob_pool_allocator;
static ncnn::PoolAllocator g_workspace_pool_allocator;

static ncnn::Mat ncnn_param;
static ncnn::Mat ncnn_bin;
static ncnn::Net ncnn_net;


extern "C" {

// public native boolean Init(byte[] param, byte[] bin, byte[] words);
JNIEXPORT jboolean JNICALL
Java_com_example_ncnn1_NcnnJni_Init(JNIEnv *env, jobject thiz, jbyteArray param, jbyteArray bin) {
    // init param
    {
        int len = env->GetArrayLength(param);
        ncnn_param.create(len, (size_t) 1u);
        env->GetByteArrayRegion(param, 0, len, (jbyte *) ncnn_param);
        int ret = ncnn_net.load_param((const unsigned char *) ncnn_param);
        __android_log_print(ANDROID_LOG_DEBUG, "NcnnJni", "load_param %d %d", ret, len);
    }

    // init bin
    {
        int len = env->GetArrayLength(bin);
        ncnn_bin.create(len, (size_t) 1u);
        env->GetByteArrayRegion(bin, 0, len, (jbyte *) ncnn_bin);
        int ret = ncnn_net.load_model((const unsigned char *) ncnn_bin);
        __android_log_print(ANDROID_LOG_DEBUG, "NcnnJni", "load_model %d %d", ret, len);
    }

    ncnn::Option opt;
    opt.lightmode = true;
    opt.num_threads = 4;
    opt.blob_allocator = &g_blob_pool_allocator;
    opt.workspace_allocator = &g_workspace_pool_allocator;

    ncnn::set_default_option(opt);

    return JNI_TRUE;
}

// public native String Detect(Bitmap bitmap);
JNIEXPORT jfloatArray JNICALL Java_com_example_ncnn1_NcnnJni_Detect(JNIEnv* env, jobject thiz, jobject bitmap)
{
    // ncnn from bitmap
    ncnn::Mat in;
    {
        AndroidBitmapInfo info;
        AndroidBitmap_getInfo(env, bitmap, &info);
        int width = info.width;
        int height = info.height;
        if (info.format != ANDROID_BITMAP_FORMAT_RGBA_8888)
            return NULL;

        void* indata;
        AndroidBitmap_lockPixels(env, bitmap, &indata);
        // 把像素转换成data，并指定通道顺序
        in = ncnn::Mat::from_pixels((const unsigned char*)indata, ncnn::Mat::PIXEL_RGBA2BGR, width, height);

        AndroidBitmap_unlockPixels(env, bitmap);
    }

    // ncnn_net
    std::vector<float> cls_scores;
    {
        // 减去均值和乘上比例
        const float mean_vals[3] = {103.94f, 116.78f, 123.68f};
        const float scale[3] = {0.017f, 0.017f, 0.017f};

        in.substract_mean_normalize(mean_vals, scale);

        ncnn::Extractor ex = ncnn_net.create_extractor();
        // 如果时不加密是使用ex.input("data", in);
        ex.input(mobilenet_v2_param_id::BLOB_data, in);

        ncnn::Mat out;
        // 如果时不加密是使用ex.extract("prob", out);
        ex.extract(mobilenet_v2_param_id::BLOB_prob, out);

        int output_size = out.w;
        jfloat *output[output_size];
        for (int j = 0; j < out.w; j++) {
            output[j] = &out[j];
        }

        jfloatArray jOutputData = env->NewFloatArray(output_size);
        if (jOutputData == nullptr) return nullptr;
        env->SetFloatArrayRegion(jOutputData, 0, output_size,
                                 reinterpret_cast<const jfloat *>(*output));  // copy

        return jOutputData;
    }
}
}
  • 在项目包com.example.ncnn1下，修改MainActivity.java中的代码，修改如下：
package com.example.ncnn1;

import android.Manifest;
import android.app.Activity;
import android.content.Intent;
import android.content.pm.PackageManager;
import android.content.res.AssetManager;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.net.Uri;
import android.os.Bundle;
import android.support.annotation.NonNull;
import android.support.annotation.Nullable;
import android.support.v4.app.ActivityCompat;
import android.support.v4.content.ContextCompat;
import android.text.method.ScrollingMovementMethod;
import android.util.Log;
import android.view.View;
import android.widget.Button;
import android.widget.ImageView;
import android.widget.TextView;
import android.widget.Toast;

import com.bumptech.glide.Glide;
import com.bumptech.glide.load.engine.DiskCacheStrategy;
import com.bumptech.glide.request.RequestOptions;

import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;


public class MainActivity extends Activity {
    private static final String TAG = MainActivity.class.getName();
    private static final int USE_PHOTO = 1001;
    private String camera_image_path;
    private ImageView show_image;
    private TextView result_text;
    private boolean load_result = false;
    private int[] ddims = {1, 3, 224, 224};
    private int model_index = 1;
    private List<String> resultLabel = new ArrayList<>();
    private NcnnJni squeezencnn = new NcnnJni();

    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        try {
            initSqueezeNcnn();
        } catch (IOException e) {
            Log.e("MainActivity", "initSqueezeNcnn error");
        }

        init_view();
        readCacheLabelFromLocalFile();
    }

    private void initSqueezeNcnn() throws IOException {
        byte[] param = null;
        byte[] bin = null;

        {
            InputStream assetsInputStream = getAssets().open("mobilenet_v2.param.bin");
            int available = assetsInputStream.available();
            param = new byte[available];
            int byteCode = assetsInputStream.read(param);
            assetsInputStream.close();
        }
        {
            InputStream assetsInputStream = getAssets().open("mobilenet_v2.bin");
            int available = assetsInputStream.available();
            bin = new byte[available];
            int byteCode = assetsInputStream.read(bin);
            assetsInputStream.close();
        }

        load_result = squeezencnn.Init(param, bin);
        Log.d("load model", "result:" + load_result);
    }

    // initialize view
    private void init_view() {
        request_permissions();
        show_image = (ImageView) findViewById(R.id.show_image);
        result_text = (TextView) findViewById(R.id.result_text);
        result_text.setMovementMethod(ScrollingMovementMethod.getInstance());
        Button use_photo = (Button) findViewById(R.id.use_photo);

        // use photo click
        use_photo.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View view) {
                if (!load_result) {
                    Toast.makeText(MainActivity.this, "never load model", Toast.LENGTH_SHORT).show();
                    return;
                }
                PhotoUtil.use_photo(MainActivity.this, USE_PHOTO);
            }
        });
    }

	// load label's name
    private void readCacheLabelFromLocalFile() {
        try {
            AssetManager assetManager = getApplicationContext().getAssets();
            BufferedReader reader = new BufferedReader(new InputStreamReader(assetManager.open("synset.txt")));
            String readLine = null;
            while ((readLine = reader.readLine()) != null) {
                resultLabel.add(readLine);
            }
            reader.close();
        } catch (Exception e) {
            Log.e("labelCache", "error " + e);
        }
    }

    @Override
    protected void onActivityResult(int requestCode, int resultCode, @Nullable Intent data) {
        String image_path;
        RequestOptions options = new RequestOptions().skipMemoryCache(true).diskCacheStrategy(DiskCacheStrategy.NONE);
        if (resultCode == Activity.RESULT_OK) {
            switch (requestCode) {
                case USE_PHOTO:
                    if (data == null) {
                        Log.w(TAG, "user photo data is null");
                        return;
                    }
                    Uri image_uri = data.getData();
                    Glide.with(MainActivity.this).load(image_uri).apply(options).into(show_image);
                    // get image path from uri
                    image_path = PhotoUtil.get_path_from_URI(MainActivity.this, image_uri);
                    // predict image
                    predict_image(image_path);
                    break;
            }
        }
    }

    //  predict image
    private void predict_image(String image_path) {
        // picture to float array
        Bitmap bmp = PhotoUtil.getScaleBitmap(image_path);
        Bitmap rgba = bmp.copy(Bitmap.Config.ARGB_8888, true);

        // resize to 227x227
        Bitmap input_bmp = Bitmap.createScaledBitmap(rgba, ddims[2], ddims[3], false);
        try {
            // Data format conversion takes too long
            // Log.d("inputData", Arrays.toString(inputData));
            long start = System.currentTimeMillis();
            // get predict result
            float[] result = squeezencnn.Detect(input_bmp);
            long end = System.currentTimeMillis();
            Log.d(TAG, "origin predict result:" + Arrays.toString(result));
            long time = end - start;
            Log.d("result length", String.valueOf(result.length));
            // show predict result and time
            int r = get_max_result(result);
            String show_text = "result：" + r + "\nname：" + resultLabel.get(r) + "\nprobability：" + result[r] + "\ntime：" + time + "ms";
            result_text.setText(show_text);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    // get max probability label
    private int get_max_result(float[] result) {
        float probability = result[0];
        int r = 0;
        for (int i = 0; i < result.length; i++) {
            if (probability < result[i]) {
                probability = result[i];
                r = i;
            }
        }
        return r;
    }

    // request permissions
    private void request_permissions() {

        List<String> permissionList = new ArrayList<>();
        if (ContextCompat.checkSelfPermission(this, Manifest.permission.CAMERA) != PackageManager.PERMISSION_GRANTED) {
            permissionList.add(Manifest.permission.CAMERA);
        }

        if (ContextCompat.checkSelfPermission(this, Manifest.permission.WRITE_EXTERNAL_STORAGE) != PackageManager.PERMISSION_GRANTED) {
            permissionList.add(Manifest.permission.WRITE_EXTERNAL_STORAGE);
        }

        if (ContextCompat.checkSelfPermission(this, Manifest.permission.READ_EXTERNAL_STORAGE) != PackageManager.PERMISSION_GRANTED) {
            permissionList.add(Manifest.permission.READ_EXTERNAL_STORAGE);
        }

        // if list is not empty will request permissions
        if (!permissionList.isEmpty()) {
            ActivityCompat.requestPermissions(this, permissionList.toArray(new String[permissionList.size()]), 1);
        }
    }

    @Override
    public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions, @NonNull int[] grantResults) {
        super.onRequestPermissionsResult(requestCode, permissions, grantResults);
        switch (requestCode) {
            case 1:
                if (grantResults.length > 0) {
                    for (int i = 0; i < grantResults.length; i++) {

                        int grantResult = grantResults[i];
                        if (grantResult == PackageManager.PERMISSION_DENIED) {
                            String s = permissions[i];
                            Toast.makeText(this, s + " permission was denied", Toast.LENGTH_SHORT).show();
                        }
                    }
                }
                break;
        }
    }
}
  • 同样在项目的包com.example.ncnn1下，创建一个NcnnJni.java类，用于提供JNI接口，代码如下：
package com.example.ncnn1;

import android.graphics.Bitmap;

public class NcnnJni
{
    public native boolean Init(byte[] param, byte[] bin);

    public native float[] Detect(Bitmap bitmap);

    static {
        System.loadLibrary("ncnn_jni");
    }
}
  • 还是在项目的包com.example.ncnn1下，创建一个PhotoUtil.java类，这个是图片的工具类，代码如下：
package com.example.ncnn1;

import android.app.Activity;
import android.content.Context;
import android.content.Intent;
import android.database.Cursor;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.net.Uri;
import android.provider.MediaStore;

import java.nio.FloatBuffer;

public class PhotoUtil {
    // get picture in photo
    public static void use_photo(Activity activity, int requestCode) {
        Intent intent = new Intent(Intent.ACTION_PICK);
        intent.setType("image/*");
        activity.startActivityForResult(intent, requestCode);
    }

    // get photo from Uri
    public static String get_path_from_URI(Context context, Uri uri) {
        String result;
        Cursor cursor = context.getContentResolver().query(uri, null, null, null, null);
        if (cursor == null) {
            result = uri.getPath();
        } else {
            cursor.moveToFirst();
            int idx = cursor.getColumnIndex(MediaStore.Images.ImageColumns.DATA);
            result = cursor.getString(idx);
            cursor.close();
        }
        return result;
    }

    // compress picture
    public static Bitmap getScaleBitmap(String filePath) {
        BitmapFactory.Options opt = new BitmapFactory.Options();
        opt.inJustDecodeBounds = true;
        BitmapFactory.decodeFile(filePath, opt);

        int bmpWidth = opt.outWidth;
        int bmpHeight = opt.outHeight;

        int maxSize = 500;

        // compress picture with inSampleSize
        opt.inSampleSize = 1;
        while (true) {
            if (bmpWidth / opt.inSampleSize < maxSize || bmpHeight / opt.inSampleSize < maxSize) {
                break;
            }
            opt.inSampleSize *= 2;
        }
        opt.inJustDecodeBounds = false;
        return BitmapFactory.decodeFile(filePath, opt);
    }
}
  • 修改启动页面的布局，修改如下：
<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">

    <LinearLayout
        android:id="@+id/btn_ll"
        android:layout_alignParentBottom="true"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:orientation="horizontal">

        <Button
            android:id="@+id/use_photo"
            android:layout_weight="1"
            android:layout_width="0dp"
            android:layout_height="wrap_content"
            android:text="相册" />

    </LinearLayout>

    <TextView
        android:layout_above="@id/btn_ll"
        android:id="@+id/result_text"
        android:textSize="16sp"
        android:layout_width="match_parent"
        android:hint="预测结果会在这里显示"
        android:layout_height="100dp" />

    <ImageView
        android:layout_alignParentTop="true"
        android:layout_above="@id/result_text"
        android:id="@+id/show_image"
        android:layout_width="match_parent"
        android:layout_height="match_parent" />

</RelativeLayout>
  • 修改APP目录下的CMakeLists.txt文件，修改如下：
# For more information about using CMake with Android Studio, read the
# documentation: https://d.android.com/studio/projects/add-native-code.html

# Sets the minimum version of CMake required to build the native library.

cmake_minimum_required(VERSION 3.4.1)

# Creates and names a library, sets it as either STATIC
# or SHARED, and provides the relative paths to its source code.
# You can define multiple libraries, and CMake builds them for you.
# Gradle automatically packages shared libraries with your APK.

set(ncnn_lib ${CMAKE_SOURCE_DIR}/src/main/jniLibs/armeabi-v7a/libncnn.a)
add_library (ncnn_lib STATIC IMPORTED)
set_target_properties(ncnn_lib PROPERTIES IMPORTED_LOCATION ${ncnn_lib})


add_library( # Sets the name of the library.
             ncnn_jni

             # Sets the library as a shared library.
             SHARED

             # Provides a relative path to your source file(s).
             src/main/cpp/ncnn_jni.cpp )

# Searches for a specified prebuilt library and stores the path as a
# variable. Because CMake includes system libraries in the search path by
# default, you only need to specify the name of the public NDK library
# you want to add. CMake verifies that the library exists before
# completing its build.

find_library( # Sets the name of the path variable.
              log-lib

              # Specifies the name of the NDK library that
              # you want CMake to locate.
              log )

# Specifies libraries CMake should link to your target library. You
# can link multiple libraries, such as libraries you define in this
# build script, prebuilt third-party libraries, or system libraries.

target_link_libraries( # Specifies the target library.
                       ncnn_jni
                       ncnn_lib
                       jnigraphics

                       # Links the target library to the log library
                       # included in the NDK.
                       ${log-lib} )
  • 修改APP目录下的build.gradle文件，修改如下：
apply plugin: 'com.android.application'

android {
    compileSdkVersion 28
    defaultConfig {
        applicationId "com.example.ncnn1"
        minSdkVersion 21
        targetSdkVersion 28
        versionCode 1
        versionName "1.0"
        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
        externalNativeBuild {
            cmake {
                cppFlags "-std=c++11 -fopenmp"
                abiFilters "armeabi-v7a"
            }
        }
    }
    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
        }
    }
    externalNativeBuild {
        cmake {
            path "CMakeLists.txt"
        }
    }

    sourceSets {
        main {
            jniLibs.srcDirs = ["src/main/jniLibs"]
            jni.srcDirs = ['src/cpp']
        }
    }
}

dependencies {
    implementation fileTree(dir: 'libs', include: ['*.jar'])
    implementation 'com.android.support:appcompat-v7:28.0.0-rc02'
    implementation 'com.android.support.constraint:constraint-layout:1.1.3'
    testImplementation 'junit:junit:4.12'
    implementation 'com.github.bumptech.glide:glide:4.3.1'
    androidTestImplementation 'com.android.support.test:runner:1.0.2'
    androidTestImplementation 'com.android.support.test.espresso:espresso-core:3.0.2'
}
  • 最后别忘了在配置文件中添加权限。
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>

最后的效果图如下：
在这里插入图片描述

代码传送门： 上面已经几乎包括所有的代码了，为了读者方便直接使用，可以在这里下载项目源代码。

参考资料

  1. https://github.com/BVLC/caffe
  2. https://github.com/Tencent/ncnn/wiki/how-to-use-ncnn-with-alexnet
  3. https://github.com/Tencent/ncnn/wiki/how-to-build
  4. https://github.com/Tencent/ncnn/tree/master/examples/squeezencnn
标题：在Android手机上使用腾讯的ncnn实现图像分类
作者：yeyupiaoling
地址：https://yeyupiaoling.cn/articles/1584973588191.html