[原创] 一个简单的TensorFlow-Serving例子

learnhard — Fri, 09 Aug 2019 03:22:14 +0000

本文展示了如何用Python训练一个简单的神经网络模型，保存为模型文件，并且用TensorFlow-Serving的Docker镜像把它加载起来，提供在线服务的过程。
环境：Ubuntu 16.04 LTS，TensorFlow 1.14.0，Python 3.6.8

[×] 训练一个简单的神经网络模型
直接看代码：

import os
import shutil
 
import numpy as np
import tensorflow as tf
 
 
def add_layer(inputs, input_size, output_size, activation_function=None):
    weights = tf.Variable(tf.random_normal([input_size, output_size]))
    biases = tf.Variable(tf.zeros([1, output_size]) + 0.1)
    wx_plus_b = tf.matmul(inputs, weights) + biases  # WX + b
    if activation_function is None:
        outputs = wx_plus_b
    else:
        outputs = activation_function(wx_plus_b)
    return outputs
 
 
# 造一些随机输入数据
num_points = 30000  # 总数据条数
feature_number = 100  # 每条输入数据有100个feature
# num_points个输入数据,每个有feature_number个feature,即输入数据的维度是(num_points,feature_number)
x_data = np.random.rand(num_points, feature_number)
y_data = np.random.randint(0, 2, (num_points, 1))  # nx1的数组, 每一行为1个数(0或1)
 
# 用于接收输入的Tensor
x_actual = tf.placeholder(tf.float32, [None, feature_number], name="myInput")
y_actual = tf.placeholder(tf.float32, [None, 1], name="myOutput")
 
# 隐层1
l1 = add_layer(x_actual, feature_number, 32, activation_function=tf.nn.relu)
# 隐层2
l2 = add_layer(l1, 32, 64, activation_function=tf.nn.tanh)
# 隐层3
l3 = add_layer(l2, 64, 32, activation_function=tf.nn.relu)
 
# 输出层
y_predict = add_layer(l3, 32, 1, activation_function=tf.nn.sigmoid)
# 损失函数
loss = -tf.reduce_mean(y_actual * tf.log(tf.clip_by_value(y_predict, 1e-10, 1.0)))
# 优化器
train_step = tf.train.AdamOptimizer(0.001).minimize(loss)
 
init = tf.global_variables_initializer()
# 迭代次数
num_iterations = 10000
with tf.Session() as sess:
    sess.run(init)
    for i in range(num_iterations):
        # 训练模型
        sess.run(train_step, feed_dict={x_actual: x_data, y_actual: y_data})
        if i % 500 == 0:
            prediction_value = sess.run(y_predict, feed_dict={x_actual: x_data})
            print(sess.run(loss, feed_dict={x_actual: x_data, y_actual: y_data}))
    # 训练完成后,以SavedModel格式保存模型文件
    model_output_dir = "./model/201908070001"
    if os.path.exists(model_output_dir):  # 目录存在
        shutil.rmtree(model_output_dir)  # 删除原目录
    tf.saved_model.simple_save(
        sess, model_output_dir, inputs={"myInput": x_actual}, outputs={"myOutput": y_predict})
 
    # 做5次预测(测试一下)
    for i in range(5):
        x_input = np.random.rand(1, feature_number)  # 1表示输入一条数据
        feed_dict = {x_actual: x_input}
        result = sess.run(y_predict, feed_dict)
        print('prediction result: %f' % result)

(据反映，有些人的浏览器看到上面的代码是没有正确缩进的，可以自己复制到 IDE 里处理一下)
数据是随机生成的，模型是随意定义的，所以不要在意其合理性。
模型要以SavedModel格式保存，否则不能用TensorFlow-Serving来serve，在这里我输出的模型目录为./model/201908070001，里面的文件大概有这些：

├── saved_model.pb

└── variables

├── variables.data-00000-of-00001

└── variables.index

文章来源：https://www.codelast.com/
[×] 准备TensorFlow-Serving的Docker镜像
首先你当然得安装Docker（略），然后下载TF-Serving的Docker镜像：

docker pull tensorflow/serving

下载完后，再 cd 到上面输出的“model”目录的上一级目录：

TESTDATA="$(pwd)/model"

docker run -t --rm -p 8501:8501 \

-v "$TESTDATA:/models/simple_fc_nn" \

-e MODEL_NAME=simple_fc_nn \

tensorflow/serving

这样就可以把模型serve起来了。其中，端口号可以自己改，simple_fc_nn是我自己起的模型名称，在后面使用REST API来访问TF-Serving服务的时候，会用到这个名称。
文章来源：https://www.codelast.com/
正常的话，命令行会输出类似于下面的日志：

tensorflow_serving/model_servers/server.cc:82] Building single TensorFlow model file config: model_name: simple_fc_nn model_base_path: /models/simple_fc_nn

tensorflow_serving/model_servers/server_core.cc:462] Adding/updating models.

tensorflow_serving/model_servers/server_core.cc:561] (Re-)adding model: simple_fc_nn

tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable {name: simple_fc_nn version: 201908070001}

tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: simple_fc_nn version: 201908070001}

tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: simple_fc_nn version: 201908070001}

external/org_tensorflow/tensorflow/contrib/session_bundle/bundle_shim.cc:363] Attempting to load native SavedModelBundle in bundle-shim from: /models/simple_fc_nn/201908070001

external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /models/simple_fc_nn/201908070001

external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }

external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA

external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.

external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 57164 microseconds.

tensorflow_serving/servables/tensorflow/saved_model_warmup.cc:103] No warmup data file found at /models/simple_fc_nn/201908070001/assets.extra/tf_serving_warmup_requests

tensorflow_serving/core/loader_harness.cc:86] Successfully loaded servable version {name: simple_fc_nn version: 201908070001}

tensorflow_serving/model_servers/server.cc:324] Running gRPC ModelServer at 0.0.0.0:8500 ...

[warn] getaddrinfo: address family for nodename not supported

[evhttp_server.cc : 239] RAW: Entering the event loop ...

tensorflow_serving/model_servers/server.cc:344] Exporting HTTP/REST API at:localhost:8501 ...

此时，我们就可以通过REST API来访问这个服务，来进行模型预测了。
文章来源：https://www.codelast.com/
[×] 通过REST API查看服务状态

curl http://localhost:8501/v1/models/simple_fc_nn

这里可以看到，URL里的simple_fc_nn就是在启动TF-Serving服务的时候指定的那个名字。
另外，这里使用的是localhost，所以必须在TF-Serving运行的同一台机器上执行该命令。
返回：

{

"model_version_status": [

{

"version": "201908070001",

"state": "AVAILABLE",

"status": {

"error_code": "OK",

"error_message": ""

}

]

}

文章来源：https://www.codelast.com/

[×] 通过REST API查看模型的元数据

curl http://localhost:8501/v1/models/simple_fc_nn/metadata

{

"model_spec": {

"name": "simple_fc_nn",

"signature_name": "",

"version": "201908070001"

"metadata": {

"signature_def": {

"serving_default": {

"inputs": {

"myInput": {

"dtype": "DT_FLOAT",

"tensor_shape": {

"dim": [

{

"size": "-1",

"name": ""

{

"size": "100",

"name": ""

}

"unknown_rank": false

"name": "myInput:0"

}

"outputs": {

"myOutput": {

"dtype": "DT_FLOAT",

"tensor_shape": {

"dim": [

{

"size": "-1",

"name": ""

{

"size": "1",

"name": ""

}

"unknown_rank": false

"name": "Sigmoid:0"

}

"method_name": "tensorflow/serving/predict"

}

文章来源：https://www.codelast.com/

[×] 通过REST API做一次模型预测

curl -d '{"instances": [[0.9255854, 0.6900963, 0.16456964, 0.83122249, 0.53394498, 0.0966489, 0.33756461, 0.39532023, 0.90302752, 0.56739237, 0.73394432, 0.85927172, 0.37721589, 0.54909182, 0.46008562, 0.96891122, 0.62722268, 0.06291056, 0.58401942, 0.16767831, 0.94320249, 0.22090671, 0.65477176, 0.1654681, 0.02658432, 0.21340836, 0.01726125, 0.34000626, 0.51242977, 0.8792962, 0.30855666, 0.77404037, 0.92957236, 0.12984285, 0.28480676, 0.20054448, 0.88745966, 0.89472512, 0.70519433, 0.52899148, 0.88639978, 0.45279248, 0.55500912, 0.74058798, 0.92649993, 0.65843703, 0.27740498, 0.3078109, 0.39115213, 0.67084904, 0.00103263, 0.19882637, 0.69135604, 0.98284994, 0.56733945, 0.14392018, 0.75021845, 0.69635296, 0.94257055, 0.16369508, 0.07440702, 0.86413908, 0.19726159, 0.59199729, 0.56901319, 0.92222904, 0.73738314, 0.08026704, 0.84337852, 0.00719291, 0.77943135, 0.47079168, 0.12110838, 0.8664636, 0.41693313, 0.50308834, 0.19198665, 0.25000135, 0.2613975, 0.48228129, 0.2522098, 0.08535753, 0.86548783, 0.74076042, 0.80560582, 0.33183196, 0.64742734, 0.71139366, 0.13395337, 0.90973808, 0.68171421, 0.5592008, 0.46866331, 0.58312591, 0.71402775, 0.82166998, 0.3564542, 0.97947134, 0.23538156, 0.9138861]]}' -X POST http://localhost:8501/v1/models/simple_fc_nn:predict

由于训练模型的时候定义了输入的每条数据为一个100维的向量，因此我造了上面的数据。
返回：

{"predictions": [[1]]}

这就算完成了一次预测过程。
文章来源：https://www.codelast.com/

[×] 通过Apache ab对TF-Serving进行性能测试

ab -n 100000 -c 50 -T 'Content-Type:application/json' -p ./post.txt http://localhost:8501/v1/models/simple_fc_nn:predict

其中，post.txt 保存的是上面请求中的JSON字符串。
ab会打印出详细的benchmark数据，包括QPS等：

Document Path: /v1/models/simple_fc_nn:predict

Document Length: 42 bytes

Concurrency Level: 50

Time taken for tests: 7.424 seconds

Complete requests: 100000

Failed requests: 0

Total transferred: 9300000 bytes

Total body sent: 138500000

HTML transferred: 4200000 bytes

Requests per second: 13469.99 [#/sec] (mean)

Time per request: 3.712 [ms] (mean)

Time per request: 0.074 [ms] (mean, across all concurrent requests)

Transfer rate: 1223.35 [Kbytes/sec] received

                        18218.68 kb/s sent

                        19442.03 kb/s total

Connection Times (ms)

              min mean[+/-sd] median max

Connect: 0 1 0.3 0 4

Processing: 1 3 1.2 3 211

Waiting: 1 3 1.2 3 211

Total: 1 4 1.2 4 212

文章来源：https://www.codelast.com/
[×] 停掉TF-Serving服务
先找到Docker容器进程：

docker container ps | grep "tensorflow/serving"

输出：

2bc9547d56b0 tensorflow/serving "/usr/bin/tf_serving…" 18 minutes ago Up 18 minutes 8500/tcp, 0.0.0.0:8501->8501/tcp strange_hopper

第一列为container id，干掉它即可：

docker container kill 2bc9547d56b0

（完）。
文章来源：https://www.codelast.com/
➤➤ 版权声明 ➤➤
转载需注明出处：codelast.com
感谢关注我的微信公众号（微信扫一扫）：

TensorFlow-Serving – 编码无悔 / Intent & Focused

[原创] 一个简单的TensorFlow-Serving例子