Tensorflow：“GraphDef不能大于2GB”。在分配变量后保存模型时发生错误

Question

我想使用预训练模型热烈启动另一个模型，但有一点区别。简单地说，我创建一个新模型，并使用预训练模型权重为变量赋予相同的名称。但是，保存模型时发生错误。

Traceback (most recent call last): File "tf_test.py", line 23, in <module> save_path = saver.save(sess, "./model.ckpt") File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1308, in save self.export_meta_graph(meta_graph_filename) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1331, in export_meta_graph graph_def=ops.get_default_graph().as_graph_def(add_shapes=True), File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2268, in as_graph_def result, _ = self._as_graph_def(from_version, add_shapes) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2231, in _as_graph_def raise ValueError("GraphDef cannot be larger than 2GB.") ValueError: GraphDef cannot be larger than 2GB.

的示例代码如下：

import tensorflow as tf 
import numpy as np 

v1 = tf.get_variable("L_enc", [400000, 1024]) 
v2 = tf.get_variable("L_dec", [400000, 1024]) 

init_op = tf.initialize_all_variables() 

saver = tf.train.Saver(tf.all_variables()) 

with tf.Session() as sess: 
    sess.run(init_op) 
    for v in tf.trainable_variables(): 
    embedding = np.random.uniform(-1, 1, (400000, 1024)) 
    sess.run(v.assign(embedding)) 
    # Save the variables to disk. 
    save_path = saver.save(sess, "./model.ckpt") 
    print("Model saved in file: %s" % save_path) 

Answer 1

有2GB的，因为在protobuf的32位带符号的大小的序列化个体张量 的硬性限制。

https://github.com/tensorflow/tensorflow/issues/4291

Answer 2

法布里奇奥correctly points out有关于协议缓冲区的大小硬2GB的限制，但为什么程序打这个限制，你可能想知道。该问题由这些线茎：

for v in tf.trainable_variables(): 
    embedding = np.random.uniform(-1, 1, (400000, 1024)) 
    sess.run(v.assign(embedding)) 

当执行命中v.assign(embedding)，新节点被添加到TensorFlow曲线图。特别是，每个embedding阵列都被转换为tf.constant()张量，这个张量将非常大（根据我的估计大约为328MB）。

避免这种情况的最佳方法是使用tf.train.Saver将以前模型中的变量直接加载到新模型中。由于模型可能具有不同的结构，因此可能需要指定从旧模型中的变量名称到新模型中的tf.Variable对象的映射。

---

解决您的问题将是值分配给每个变量预先创建一个tf.placeholder()运的另一种方式。这可能需要你的代码还多的重组，但对我下面的工作：

v1 = tf.get_variable("L_enc", [400000, 1024]) 
v2 = tf.get_variable("L_dec", [400000, 1024]) 

# Define a separate placeholder and assign op for each variable, so 
# that we can feed the initial value without adding it to the graph. 
vars = [v1, v2] 
placeholders = [tf.placeholder(tf.float32, shape=[400000, 1024]) for v in vars] 
assign_ops = [v.assign(p) for (v, p) in zip(vars, placeholders)] 

init_op = tf.global_variables_initializer() 

saver = tf.train.Saver(tf.all_variables()) 

with tf.Session() as sess: 
    sess.run(init_op) 
    for p, assign_op in zip(placeholders, assign_ops): 
    embedding = np.random.uniform(-1, 1, (400000, 1024)) 
    sess.run(assign_op, {p: embedding}) 

    # Save the variables to disk. 
    save_path = saver.save(sess, "./model.ckpt") 
    print("Model saved in file: %s" % save_path)