在python tensorflow中划分批处理

Question

我试图设置批处理大小并运行Autoencoder程序，因为没有足够的内存来使用完整批处理。所以我试图使用tf.train.batch。但由于函数的参数是一个张量，我试图用tf.convert_to_tensor将np数组转换为张量。但是内存超过2GB，无法变成张量。我怎样才能用小批量培训？下面是 是我的代码。

N_img=47000000 
batch_size=100 
X_train = np.zeros(shape=(N_img, Freq_LEN, LOOK_LEN, 1), dtype='float32') 
x = tf.placeholder(tf.float32, [None, FRM_LEN/2,FRM_LEN/2,1]) #FRM_LEN=256 
y = tf.placeholder(tf.float32, [None, FRM_LEN/2,FRM_LEN/2,1]) 
X_train=tf.convert_to_tensor(X_train) 
X_train_batch= tf.train.batch(X_train,batch_size=batch_size) 

print("Start training..") 

for step in range(n_iters): 
    sess.run(optm, feed_dict={x: X_train_batch, y: X_train_batch, keepprob: 0.7}) 
    if step % 100 == 0: 
     print(step,sess.run(cost, feed_dict={x: X_train_batch, y: X_train_batch, keepprob: 1})) 

print("finish training") 

Answer 1

尝试创建不需要张量参数（避免tf.convert_to_tensor操作）的自定义功能generate_batch，例如：

import numpy as np 
batch_size = 100 

X_train = np.zeros(shape=(N_img, Freq_LEN, LOOK_LEN, 1), dtype='float32') 
y_train = np.zeros(shape=(N_img, Freq_LEN, LOOK_LEN, 1), dtype='float32') 

data_index = 0 

def generate_batch(batch_size): 
    global data_index 
    batch = np.ndarray(shape=(batch_size, Freq_LEN, LOOK_LEN, 1), dtype=np.float32) #the same shapes as train data 
    labels = np.ndarray(shape=(batch_size, Freq_LEN, LOOK_LEN, 1), dtype=np.float32) 
    for i in range(batch_size): 
     batch[i] = X_train[data_index] 
     labels[i] = y_train[data_index] 
     data_index = (data_index + 1) % len(X_train) 
    return batch, labels 

for step in range(n_iters): 
    X_train_batch, X_train_batch = generate_batch(batch_size) 
    sess.run(optm, feed_dict={x: X_train_batch, y: X_train_batch, keepprob: 0.7}) 
    if step % 100 == 0: 
     print(step,sess.run(cost, feed_dict={x: X_train_batch, y: X_train_batch, keepprob: 1})) 

Answer 2

力批次处理的CPU上发生：

.... 
with tf.device('/cpu:0'): 
    x = tf.placeholder(tf.float32, [None, FRM_LEN/2,FRM_LEN/2,1]) #FRM_LEN=256 
    y = tf.placeholder(tf.float32, [None, FRM_LEN/2,FRM_LEN/2,1]) 
    X_train=tf.convert_to_tensor(X_train) 
    X_train_batch= tf.train.batch(X_train,batch_size=batch_size) 
print("Start training..") 
....