势头和（小）批量训练是否兼容？

Question

我有一个反向传播模型工作得很好，但我想实施批量训练。

一批培训（反向传播功能），伪代码之前代码：

forevery(connection in this.connections.in){ 
    // Adjust weight 
    var deltaWeight = rate * gradient + momentum * connection.previousDeltaWeight; 
    connection.weight += deltaWeight; 
    connection.previousDeltaWeight = deltaWeight; 
} 

// Adjust bias 
var deltaBias = rate * this.error.responsibility + momentum * this.previousDeltaBias; 
this.bias += deltaBias; 

this.previousDeltabias = deltaBias; 

而且新的代码是：

forevery(connection in this.connections.in){ 
    // Adjust weight 
    var deltaWeight = rate * gradient * this.mask + momentum * connection.previousDeltaWeight; 
    connection.totalDeltaWeight += deltaWeight; 
    if(update){ 
    connection.weight += connection.totalDeltaWeight; 
    connection.previousDeltaWeight = connection.totalDeltaWeight; 
    connection.totalDeltaWeight = 0; 
    } 
} 

// Adjust bias 
var deltaBias = rate * this.error.responsibility + momentum * this.previousDeltaBias; 
this.totalDeltaBias += deltaBias; 
if(update){ 
    this.bias += this.totalDeltaBias; 
    this.previousDeltaBias = this.totalDeltaBias; 
    this.totalDeltaBias = 0; 
} 

因此，如果批量大小为4，反传被调用用3X update=false和第4次与update=true。批量训练工作正常，但是当我开启动力(=0.9)时，所有值都开始溢出。可能是什么问题呢？

Answer 1

哇。我积累了错误的动力。我把它包括在内batch_size这是错误的，所以它现在只包含一次。

forevery(connection in this.connections.in){ 
    // Adjust weight 
    var deltaWeight = rate * gradient * this.mask; 
    connection.totalDeltaWeight += deltaWeight; 
    if(update){ 
    connection.totalDeltaWeight += momentum * connection.previousDeltaWeight; 
    connection.weight += connection.totalDeltaWeight; 
    connection.previousDeltaWeight = connection.totalDeltaWeight; 
    connection.totalDeltaWeight = 0; 
    } 
} 

// note: MINI_BATCH SHALL BE OPTIMIZED SOON 

// Adjust bias 
var deltaBias = rate * this.error.responsibility; 
this.totalDeltaBias += deltaBias; 
if(update){ 
    this.totalDeltaBias += momentum * this.previousDeltaBias; 
    this.bias += this.totalDeltaBias; 
    this.previousDeltaBias = this.totalDeltaBias; 
    this.totalDeltaBias = 0; 
}