0
我想训练这个神经网络来对某些数据做出预测。 我在一个小数据集(大约100条记录)上试过它,它的功能就像一个魅力。然后插入新的数据集,我发现NN收敛到0输出,误差近似收敛到正例的数量和例子总数之间的比例。神经网络收敛到零输出
我的数据集由yes/no features(1.0/0.0)组成,地面实况为yes/no。
我的推测:
1)有一个局部最小值与输出0(但我试图与学习率和init权的许多价值观,似乎收敛总是存在的)
2)我的体重更新是错误的(但看起来不错)
3)这只是一个输出缩放问题。我尝试缩放输出(即输出/最大(输出)和输出/平均(输出)),但结果不如您在下面提供的代码中看到的那样好。我应该以不同的方式进行缩放吗? SOFTMAX?
这里是代码:
import pandas as pd
import numpy as np
import pickle
import random
from collections import defaultdict
alpha = 0.1
N_LAYERS = 10
N_ITER = 10
#N_FEATURES = 8
INIT_SCALE = 1.0
train = pd.read_csv("./data/prediction.csv")
y = train['y_true'].as_matrix()
y = np.vstack(y).astype(float)
ytest = y[18000:]
y = y[:18000]
X = train.drop(['y_true'], axis = 1).as_matrix()
Xtest = X[18000:].astype(float)
X = X[:18000]
def tanh(x,deriv=False):
if(deriv==True):
return (1 - np.tanh(x)**2) * alpha
else:
return np.tanh(x)
def sigmoid(x,deriv=False):
if(deriv==True):
return x*(1-x)
else:
return 1/(1+np.exp(-x))
def relu(x,deriv=False):
if(deriv==True):
return 0.01 + 0.99*(x>0)
else:
return 0.01*x + 0.99*x*(x>0)
np.random.seed()
syn = defaultdict(np.array)
for i in range(N_LAYERS-1):
syn[i] = INIT_SCALE * np.random.random((len(X[0]),len(X[0]))) - INIT_SCALE/2
syn[N_LAYERS-1] = INIT_SCALE * np.random.random((len(X[0]),1)) - INIT_SCALE/2
l = defaultdict(np.array)
delta = defaultdict(np.array)
for j in xrange(N_ITER):
l[0] = X
for i in range(1,N_LAYERS+1):
l[i] = relu(np.dot(l[i-1],syn[i-1]))
error = (y - l[N_LAYERS])
e = np.mean(np.abs(error))
if (j% 1) == 0:
print "\nIteration " + str(j) + " of " + str(N_ITER)
print "Error: " + str(e)
delta[N_LAYERS] = error*relu(l[N_LAYERS],deriv=True) * alpha
for i in range(N_LAYERS-1,0,-1):
error = delta[i+1].dot(syn[i].T)
delta[i] = error*relu(l[i],deriv=True) * alpha
for i in range(N_LAYERS):
syn[i] += l[i].T.dot(delta[i+1])
pickle.dump(syn, open('neural_weights.pkl', 'wb'))
# TESTING with f1-measure
# RECALL = TRUE POSITIVES/(TRUE POSITIVES + FALSE NEGATIVES)
# PRECISION = TRUE POSITIVES/(TRUE POSITIVES + FALSE POSITIVES)
l[0] = Xtest
for i in range(1,N_LAYERS+1):
l[i] = relu(np.dot(l[i-1],syn[i-1]))
out = l[N_LAYERS]/max(l[N_LAYERS])
tp = float(0)
fp = float(0)
fn = float(0)
tn = float(0)
for i in l[N_LAYERS][:50]:
print i
for i in range(len(ytest)):
if out[i] > 0.5 and ytest[i] == 1:
tp += 1
if out[i] <= 0.5 and ytest[i] == 1:
fn += 1
if out[i] > 0.5 and ytest[i] == 0:
fp += 1
if out[i] <= 0.5 and ytest[i] == 0:
tn += 1
print "tp: " + str(tp)
print "fp: " + str(fp)
print "tn: " + str(tn)
print "fn: " + str(fn)
print "\nprecision: " + str(tp/(tp + fp))
print "recall: " + str(tp/(tp + fn))
f1 = 2 * tp /(2 * tp + fn + fp)
print "\nf1-measure:" + str(f1)
,这是输出:
Iteration 0 of 10
Error: 0.222500767998
Iteration 1 of 10
Error: 0.222500771157
Iteration 2 of 10
Error: 0.222500774321
Iteration 3 of 10
Error: 0.22250077749
Iteration 4 of 10
Error: 0.222500780663
Iteration 5 of 10
Error: 0.222500783841
Iteration 6 of 10
Error: 0.222500787024
Iteration 7 of 10
Error: 0.222500790212
Iteration 8 of 10
Error: 0.222500793405
Iteration 9 of 10
Error: 0.222500796602
[ 0.]
[ 0.]
[ 5.58610895e-06]
[ 0.]
[ 0.]
[ 0.]
[ 0.]
[ 0.]
[ 4.62182626e-06]
[ 0.]
[ 0.]
[ 0.]
[ 0.]
[ 5.58610895e-06]
[ 0.]
[ 0.]
[ 0.]
[ 0.]
[ 4.62182626e-06]
[ 0.]
[ 0.]
[ 5.04501079e-10]
[ 5.58610895e-06]
[ 0.]
[ 0.]
[ 0.]
[ 0.]
[ 0.]
[ 0.]
[ 0.]
[ 0.]
[ 0.]
[ 0.]
[ 0.]
[ 5.04501079e-10]
[ 0.]
[ 0.]
[ 4.62182626e-06]
[ 0.]
[ 5.58610895e-06]
[ 0.]
[ 0.]
[ 0.]
[ 5.58610895e-06]
[ 0.]
[ 0.]
[ 0.]
[ 5.58610895e-06]
[ 0.]
[ 1.31432294e-05]
tp: 28.0
fp: 119.0
tn: 5537.0
fn: 1550.0
precision: 0.190476190476
recall: 0.0177439797212
f1-measure:0.0324637681159
10次迭代是_nothing_。把它扩展到至少1000.我不确定它会解决你的问题,因为你的错误实际上是通过迭代得到更高的。但我建议你改变它。 –
是的,这只是一个测试。错误增加是因为学习速度有点太大。无论如何,该数据集是27k的例子,所以我认为没有太多的迭代是必需的。 – RobiNoob
27k样品!只有在27k个样本具有相同的样本并且没有任何噪声的情况下,您的网络在10次迭代中绝不会收敛。缩放迭代次数,并显示结果。 –