R中回归公式的执行

Question

我想R中使用随机梯度下降，以建立自己的回归函数，但我现在所拥有的使权重成长过程中没有约束，因此从来没有停止：

# Logistic regression 
# Takes training example vector, output vector, learn rate scalar, and convergence delta limit scalar 
my_logr <- function(training_examples,training_outputs,learn_rate,conv_lim) { 
    # Initialize gradient vector 
    gradient <- as.vector(rep(0,NCOL(training_examples))) 
    # Difference between weights 
    del_weights <- as.matrix(1) 
    # Weights 
    weights <- as.matrix(runif(NCOL(training_examples))) 
    weights_old <- as.matrix(rep(0,NCOL(training_examples))) 

    # Compute gradient 
    while(norm(del_weights) > conv_lim) { 

    for (k in 1:NROW(training_examples)) { 
     gradient <- gradient + 1/NROW(training_examples)* 
     ((t(training_outputs[k]*training_examples[k,] 
      /(1+exp(training_outputs[k]*t(weights)%*%as.numeric(training_examples[k,])))))) 
    } 

    # Update weights 
    weights <- weights_old - learn_rate*gradient 
    del_weights <- as.matrix(weights_old - weights) 
    weights_old <- weights 

    print(weights) 
    } 
    return(weights) 
} 

的功能可以用下面的代码进行测试：

data(iris) # Iris data already present in R  
# Dataset for part a (first 50 vs. last 100) 
iris_a <- iris 
iris_a$Species <- as.integer(iris_a$Species) 
# Convert list to binary class 
for (i in 1:NROW(iris_a$Species)) {if (iris_a$Species[i] != "1") {iris_a$Species[i] <- -1}}  
random_sample <- sample(1:NROW(iris),50) 

weights_a <- my_logr(iris_a[random_sample,1:4],iris_a$Species[random_sample],1,.1) 

我双重检查我的针对Abu-Mostafa's算法，其如下：

1. 初始化权重向量
2. 对于每个时间段计算梯度：
   gradient <- -1/N * sum_{1 to N} (training_answer_n * training_Vector_n/(1 + exp(training_answer_n * dot(weight,training_vector_n))))
3. weight_new <- weight - learn_rate*gradient
4. 重复，直到体重增量足够小

我失去了一些东西在这里？

Answer 1

从数学的角度来看，所述权重向量的无约束大小不产生独特的解决方案。当我将这两行的分类功能，它融合在两个步骤：

# Normalize 
weights <- weights/norm(weights) 

...

# Update weights 
weights <- weights_old - learn_rate*gradient 
weights <- weights/norm(weights) 

我不能让@ SimonO101的工作，我没有使用此真正的工作代码（有嵌入像glm），所以它足以做我理解的循环。 整个功能如下：

# Logistic regression 
# Takes training example vector, output vector, learn rate scalar, and convergence delta limit scalar 
my_logr <- function(training_examples,training_outputs,learn_rate,conv_lim) { 
    # Initialize gradient vector 
    gradient <- as.vector(rep(0,NCOL(training_examples))) 
    # Difference between weights 
    del_weights <- as.matrix(1) 
    # Weights 
    weights <- as.matrix(runif(NCOL(training_examples))) 
    weights_old <- as.matrix(rep(0,NCOL(training_examples))) 

    # Normalize 
    weights <- weights/norm(weights) 

    # Compute gradient 
    while(norm(del_weights) > conv_lim) { 

    for (k in 1:NCOL(training_examples)) { 
     gradient <- gradient - 1/NROW(training_examples)* 
     ((t(training_outputs[k]*training_examples[k,] 
      /(1+exp(training_outputs[k]*t(weights)%*%as.numeric(training_examples[k,])))))) 
    } 
#  gradient <- -1/NROW(training_examples) * sum(training_outputs * training_examples/(1 + exp(training_outputs * weights%*%training_outputs))) 

    # Update weights 
    weights <- weights_old - learn_rate*gradient 
    weights <- weights/norm(weights) 
    del_weights <- as.matrix(weights_old - weights) 
    weights_old <- weights 

    print(weights) 
    } 
    return(weights) 
} 

Answer 2

有几个问题。首先，你可以更好地使用R的矢量化方法。其次，我不是随机梯度下降的专家，但是您在问题下面给出的算法与您在函数中如何计算梯度不相符。仔细检查这个代码，但它似乎收敛，并且我认为它遵循Abu-Mostfafa's。我收集你想要计算这个梯度;

gradient <- -1/N * sum(training_outputs * training_examples/(1 + exp(training_outputs * dot(weights ,training_outputs)))) 

那么你的算法应该阅读这部分...

while(norm(del_weights) > conv_lim) { 
gradient <- -1/NROW(iris_a) * sum(training_outputs * training_examples/(1 + exp(training_outputs * as.matrix(training_examples) %*% weights))) 

# Update weights 
weights <- weights_old - learn_rate*gradient 
del_weights <- as.matrix(weights_old - weights) 
weights_old <- weights 
print(weights) 

}

您可以创建更容易物种变量的二元分类使用：

iris_a$Species <- as.numeric(iris_a$Species) 
iris_a$Species[ iris_a$Species != 1 ] <- -1  

我无法告诉您返回的结果是否合理，但该代码应该按照步骤2进行。检查每个步骤仔细，记住R是矢量化的，所以你可以在没有循环的矢量上做元素明智的操作。例如为：

x <- 1:5 
y <- 1:5 
x*y 
#[1] 1 4 9 16 25