2017-02-26 125 views
1

我有一个Kubernetes 1.5.3集群,它使用kubeadm(详情如下)在两个节点上启动。我一直在试图启动今晚的prometheus-operator。在我删除了一种类型=普罗米修斯的资源之后,我的集群进入了一个非常奇怪的状态,当我试图重新创建普罗米修斯资源时,虽然TPR存在,但没有创建任何Pod。调查使我:kube-apiserver无法正常运行,“名称命名空间不存在索引”

[email protected]:~$ kubectl logs -n kube-system kube-apiserver-hypervisor01 | tail -10 W0226 03:57:34.641244 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:57:34.645073 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:58:04.642150 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:58:04.647953 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:58:34.642118 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:58:34.646427 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:59:04.642189 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:59:04.647978 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:59:34.646129 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:59:34.666355 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist

解决办法在此rancher github issue是简单地重新启动API服务器,这将是很好,除了这并没有解决问题。否则,它会指向etcd上的高负载,正如您在下面看到的那样,该负载是托管为一个容器,位于当前负载小于1的主机上。etcdctl命令运行成功,导致我相信etcd本身很好。为了好的措施,我碰到了etcd docker容器,它仍然正常工作。但是,从这里开始,我迷路了,需要一些帮助让我的集群恢复正常工作状态。

群集概述:

[email protected]:~$ kubectl get no NAME STATUS AGE hypervisor01 Ready,master 2d kubes01.pss9.kickass.systems Ready 2d [email protected]:~$ kubectl get -n kube-system po NAME READY STATUS RESTARTS AGE dummy-2088944543-2p3zf 1/1 Running 1 2d etcd-hypervisor01 1/1 Running 2 2d kube-apiserver-hypervisor01 1/1 Running 3 2d kube-controller-manager-hypervisor01 1/1 Running 1 2d kube-discovery-1769846148-v8h50 1/1 Running 1 2d kube-dns-2924299975-3s26d 4/4 Running 4 2d kube-proxy-vpw73 1/1 Running 1 2d kube-proxy-zfh13 1/1 Running 0 2d kube-registry-proxy-6hhk9 1/1 Running 1 2d kube-registry-proxy-nl1s1 1/1 Running 0 2d kube-registry-v0-4d94t 1/1 Running 0 2d kube-scheduler-hypervisor01 1/1 Running 1 2d [email protected]:~$ kubectl get po NAME READY STATUS RESTARTS AGE kube-flannel-ds-589fw 2/2 Running 3 2d kube-flannel-ds-7f5sx 2/2 Running 0 2d kube-state-metrics-3229993571-20nbc 1/1 Running 0 1h node-exporter-1cdlj 1/1 Running 0 1h node-exporter-jc54s 1/1 Running 0 1h prometheus-operator-996254120-0wzg6 1/1 Running 0 2h

+0

https://github.com/kubernetes/kubernetes/blob/v1.5.3/pkg/client/cache/listers.go#L68该代码似乎暗示事情应该继续在这里成功,所以也许这个日志是一个红鲱鱼... –

回答

1

E0226 16:20:47.861762 1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-k8s, requeuing: Failed to create prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value I0226 16:20:47.862793 1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-k8s", UID:"272d42fd-fbd1-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304212", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0, error: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value E0226 16:20:47.865352 1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-services, requeuing: Failed to create prometheus-prometheus-services-db-prometheus-prometheus-services-0: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value I0226 16:20:47.865472 1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-services", UID:"1899b833-fbd2-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304733", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-services-db-prometheus-prometheus-services-0, error: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value E0226 16:21:17.854692 1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-k8s, requeuing: Failed to create prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value I0226 16:21:17.855043 1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-k8s", UID:"272d42fd-fbd1-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304212", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0, error: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value E0226 16:21:17.858436 1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-services, requeuing: Failed to create prometheus-prometheus-services-db-prometheus-prometheus-services-0: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value I0226 16:21:17.858561 1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-services", UID:"1899b833-fbd2-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304733", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-services-db-prometheus-prometheus-services-0, error: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value

用户错误捂脸,没想到在管理日志看。

相关问题