Openshift Catalog证书更新

2021-10-08 约 454 字预计阅读 1 分钟

故障排查

controller-manager 一直在BackOff状态：

$ oc -n kube-service-catalog get pods
...
controller-manager-x5dr8   0/1       CrashLoopBackOff   21         1h

查看日志：

$ oc -n kube-service-catalog logs -f --tail=30 controller-manager-x5dr8
I1008 06:24:21.655748       1 feature_gate.go:194] feature gates: map[OriginatingIdentity:true]
I1008 06:24:21.655957       1 feature_gate.go:194] feature gates: map[OriginatingIdentity:true AsyncBindingOperations:true]
I1008 06:24:21.655983       1 feature_gate.go:194] feature gates: map[OriginatingIdentity:true AsyncBindingOperations:true NamespacedServiceBroker:true]
I1008 06:24:21.656012       1 hyperkube.go:192] Service Catalog version v3.11.0-0.1.35+8d4f895-2;Upstream:v0.1.35 (built 2019-01-08T23:12:26Z)
I1008 06:24:21.659263       1 leaderelection.go:185] attempting to acquire leader lease  kube-service-catalog/service-catalog-controller-manager...
I1008 06:24:21.677905       1 leaderelection.go:194] successfully acquired lease kube-service-catalog/service-catalog-controller-manager
I1008 06:24:21.678992       1 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-service-catalog", Name:"service-catalog-controller-manager", UID:"f4993f8c-93f0-11e9-9c59-00163e0a2de7", APIVersion:"v1", ResourceVersion:"189814138", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' controller-manager-x5dr8-external-service-catalog-controller became leader
F1008 06:24:21.726721       1 controller_manager.go:237] error running controllers: failed to get api versions from server: failed to get supported resources from server: unable to retrieve the complete list of server APIs: servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request

这个报错有点奇怪的，去对应代码看了看，就是APIServer那边返回的，我直接用cURL看了一下：

$ TOKEN=$(oc whoami -t)
$ curl  -X GET -H "Authorization: Bearer ${TOKEN}" 'https://1.2.3.4:8443/apis/servicecatalog.k8s.io/v1beta1'
Error: 'x509: certificate has expired or is not yet valid'

原来是catalog服务的证书到期了。

解决方法

使用OpenShift-Ansible来重新部署更新证书。

复制一份playbooks/redeploy-certificates.yml，把playbooks/redeploy-certificates.yml中其他项目的playbook注释了，只保留init和catalog的。

$ cd openshift-ansible
$ cp -a  playbooks/redeploy-certificates.yml playbooks/redeploy-certificates-catalog.yml

$ cat playbooks/redeploy-certificates-catalog.yml
---
- import_playbook: init/main.yml

- import_playbook: openshift-service-catalog/private/redeploy-certificates.yml
  when: openshift_enable_service_catalog | default(true) | bool

重新部署：

$ ansible-playbook -i </path/to/inventory/file>  playbooks/redeploy-certificates-catalog.yml

部署完后确认：

curl  -I -X GET -H "Authorization: Bearer ${TOKEN}" 'https://1.2.3.4:8443/apis/servicecatalog.k8s.io/v1beta1'
HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: application/json
Date: Fri, 08 Oct 2021 06:37:39 GMT
Transfer-Encoding: chunked

目录

Openshift Catalog证书更新

故障排查

解决方法