kubernetes下搭建harbor企业级docker仓库

Java007 · · 2409 次点击 · 开始浏览    置顶
这是一个创建于 的主题,其中的信息可能已经有所发展或是发生改变。
### 前言 harbor是由vmware开源的企业级docker repository,提供私有仓库,安全认证,权限管理,漏洞扫描及仓库复制等多种功能,因为之前搭建的harbor在线上环境存在各种各样的问题(之前搭建的方式目前发现已被官方宣布弃用[https://github.com/vmware/harbor/blob/master/docs/kubernetes_deployment.md](https://github.com/vmware/harbor/blob/master/docs/kubernetes_deployment.md),如ui显示不正常,权限认证不正常),于是近期抽空用官方推荐的helm方式对harbor进行重新部署,并替换掉线上harbor ### 准备 1. Kubernetes集群1.8+ 2. Kubernetes Ingress Controller已启用(这里我们选用的traefik-ingress) 3. kubectl 客户端 1.8+ 4. 可选的持久化功能需要准备PV或者SC(我们这里使用nfs创建的PV和PVC,如果有条件使用SC的会更加方便) ### 部署过程 1. 首先按照[官方教程](https://github.com/kubernetes/helm#install)安装[Helm](https://github.com/kubernetes/helm#install),然后初始化Helm。 `注意`: 初始化需要使用下面命令使用canary镜像,否则无法正常安装,会报错helm部署文件的格式不正确,目前这是一个已知issue:[https://github.com/vmware/harbor/issues/4484](https://github.com/vmware/harbor/issues/4484) ``` helm init --canary-image ``` 如果你和我一样,在这之前已经用官方教程安装好了helm,则可使用下面命令更新helm server: ``` helm init --canary-image --upgrade ``` 2. 下载helm部署代码并进入harbor helm目录。 ``` git clone https://github.com/vmware/harbor cd harbor/contrib/helm/harbor ``` 推荐使用这个连接下载指定的文件夹,否则在网络不佳的情况下拉取全部代码会比较耗时: ``` https://minhaskamal.github.io/DownGit/#/home ``` 3. 更新helm dependency harbor的helm部署依赖了postgresql的helm,在官方的安装文档没有明确说明,因为我也是第一次使用,直接按照官方文档说明安装,就会缺失postgresql的部署,导致整个服务无法启动 ``` helm dependency update ``` 4. 安装harbor 这里官方提供了两种方式,Insecure和Secure,我这里选用的是Secure安全的部署方式,让harbor自己生成CA和SSL,简单方便。执行如下命令安装harbor: ``` helm install . --debug --name hub --set externalDomain=harbor.my.domain ``` externalDomain为外部能访问到harbor的域名,到目前为止,你可以在本地/etc/hosts中添加域名解析,从本地访问进行测试,待测试完成后再加入traefik-ingress中,当然你也可以直接在traefik-ingress中添加域名解析。 5. 添加traefik-ingress的域名解析 ``` apiVersion: extensions/v1beta1 kind: Ingress metadata: name: traefik-default-ingress namespace: default annotations: kubernetes.io/ingress.class: "traefik" spec: rules: - host: harbor.my.domain http: paths: - path: / backend: serviceName: my-release-harbor-ui servicePort: 80 ``` ### helm harbor自定义配置 注意: 我这里把所有自定义配置放到后面的附录中,供大家参考,同时可以点击[https://github.com/Anteoy/harbor-helm/commit/226b296d130b4f956f8463eecf2aa473bc1e844c](https://github.com/Anteoy/harbor-helm/commit/226b296d130b4f956f8463eecf2aa473bc1e844c),查看我上传到github的自定义配置,从github阅读更清晰。 1. 重要:在生产环境中需要持久化数据存储,否则pod重启或重建会造成数据丢失。如果有创建好的storageClass可以直接在values.yaml配置,如果没有或暂时不能使用SC的,比如我这里只有nfs,则需要像我一样多修改一些配置。另外,nfs其实在社区也有storageClass的驱动库,不过我看了下使用起来较为繁琐,于是这里仍然修改为使用nfs。 2. 修改templates默认的namespace,注意需要修改依赖的chart文件夹下的postgresql-0.9-1.tgz压缩文件中的部署yaml模板,否则harbor的ui,register,mysql和postgresql不在一个namespace下,不能正常安装 3. 注意修改postgresql的持久化数据卷,可参考[https://github.com/kubernetes/charts/tree/master/stable/postgresql](https://github.com/kubernetes/charts/tree/master/stable/postgresql) ### 附录 ``` templates/adminserver/adminserver-cm.yaml @@ -1,6 +1,7 @@ apiVersion: v1 kind: ConfigMap metadata: + namespace: class100-ops name: "{{ template "harbor.fullname" . }}-adminserver" labels: {{ include "harbor.labels" . | indent 4 }} 1 templates/adminserver/adminserver-secrets.yaml @@ -2,6 +2,7 @@ apiVersion: v1 kind: Secret metadata: name: "{{ template "harbor.fullname" . }}-adminserver" + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} component: adminserver 5 templates/adminserver/adminserver-ss.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1beta2 kind: StatefulSet metadata: name: "{{ template "harbor.fullname" . }}-adminserver" + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} component: adminserver @@ -42,13 +43,15 @@ spec: volumeMounts: - name: adminserver-config mountPath: /etc/adminserver/config + subPath: harbor-v1/adminserver/ - name: adminserver-key mountPath: /etc/adminserver/key subPath: key volumes: {{- if not .Values.persistence.enabled }} - name: adminserver-config - emptyDir: {} + persistentVolumeClaim: + claimName: harbor-pvc {{- end }} - name: adminserver-key secret: 1 templates/adminserver/adminserver-svc.yaml @@ -2,6 +2,7 @@ apiVersion: v1 kind: Service metadata: name: "{{ template "harbor.fullname" . }}-adminserver" + namespace: class100-ops spec: ports: - port: 80 1 templates/clair/clair-cm.yaml @@ -3,6 +3,7 @@ apiVersion: v1 kind: ConfigMap metadata: name: {{ template "harbor.fullname" . }}-clair + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} component: clair 1 templates/clair/clair-dpl.yaml @@ -3,6 +3,7 @@ apiVersion: extensions/v1beta1 kind: Deployment metadata: name: {{ template "harbor.fullname" . }}-clair + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} component: clair 1 templates/clair/clair-svc.yaml @@ -6,6 +6,7 @@ apiVersion: v1 kind: Service metadata: name: clair + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} spec: 1 templates/ingress/ingress.yaml @@ -2,6 +2,7 @@ apiVersion: extensions/v1beta1 kind: Ingress metadata: name: "{{ template "harbor.fullname" . }}-ingress" + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} annotations: 1 templates/ingress/secret.yaml @@ -5,6 +5,7 @@ apiVersion: v1 kind: Secret metadata: name: "{{ template "harbor.fullname" . }}-ingress" + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} type: kubernetes.io/tls 1 templates/jobservice/jobservice-cm.yaml @@ -2,6 +2,7 @@ apiVersion: v1 kind: ConfigMap metadata: name: "{{ template "harbor.fullname" . }}-jobservice" + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} data: 1 templates/jobservice/jobservice-dpl.yaml @@ -2,6 +2,7 @@ apiVersion: extensions/v1beta1 kind: Deployment metadata: name: "{{ template "harbor.fullname" . }}-jobservice" + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} component: jobservice 1 templates/jobservice/jobservice-secrets.yaml @@ -2,6 +2,7 @@ apiVersion: v1 kind: Secret metadata: name: "{{ template "harbor.fullname" . }}-jobservice" + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} type: Opaque 1 templates/jobservice/jobservice-svc.yaml @@ -2,6 +2,7 @@ apiVersion: v1 kind: Service metadata: name: "{{ template "harbor.fullname" . }}-jobservice" + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} spec: 1 templates/mysql/mysql-secret.yaml @@ -2,6 +2,7 @@ apiVersion: v1 kind: Secret metadata: name: "{{ template "harbor.fullname" . }}-mysql" + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} type: Opaque 7 templates/mysql/mysql-ss.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1beta2 kind: StatefulSet metadata: name: "{{ template "harbor.fullname" . }}-mysql" + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} component: mysql @@ -33,10 +34,12 @@ spec: volumeMounts: - name: mysql-data mountPath: /var/lib/mysql + subPath: harbor-v1/mysql-data/ {{- if not .Values.persistence.enabled }} volumes: - - name: "mysql-data" - emptyDir: {} + - name: mysql-data + persistentVolumeClaim: + claimName: harbor-pvc {{- end -}} {{- if .Values.persistence.enabled }} volumeClaimTemplates: 1 templates/mysql/mysql-svc.yaml @@ -2,6 +2,7 @@ apiVersion: v1 kind: Service metadata: name: "{{ template "harbor.fullname" . }}-mysql" + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} spec: 1 templates/registry/registry-cm.yaml @@ -2,6 +2,7 @@ apiVersion: v1 kind: ConfigMap metadata: name: "{{ template "harbor.fullname" . }}-registry" + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} data: 1 templates/registry/registry-secret.yaml @@ -2,6 +2,7 @@ apiVersion: v1 kind: Secret metadata: name: "{{ template "harbor.fullname" . }}-registry" + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} type: Opaque 5 templates/registry/registry-ss.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1beta2 kind: StatefulSet metadata: name: "{{ template "harbor.fullname" . }}-registry" + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} component: registry @@ -37,6 +38,7 @@ spec: volumeMounts: - name: registry-data mountPath: /var/lib/registry + subPath: harbor-v1/registry-data/ - name: registry-root-certificate mountPath: /etc/registry/root.crt subPath: root.crt @@ -47,7 +49,8 @@ spec: {{- if not .Values.registry.objectStorage }} {{- if not .Values.persistence.enabled }} - name: registry-data - emptyDir: {} + persistentVolumeClaim: + claimName: harbor-pvc {{- end }} {{- end }} - name: registry-root-certificate 1 templates/registry/registry-svc.yaml @@ -2,6 +2,7 @@ apiVersion: v1 kind: Service metadata: name: "{{ template "harbor.fullname" . }}-registry" + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} spec: 1 templates/ui/ui-cm.yaml @@ -2,6 +2,7 @@ apiVersion: v1 kind: ConfigMap metadata: name: "{{ template "harbor.fullname" . }}-ui" + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} data: 3 templates/ui/ui-dpl.yaml @@ -2,6 +2,7 @@ apiVersion: extensions/v1beta1 kind: Deployment metadata: name: "{{ template "harbor.fullname" . }}-ui" + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} component: ui @@ -54,8 +55,10 @@ spec: subPath: private_key.pem - name: ca-download mountPath: /etc/ui/ca + subPath: harbor-v1/ui-ca/ - name: psc mountPath: /etc/ui/token + subPath: harbor-v1/ui-psc/ volumes: - name: ui-config configMap: 1 templates/ui/ui-secrets.yaml @@ -2,6 +2,7 @@ apiVersion: v1 kind: Secret metadata: name: "{{ template "harbor.fullname" . }}-ui" + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} type: Opaque 1 templates/ui/ui-svc.yaml @@ -2,6 +2,7 @@ apiVersion: v1 kind: Service metadata: name: "{{ template "harbor.fullname" . }}-ui" + namespace: class100-ops labels: {{ include "harbor.labels" . | indent 4 }} spec: 3 values.yaml @@ -287,4 +287,5 @@ postgresql: postgresPassword: not-a-secure-password postgresDatabase: clair persistence: - enabled: false + enabled: true + existingClaim: harbor-pvc ``` ### 参考 1. [https://github.com/vmware/harbor/issues/4484](https://github.com/vmware/harbor/issues/4484) 2. [https://github.com/vmware/harbor/tree/master/contrib/helm/harbor](https://github.com/vmware/harbor/tree/master/contrib/helm/harbor) 3. [https://github.com/vmware/harbor/issues/4481](https://github.com/vmware/harbor/issues/4481) 4. [https://kubernetes.io/docs/concepts/storage/storage-classes/#introduction](https://kubernetes.io/docs/concepts/storage/storage-classes/#introduction)
2409 次点击  
加入收藏 微博
暂无回复
添加一条新回复 (您需要 登录 后才能回复 没有账号 ?)
  • 请尽量让自己的回复能够对别人有帮助
  • 支持 Markdown 格式, **粗体**、~~删除线~~、`单行代码`
  • 支持 @ 本站用户;支持表情(输入 : 提示),见 Emoji cheat sheet
  • 图片支持拖拽、截图粘贴等方式上传