Joen's Blog

CICD Renewal

coding五月 27, 20161mins

CI CD DevOps

持续集成H2

依然使用Gitlab CI作为CI工具，一些重要的特性

通过service 使用dind
通过dotenv传递任务间的环境变量
强大的rules设置：仅PR时触发CI

阶段和任务H3

build
- build gradle/npm build
test
- code_analysis(code_quality) sonarqube代码质量检查
publish
- public_registry 发布到Maven/NPM/镜像仓库

示例H3

bash
variables:
  CI_REGISTRY: harbor.dev.hithinksoft.com
  CI_REGISTRY_IMAGE: ${CI_REGISTRY}/${CI_PROJECT_NAMESPACE}/${CI_PROJECT_NAME}
  SONAR_URL: http://sonarqube.dev.hithinksoft.com/
  GRADLE_USER_HOME: ${CI_PROJECT_DIR}/.gradle
  NPM_CONFIG_REGISTRY: https://nexus.dev.hithinksoft.com/repository/npm-public/
  NPM_CONFIG_CACHE: /cache/.npm
  YARN_CACHE_FOLDER: /cache/.yarn-cache
  ELECTRON_MIRROR: "https://cdn.npm.taobao.org/dist/electron/"
  ELECTRON_BUILDER_BINARIES_MIRROR: "https://nexus.dev.hithinksoft.com/repository/raw-public/"
  ELECTRON_CACHE: "/cache/electron"
  ELECTRON_BUILDER_CACHE: "/cache/electron-builder"
  SASS_BINARY_SITE: "http://npm.taobao.org/mirrors/node-sass"
  DOCKER_TLS_CERTDIR: "/certs"
  TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX: harbor.dev.hithinksoft.com/docker-proxy/
default:
  image: harbor.dev.hithinksoft.com/library/eclipse-temurin:17-docker
  services:
    - name: harbor.dev.hithinksoft.com/docker-proxy/docker:24.0.5-dind
      alias: docker
cache:
  paths:
    - .gradle/caches
    - .gradle/wrapper
stages:
  - .pre
  - build
  - test
  - publish
  - .post
.build_commit_tag: &build_commit_tag |
  C7N_COMMIT_TIMESTAMP=$(git log -1 --pretty=format:"%ci"| awk '{print $1$2}' | sed 's/[-:]//g')
  C7N_COMMIT_YEAR=${C7N_COMMIT_TIMESTAMP:0:4}
  C7N_COMMIT_MONTH=$(echo ${C7N_COMMIT_TIMESTAMP:4:2} | sed s'/^0//')
  C7N_COMMIT_DAY=$(echo ${C7N_COMMIT_TIMESTAMP:6:2} | sed s'/^0//')
  C7N_COMMIT_HOURS=${C7N_COMMIT_TIMESTAMP:8:2}
  C7N_COMMIT_MINUTES=${C7N_COMMIT_TIMESTAMP:10:2}
  C7N_COMMIT_SECONDS=${C7N_COMMIT_TIMESTAMP:12:2}
  export C7N_COMMIT_TIME=$C7N_COMMIT_YEAR.$C7N_COMMIT_MONTH.$C7N_COMMIT_DAY-$C7N_COMMIT_HOURS$C7N_COMMIT_MINUTES$C7N_COMMIT_SECONDS
  # SHA value
  export C7N_COMMIT_SHA=$(git log -1 --pretty=format:"%H" | awk '{print substr($1,1,8)}')
  # Branch name
  if [ $CIRCLECI ]; then
    export C7N_BRANCH=$(echo $CIRCLE_BRANCH | tr '[A-Z]' '[a-z]' | tr '[:punct:]' '-')
  elif [ $GITLAB_CI ]; then
    export C7N_BRANCH=$CI_COMMIT_REF_SLUG
  fi
  # Default version
  if [ $CI_COMMIT_TAG ]; then
    export C7N_VERSION=$CI_COMMIT_TAG
  elif [ $CIRCLE_TAG ]; then
    export C7N_VERSION=$CIRCLE_TAG
  else
    export C7N_VERSION=$C7N_COMMIT_TIME-$C7N_BRANCH
  fi
.build_prepare: &build_prepare |
  if [[ -z "$CI_COMMIT_TAG" ]]; then
    export CI_APPLICATION_REPOSITORY=${CI_APPLICATION_REPOSITORY:-$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG}
    export CI_APPLICATION_TAG=${CI_APPLICATION_TAG:-$CI_COMMIT_SHA}
  else
    export CI_APPLICATION_REPOSITORY=${CI_APPLICATION_REPOSITORY:-$CI_REGISTRY_IMAGE}
    export CI_APPLICATION_TAG=${CI_APPLICATION_TAG:-$CI_COMMIT_TAG}
  fi
.docker_auth: &docker_auth |
  export DOCKER_CONFIG=/root/.docker
  mkdir -p $DOCKER_CONFIG
  echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n $CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD | base64)\"}}}" > $DOCKER_CONFIG/config.json
build_env:
  stage: .pre
  image: bitnami/git@sha256:8f92cadd2c82668333628d6feb3e0a4d71738fab317cbebd44e18f57845f62cf
  script:
    - *build_commit_tag
    - *build_prepare
    - echo "CI_APPLICATION_REPOSITORY=${CI_APPLICATION_REPOSITORY}" > build.env
    - echo "CI_APPLICATION_TAG=${CI_APPLICATION_TAG}" >> build.env
  artifacts:
    reports:
      dotenv: build.env
build:
  stage: build
  image: 'harbor.dev.hithinksoft.com/docker-proxy/node@sha256:e36ac0440a12839563ad011aabdd3152d6101a9d285126f86b2de5cd7f667712'
  script:
    - npm install
    - npx lerna run test:run
    - npx lerna run build
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_COMMIT_BRANCH == 'dev'
code_analysis:
  stage: test
  image: 'harbor.dev.hithinksoft.com/docker-proxy/sonarsource/sonar-scanner-cli@sha256:494ecc3b5b1ee1625bd377b3905c4284e4f0cc155cff397805a244dee1c7d575'
  script:
    - >-
      sonar-scanner -Dsonar.token=${SONAR_TOKEN} -Dsonar.pullrequest.key=${CI_MERGE_REQUEST_ID}
      -Dsonar.pullrequest.branch=${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME}
      -Dsonar.pullrequest.base=${CI_MERGE_REQUEST_TARGET_BRANCH_NAME}
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
code_quality:
  stage: test
  script:
    - sonar-scanner -Dsonar.token=${SONAR_TOKEN} -Dsonar.branch.name=${CI_COMMIT_REF_NAME}
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_COMMIT_BRANCH == 'dev'
package:
  stage: publish
  image:
    name: 'harbor.dev.hithinksoft.com/library/kaniko@sha256:8a109b04990678bbc60335f2e4fdf003b371a6334a26048bd24d8c8cc3783671'
    entrypoint: [""]
  before_script:
    - *docker_auth
  script:
    - /kaniko/executor -c $CI_PROJECT_DIR -f $CI_PROJECT_DIR/packages/system/Dockerfile -d ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} --cache=true --cache-dir=/cache/kaniko
  rules:

image使用sha值不是tag避免查询meta数据导致的401
build_env输出构建镜像需要的环境变量，使用dotenv可以解决镜像中缺少git等工具的问题，一个镜像实现一个步骤
feature分支在PR中才触发构建
PR中使用sonar decoratorchore: 🔧 sonar related config (!1) · Merge requests · Hithink Way / hithinkway · JiHu GitLab (hithinksoft.com)

持续部署H2

使用ArgoCD作为Gitops工具，Argo CD 被实现为一个 kubernetes 控制器，它持续监控正在运行的应用程序，并将当前的实时状态与所需的目标状态（如 Git 存储库中指定）进行比较。

![[Pasted image 20240108162954.png]]

ArgoCDH3

ArgoCD Application来描述部署仓库，即Ops仓库。

bash
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: HEAD
    path: guestbook
  destination:
    server: https://kubernetes.default.svc
    namespace: guestbook

其中source定义了ops仓库的位置以及目录，destination 定义了部署的目标位置，https://kubernetes.default.svc代表当前集群内(ArgoCD做在的集群)

部署仓库支持多种工具来描述部署内容，包括

Manifest
Kustomize
Helm 推荐Helm，Helm的支持模板语法，能够较好的解决资源重名的问题(Kustomize也可以，但是面对Apps of App的场景，不方便传递)

参数的传递H4

多个运行环境要要传递不同的运行时参数，ArgoCD可以通过在Application传入不同的工具的配置参数，例如 Helm

bash
source:
  helm:
    valueFiles:
    - values-production.yaml

Kustomize

bash
source:
    path: guestbook
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: master
    kustomize:
      patches:
        - target:
            kind: Deployment
            name: guestbook-ui
          patch: |-
            - op: replace
              path: /spec/template/spec/containers/0/ports/0/containerPort
              value: 443```

Parameter Overrides 还可以通过parameters来覆盖工具定义的参数

bash
argocd app set guestbook -p ingress.enabled=true
argocd app set guestbook -p ingress.hosts[0]=guestbook.myclusterurl

这其实并不符合GitOps的初衷，但是一些特殊情况下还以比较方便的，例如镜像的Tag的更新，Kustomize的prefix、suffix的设置。

镜像版本的更新H4

对于Dev分支，使用Image Updater自动覆盖Parameter

bash
metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: ajjtszh-jcjg-api=registry.choerodon.dev.hithinksoft.com/hithinksoft-ajjtszh/ajjtszh-jcjg-api
    argocd-image-updater.argoproj.io/ajjtszh-jcjg-api.pull-secret: harbor-secret
    argocd-image-updater.argoproj.io/ajjtszh-jcjg-api.update-strategy: latest
    argocd-image-updater.argoproj.io/ajjtszh-jcjg-api.allow-tags: regexp:feature-dev

对于Main分支，通过PR手动更新Ops仓库中定义的版本号。

App Of AppsH4

理想中的Gitops模型，不仅仅要定义要App本身的部署模型，还要定义他所依赖的软件环境，例如数据库、中间件等。

ArgoCD中可以通过在部署仓库再定义一个Application对象来引用其他的依赖。

bash
├── Chart.yaml
├── templates
│   ├── guestbook.yaml
│   ├── helm-dependency.yaml
│   ├── helm-guestbook.yaml
│   └── kustomize-guestbook.yaml
└── values.yaml

templates

apiVersion:
kind: Application
metadata:
  name: guestbook
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: argocd
    server: {{ .Values.spec.destination.server }}
  project: default
  source:
    path: guestbook
    repoURL: https://github.com/argoproj/argocd-example-apps
    targetRevision: HEAD

其他H4

通过Project隔离Application，Project里还可以设置统一约束、权限控制等。

bash
project: ajjtszh

资源对象的命名冲突

Helm使用Helm Release值来作为模板中的值
Kustomize使用nameSuffix、namePrefix

最佳实践H3

目录结构

bash
├── apps // 定义应用
│   ├── base 
│   │   └── webapp // 按应用分目录
│   │       ├── application.yaml   //1
│   │       └── kustomization.yaml //2 
│   └── staging // 环境配置
│       ├── kustomization.yaml 3
│       └── values-webapp.yaml 4
├── infrastucture // 定义基础环境
│   ├── keycloak // 按应用分目录
│   │   └── application.yaml 5
│   └── kustomization.yaml 6
├── clusters // 最终部署对象，引用apps以及infrastucture
│   └── staging
│       ├── kustomization.yaml 7
│       └── values-keycloak.yaml 8
└── keycloak 
    ├── Chart.yaml 9
    ├── values.yaml 
    └── templates
        └── resouce.yaml 10

1:apps/base/webapp/application.yaml

bash
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  namespace: argocd
  name: webapp
spec:
  source:
    path: chart
    repoURL: http://gitlab.choerodon.dev.hithinksoft.com/demo/webapp.git
    targetRevision: master

2:apps/base/webapp/application.yaml

bash
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - application.yaml

3:apps/staging/kustomization.yaml

bash
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base/webapp
patches:
  - path: values-webapp.yaml
    target:
      kind: Application
      name: webapp

4:apps/staging/values-webapp.yaml

bash
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: webapp=registry.choerodon.dev.hithinksoft.com/demo/webapp
    argocd-image-updater.argoproj.io/webapp.pull-secret: harbor-secret
    argocd-image-updater.argoproj.io/webapp.update-strategy: latest
    argocd-image-updater.argoproj.io/webapp.allow-tags: regexp:feature-dev
  name: webapp
spec:
  destination:
    namespace: demo
    server: https://rancher.dev.hithinksoft.com/k8s/clusters/c-m-qbpbbttz
  project: demo
  source:
    helm:
      valuesObject:
        image:
          repository: demo/webapp
          tag: master
        service:
          enabled: true
          annotations: {}
          name: web
          type: ClusterIP
          externalPort: 80
          internalPort: 8080

5:infrastucture/keycloak/application.yaml

bash
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: keycloak
  namespace: argocd
spec:
  source:
    path: keycloak
    repoURL: http://gitlab.choerodon.dev.hithinksoft.com/demo/ops.git
    targetRevision: master

6:infrastucture/kustomization.yaml

bash
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - keycloak/application.yaml

7:clusters/staging/kustomization.yaml

bash
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
nameSuffix: -staging
resources:
  - ../../apps/staging/
  - ../../infrastructure/
patches:
  - path: values-keycloak.yaml
    target:
      kind: Application
      name: keycloak

[!tldr] NOTE 不同的环境定义不同的nameSuffix，避免资源名冲突。在最终应用该文件的Application可以通过Parameter再一定义nameSuffix，再次区分不同的staging。

8:clusters/staging/values-keycloak.yaml

bash
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: keycloak
spec:
  destination:
    namespace: demo
    server: https://rancher.dev.hithinksoft.com/k8s/clusters/c-m-qbpbbttz
  project: demo
  source:
    helm:
      valuesObject: 
        keycloak:
          ingress:
            enabled: true
            hostname: keycloak.k8s.hithinksoft.com
          auth:
            adminUser: admin
            adminPassword: admin
          postgresql:
            auth:
              password: admin
          proxy: edge
          hostname-strict: false

9:keycloak/Chart.yaml

bash
apiVersion: v2
name: keycloak
descriptin: A helm chart for keycloak
version: 0.0.1
apiVersion: "1.0"
dependencies:
- name: keycloak
  repository: https://charts.bitnami.com/bitnami
  version: 18.0.0

[!tldr] NOTE 通过dependencies来定义可以省去再定义一层Application

10:keycloak/templates/resource.yaml 额外定义Chart中没有定义的资源，例如cert-manager的ClusterIssuer

针对资源对象的命名冲突的问题：Application尽量使用Helm来定义，顶级的Application Of Application(即Apps、Infrastructure、Clusters目录)使用Kustomize，方便单独抽出定义Value值的文件。

CICD Renewal

目录

持续集成H2

阶段和任务H3

示例H3

持续部署H2

ArgoCDH3

参数的传递H4

镜像版本的更新H4

App Of AppsH4

其他H4

最佳实践H3

评论

新的评论

CICD Renewal

目录

阶段和任务H3

示例H3

持续部署H2

ArgoCDH3

参数的传递H4

镜像版本的更新H4

App Of AppsH4

其他H4

最佳实践H3

评论

新的评论

.css-1hnz6hu{position:static;}.css-1hnz6hu::before{content:'';cursor:inherit;display:block;position:absolute;top:0px;left:0px;z-index:0;width:100%;height:100%;}使用coveralls统计测试覆盖率

代码测试总结

使用coveralls统计测试覆盖率