K8S架构+best

What are Kubernetes architecture components?

The main components of a Kubernetes cluster include:

Nodes: Nodes are VMs or physical servers that host containerized applications. Each node in a cluster can run one or more application instance. There can be as few as one node, however, a typical Kubernetes cluster will have several nodes (and deployments with hundreds or more nodes are not uncommon).

Image Registry: Container images are kept in the registry and transferred to nodes by the control plane for execution in container pods.

Pods: Pods are where containerized applications run. They can include one or more containers and are the smallest unit of deployment for applications in a Kubernetes cluster.

What is Kubernetes Control Plane architecture?

A Kubernetes control plane is the control plane for a Kubernetes cluster. Its components include:

  • kube-apiserver. As its name suggests the API server exposes the Kubernetes API, which is communications central. External communications via command line interface (CLI) or other user interfaces (UI) pass to the kube-apiserver, and all control planes to node communications also goes through the API server.
  • etcd: The key value store where all data relating to the cluster is stored. etcd is highly available and consistent since all access to etcd is through the API server. Information in etcd is generally formatted in human-readable YAML (which stands for the recursive “YAML Ain’t Markup Language”).
  • kube-scheduler: When a new Pod is created, this component assigns it to a node for execution based on resource requirements, policies, and ‘affinity’ specifications regarding geolocation and interference with other workloads.
  • kube-controller-manager: Although a Kubernetes cluster has several controller functions, they are all compiled into a single binary known as kube-controller-manager.

Controller functions included in this process include:

  • Replication controller: Ensures the correct number of pods is in existence for each replicated pod running in the cluster
  • Node controller: Monitors the health of each node and notifies the cluster when nodes come online or become unresponsive
  • Endpoints controller: Connects Pods and Services to populate the Endpoints object
  • Service Account and Token controllers: Allocates API access tokens and default accounts to new namespaces in the cluster
  • cloud-controller-manager: If the cluster is partly or entirely cloud-based, the cloud controller manager links the cluster to the cloud provider’s API. Only those controls specific to the cloud provider will run. The cloud controller manager does not exist on clusters that are entirely on-premises. More than one cloud controller manager can be running in a cluster for fault tolerance or to improve overall cloud performance.

Elements of the cloud controller manager include:

  • Node controller: Determines status of a cloud-based node that has stopped responding, i.e., if it has been deleted
  • Route controller: Establishes routes in the cloud provider infrastructure
  • Service controller: Manages cloud provider’s load balancers

What is Kubernetes node architecture?

Nodes are the machines, either VMs or physical servers, where Kubernetes place Pods to execute. Node components include:

kubelet: Every node has an agent called kubelet. It ensures that the container described in PodSpecs are up and running properly. 

kube-proxy: A network proxy on each node that maintains network nodes which allows for the communication from Pods to network sessions, whether inside or outside the cluster, using operating system (OS) packet filtering if available.

container runtime: Software responsible for running the containerized applications. Although Docker is the most popular, Kubernetes supports any runtime that adheres to the Kubernetes CRI (Container Runtime Interface).

What are other Kubernetes infrastructure components?

Pods: By encapsulating one (or more) application containers, pods are the most basic execution unit of a Kubernetes application. Each Pod contains the code and storage resources required for execution and has its own IP address. Pods include configuration options as well. Typically, a Pod contains a single container or few containers that are coupled into an application or business function and that share a set of resources and data.

Deployments: A method of deploying containerized application Pods. A desired state described in a Deployment will cause controllers to change the actual state of the cluster to achieve that state in an orderly manner. Learn more about Kubernetes Deployments.

ReplicaSet: Ensures that a specified number of identical Pods are running at any given point in time.

Cluster DNS: serves DNS records needed to operate Kubernetes services.

Container Resource Monitoring: Captures and records container metrics in a central database.

What are Kubernetes architecture best practices and design principles?

Gartner’s Container Best Practices suggest a platform strategy that considers security, governance, monitoring, storage, networking, container lifecycle management and orchestration like Kubernetes.

Here are some best practices for architecting Kubernetes clusters:

  • Ensure you have updated to the latest Kubernetes version (1.18 as of this writing).
  • Invest up-front in training for developer and operations teams.
  • Establish governance enterprise-wide. Ensure tools and vendors are aligned and integrated with Kubernetes orchestration.
  • Enhance security by integrating image-scanning processes as part of your CI/CD process, scanning during build and run phases. Open-source code pulled from a Github repository should always be considered suspect.
  • Adopt role-based access control (RBAC) across the cluster. Least privilege, zero-trust models should be the standard.
  • Further secure containers by using only non-root users and making the file system read-only.
  • Avoid use of default value, since simple declaratives are less error-prone and demonstrate intent more clearly.
  • Be careful when using basic Docker Hub images, which can contain malware or be bloated with unnecessary code. Start with lean, clean code and build packages up from there. Small images build faster, are smaller on disk, and image pulls are faster as well.
  • Keep containers simple. One process per container will let the orchestrator report if that one process is healthy or not. 
  • When in doubt, crash. Kubernetes will restart a failed container, so do not restart on failure.
  • Be verbose. Descriptive labels help current developers and will be invaluable to developers to follow in their footsteps.
  • Don’t get too granular with microservices. Not every function within a logical code component need be its own microservice.
  • Automate, where it makes sense. Automating CI/CD pipeline lets you avoid manual Kubernetes deployments entirely.
  • Use livenessProbe and readinessProbe to help manage Pod lifecycles, or pods may end up being terminated while initializing or begin receiving user requests before they are ready.

Kubernetes architecture is straightforward and intuitive. The loose coupling between control plane and node allows for nearly infinite flexibility and the ability for an application to scale out virtually instantaneously to meet changing needs, to migrate users to new builds, and to support migration from on-premises to cloud-based nodes or between multiple clouds to take advantage of desired features of each cloud provider.

架构管理2:

kube-apiserver

kube-apiserver 是 Kubernetes 最重要的核心组件之一,主要提供以下的功能

  • 提供集群管理的 REST API 接口,包括认证授权、数据校验以及集群状态变更等
  • 提供其他模块之间的数据交互和通信的枢纽(其他模块通过 API Server 查询或修改数据,只有 API Server 才直接操作 etcd)

REST API

kube-apiserver 支持同时提供 https(默认监听在 6443 端口)和 http API(默认监听在 127.0.0.1 的 8080 端口),其中 http API 是非安全接口,不做任何认证授权机制,不建议生产环境启用。两个接口提供的 REST API 格式相同,参考 Kubernetes API Reference 查看所有 API 的调用格式。

img

(图片来自 OpenShift Blog

在实际使用中,通常通过 kubectl 来访问 apiserver,也可以通过 Kubernetes 各个语言的 client 库来访问 apiserver。在使用 kubectl 时,打开调试日志也可以看到每个 API 调用的格式,比如

$ kubectl --v=8 get pods

可通过 kubectl api-versionskubectl api-resources 查询 Kubernetes API 支持的 API 版本以及资源对象。

$ kubectl api-versionsadmissionregistration.k8s.io/v1beta1apiextensions.k8s.io/v1beta1apiregistration.k8s.io/v1apiregistration.k8s.io/v1beta1apps/v1apps/v1beta1apps/v1beta2authentication.k8s.io/v1authentication.k8s.io/v1beta1authorization.k8s.io/v1authorization.k8s.io/v1beta1autoscaling/v1autoscaling/v2beta1batch/v1batch/v1beta1certificates.k8s.io/v1beta1events.k8s.io/v1beta1extensions/v1beta1metrics.k8s.io/v1beta1networking.k8s.io/v1policy/v1beta1rbac.authorization.k8s.io/v1rbac.authorization.k8s.io/v1beta1scheduling.k8s.io/v1beta1storage.k8s.io/v1storage.k8s.io/v1beta1v1​$ kubectl api-resources --api-group=storage.k8s.ioNAME                SHORTNAMES   APIGROUP         NAMESPACED   KINDstorageclasses      sc           storage.k8s.io   false        StorageClassvolumeattachments                storage.k8s.io   false        VolumeAttachment

OpenAPI 和 Swagger

通过 /swaggerapi 可以查看 Swagger API,/openapi/v2 查看 OpenAPI。

开启 --enable-swagger-ui=true 后还可以通过 /swagger-ui 访问 Swagger UI。

根据 OpenAPI 也可以生成各种语言的客户端,比如可以用下面的命令生成 Go 语言的客户端:

git clone https://github.com/kubernetes-client/gen /tmp/gencat >go.settings <<EOF# Kubernetes branch nameexport KUBERNETES_BRANCH="release-1.11"​# client version for packaging and releasing.export CLIENT_VERSION="1.0"​# Name of the release packageexport PACKAGE_NAME="client-go"EOF​/tmp/gen/openapi/go.sh ./client-go ./go.settings

访问控制

Kubernetes API 的每个请求都会经过多阶段的访问控制之后才会被接受,这包括认证、授权以及准入控制(Admission Control)等。

认证

开启 TLS 时,所有的请求都需要首先认证。Kubernetes 支持多种认证机制,并支持同时开启多个认证插件(只要有一个认证通过即可)。如果认证成功,则用户的 username 会传入授权模块做进一步授权验证;而对于认证失败的请求则返回 HTTP 401。

Kubernetes 不直接管理用户

虽然 Kubernetes 认证和授权用到了 username,但 Kubernetes 并不直接管理用户,不能创建 user 对象,也不存储 username。

更多认证模块的使用方法可以参考 Kubernetes 认证插件

授权

认证之后的请求就到了授权模块。跟认证类似,Kubernetes 也支持多种授权机制,并支持同时开启多个授权插件(只要有一个验证通过即可)。如果授权成功,则用户的请求会发送到准入控制模块做进一步的请求验证;而对于授权失败的请求则返回 HTTP 403.

更多授权模块的使用方法可以参考 Kubernetes 授权插件

准入控制

准入控制(Admission Control)用来对请求做进一步的验证或添加默认参数。不同于授权和认证只关心请求的用户和操作,准入控制还处理请求的内容,并且仅对创建、更新、删除或连接(如代理)等有效,而对读操作无效。准入控制也支持同时开启多个插件,它们依次调用,只有全部插件都通过的请求才可以放过进入系统。

更多准入控制模块的使用方法可以参考 Kubernetes 准入控制

启动 apiserver 示例

kube-apiserver --feature-gates=AllAlpha=true --runtime-config=api/all=true \    --requestheader-allowed-names=front-proxy-client \    --client-ca-file=/etc/kubernetes/pki/ca.crt \    --allow-privileged=true \    --experimental-bootstrap-token-auth=true \    --storage-backend=etcd3 \    --requestheader-username-headers=X-Remote-User \    --requestheader-extra-headers-prefix=X-Remote-Extra- \    --service-account-key-file=/etc/kubernetes/pki/sa.pub \    --tls-cert-file=/etc/kubernetes/pki/apiserver.crt \    --tls-private-key-file=/etc/kubernetes/pki/apiserver.key \    --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt \    --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt \    --insecure-port=8080 \    --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds \    --requestheader-group-headers=X-Remote-Group \    --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key \    --secure-port=6443 \    --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \    --service-cluster-ip-range=10.96.0.0/12 \    --authorization-mode=RBAC \    --advertise-address=192.168.0.20 --etcd-servers=http://127.0.0.1:2379

工作原理

kube-apiserver 提供了 Kubernetes 的 REST API,实现了认证、授权、准入控制等安全校验功能,同时也负责集群状态的存储操作(通过 etcd)。

/apis/batch/v2alpha1/jobs 为例,GET 请求的处理过程如下图所示:

img

POST 请求的处理过程为:

img

(图片来自 OpenShift Blog

API 访问

有多种方式可以访问 Kubernetes 提供的 REST API:

kubectl

kubectl get --raw /api/v1/namespaceskubectl get --raw /apis/metrics.k8s.io/v1beta1/nodeskubectl get --raw /apis/metrics.k8s.io/v1beta1/pods

kubectl proxy

$ kubectl proxy --port=8080 &​$ curl http://localhost:8080/api/{  "versions": [    "v1"  ]}

curl

# In Pods with service account.$ TOKEN=$(cat /run/secrets/kubernetes.io/serviceaccount/token)$ CACERT=/run/secrets/kubernetes.io/serviceaccount/ca.crt$ curl --cacert $CACERT --header "Authorization: Bearer $TOKEN"  https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT/api{  "kind": "APIVersions",  "versions": [    "v1"  ],  "serverAddressByClientCIDRs": [    {      "clientCIDR": "0.0.0.0/0",      "serverAddress": "10.0.1.149:443"    }  ]}
# Outside of Pods.$ APISERVER=$(kubectl config view | grep server | cut -f 2- -d ":" | tr -d " ")$ TOKEN=$(kubectl describe secret $(kubectl get secrets | grep default | cut -f1 -d ' ') | grep -E '^token'| cut -f2 -d':'| tr -d '\t')$ curl $APISERVER/api --header "Authorization: Bearer $TOKEN" --insecure{  "kind": "APIVersions",  "versions": [    "v1"  ],  "serverAddressByClientCIDRs": [    {      "clientCIDR": "0.0.0.0/0",      "serverAddress": "10.0.1.149:443"    }  ]}

API 参考文档

最近 3 个稳定版本的 API 参考文档为:

发表回复

您的电子邮箱地址不会被公开。 必填项已用*标注