{"url":"https://api.github.com/gists/c213ded09dd5cd84bf87352782bca001","forks_url":"https://api.github.com/gists/c213ded09dd5cd84bf87352782bca001/forks","commits_url":"https://api.github.com/gists/c213ded09dd5cd84bf87352782bca001/commits","id":"c213ded09dd5cd84bf87352782bca001","node_id":"MDQ6R2lzdGMyMTNkZWQwOWRkNWNkODRiZjg3MzUyNzgyYmNhMDAx","git_pull_url":"https://gist.github.com/c213ded09dd5cd84bf87352782bca001.git","git_push_url":"https://gist.github.com/c213ded09dd5cd84bf87352782bca001.git","html_url":"https://gist.github.com/michaellihs/c213ded09dd5cd84bf87352782bca001","files":{"kubernetes-the-hard-way.md":{"filename":"kubernetes-the-hard-way.md","type":"text/markdown","language":"Markdown","raw_url":"https://gist.githubusercontent.com/michaellihs/c213ded09dd5cd84bf87352782bca001/raw/c4d7b45867c0255af7666bc842236d35f02ddd44/kubernetes-the-hard-way.md","size":11000,"truncated":false,"content":"Prerequisites\n=============\n\n[TOC]: # \" \"\n\n- [Glossary](#glossary)\n- [Set up `gcloud`](#set-up-gcloud)\n- [Networking](#networking)\n- [Compute](#compute)\n- [Provisioning a CA and Generating TLS Certificates](#provisioning-a-ca-and-generating-tls-certificates)\n- [Generating Kubernetes Configuration Files for Authentication](#generating-kubernetes-configuration-files-for-authentication)\n- [Encryption](#encryption)\n- [etcd cluster](#etcd-cluster)\n- [Bootstrapping Kubernetes Control Plane](#bootstrapping-kubernetes-control-plane)\n    - [Enable HTTP Health Checks](#enable-http-health-checks)\n    - [Configure RBAC for Kubelet Authorization](#configure-rbac-for-kubelet-authorization)\n    - [The Kubernetes Frontend Load Balancer](#the-kubernetes-frontend-load-balancer)\n- [Bootstrapping the Kubernetes Worker Nodes](#bootstrapping-the-kubernetes-worker-nodes)\n    - [Verification](#verification)\n- [Configure kubectl](#configure-kubectl)\n- [Provisioning Pod Network Routes](#provisioning-pod-network-routes)\n- [Deploying the DNS Cluster Add-on](#deploying-the-dns-cluster-add-on)\n\n\n\n\nGlossary\n--------\n\n* **Kubernetes Control Plane**\n* **Worker Nodes**\n* **Controller Nodes**\n* **Kubernetes Cluster**\n* **Virtual Private Cloud (VPC) Network** created for a Kubernetes cluster. Within this VPC a **subnet** is created\n\n\nSet up `gcloud`\n---------------\n\n```bash\n# download from https://cloud.google.com/sdk/docs/quickstart-mac-os-x and extract\nmv ~/Downloads/google-cloud-sdk /usr/local/bin\n/usr/local/bin/google-cloud-sdk/install.sh\nln -s /usr/local/bin/google-cloud-sdk/bin/gcloud /usr/local/bin/gcloud\n\n# now test with\ngcloud\n```\n\nCompute Engine API needs to be activated in the GCP UI (requires billing). Afterwards `gcloud init` can be run. Selected `europe-west3-a` as region.\n\n\nSet Up Kubernetes\n=================\n\n\nNetworking\n----------\n\n1. Create a VPC (Virtual Private Cloud) Network\n2. Create a subnet within this VPC Network\n3. Create Firewall rules that allow communiction inside this network\n4. Create a static IP to access the K8S API using an external load balancer. Question: is the load balancer created or just configured to use the static IP?\n\n\nCompute\n-------\n\n1. Create 3 VMs for the controller nodes\n2. Create 3 VMs for the worker nodes\n3. Generate a pair of ssh keys upon first connection to the VMs\n\n\nProvisioning a CA and Generating TLS Certificates\n-------------------------------------------------\n\n1. Generate a CA\n2. Create admin client certificates\n3. Create Kubelet (worker) client certificates\n4. Create Controller client certificate\n5. Create Kube Proxy client certificate\n6. Create Scheduler client certificate\n7. Distribute client and server certificates\n\n\nGenerating Kubernetes Configuration Files for Authentication\n------------------------------------------------------------\n\n\n1. Read Kubernetes public IP address into `KUBERNETES_PUBLIC_ADDRESS`\n2. Generate config files for workers\n\n    ``` bash\n    for instance in worker-0 worker-1 worker-2; do\n      kubectl config set-cluster kubernetes-the-hard-way \\\n        --certificate-authority=../certs/ca.pem \\\n        --embed-certs=true \\\n        --server=https://${KUBERNETES_PUBLIC_ADDRESS}:6443 \\\n        --kubeconfig=${instance}.kubeconfig\n    \n      kubectl config set-credentials system:node:${instance} \\\n        --client-certificate=../certs/${instance}.pem \\\n        --client-key=../certs/${instance}-key.pem \\\n        --embed-certs=true \\\n        --kubeconfig=${instance}.kubeconfig\n    \n      kubectl config set-context default \\\n        --cluster=kubernetes-the-hard-way \\\n        --user=system:node:${instance} \\\n        --kubeconfig=${instance}.kubeconfig\n    \n      kubectl config use-context default --kubeconfig=${instance}.kubeconfig\n    done\n    ```\n\n3. Generate config files for kube-proxy\n\n    ``` bash\n    kubectl config set-cluster kubernetes-the-hard-way \\\n        --certificate-authority=../certs/ca.pem \\\n        --embed-certs=true \\\n        --server=https://${KUBERNETES_PUBLIC_ADDRESS}:6443 \\\n        --kubeconfig=kube-proxy.kubeconfig\n    \n    kubectl config set-credentials system:kube-proxy \\\n        --client-certificate=../certs/kube-proxy.pem \\\n        --client-key=../certs/kube-proxy-key.pem \\\n        --embed-certs=true \\\n        --kubeconfig=kube-proxy.kubeconfig\n    \n    kubectl config set-context default \\\n        --cluster=kubernetes-the-hard-way \\\n        --user=system:kube-proxy \\\n        --kubeconfig=kube-proxy.kubeconfig\n    \n    kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig\n    ```\n\n4. Generate the kube-controller-manager Kubernetes Configuration File\n\n    ``` bash\n    kubectl config set-cluster kubernetes-the-hard-way \\\n    --certificate-authority=../certs/ca.pem \\\n    --embed-certs=true \\\n    --server=https://127.0.0.1:6443 \\\n    --kubeconfig=kube-controller-manager.kubeconfig\n\n    kubectl config set-credentials system:kube-controller-manager \\\n    --client-certificate=../certs/kube-controller-manager.pem \\\n    --client-key=../certs/kube-controller-manager-key.pem \\\n    --embed-certs=true \\\n    --kubeconfig=kube-controller-manager.kubeconfig\n\n    kubectl config set-context default \\\n    --cluster=kubernetes-the-hard-way \\\n    --user=system:kube-controller-manager \\\n    --kubeconfig=kube-controller-manager.kubeconfig\n\n    kubectl config use-context default --kubeconfig=kube-controller-manager.kubeconfig\n    ```\n\n5. Generate the kube-scheduler Kubernetes Configuration File\n\n    ``` bash\n      kubectl config set-cluster kubernetes-the-hard-way \\\n        --certificate-authority=../certs/ca.pem \\\n        --embed-certs=true \\\n        --server=https://127.0.0.1:6443 \\\n        --kubeconfig=kube-scheduler.kubeconfig\n    \n      kubectl config set-credentials system:kube-scheduler \\\n        --client-certificate=../certs/kube-scheduler.pem \\\n        --client-key=../certs/kube-scheduler-key.pem \\\n        --embed-certs=true \\\n        --kubeconfig=kube-scheduler.kubeconfig\n    \n      kubectl config set-context default \\\n        --cluster=kubernetes-the-hard-way \\\n        --user=system:kube-scheduler \\\n        --kubeconfig=kube-scheduler.kubeconfig\n    \n      kubectl config use-context default --kubeconfig=kube-scheduler.kubeconfig\n    ```\n\n6. Generate the admin Kubernetes Configuration File\n\n    ``` bash\n      kubectl config set-cluster kubernetes-the-hard-way \\\n        --certificate-authority=../certs/ca.pem \\\n        --embed-certs=true \\\n        --server=https://127.0.0.1:6443 \\\n        --kubeconfig=admin.kubeconfig\n    \n      kubectl config set-credentials admin \\\n        --client-certificate=../certs/admin.pem \\\n        --client-key=../certs/admin-key.pem \\\n        --embed-certs=true \\\n        --kubeconfig=admin.kubeconfig\n    \n      kubectl config set-context default \\\n        --cluster=kubernetes-the-hard-way \\\n        --user=admin \\\n        --kubeconfig=admin.kubeconfig\n    \n      kubectl config use-context default --kubeconfig=admin.kubeconfig\n    ```\n\n7. Distribute the Kubernetes Configuration Files\n\n\nEncryption\n----------\n\n1. Generate an encryption key\n2. Create an encryption configuration\n3. Distribute encryption config to controller nodes\n\n\netcd cluster\n------------\n\nCommands must be run on all controller nodes, therefore use tmux with synchronized panes\n\n* Create new panes with `ctrl + b` then `\"`\n* Enable synchronize-panes: `ctrl + b` then `shift :`. Then type `set synchronize-panes on` at the prompt\n* To disable synchronization: `set synchronize-panes off`\n\nInstall etcd via\n\n1. Download the and install official etcd binaries\n2. Retrieve internal IP of VM\n3. Create etcd system.d config file in `/etc/systemd/system/etcd.service`\n4. Start the etcd server\n5. Verify etcd configuration and cluster members\n\n\nBootstrapping Kubernetes Control Plane\n--------------------------------------\n\nOn all controllers:\n\n1. Create Kubernetes configuration directory\n2. Download the Kubernetes Controller Binaries\n3. Install the Kubernetes Controller Binaries\n4. Configure the Kubernetes API server\n5. Create the `kube-apiserver.service` systemd unit file\n6. Configure the Kubernetes Controller Manager\n7. Create the `kube-controller-manager.service` systemd unit file\n8. Configure the Kubernetes Scheduler\n9. Create the `kube-scheduler.yaml` configuration file\n10. Create the kube-scheduler.service systemd unit file\n11. Start the Controller Services\n\n\n### Enable HTTP Health Checks\n\nA Google Network Load Balancer will be used to distribute traffic across the three API servers and allow each API server to terminate TLS connections and validate client certificates. The network load balancer only supports HTTP health checks which means the HTTPS endpoint exposed by the API server cannot be used. As a workaround the nginx webserver can be used to proxy HTTP health checks. In this section nginx will be installed and configured to accept HTTP health checks on port `80` and proxy the connections to the API server on https://127.0.0.1:6443/healthz`.\n\n1. Install nginx on all controllers\n2. Configure nginx to handle basic health checks\n3. Enable and restart nginx\n4. Verify health via `kubectl` and the nginx healthz endpoint - before you can use `kubectl` you have to run\n\n   ``` bash\n   kubectl create clusterrolebinding root-cluster-admin-binding --clusterrole=cluster-admin --user=admin\n   ```\n\n### Configure RBAC for Kubelet Authorization\n\n1. ssh to controller-0\n2. Create the `system:kube-apiserver-to-kubelet` ClusterRole\n3. Bind the `system:kube-apiserver-to-kubelet` ClusterRole to the `kubernetes` user\n\n\n### The Kubernetes Frontend Load Balancer\n\nYou will provision an external load balancer to front the Kubernetes API Servers. The `kubernetes-the-hard-way` static IP address will be attached to the resulting load balancer.\n\n> Run the following steps from the machine where you created the compute instances, not from the compute instances themselves\n\n1. Provision a Network Load Balancer\n2. Verify connection from your workstation (make sure to cd into the directory with the certificates)\n\n\nBootstrapping the Kubernetes Worker Nodes\n-----------------------------------------\n\nOn each worker node, run\n\n1. apt-get update\n2. apt-get -y install socat conntrack ipset\n3. Install worker binaries\n4. Create installation directories\n5. Install the worker binaries\n6. Configure CNI networking\n7. Configure containerd\n8. Configure the Kubelet\n9. Configure the Kubernetes Proxy\n10. Start the worker services\n\n\n### Verification\n\nRun the verification command from your workstation\n\n\nConfigure kubectl\n-----------------\n\nFrom within the directory that contains the certificates:\n\n1. Generate the admin kubernetes configuration file\n2. verify configuration\n\n\nProvisioning Pod Network Routes\n-------------------------------\n\nFrom your workstation, create the routes that enable pods to communicate.\n\n\nDeploying the DNS Cluster Add-on\n--------------------------------\n\n1. Deploy coredns cluster add-on\n2. \n\nReferences\n==========\n\n* https://github.com/kelseyhightower/kubernetes-the-hard-way","encoding":"utf-8"}},"public":true,"created_at":"2019-01-23T14:59:27Z","updated_at":"2019-07-02T15:15:31Z","description":"Kubernetes the Hard Way","comments":0,"user":null,"comments_enabled":true,"comments_url":"https://api.github.com/gists/c213ded09dd5cd84bf87352782bca001/comments","owner":{"login":"michaellihs","id":575011,"node_id":"MDQ6VXNlcjU3NTAxMQ==","avatar_url":"https://avatars.githubusercontent.com/u/575011?v=4","gravatar_id":"","url":"https://api.github.com/users/michaellihs","html_url":"https://github.com/michaellihs","followers_url":"https://api.github.com/users/michaellihs/followers","following_url":"https://api.github.com/users/michaellihs/following{/other_user}","gists_url":"https://api.github.com/users/michaellihs/gists{/gist_id}","starred_url":"https://api.github.com/users/michaellihs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaellihs/subscriptions","organizations_url":"https://api.github.com/users/michaellihs/orgs","repos_url":"https://api.github.com/users/michaellihs/repos","events_url":"https://api.github.com/users/michaellihs/events{/privacy}","received_events_url":"https://api.github.com/users/michaellihs/received_events","type":"User","user_view_type":"public","site_admin":false},"forks":[],"history":[{"user":{"login":"michaellihs","id":575011,"node_id":"MDQ6VXNlcjU3NTAxMQ==","avatar_url":"https://avatars.githubusercontent.com/u/575011?v=4","gravatar_id":"","url":"https://api.github.com/users/michaellihs","html_url":"https://github.com/michaellihs","followers_url":"https://api.github.com/users/michaellihs/followers","following_url":"https://api.github.com/users/michaellihs/following{/other_user}","gists_url":"https://api.github.com/users/michaellihs/gists{/gist_id}","starred_url":"https://api.github.com/users/michaellihs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaellihs/subscriptions","organizations_url":"https://api.github.com/users/michaellihs/orgs","repos_url":"https://api.github.com/users/michaellihs/repos","events_url":"https://api.github.com/users/michaellihs/events{/privacy}","received_events_url":"https://api.github.com/users/michaellihs/received_events","type":"User","user_view_type":"public","site_admin":false},"version":"011c6e72a80b7dd4fc4366165207f1b5f9390e4a","committed_at":"2019-01-25T16:00:18Z","change_status":{"total":306,"additions":305,"deletions":1},"url":"https://api.github.com/gists/c213ded09dd5cd84bf87352782bca001/011c6e72a80b7dd4fc4366165207f1b5f9390e4a"},{"user":{"login":"michaellihs","id":575011,"node_id":"MDQ6VXNlcjU3NTAxMQ==","avatar_url":"https://avatars.githubusercontent.com/u/575011?v=4","gravatar_id":"","url":"https://api.github.com/users/michaellihs","html_url":"https://github.com/michaellihs","followers_url":"https://api.github.com/users/michaellihs/followers","following_url":"https://api.github.com/users/michaellihs/following{/other_user}","gists_url":"https://api.github.com/users/michaellihs/gists{/gist_id}","starred_url":"https://api.github.com/users/michaellihs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaellihs/subscriptions","organizations_url":"https://api.github.com/users/michaellihs/orgs","repos_url":"https://api.github.com/users/michaellihs/repos","events_url":"https://api.github.com/users/michaellihs/events{/privacy}","received_events_url":"https://api.github.com/users/michaellihs/received_events","type":"User","user_view_type":"public","site_admin":false},"version":"df7a266e904f1036c0fc00bc05d605b590a37886","committed_at":"2019-01-23T15:30:01Z","change_status":{"total":5,"additions":3,"deletions":2},"url":"https://api.github.com/gists/c213ded09dd5cd84bf87352782bca001/df7a266e904f1036c0fc00bc05d605b590a37886"},{"user":{"login":"michaellihs","id":575011,"node_id":"MDQ6VXNlcjU3NTAxMQ==","avatar_url":"https://avatars.githubusercontent.com/u/575011?v=4","gravatar_id":"","url":"https://api.github.com/users/michaellihs","html_url":"https://github.com/michaellihs","followers_url":"https://api.github.com/users/michaellihs/followers","following_url":"https://api.github.com/users/michaellihs/following{/other_user}","gists_url":"https://api.github.com/users/michaellihs/gists{/gist_id}","starred_url":"https://api.github.com/users/michaellihs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaellihs/subscriptions","organizations_url":"https://api.github.com/users/michaellihs/orgs","repos_url":"https://api.github.com/users/michaellihs/repos","events_url":"https://api.github.com/users/michaellihs/events{/privacy}","received_events_url":"https://api.github.com/users/michaellihs/received_events","type":"User","user_view_type":"public","site_admin":false},"version":"ed7fd10052b955822d7f555a3f8b89219d6a8f04","committed_at":"2019-01-23T15:05:22Z","change_status":{"total":1,"additions":1,"deletions":0},"url":"https://api.github.com/gists/c213ded09dd5cd84bf87352782bca001/ed7fd10052b955822d7f555a3f8b89219d6a8f04"},{"user":{"login":"michaellihs","id":575011,"node_id":"MDQ6VXNlcjU3NTAxMQ==","avatar_url":"https://avatars.githubusercontent.com/u/575011?v=4","gravatar_id":"","url":"https://api.github.com/users/michaellihs","html_url":"https://github.com/michaellihs","followers_url":"https://api.github.com/users/michaellihs/followers","following_url":"https://api.github.com/users/michaellihs/following{/other_user}","gists_url":"https://api.github.com/users/michaellihs/gists{/gist_id}","starred_url":"https://api.github.com/users/michaellihs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaellihs/subscriptions","organizations_url":"https://api.github.com/users/michaellihs/orgs","repos_url":"https://api.github.com/users/michaellihs/repos","events_url":"https://api.github.com/users/michaellihs/events{/privacy}","received_events_url":"https://api.github.com/users/michaellihs/received_events","type":"User","user_view_type":"public","site_admin":false},"version":"49a8c918a528f531e4afa7bfab18d2384b88320f","committed_at":"2019-01-23T15:05:06Z","change_status":{"total":26,"additions":25,"deletions":1},"url":"https://api.github.com/gists/c213ded09dd5cd84bf87352782bca001/49a8c918a528f531e4afa7bfab18d2384b88320f"},{"user":{"login":"michaellihs","id":575011,"node_id":"MDQ6VXNlcjU3NTAxMQ==","avatar_url":"https://avatars.githubusercontent.com/u/575011?v=4","gravatar_id":"","url":"https://api.github.com/users/michaellihs","html_url":"https://github.com/michaellihs","followers_url":"https://api.github.com/users/michaellihs/followers","following_url":"https://api.github.com/users/michaellihs/following{/other_user}","gists_url":"https://api.github.com/users/michaellihs/gists{/gist_id}","starred_url":"https://api.github.com/users/michaellihs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaellihs/subscriptions","organizations_url":"https://api.github.com/users/michaellihs/orgs","repos_url":"https://api.github.com/users/michaellihs/repos","events_url":"https://api.github.com/users/michaellihs/events{/privacy}","received_events_url":"https://api.github.com/users/michaellihs/received_events","type":"User","user_view_type":"public","site_admin":false},"version":"38427951ea58c56458f5e2f710ec5302ea615b84","committed_at":"2019-01-23T14:59:26Z","change_status":{"total":1,"additions":1,"deletions":0},"url":"https://api.github.com/gists/c213ded09dd5cd84bf87352782bca001/38427951ea58c56458f5e2f710ec5302ea615b84"}],"truncated":false}