GitLab upgraded to 13.6.1-ee.0 - changelog: https://gitlab.com/gitlab-org/gitlab/blob/master/CHANGELOG-EE.md

README.md 12.1 KB
Newer Older
Maxime Guyot's avatar
Maxime Guyot committed
1
# Deploy a Production Ready Kubernetes Cluster
Smaine Kahlouch's avatar
Smaine Kahlouch committed
2

Maxime Guyot's avatar
Maxime Guyot committed
3
![Kubernetes Logo](https://raw.githubusercontent.com/kubernetes-sigs/kubespray/master/docs/img/kubernetes-logo.png)
Smaine Kahlouch's avatar
Smaine Kahlouch committed
4

5
If you have questions, check the documentation at [kubespray.io](https://kubespray.io) and join us on the [kubernetes slack](https://kubernetes.slack.com), channel **\#kubespray**.
mgsergio's avatar
mgsergio committed
6
You can get your invite [here](http://slack.k8s.io/)
Smaine Kahlouch's avatar
Smaine Kahlouch committed
7

8
- Can be deployed on **[AWS](docs/aws.md), GCE, [Azure](docs/azure.md), [OpenStack](docs/openstack.md), [vSphere](docs/vsphere.md), [Packet](docs/packet.md) (bare metal), Oracle Cloud Infrastructure (Experimental), or Baremetal**
Maxime Guyot's avatar
Maxime Guyot committed
9 10 11 12
- **Highly available** cluster
- **Composable** (Choice of the network plugin for instance)
- Supports most popular **Linux distributions**
- **Continuous integration tests**
Smaine Kahlouch's avatar
Smaine Kahlouch committed
13

Maxime Guyot's avatar
Maxime Guyot committed
14
## Quick Start
Smaine Kahlouch's avatar
Smaine Kahlouch committed
15 16 17

To deploy the cluster you can use :

18
### Ansible
19

20 21
#### Usage

Maxime Guyot's avatar
Maxime Guyot committed
22 23
```ShellSession
# Install dependencies from ``requirements.txt``
24
sudo pip3 install -r requirements.txt
25

Maxime Guyot's avatar
Maxime Guyot committed
26 27
# Copy ``inventory/sample`` as ``inventory/mycluster``
cp -rfp inventory/sample inventory/mycluster
28

Maxime Guyot's avatar
Maxime Guyot committed
29 30
# Update Ansible inventory file with inventory builder
declare -a IPS=(10.10.1.3 10.10.1.4 10.10.1.5)
31
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
32

Maxime Guyot's avatar
Maxime Guyot committed
33 34 35
# Review and change parameters under ``inventory/mycluster/group_vars``
cat inventory/mycluster/group_vars/all/all.yml
cat inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml
36

Maxime Guyot's avatar
Maxime Guyot committed
37 38 39 40
# Deploy Kubespray with Ansible Playbook - run the playbook as root
# The option `--become` is required, as for example writing SSL keys in /etc/,
# installing packages and interacting with various systemd daemons.
# Without --become the playbook will fail to run!
41
ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=root cluster.yml
Maxime Guyot's avatar
Maxime Guyot committed
42
```
43

44 45
Note: When Ansible is already installed via system packages on the control machine, other python packages installed via `sudo pip install -r requirements.txt` will go to a different directory tree (e.g. `/usr/local/lib/python2.7/dist-packages` on Ubuntu) from Ansible's (e.g. `/usr/lib/python2.7/dist-packages/ansible` still on Ubuntu).
As a consequence, `ansible-playbook` command will fail with:
Maxime Guyot's avatar
Maxime Guyot committed
46 47

```raw
48 49
ERROR! no action detected in task. This often indicates a misspelled module name, or incorrect module path.
```
Maxime Guyot's avatar
Maxime Guyot committed
50

51 52 53 54 55
probably pointing on a task depending on a module present in requirements.txt (i.e. "unseal vault").

One way of solving this would be to uninstall the Ansible package and then, to install it via pip but it is not always possible.
A workaround consists of setting `ANSIBLE_LIBRARY` and `ANSIBLE_MODULE_UTILS` environment variables respectively to the `ansible/modules` and `ansible/module_utils` subdirectories of pip packages installation location, which can be found in the Location field of the output of `pip show [package]` before executing `ansible-playbook`.

56 57
### Vagrant

58 59 60
For Vagrant we need to install python dependencies for provisioning tasks.
Check if Python and pip are installed:

Maxime Guyot's avatar
Maxime Guyot committed
61 62 63
```ShellSession
python -V && pip -V
```
64 65 66 67

If this returns the version of the software, you're good to go. If not, download and install Python from here <https://www.python.org/downloads/source/>
Install the necessary requirements

Maxime Guyot's avatar
Maxime Guyot committed
68 69 70 71 72 73 74 75 76 77
```ShellSession
sudo pip install -r requirements.txt
vagrant up
```

## Documents

- [Requirements](#requirements)
- [Kubespray vs ...](docs/comparisons.md)
- [Getting started](docs/getting-started.md)
78
- [Setting up your first cluster](docs/setting-up-your-first-cluster.md)
Maxime Guyot's avatar
Maxime Guyot committed
79 80 81 82 83 84 85
- [Ansible inventory and tags](docs/ansible.md)
- [Integration with existing ansible repo](docs/integration.md)
- [Deployment data variables](docs/vars.md)
- [DNS stack](docs/dns-stack.md)
- [HA mode](docs/ha-mode.md)
- [Network plugins](#network-plugins)
- [Vagrant install](docs/vagrant.md)
86
- [Flatcar Container Linux bootstrap](docs/flatcar.md)
spaced's avatar
spaced committed
87
- [Fedora CoreOS bootstrap](docs/fcos.md)
Maxime Guyot's avatar
Maxime Guyot committed
88 89 90 91 92 93 94 95 96 97
- [Debian Jessie setup](docs/debian.md)
- [openSUSE setup](docs/opensuse.md)
- [Downloaded artifacts](docs/downloads.md)
- [Cloud providers](docs/cloud.md)
- [OpenStack](docs/openstack.md)
- [AWS](docs/aws.md)
- [Azure](docs/azure.md)
- [vSphere](docs/vsphere.md)
- [Packet Host](docs/packet.md)
- [Large deployments](docs/large-deployments.md)
98
- [Adding/replacing a node](docs/nodes.md)
Maxime Guyot's avatar
Maxime Guyot committed
99
- [Upgrades basics](docs/upgrades.md)
100
- [Air-Gap installation](docs/offline-environment.md)
Maxime Guyot's avatar
Maxime Guyot committed
101 102 103 104
- [Roadmap](docs/roadmap.md)

## Supported Linux Distributions

105
- **Flatcar Container Linux by Kinvolk**
Maxime Guyot's avatar
Maxime Guyot committed
106
- **Debian** Buster, Jessie, Stretch, Wheezy
107
- **Ubuntu** 16.04, 18.04, 20.04
108
- **CentOS/RHEL** 7, 8 (experimental: see [centos 8 notes](docs/centos8.md))
109
- **Fedora** 31, 32
110
- **Fedora CoreOS** (experimental: see [fcos Note](docs/fcos.md))
Maxime Guyot's avatar
Maxime Guyot committed
111
- **openSUSE** Leap 42.3/Tumbleweed
112
- **Oracle Linux** 7, 8 (experimental: [centos 8 notes](docs/centos8.md) apply)
Smaine Kahlouch's avatar
Smaine Kahlouch committed
113

Bogdan Dobrelya's avatar
Bogdan Dobrelya committed
114 115
Note: Upstart/SysV init based OS types are not supported.

Maxime Guyot's avatar
Maxime Guyot committed
116 117 118
## Supported Components

- Core
119
  - [kubernetes](https://github.com/kubernetes/kubernetes) v1.19.3
120
  - [etcd](https://github.com/coreos/etcd) v3.4.3
121
  - [docker](https://www.docker.com/) v19.03 (see note)
122
  - [containerd](https://containerd.io/) v1.3.7
Hans Feldt's avatar
Hans Feldt committed
123
  - [cri-o](http://cri-o.io/) v1.19 (experimental: see [CRI-O Note](docs/cri-o.md). Only on fedora, ubuntu and centos based OS)
Maxime Guyot's avatar
Maxime Guyot committed
124
- Network Plugin
125
  - [cni-plugins](https://github.com/containernetworking/plugins) v0.8.7
126
  - [calico](https://github.com/projectcalico/calico) v3.16.5
Maxime Guyot's avatar
Maxime Guyot committed
127
  - [canal](https://github.com/projectcalico/canal) (given calico/flannel versions)
128
  - [cilium](https://github.com/cilium/cilium) v1.8.5
Maxime Guyot's avatar
Maxime Guyot committed
129
  - [contiv](https://github.com/contiv/install) v1.2.1
130
  - [flanneld](https://github.com/coreos/flannel) v0.13.0
131
  - [kube-ovn](https://github.com/alauda/kube-ovn) v1.3.0
132
  - [kube-router](https://github.com/cloudnativelabs/kube-router) v1.1.0
133
  - [multus](https://github.com/intel/multus-cni) v3.6.0
134
  - [ovn4nfv](https://github.com/opnfv/ovn4nfv-k8s-plugin) v1.1.0
135
  - [weave](https://github.com/weaveworks/weave) v2.7.0
Maxime Guyot's avatar
Maxime Guyot committed
136
- Application
Alvaro's avatar
Alvaro committed
137
  - [ambassador](https://github.com/datawire/ambassador): v1.5
Maxime Guyot's avatar
Maxime Guyot committed
138 139
  - [cephfs-provisioner](https://github.com/kubernetes-incubator/external-storage) v2.1.0-k8s1.11
  - [rbd-provisioner](https://github.com/kubernetes-incubator/external-storage) v2.1.1-k8s1.11
140
  - [cert-manager](https://github.com/jetstack/cert-manager) v0.16.1
141
  - [coredns](https://github.com/coredns/coredns) v1.7.0
142
  - [ingress-nginx](https://github.com/kubernetes/ingress-nginx) v0.40.2
Smaine Kahlouch's avatar
Smaine Kahlouch committed
143

144
Note: The list of validated [docker versions](https://kubernetes.io/docs/setup/production-environment/container-runtimes/#docker) is 1.13.1, 17.03, 17.06, 17.09, 18.06, 18.09 and 19.03. The recommended docker version is 19.03. The kubelet might break on docker's non-standard version numbering (it no longer uses semantic versioning). To ensure auto-updates don't break your cluster look into e.g. yum versionlock plugin or apt pin).
145

Maxime Guyot's avatar
Maxime Guyot committed
146 147
## Requirements

148
- **Minimum required version of Kubernetes is v1.17**
149
- **Ansible v2.9+, Jinja 2.11+ and python-netaddr is installed on the machine that will run Ansible commands**
150
- The target servers must have **access to the Internet** in order to pull docker images. Otherwise, additional configuration is required (See [Offline Environment](docs/offline-environment.md))
Maxime Guyot's avatar
Maxime Guyot committed
151 152 153
- The target servers are configured to allow **IPv4 forwarding**.
- **Your ssh key must be copied** to all the servers part of your inventory.
- The **firewalls are not managed**, you'll need to implement your own rules the way you used to.
154
    in order to avoid any issue during deployment you should disable your firewall.
Maxime Guyot's avatar
Maxime Guyot committed
155
- If kubespray is ran from non-root user account, correct privilege escalation method
156 157
    should be configured in the target servers. Then the `ansible_become` flag
    or command parameters `--become or -b` should be specified.
Smaine Kahlouch's avatar
Smaine Kahlouch committed
158

159
Hardware:
160
These limits are safe guarded by Kubespray. Actual requirements for your workload can differ. For a sizing guide go to the [Building Large Clusters](https://kubernetes.io/docs/setup/cluster-large/#size-of-master-and-master-components) guide.
161

Maxime Guyot's avatar
Maxime Guyot committed
162 163 164 165
- Master
  - Memory: 1500 MB
- Node
  - Memory: 1024 MB
166

Maxime Guyot's avatar
Maxime Guyot committed
167
## Network Plugins
168

169
You can choose between 10 network plugins. (default: `calico`, except Vagrant uses `flannel`)
Smaine Kahlouch's avatar
Smaine Kahlouch committed
170

Maxime Guyot's avatar
Maxime Guyot committed
171
- [flannel](docs/flannel.md): gre/vxlan (layer 2) networking.
Smaine Kahlouch's avatar
Smaine Kahlouch committed
172

173 174 175 176
- [Calico](https://docs.projectcalico.org/latest/introduction/) is a networking and network policy provider. Calico supports a flexible set of networking options
    designed to give you the most efficient networking across a range of situations, including non-overlay
    and overlay networks, with or without BGP. Calico uses the same engine to enforce network policy for hosts,
    pods, and (if using Istio and Envoy) applications at the service mesh layer.
Smaine Kahlouch's avatar
Smaine Kahlouch committed
177

Maxime Guyot's avatar
Maxime Guyot committed
178
- [canal](https://github.com/projectcalico/canal): a composition of calico and flannel plugins.
Smaine Kahlouch's avatar
Smaine Kahlouch committed
179

Maxime Guyot's avatar
Maxime Guyot committed
180
- [cilium](http://docs.cilium.io/en/latest/): layer 3/4 networking (as well as layer 7 to protect and secure application protocols), supports dynamic insertion of BPF bytecode into the Linux kernel to implement security services, networking and visibility logic.
melkosoft's avatar
melkosoft committed
181

Maxime Guyot's avatar
Maxime Guyot committed
182
- [contiv](docs/contiv.md): supports vlan, vxlan, bgp and Cisco SDN networking. This plugin is able to
183
    apply firewall policies, segregate containers in multiple network and bridging pods onto physical networks.
unclejack's avatar
unclejack committed
184

185 186
- [ovn4nfv](docs/ovn4nfv.md): [ovn4nfv-k8s-plugins](https://github.com/opnfv/ovn4nfv-k8s-plugin) is the network controller, OVS agent and CNI server to offer basic SFC and OVN overlay networking.

Maxime Guyot's avatar
Maxime Guyot committed
187
- [weave](docs/weave.md): Weave is a lightweight container overlay network that doesn't require an external K/V database cluster.
188
    (Please refer to `weave` [troubleshooting documentation](https://www.weave.works/docs/net/latest/troubleshooting/)).
Smaine Kahlouch's avatar
Smaine Kahlouch committed
189

Maxime Guyot's avatar
Maxime Guyot committed
190
- [kube-ovn](docs/kube-ovn.md): Kube-OVN integrates the OVN-based Network Virtualization with Kubernetes. It offers an advanced Container Network Fabric for Enterprises.
191

Maxime Guyot's avatar
Maxime Guyot committed
192
- [kube-router](docs/kube-router.md): Kube-router is a L3 CNI for Kubernetes networking aiming to provide operational
193 194 195 196
    simplicity and high performance: it uses IPVS to provide Kube Services Proxy (if setup to replace kube-proxy),
    iptables for network policies, and BGP for ods L3 networking (with optionally BGP peering with out-of-cluster BGP peers).
    It can also optionally advertise routes to Kubernetes cluster Pods CIDRs, ClusterIPs, ExternalIPs and LoadBalancerIPs.

Maxime Guyot's avatar
Maxime Guyot committed
197
- [macvlan](docs/macvlan.md): Macvlan is a Linux network driver. Pods have their own unique Mac and Ip address, connected directly the physical (layer 2) network.
198

Maxime Guyot's avatar
Maxime Guyot committed
199
- [multus](docs/multus.md): Multus is a meta CNI plugin that provides multiple network interface support to pods. For each interface Multus delegates CNI calls to secondary CNI plugins such as Calico, macvlan, etc.
200

Bogdan Dobrelya's avatar
Bogdan Dobrelya committed
201 202 203
The choice is defined with the variable `kube_network_plugin`. There is also an
option to leverage built-in cloud provider networking instead.
See also [Network checker](docs/netcheck.md).
Smaine Kahlouch's avatar
Smaine Kahlouch committed
204

Alvaro's avatar
Alvaro committed
205 206 207 208 209 210
## Ingress Plugins

- [ambassador](docs/ambassador.md): the Ambassador Ingress Controller and API gateway.

- [nginx](https://kubernetes.github.io/ingress-nginx): the NGINX Ingress Controller.

Maxime Guyot's avatar
Maxime Guyot committed
211
## Community docs and resources
212

Maxime Guyot's avatar
Maxime Guyot committed
213 214 215
- [kubernetes.io/docs/setup/production-environment/tools/kubespray/](https://kubernetes.io/docs/setup/production-environment/tools/kubespray/)
- [kubespray, monitoring and logging](https://github.com/gregbkr/kubernetes-kargo-logging-monitoring) by @gregbkr
- [Deploy Kubernetes w/ Ansible & Terraform](https://rsmitty.github.io/Terraform-Ansible-Kubernetes/) by @rsmitty
roc's avatar
roc committed
216
- [Deploy a Kubernetes Cluster with Kubespray (video)](https://www.youtube.com/watch?v=CJ5G4GpqDy0)
217

Maxime Guyot's avatar
Maxime Guyot committed
218
## Tools and projects on top of Kubespray
219

Maxime Guyot's avatar
Maxime Guyot committed
220 221
- [Digital Rebar Provision](https://github.com/digitalrebar/provision/blob/v4/doc/integrations/ansible.rst)
- [Terraform Contrib](https://github.com/kubernetes-sigs/kubespray/tree/master/contrib/terraform)
222

Maxime Guyot's avatar
Maxime Guyot committed
223
## CI Tests
Smaine Kahlouch's avatar
Smaine Kahlouch committed
224

225 226 227
[![Build graphs](https://gitlab.com/kargo-ci/kubernetes-sigs-kubespray/badges/master/pipeline.svg)](https://gitlab.com/kargo-ci/kubernetes-sigs-kubespray/pipelines)

CI/end-to-end tests sponsored by: [CNCF](https://cncf.io), [Packet](https://www.packet.com/), [OVHcloud](https://www.ovhcloud.com/), [ELASTX](https://elastx.se/).
Smaine Kahlouch's avatar
Smaine Kahlouch committed
228

Bogdan Dobrelya's avatar
Bogdan Dobrelya committed
229
See the [test matrix](docs/test_cases.md) for details.