# NU.nl ## Some numbers * Biggest news site in NL * 2mln unique daily visitors * 7mln unique monthly visitors * API: ~3000 rps * Development: * 30 ppl * 5 teams * \+ Centralised teams (tooling, media platform, account, etc.)

# /me ## Tibo Beijen * Lead dev/architect at NU.nl * DevOps, Python and JS (not limited to) * UX enthusiast * Coffee addict * Loud music aficionado * Dad * Runner

# DevOps ## What, where and how we run software * AWS * Terraform + Ansible * Python + Nodejs * Applications include: * REST API * CMS * Frontend * Commenting platform * GraphQL API * Other APIs * Dashboards * Misc.

Leading up to Kubernetes

# Challenges ## Early 2018, infra as code, but also held back by... * No scaling * AWS setup lift & shift from old pre- public-cloud stack * Ansible * Nice at first but increasingly complex on long-lived infra * Capistrano 'scripted' deploys (via Webistrano web interface) * Complex. Slow deploys. Inefficient. Error-prone. * Chicken - egg * Updating infra requires changing deploy mechanism * Changing deploy mechanism requires portable build results * Zipped virtualenv folders are hardly portable * Are EC2s concistent accross stages? * Are EC2s consistent with build environment?

# 2018 ## Launching NUjij v2 * We need to set up a new application * Coral Talk * Node.js based application * Vanilla version readily available as container * We need to add customisations via plugins * Containers look like a good fit. * Light-weight * Versatile * Well-established * Kubernetes looks like a good way to run containers. * Out of early-adopters phase

# Options ## Can we run Kubernetes? * re:Invent 2017: Anouncing EKS * ... Not in europe yet * Google GKE * Tempting at a first glance. Not really an option (for us): * Networking * Integrations * Billing * Support plans * Etc. etc. * Run your own * Kops, Kubespray, kubeadm

Kops

# Kubernetes v1 ## KOPS Characteristics: * Command-line tool * Sets up HA control plane (master nodes) * Options * Directly provision * Create terraform manifests * Performs Kubernetes rolling updates * Add node, drain, remove, repeat Our implementation: * One cluster per AWS account * Kops templating (= Go templating) for kube-system components * Bash script for common tasks

# Kubernetes v1 ## Learnings * It should be possible to run more than a single cluster per AWS account * **Note\:** Has to do with our setup, not a problem in general * Still not having a 'good' way to deploy updates. * Merely bumping an image version is way too limited. * ETCd * Still a black box. Didn't learn, that's a problem. * Backup/restore? * Not a good combination with single cluster per account restriction. * Things can break * Don't accidentally have elastic-curator load a 90Gb ElasticSearch index in memory. * How a changed default can ruin your ~~day~~ evening. * Requests and limits are important * Clusters should not be pets

# 2019 ## IT landscape changes * EKS generally available * New options to setup EKS: * eksctl * Terraform modules

# 2019 ## So, Kubernetes... * Pros * Very fast feedback loop * helm template, helm diff: **seconds** * terraform plan/apply: **minutes** * Some problems only appearing at apply-time. * Similar to `pytest -k my-test` * Versatile * Init containers * Mount configmaps as files * No _need_ to build own nginx image * Easy to navigate * What's running? * Logs * Checking config * Exec into pods if needed for troubleshooting * Cons * Not 'easy', quite some choices to be made * Less easily integrated with rest of AWS * Either not (yet) possible or more complex * IAM roles (via service accounts) * Security groups (sept. 2020, via service accounts)

# 12factor ## Getting applications in a healthy state ... By means of containerising Benefits include: * Scalable * Easy to automate CI/CD * Consistent operations (deploying, observing) * Build artifact + config = running application Target platforms: * **Kubernetes** * ECS (EC2 / Fargate) * Docker swarm * Docker on EC2 * Serverless (ok, that's not containers)

Moving to EKS

# Why EKS? ## 1/2 * Goal is using Kubernetes, not operating it * Less operational responsibility * ETCd * Right-sizing a HA control plane * Cheaper * 3x m5.large (eu-west-1) * On-demand: 3x ~$78/mo = **#$236/mo** * Reserved: 3x ~$49/mo = **$147/mo** * EKS control plane * $0.10/hr = **$72/mo** * Positive signals * Introduction of IRSA * Introduction of managed nodes * Introduction of EKS Fargate * Increased adoption, no horror stories

# Why EKS? ## 2/2 * Only accessible via IAM * IAM roles easily mapped to RBAC roles * No un/pw backdoors * Reasons to _not_ run EKS could be: * Specific network requirements that are not compatible with AWS VPC CNI plugin * Multi-tenancy options EKS as well: * Multiple Clusters * Calico * Recently indroduced security groups for pods * Needing parity or integration with clusters elsewhere (multi- or hybrid cloud) * Needing to run very recent Kubernetes version not yet available on EKS * Waiting might be an option too... * Requires scale or strict requirements for ROI

EKS

# EKS Setup ## Tools * Terraform * terraform-aws-eks * Helm (v3) * Sops * Secrets using AWS KMS * Helmfile * Jenkins pipeline for helmfile * Enforces actual config to be in sync with codebase

EKS Cluster

Ingress

Multiple clusters

Flexibility

# Components ## Many options * Skipper (ingress) * Fluentd-cloudwatch (logs) * Cluster-autoscaler * Newrelic agent (logs & metrics) * Prometheus operator (metrics)

# Prometheus operator ## Metrics for clusters and applications * Operator that updates Prometheus configuration based on: * Servicemonitor * Scrape targets * Scrape interval * Supported by quite a number of helm charts * Alertrule * Promql queries, if condition is not met -> alert * Comes bundled with * Grafana + dashboards * Preconfigured alerts * Easy to create custom dashboards (as code) for applications that expose metrics

Prometheus

Dashboards

Prometheus

Alerts

# Prometheus ## Who monitors the monitor? * New Relic infrastructure agent * ELK stack * Prometheus federation

Summary & future

Comparison

Evolution

	v1	v2
Provisioning	Kops	Terraform
Kube-system components	Kops-based templating	Helm / Helmfile
Control plane	self-managed	EKS
Max. clusters per AWS account	1	Unlimited
Logging	ELK (Filebeat)	Cloudwatch (Fluentd)1
Metrics	ELK (Metricbeat) Prometheus	New Relic Prometheus operator
Ingress controller	Skipper + ip annotations	Skipper 2x (public/restricted)
Ingress AWS ALBs	kube-ingress-aws-controller + Cloudformation	Terraform provisioned

¹Becoming increasingly expensive. Search capabilities of insights is not as good as Kibana.

# Future ## Exploring * Either: * Explore managed nodes * Consider spot instances * Benefits: * Cost reduction (operations or AWS bill) * Chaos Monkey-ish: Drain, reschedule, all should have 0 effect. * EKS Fargate * Fast scaling * GitOps * Instead of: * Grant CI/CD near-total access to cluster (possibly SaaS) * Do: * Grant cluster access to code repository

?

twitter.com/TBeijen

www.linkedin.com/tibobeijen

www.tibobeijen.nl