NU.nl

Some numbers

  • Biggest news site in NL

  • 2mln unique daily visitors

  • 7mln unique monthly visitors

  • API: ~3000 rps

  • Development:

    • 30 ppl
    • 5 teams
    • + Centralised teams (tooling, media platform, account, etc.)

/me

Tibo Beijen

  • Lead dev/architect at NU.nl
  • DevOps, Python and JS (not limited to)
  • UX enthusiast
  • Coffee addict
  • Loud music aficionado
  • Dad
  • Runner

DevOps

What, where and how we run software

  • AWS
  • Terraform + Ansible
  • Python + Nodejs
  • Applications include:
    • REST API
    • CMS
    • Frontend
    • Commenting platform
    • GraphQL API
    • Other APIs
    • Dashboards
    • Misc.
Leading up to Kubernetes

Challenges

Early 2018, infra as code, but also held back by...

  • No scaling

    • AWS setup lift & shift from old pre- public-cloud stack

  • Ansible

    • Nice at first but increasingly complex on long-lived infra

  • Capistrano 'scripted' deploys (via Webistrano web interface)

    • Complex. Slow deploys. Inefficient. Error-prone.

  • Chicken - egg

    • Updating infra requires changing deploy mechanism

    • Changing deploy mechanism requires portable build results

    • Zipped virtualenv folders are hardly portable

      • Are EC2s concistent accross stages?
      • Are EC2s consistent with build environment?

2018

Launching NUjij v2

  • We need to set up a new application

    • Coral Talk
    • Node.js based application
    • Vanilla version readily available as container
    • We need to add customisations via plugins

  • Containers look like a good fit.

    • Light-weight
    • Versatile
    • Well-established

  • Kubernetes looks like a good way to run containers.

    • Out of early-adopters phase

Options

Can we run Kubernetes?

  • re:Invent 2017: Anouncing EKS

    • ... Not in europe yet

  • Google GKE

    • Tempting at a first glance. Not really an option (for us):

      • Networking
      • Integrations
      • Billing
      • Support plans
      • Etc. etc.

  • Run your own

    • Kops, Kubespray, kubeadm
Kops

Kubernetes v1

KOPS

Characteristics:

  • Command-line tool
  • Sets up HA control plane (master nodes)
  • Options
    • Directly provision
    • Create terraform manifests
  • Performs Kubernetes rolling updates
    • Add node, drain, remove, repeat

Our implementation:

  • One cluster per AWS account
  • Kops templating (= Go templating) for kube-system components
  • Bash script for common tasks

Kubernetes v1

Learnings

  • It should be possible to run more than a single cluster per AWS account

    • Note: Has to do with our setup, not a problem in general

  • Still not having a 'good' way to deploy updates.

    • Merely bumping an image version is way too limited.

  • ETCd

    • Still a black box. Didn't learn, that's a problem.
    • Backup/restore?
    • Not a good combination with single cluster per account restriction.

  • Things can break

    • Don't accidentally have elastic-curator load a 90Gb ElasticSearch index in memory.
      • How a changed default can ruin your day evening.
    • Requests and limits are important

  • Clusters should not be pets

2019

IT landscape changes

  • EKS generally available

  • New options to setup EKS:

    • eksctl
    • Terraform modules

2019

So, Kubernetes...

  • Pros
    • Very fast feedback loop
      • helm template, helm diff: seconds
      • terraform plan/apply: minutes
        • Some problems only appearing at apply-time.
      • Similar to pytest -k my-test
    • Versatile
      • Init containers
      • Mount configmaps as files
        • No need to build own nginx image
    • Easy to navigate
      • What's running?
      • Logs
      • Checking config
      • Exec into pods if needed for troubleshooting
  • Cons
    • Not 'easy', quite some choices to be made
    • Less easily integrated with rest of AWS
      • Either not (yet) possible or more complex
        • IAM roles (via service accounts)
        • Security groups (sept. 2020, via service accounts)

12factor

Getting applications in a healthy state

... By means of containerising

Benefits include:

  • Scalable
  • Easy to automate CI/CD
  • Consistent operations (deploying, observing)
  • Build artifact + config = running application

Target platforms:

  • Kubernetes
  • ECS (EC2 / Fargate)
  • Docker swarm
  • Docker on EC2
  • Serverless (ok, that's not containers)
Moving to EKS

Why EKS?

1/2

  • Goal is using Kubernetes, not operating it

  • Less operational responsibility

    • ETCd
    • Right-sizing a HA control plane

  • Cheaper

    • 3x m5.large (eu-west-1)

      • On-demand: 3x ~$78/mo = #$236/mo
      • Reserved: 3x ~$49/mo = $147/mo

    • EKS control plane

      • $0.10/hr = $72/mo

  • Positive signals

    • Introduction of IRSA
    • Introduction of managed nodes
    • Introduction of EKS Fargate
    • Increased adoption, no horror stories

Why EKS?

2/2

  • Only accessible via IAM

    • IAM roles easily mapped to RBAC roles
    • No un/pw backdoors

  • Reasons to not run EKS could be:

    • Specific network requirements that are not compatible with AWS VPC CNI plugin

      • Multi-tenancy options EKS as well:

        • Multiple Clusters
        • Calico
        • Recently indroduced security groups for pods

    • Needing parity or integration with clusters elsewhere (multi- or hybrid cloud)

    • Needing to run very recent Kubernetes version not yet available on EKS

      • Waiting might be an option too...

    • Requires scale or strict requirements for ROI

EKS

EKS Setup

Tools

  • Terraform

    • terraform-aws-eks

  • Helm (v3)

  • Sops

    • Secrets using AWS KMS

  • Helmfile

  • Jenkins pipeline for helmfile

    • Enforces actual config to be in sync with codebase

EKS Cluster

Ingress

Multiple clusters

Flexibility

Components

Many options

  • Skipper (ingress)
  • Fluentd-cloudwatch (logs)
  • Cluster-autoscaler
  • Newrelic agent (logs & metrics)
  • Prometheus operator (metrics)

Prometheus operator

Metrics for clusters and applications

  • Operator that updates Prometheus configuration based on:

    • Servicemonitor

      • Scrape targets
      • Scrape interval
      • Supported by quite a number of helm charts

    • Alertrule

      • Promql queries, if condition is not met -> alert

  • Comes bundled with

    • Grafana + dashboards
    • Preconfigured alerts

  • Easy to create custom dashboards (as code) for applications that expose metrics

Prometheus

Dashboards

Prometheus

Alerts

Prometheus

Who monitors the monitor?

  • New Relic infrastructure agent
  • ELK stack
  • Prometheus federation
Summary & future

Comparison

Evolution

v1v2
ProvisioningKopsTerraform
Kube-system componentsKops-based templatingHelm / Helmfile
Control planeself-managedEKS
Max. clusters per AWS account1Unlimited
LoggingELK (Filebeat)Cloudwatch (Fluentd)1
MetricsELK (Metricbeat)
Prometheus		New Relic
Prometheus operator
Ingress controllerSkipper + ip annotationsSkipper 2x (public/restricted)
Ingress AWS ALBskube-ingress-aws-controller +
Cloudformation		Terraform provisioned



1Becoming increasingly expensive. Search capabilities of insights is not as good as Kibana.

Future

Exploring

  • Either:

    • Explore managed nodes

    • Consider spot instances

    • Benefits:

      • Cost reduction (operations or AWS bill)
      • Chaos Monkey-ish: Drain, reschedule, all should have 0 effect.

  • EKS Fargate

    • Fast scaling

  • GitOps

    • Instead of:

      • Grant CI/CD near-total access to cluster (possibly SaaS)

    • Do:

      • Grant cluster access to code repository

?

twitter.com/TBeijen

www.linkedin.com/tibobeijen

www.tibobeijen.nl