• Backup and restore a K3S cluster

    3 min read

    k3s backup restore etcd

    If we have a K3S Kubernetes cluster that we want to create a backup of, we can use the k3s etcd-snapshot, but that's just going to backup the information related to Pods and other Kubernetes objects, it won't backup data that resides outside of the cluster such as disks (PersistentVolumes, emptyDirs, ...), or even it's state.

    Having clarified that we are just going to backup some of the data, let's take a look how to do it.

    06/05/2022

    Read more...
  • Kubernetes: volume node affinity conflict

    4 min read

    While trying to deploy Pods we might notice the on the Events section that Pod cannot be scheduled due to a volume node affinity conflict:

    $ kubectl describe pod website-365-flask-ampa2-ha-member-1 -n website-365 
    Name:           website-365-flask-ampa2-ha-member-1
    Namespace:      website-365
    Priority:       0
    Node:           <none>
    Labels:         (...)
    Annotations:    (...)
    Status:         Pending
    IP:             
    IPs:            <none>
    Controlled By:  StatefulSet/website-365-flask-ampa2-ha-member
    Init Containers:
    (...)
    Containers:
    (...)
    Conditions:
      Type           Status
      PodScheduled   False 
    Volumes:
      volume:
        Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
        ClaimName:  volume-website-365-flask-ampa2-ha-member-1
        ReadOnly:   false
    (...)
    Events:
      Type     Reason             Age                      From                Message
      ----     ------             ----                     ----                -------
      Normal   NotTriggerScaleUp  31m (x20835 over 7d19h)  cluster-autoscaler  pod didn't trigger scale-up: 2 node(s) had taint {pti/role: system}, that the pod didn't tolerate, 1 node(s) had volume node affinity conflict
      Normal   NotTriggerScaleUp  95s (x46144 over 7d19h)  cluster-autoscaler  pod didn't trigger scale-up: 1 node(s) had volume node affinity conflict, 2 node(s) had taint {pti/role: system}, that the pod didn't tolerate
      Warning  FailedScheduling   64s (x2401 over 43h)     default-scheduler   0/4 nodes are available: 2 node(s) had taint {pti/role: system}, that the pod didn't tolerate, 2 node(s) had volume node affinity conflict.
    

    27/04/2022

    Read more...
  • CoreDNS on Kubernetes: Allow DNS zone transfer

    3 min read

    kubernetes CoreDNS

    Kubernetes, by default, registers all the Pods and services using the cluster.local DNS zone. At some point we might want to be able to take a look at this zone. Zone transfers are going to be restricted by default:

    dnstools# dig axfr cluster.local
    
    ; <<>> DiG 9.11.3 <<>> axfr cluster.local
    ;; global options: +cmd
    ; Transfer failed.
    

    But if we are using CoreDNS, we can configure it to temporally allow zone transfers to be able to take a look at it

    25/04/2022

    Read more...
  • Keeping the terraform state in a Kubernetes Secret

    2 min read

    terraform state Kubernetes Secret

    Storing the terraform state into a S3 bucket with dynamoDB for locking has become the de facto standard for being able to share the state across an organization. Nevertheless, there are interesting alternatives: We can use a Kubernetes Secret

    19/04/2022

    Read more...
  • Kubernetes: Autoscaling using Prometheus as a external metrics provider

    5 min read

    kubernetes hpa prometheus external metrics

    Using an external metrics provider (Kubernetes 1.10+) we can use an HorizontalPodAutoscaler to automatically scale applications using any metric collected by Prometheus. Let's take a look on how to configure it

    05/04/2022

    Read more...

More recent...

Older content...

Kubernetes:
container orchestration
kubernetes
tags related to this category
yq kubectl Linkerd Argo Rollouts Rollouts Capsule Pod MutatingAdmissionPolicy MutatingAdmissionPolicyBinding kind plugin custom command Argo Workflows CronWorkflow StatefulSet Workflow Kaniko WorkflowTemplate install kubernetes security Pod Security Standards port-forward socat operator-sdk golang Pushgateway RBAC Rule troubleshooting APIRequestCount affinity topologySpreadConstraints Route ExternalSecret Secret jsonpath ServiceAccount Ingress k3s letsencrypt tcpdump ssh CRD additionalPrinterColumns Velero query PV Operator Role ClusterRole web-console operator oc-mirror Secrets Manager tekton context Policy enforcement Rules Project ConfigMap Environment ROSA IngressRoute redirect RDS psql API server S3 patch file apply selector minikube arm64 colima EKS-connector SecurityContextConstraint SecretStore scripting CRC credentials Deployment valueFrom setup helm StorageClass tagging EBS externalDNS ALB HPA convert API version example shipwright ECR imagePullSecrets ENI subnet krew blame cloud provider etcd availability zones CoreDNS backend state podAntiAffinity Composite images GKE activeDeadlineSeconds Job lifetime bestby IRSA label annotation PersistentVolume Volume fsGroup vpa cluster autoscaler Karpenter provider kubernetes_manifest fsGroupChangePolicy container escape spot instances termination handler persistentVolumeReclaimPolicy fieldPath upgrade privileged network NetworkPolicy bash ps longhorn ASCP QoD raspberry pi drain evict uncordon kubeconfig config view logs admission controller hook postStart preStop deprecations gp3 get-all taints securityGroup probe readinessProbe livenessProbe tolerations explain MutatingWebhook startupProbe RollingUpdate Recreate PDB emptyDir netstat ss autoscale Kubeconfig initContainers DNS tree DaemonSet stern tail LimitRange resource limits restartPolicy system-upgrade-controller rolling update history undo Volumes awsElasticBlockStore change-cause set image imperative hostAliases imagePullPolicy metrics-server Service overlay agent nodes declarative ELB HTTPS alpine package nodeSelector scheduler kubie api-versions events multiple containers SecretKeyRef ReplicaSet NodePort Pod restart rollout deployment nginx-contoller ValidatingWebhookConfiguration error recovery httpHeaders uid securityContext exec interactive LoadBalancer IAM scale replicas nodeName externalName namespace Cronjob multinode template yaml unused-volumes diff