The goal - Leveraging PowerShell doesn't mean you have to use PowerShell commands

PowerShell is a shell just like bash, zsh, fish, etc. Which means it can also be used for running native commands or standalone executables... like kubectl.

This post is all about using Notebooks as a form of documentation. To do this, I've taken an actual Kubernetes docs page and converted it into a Notebook.

You should be able to download this ipynb file, open it and follow along by clicking play next to each code block (assuming you have the prereqs).

Here's a screenshot of this Notebook in Azure Data Studio: screenshot of Azure Data Studio

Prereqs

  • Jupyter and a jupyter client - I used Azure Data Studio (insiders) for this but you could use Jupyter Lab or nteract. Those are great too!
  • .NET Interactive - A .NET Jupyter Kernel that supports C#, F# and PowerShell
    • This comes with a PowerShell runtime so you don't even have to install PowerShell on your machine... just the kernel
    • It's cross-platform - Windows, macOS, and Linux

Since this article is about Kubernetes, you should also have a Kubernetes cluster and kubectl available.

Install powershell-yaml

We're also going to install this PowerShell for working with YAML. It is solely used to mutate the YAML example. You don't need this in the wild... but it helps!

What's nice is that we can simply install it with a code cell like so:

Install-Module -Name powershell-yaml

Now the "demo"...

The rest of this article comes straight from this markdown file that I've converted into a Notebook by taking the code blocks and turning them into Notebook code cells.

Using kubectl describe pod to fetch details about pods

For this example we'll use a Deployment to create two pods.

application/nginx-with-request.yaml

apiVersion:apps/v1kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 80

Create deployment by running following command:

kubectl apply -f https://k8s.io/examples/application/nginx-with-request.yaml

deployment.apps/nginx-deployment created

Check pod status by following command:

kubectl get pods

NAME                               READY   STATUS    RESTARTS   AGE
nginx-deployment-8df4655b6-6cznf   1/1     Running   0          22s
nginx-deployment-8df4655b6-nzw89   1/1     Running   0          22s

We can retrieve a lot more information about each of these pods using kubectl describe pod. For example:

# Get the Pod name
$result = kubectl get pods -o json | ConvertFrom-Json
$podName = $result.items[0].metadata.name

kubectl describe pod $podName

Name:           nginx-deployment-8df4655b6-6cznf
Namespace:      default
Priority:       0
Node:           docker-desktop/192.168.65.3
Start Time:     Sat, 02 May 2020 17:55:46 -0700
Labels:         app=nginx
                pod-template-hash=8df4655b6
Annotations:    <none>
Status:         Running
IP:             10.1.0.142
Controlled By:  ReplicaSet/nginx-deployment-8df4655b6
Containers:
  nginx:
    Container ID:   docker://76faabd5f8cdc6bad07dc188be00b97c790dd84db360449c2e1c5fbb2005981d
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:86ae264c3f4acb99b2dee4d0098c40cb8c46dcf9e1148f05d3a51c4df6758c12
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Sat, 02 May 2020 17:55:54 -0700
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:        500m
      memory:     128Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-v495d (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-v495d:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-v495d
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age    From                     Message
  ----    ------     ----   ----                     -------
  Normal  Scheduled  3m51s  default-scheduler        Successfully assigned default/nginx-deployment-8df4655b6-6cznf to docker-desktop
  Normal  Pulling    3m50s  kubelet, docker-desktop  Pulling image "nginx"
  Normal  Pulled     3m43s  kubelet, docker-desktop  Successfully pulled image "nginx"
  Normal  Created    3m43s  kubelet, docker-desktop  Created container nginx
  Normal  Started    3m43s  kubelet, docker-desktop  Started container nginx

Here you can see configuration information about the container(s) and Pod (labels, resource requirements, etc.), as well as status information about the container(s) and Pod (state, readiness, restart count, events, etc.).

The container state is one of Waiting, Running, or Terminated. Depending on the state, additional information will be provided -- here you can see that for a container in Running state, the system tells you when the container started.

Ready tells you whether the container passed its last readiness probe. (In this case, the container does not have a readiness probe configured; the container is assumed to be ready if no readiness probe is configured.)

Restart Count tells you how many times the container has been restarted; this information can be useful for detecting crash loops in containers that are configured with a restart policy of 'always.'

Currently the only Condition associated with a Pod is the binary Ready condition, which indicates that the pod is able to service requests and should be added to the load balancing pools of all matching services.

Lastly, you see a log of recent events related to your Pod. The system compresses multiple identical events by indicating the first and last time it was seen and the number of times it was seen. "From" indicates the component that is logging the event, "SubobjectPath" tells you which object (e.g. container within the pod) is being referred to, and "Reason" and "Message" tell you what happened.

Example: debugging Pending Pods

A common scenario that you can detect using events is when you've created a Pod that won't fit on any node. For example, the Pod might request more resources than are free on any node, or it might specify a label selector that doesn't match any nodes. Let's say we created the previous Deployment with 5 replicas (instead of 2) and requesting 600 millicores instead of 500, on a four-node cluster where each (virtual) machine has 1 CPU. In that case one of the Pods will not be able to schedule. (Note that because of the cluster addon pods such as fluentd, skydns, etc., that run on each node, if we requested 1000 millicores then none of the Pods would be able to schedule.)

Let's modify the above example to make this happen...

# Make changes to YAML
$yaml = Invoke-RestMethod https://k8s.io/examples/application/nginx-with-request.yaml | ConvertFrom-Yaml
$yaml.spec.replicas = 5

# NOTE: for my Docker for Mac setup, 2000m is what it took to get some Pending.
$yaml.spec.template.spec.containers[0].resources.limits.cpu = "2000m"

# Apply those
$yaml | ConvertTo-Yaml | kubectl apply -f -

deployment.apps/nginx-deployment configured

Ok let's see how our pods are doing...

kubectl get pods

NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-6c4956f7bc-b8pg5   0/1     Pending   0          109s
nginx-deployment-6c4956f7bc-mb5pc   1/1     Running   0          2m4s
nginx-deployment-6c4956f7bc-nkx2r   0/1     Pending   0          2m4s
nginx-deployment-6c4956f7bc-rqhdw   1/1     Running   0          2m4s
nginx-deployment-6c4956f7bc-zt92l   0/1     Pending   0          2m1s
nginx-deployment-bc849747-269t7     1/1     Running   0          2m35s
nginx-deployment-bc849747-whpvx     1/1     Running   0          2m35s

To find out why all of the pods are not running, we can use kubectl describe pod on a "pending" Pod and look at its events:

# This just grabs the first 'Pending' pod's name
$result = kubectl get pods --field-selector status.phase=Pending -o json | ConvertFrom-Json
$podName = $result.items[0].metadata.name

kubectl describe pod $podName

Name:           nginx-deployment-6c4956f7bc-b8pg5
Namespace:      default
Priority:       0
Node:           <none>
Labels:         app=nginx
                pod-template-hash=6c4956f7bc
Annotations:    <none>
Status:         Pending
IP:             
Controlled By:  ReplicaSet/nginx-deployment-6c4956f7bc
Containers:
  nginx:
    Image:      nginx
    Port:       80/TCP
    Host Port:  0/TCP
    Limits:
      cpu:     2
      memory:  128Mi
    Requests:
      cpu:        2
      memory:     128Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-v495d (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  default-token-v495d:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-v495d
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  68s (x14 over 19m)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu.

Here you can see the event generated by the scheduler saying that the Pod failed to schedule for reason FailedScheduling (and possibly others). The message tells us that there were not enough resources for the Pod on any of the nodes.

To correct this situation, you can use kubectl scale to update your Deployment to specify four or fewer replicas. (Or you could just leave the one Pod pending, which is harmless.)

Events such as the ones you saw at the end of kubectl describe pod are persisted in etcd and provide high-level information on what is happening in the cluster. To list all events you can use

kubectl get events --field-selector reason=FailedScheduling

LAST SEEN   TYPE      REASON             OBJECT                                  MESSAGE
8s          Warning   FailedScheduling   pod/nginx-deployment-6c4956f7bc-b8pg5   0/1 nodes are available: 1 Insufficient cpu.
28m         Warning   FailedScheduling   pod/nginx-deployment-6c4956f7bc-mb5pc   0/1 nodes are available: 1 Insufficient cpu.
8s          Warning   FailedScheduling   pod/nginx-deployment-6c4956f7bc-nkx2r   0/1 nodes are available: 1 Insufficient cpu.
8s          Warning   FailedScheduling   pod/nginx-deployment-6c4956f7bc-zt92l   0/1 nodes are available: 1 Insufficient cpu.
29m         Warning   FailedScheduling   pod/nginx-deployment-bc849747-b87vx     0/1 nodes are available: 1 Insufficient cpu.

but you have to remember that events are namespaced. This means that if you're interested in events for some namespaced object (e.g. what happened with Pods in namespace my-namespace) you need to explicitly provide a namespace to the command:

kubectl get events --namespace=my-namespace

No resources found.

To see events from all namespaces, you can use the --all-namespaces argument.

In addition to kubectl describe pod, another way to get extra information about a pod (beyond what is provided by kubectl get pod) is to pass the -o yaml output format flag to kubectl get pod. This will give you, in YAML format, even more information than kubectl describe pod--essentially all of the information the system has about the Pod. Here you will see things like annotations (which are key-value metadata without the label restrictions, that is used internally by Kubernetes system components), restart policy, ports, and volumes.

kubectl get pod $podName -o yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2020-05-03T01:25:37Z"
  generateName: nginx-deployment-6c4956f7bc-
  labels:
    app: nginx
    pod-template-hash: 6c4956f7bc
  name: nginx-deployment-6c4956f7bc-b8pg5
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: nginx-deployment-6c4956f7bc
    uid: af67aea9-fae6-4368-a264-afc20cf27e2e
  resourceVersion: "2275800"
  selfLink: /api/v1/namespaces/default/pods/nginx-deployment-6c4956f7bc-b8pg5
  uid: 6c4f5268-da0b-4abf-8987-5810481adeb3
spec:
  containers:
  - image: nginx
    imagePullPolicy: Always
    name: nginx
    ports:
    - containerPort: 80
      protocol: TCP
    resources:
      limits:
        cpu: "2"
        memory: 128Mi
      requests:
        cpu: "2"
        memory: 128Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-v495d
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: default-token-v495d
    secret:
      defaultMode: 420
      secretName: default-token-v495d
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-05-03T01:25:37Z"
    message: '0/1 nodes are available: 1 Insufficient cpu.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Guaranteed

The end

The rest of the document requires spinning up multiple nodes that I don't need to to right now 😅 I think what I've shown so far should prove my point.

Example repo

Here's an example repo of a bunch of Notebooks and a Dockerfile that works on MyBinder (a free sandbox envirnment for you to play in) and locally!

https://github.com/TylerLeonhardt/JupyterNotebooks

Remember what the goal of this was

You should be able to download this ipynb file, open it in either Jupyter Lab, nteract, or Azure Data Studio and follow along by clicking play next to each code block (assuming you have the prereqs).

Try exploring with Notebooks for your onboarding documentation or troubleshooting guides. If you have other fun usecase for Notebooks-as-a-shell, let me know on Twitter!