Getting Started with Docker

In this article, we’ll take some introductory steps to using containers with Docker.

To use Docker we need to download and install a Docker engine. Go to https://www.docker.com/products/docker-desktop and download & install the relevant package for your operating system. (Mac, Windows & Linux are all supported)

Once Docker is installed, fire up a command line and type “docker -v” to check if the docker command has been added to your search path. (You may need to logout and back in again to activate your new search path)

1 – Our First Container

In a terminal window, execute the following command:

$ docker run ubuntu

On the day I ran this, I got the following output:

Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
5d3b2c2d21bb: Pull complete
3fc2062ea667: Pull complete
75adf526d75b: Pull complete
Digest: sha256:b4f9e18267eb98998f6130342baacaeb9553f136142d40959a1b46d6401f0f2b
Status: Downloaded newer image for ubuntu:latest
$

That didn’t appear to do much, did it? Maybe our container is running in the background? Let’s see what Docker is doing:

$ docker container ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

Nope, nothing there.

So what’s happened?

First off, we asked Docker to run the container image called “ubuntu”. As there was no image called “ubuntu” locally (this is the first time we’ve run a docker command, after all) Docker went and downloaded the image from the Docker image repository: hub.docker.com

After that, Docker started the Ubuntu container. But nothing appeared to happen?

What actually happened, was that Docker started a container with the Ubuntu image but the container had nothing to do, so it exited immediately. If we add the -a parameter to the docker container ps command, we now get:

$ docker container ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a430f9ceefd4 ubuntu "/bin/bash" 17 seconds ago Exited (0) 15 seconds ago vigilant_galileo

Our container only lasted 15 seconds.

Let’s run our ubuntu image again, but this time specify that we wish to connect to a tty in the container:

$ docker run -it ubuntu
root@ec72d3ef6f04:/#

Ah – something different. Fire up a separate terminal window and execute the docker container ps command again:

$ docker container ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ec72d3ef6f04 ubuntu "/bin/bash" About a minute ago Up About a minute musing_carson

We can now see our container is running. If we return to the first window with the running Docker container and enter “CTRL-D” and then re-run the docker container ps command, you’ll see that the container has exited.

2 – Our First Containerized Application

Running an interactive Bash shell in a container isn’t really what we’re after: We want to run programs.

In an empty directory, create a file called idle.sh with the following contents:

#!/usr/bin/bash

while true; do
   sleep 10
done

This is a trivial bash script that repeatedly sleeps for 10 seconds. i.e. It does nothing.

Now we need to create a container image with this program. To do this we use a Dockerfile to tell docker how to build our own image.

In the same directory as our idle.sh script, create a file called Dockerfile with the following contents:

FROM ubuntu:latest
RUN mkdir /local
COPY idle.sh /local
CMD ["bash", "/local/idle.sh"]

This simple four line Dockerfile does the following:

  • Base our image on the latest version of Ubuntu
  • Run the command mkdir /local
  • Copy the file idle.sh from the current directory to the directory /local in the container.
  • Tell the container manger to run the command bash /local/idle.sh when the container is started.

Next we have to build our image using this configuration file. Execute the following command:

$ docker build .
[+] Building 0.2s (8/8) FINISHED                                                
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 36B                                        0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest           0.0s
 => [1/3] FROM docker.io/library/ubuntu:latest                             0.0s
 => [internal] load build context                                          0.0s
 => => transferring context: 84B                                           0.0s
 => CACHED [2/3] RUN mkdir /local                                          0.0s
 => [3/3] COPY idle.sh /local                                              0.0s
 => exporting to image                                                     0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:0dda2a8acc5008ce884d67b3a3ae6ea7c3689afb07e19  0.0s

The sha256 value will probably be different.

Now we want to run this image. But how? For the moment, we need to find the image ID.

$ docker image list
REPOSITORY   TAG       IMAGE ID       CREATED             SIZE
<none>       <none>    0dda2a8acc50   6 minutes ago       72.9MB
ubuntu       latest    4dd97cefde62   9 days ago          72.9MB

Now let’s run this image:

$ docker run 0dda2a8acc50

And when we do this, we find that the command doesn’t return. Our container appears to be running our command! If we run the docker container ps command again (in another terminal) we see:

$ docker container ps
CONTAINER ID   IMAGE          COMMAND                 CREATED         STATUS         PORTS     NAMES
d427b3993817   0dda2a8acc50   "bash /local/idle.sh"   6 seconds ago   Up 6 seconds             epic_sanderson

So our container is definitely running. So let’s terminate this before moving on. If we try CTRL-C or CTRL-Z on the command prompt running the docker run command, we find it has no effect. Instead, we have to use the command docker container stop:

$ docker container stop d427b3993817
d427b3993817
$ 

3 – Odds and Ends

There are a couple of small tasks we can do to tidy up.

Cleaning up terminated containers

As we discovered earlier, we can use the command docker container ps -a to list all exited containers. Sometimes we want to examine a container once it’s exited to see what happened. For the moment, we don’t care, so we want to clean up and delete these exited containers. The command docker container prune will remove all stopped containers.

Naming Images

When we built our image before, we had to refer to our image by its id. It would be much nicer if we could just refer to it by a name. To do that, we add the -t <NAME> parameter to our docker build command.

$ docker build -t myimage .

If we then look at the list of images, we can now see our image.

$ docker image list
REPOSITORY   TAG       IMAGE ID       CREATED       SIZE
myimage      latest    0dda2a8acc50   2 hours ago   72.9MB
ubuntu       latest    4dd97cefde62   9 days ago    72.9MB

Notice that although we’ve re-run the docker build command to re-build our image, the CREATED timestamp field is from when we first built the image. We’ll explain this in the next installment.

Now our image has a name, we can run it by name:

$ docker run myimage

That’s it for now. In the next installment, we’ll dig into a little bit more detail about images and open the doors to networking and storage.

Introduction to Containerization

In this article, I’m going to give some background to containerization.

In The Beginning

When I started in IT back in the, *ahem*, we were slowly growing racks of servers & PCs in our machine room. The problem, however, was that most of the time they were idle. We couldn’t merge them as the software they were running got unhappy if they didn’t have exclusive control of the operating system. With some work you could force some applications onto the same machine, but it was fragile and external support never liked it.

So we had racks of machines burning electricity doing very little most of the time.

And then we started running out of space in the racks in the machine room. Getting more machine room space was cost prohibitive. (We’d already knocked through to the adjacent rooms!)

We looked at very small form factor servers (e.g. four or more mini servers in a 1U chassis). Great for space but the spaghetti was crazy.

We also looked at mainstream blade systems. Less of a cabling issue (Well, except the higher power feeds needed for them) but the costs were high compared to standard servers. Plus you had massive vendor lock-in. Buy a HP blade chassis? You can only use HP servers.

Then we were introduced to VMware and its virtualization software.

In computing, the adage that nothing is new, just a rehash, is once again proven in the area of virtualization.

Whilst VMware’s virtualization came to prominence in the naughties, it’s a technology that IBM first pioneered in the 1960s.

But what is virtualization?

Virtualization is the method where software called a hypervisor can carve up a computers resources (CPU, memory, disk, etc) into many virtual computers. On each of these virtual computers (called virtual machines, abbreviated to VMs), you install your operating system (Windows, Linux, etc) If the hypervisor is doing a good job, the operating system thinks it’s got a computer all to itself. It won’t know that it’s running on the same CPU as other operating systems.

Suddenly, a single 1 or 2U server could replace many physical machines. This saved not only the cost of purchasing the hardware, but also electricity & cooling costs. One downside, however, was the cost of the hypervisor. VMware is not a cheap solution. But a VMware licenses for one physical computer (called a host) is substantially less than the cost of half a dozen physical computers. And when you end up running 20 or more virtual computers on that one host, the cost pendulum swings back in VMWare’s favor.

But there was another unforeseen problem with virtualization: The ease of creating a virtual machine. With virtual machine systems such as VMWare, you could create templates for virtual machines and spin up a new instance of a machine (complete with operating system and applications) in seconds.

This lead to patching & licensing hell. Keeping track of all the virtual machines and making sure they were patched and the installed operating system & applications were licensed correctly.

Once system administrators got a handle on the patching and licensing problems, another problem appeared. Every time you wanted a new application, you had to spin up another full operating system, even though you were only going to use a tiny fraction of it. You now had lots of virtual machines eating CPU, memory & disk just to run an application.

Containers

The next step in virtualization is where the operating system kernel creates virtual user space environments. One of the early versions of this system is the chroot jail feature in many unix variants.

Although there are many versions of this containerization feature (chroot jails, Solaris containers, OpenVZ, etc) we’re going to be looking at the Docker containerization system.

Apart from leaner systems (and being the latest IT fad!), what other reasons are there to use containers?

In my experience, there are several reasons to favour containers.

Firstly, it forces you to keep your application & configuration clearly separate from the operating system. This separation then makes updating the base (e.g. operating system of the container) much simpler.

The leads on to another reason: Upgradeability. It is trivial to update the core for the container as there is clean separation between your application and its operating environment.

My final reason is deployability. It is easy to develop software on your local workstation and then deploy it into production. No need to worry about your server and workstation running different operating systems or versions. Your container configuration specifies exactly what versions you’re going to use – and it’s the same as you’ve developed on.

Footnotes

  • There is nothing to stop containerization systems running on top of virtualization systems.
  • In around 2018, IBM announced the release of an ultravisor: A hypervisor to run hypervisor. It’s turtles all the way down…

Django & React

React is a popular Javascript library for writing user interfaces. There are lots of websites on how to get started with React and only a few which talk about how to configure React & Django together. It is perfectly possible to keep the user interface and backend as separate projects but for smaller projects this is may not be applicable.

This is my first stab at writing down how to get them working together. Much of this is taken from Valentino Gagliardi’s blog. I’ve had to tweak some bits which seem to have changed since Valentino wrote his articles.

The Problem

Normally, when you want to create a React application you execute the command:

npx create-react-app my-react-app

But if you try and do this inside a Django application you’ll probably encounter the error:

The directory <> contains files that could conflict

One way to get around this is to put the React UI in a subdirectory of your Django application. Another way is to integrate it into your Django app manually.

Getting Started

I’m going to assume you’ve already got Django & npm installed.

As I’ve already written how to get started with Django so I’ll just list the commands to create the bare bones Django application:

$ mkdir PROJECT_NAME
$ cd PROJECT_NAME
$ virtualenv -p python3 virtualenv
$ source virtualenv/bin/activate
$ pip install django
$ django-admin startproject my_project .
$ django-admin startapp main

Then edit the file my_project/settings.py and in the section INSTALLED_APPS add the line:

'main',

Now we’ll start to add React.

First off, we need to create some directories for React in our application:

$ cd main
$ mkdir -p ./src/components
$ mkdir -p ./{static,templates}/frontend

Now we create a package.json file to tell npm what modules we want:

$ cat <<EOF >package.json
{
  "name": "frontend",
  "version": "1.0.0",
  "description": "My React Application",
  "main": "index.js",
  "scripts": {
    "dev": "webpack --mode development",
    "build": "webpack --mode production"
  },
  "keywords": [],
  "author": "John Smith",
  "license": "ISC",
  "devDependencies": {
    "@babel/core": "^7.12.3",
    "@babel/preset-env": "^7.12.1",
    "@babel/preset-react": "^7.12.1",
    "babel-loader": "^8.1.0",
    "webpack": "^5.2.0",
    "webpack-cli": "^4.1.0"
  },
  "dependencies": {
    "react": "^16.13.1",
    "react-dom": "^16.13.1",
    "react-router": "^5.2.0"
  }
}
EOF

Next we need to create a configuration file for something called Babel. Babel allows Javascript programmers to ignore differences between browser Javascript engines.

$ cat <<EOF >babel.config.json
{
    "presets": [
        "@babel/preset-env",
        "@babel/preset-react",
    ],
    "plugins": [
        "@babel/plugin-proposal-class-properties"
    ]
}
EOF

Up next is a Webpack configuration file. Webpack is used to package Javascript code.

$ cat <<EOF >webpack.config.js
const path = require('path');
 
module.exports = {
  entry: './src/index.js',
  output: {
    path: path.resolve(__dirname, 'static/frontend'),
    filename: 'main.js'
  },
  module: {
    rules: [
      {
        loader: 'babel-loader',
        test: /\.js$/,
        exclude: /node_modules/,
      }
    ]
  }
};
EOF

Now we’ve created our configuration files, we can start to write some React.

$ cat <<EOF >src/index.js
import React from "react";
import ReactDOM from 'react-dom';
import App from "./components/App";
 
ReactDOM.render(
  <React.StrictMode>
    <App />
  </React.StrictMode>,
  document.getElementById('app')
);
EOF

$ cat <<EOF >src/components/App.js
import React, { Component } from "react";
import { render } from "react-dom";
 
export default class App extends Component {
  constructor(props) {
    super(props);
  }
 
  render() {
    return (
       <p>Hello World from React</p>
    );
  }
}
EOF

Now we can create a view to invoke React:

$ cat <<EOF >templates/frontend/index.html
<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Django REST with React</title>
</head>
<body>
<div id="app">
    <!-- React will load here -->
</div>
{% load static %}
<script src="{% static "frontend/main.js" %}"></script>
</body>
</html>
EOF

$ cat <<EOF >>views.py
def index(request):
    return render(request, 'frontend/index.html')
EOF

Finally, we need to update urls.py in the Django Project directory to reference this view:

$ cd ../my_project
$ cat <<EOF >urls.py
from django.urls import path
 
urlpatterns = [
    path('', views.index),
]
EOF

Finally, go back to the application directory and let’s build our Javascript library.

$ cd ../main
$ npm run dev

Once npm has finished, you can start your Django application and fire up a web browser

$ cd ..
$ ./manage.py runserver

PXE Booting Ubuntu 20.04 Installer

With Ubuntu 20.04, Canonical switched from the Debian installer to their own autoinstaller. This means that you have to build a new PXEBoot environment.

When I started setting up my environment I was surprised at the sparseness of the Ubuntu documentation. Here is how I got it working.

0 – Getting Ready

Before you start, you need to have:

  • Working TFTP Server
  • Working HTTP Server
  • Working DHCP Server
  • Ubuntu 20.04 ISO
  • Ubuntu 20.04 running VM. (Yes, you need a 20.04 VM to build a PXEBoot environment.)

Your TFTP/HTTP/DHCP servers don’t have to be on Ubuntu 20.04. Nor do they all have to be on the same host, but for these instructions I will assume they are as it makes it easier to write…

In your 20.04 VM install the following two packages

$ sudo apt-get install syslinux-common pxelinux

Then copy the file ldlinux.c32 & pxelinux.0 files to to your TFTP server:

$ scp /usr/lib/syslinux/modules/bios/ldlinux.c32 <SERVER>:/tmp
$ scp /usr/lib/PXELINUX/pxelinux.0 <SERVER>:/tmp

Whilst you’re logged into your 20.04 copy off the file /var/log/installer/autoinstall-user-data:

$ sudo scp /var/log/installer/autoinstall-user-data <SERVER>:/tmp

We’ll discuss this file later on.

1 – HTTP Server Prep.

Put the 20.04 ISO into a directory on your HTTP server:

# mkdir -p /var/www/html/ubuntu2004/autoinstall/test/
# mv /path/to/iso/ubuntu-20.04-live-server-amd64.iso /var/www/html/ubuntu2004

We also need to create two files in our newly created directory. One is empty, the other has minimal content:

# touch /var/www/html/ubuntu2004/autoinstall/test/meta-data
# cat >/var/www/html/ubuntu2004/autoinstall/test/user-data <<EOF
#cloud-config
autoinstall:
  version: 1
  identity:
    hostname: ubuntu2004
    password: "$6$exDY1mhS4KUYCE/2$zmn9ToZwTKLhCw.b4/b.ZRTIZM30JZ4QrOQ2aOXJ8yk96
xpcCof0kxKwuX1kqLG/ygbJ1f8wxED22bTL4F46P0"
    username: ubuntu
EOF

Note that the password is one line, not split over two.

2 – TFTP Server Prep

In the directory of your TFTP server services, create a directory to hold our 20.04 PXE Boot environment:

# mkdir /var/lib/tftpboot/ubuntu2004

Move the ldlinux.c32 & pxelinux.0 files into this directory:

# mv /tmp/ldlinux.c32 /var/lib/tftpboot/ubuntu2004
# mv /tmp/pxelinux.0 /var/lib/tftpboot/ubuntu2004

Mount the 20.04 ISO and copy off the kernel & root filesystem:

# mount -o ro /var/www/html/ubuntu2004/ubuntu-20.04-live-server-amd64.iso /mnt
# cp /mnt/casper/initrd /var/lib/tftpboot/ubuntu2004
# cp /mnt/casper/vmlinuz /var/lib/tftpboot/ubuntu2004
# umount /mnt

Create a starter PXEBoot configuration file:

# mkdir /var/lib/tftpboot/ubuntu2004/pxelinux.cfg

In that directory, create a file called “default” and give it the following contents:

# cat >/var/lib/tftpboot/ubuntu2004/pxelinux.cfg/default <<EOF
DEFAULT install
 LABEL install
 KERNEL vmlinuz
 INITRD initrd
 APPEND autoinstall ip=dhcp ds=nocloud-net;s=http://<IP_ADDRESS_SERVER>/ubuntu2004/autoinstall/test/ url=http://<IP_ADDRESS_SERVER>/ubuntu2004/ubuntu-20.04-live-server-amd64.iso
EOF

Replace the <IP_ADDRESS_SERVER> string with the IP address of your HTTP server.

3 – DHCP Server Prep

Edit your dhcpd.conf file and add an entry for your machine:

# cat >>/etc/dhcp/dhcpd.conf <EOF

host ubuntu2004 {
  hardware ethernet xx:xx:xx:xx:xx:xx;
  fixed-address y.y.y.y;
  next-server z.z.z.z;
  filename "ubuntu2004/pxelinux.0" 
}
EOF
  • Replace the “xx:xx:xx:xx:xx:xx” string with the MAC address of your machine.
  • Replace the “y.y.y.y” string with the IP address you want the machine to have
  • Replace the “z.z.z.z” string with the IP address of your TFTP Server.

Restart the DHCP service:

# systemctl restart isc-dhcp-server

4 – GO!

You should be able to power-on your (virtual) machine and it should boot the installer. Places to check if it doesn’t start the installer:

  • DHCP Server log (/var/log/dhcpd.log) to check that the machine is getting its IP address and boot settings.
  • The TFTP server log to make sure the machine is getting its PXEBoot files (tail -f /var/log/syslog | grep tftp )
  • The HTTP server log to make sure the machine is getting its ISO and configuration files.

Once the installer has completed, you should have an Ubuntu 20.04 machine with a user called “ubuntu” and password “ubuntu”

5 – Advanced.

Other than getting the PXEBoot environment setup, our installer hasn’t really done much. This is where we need to configure the installer. In the old Debian based installer this was done by editing the preseed file. With 20.04’s autoinstaller, we have to configure the user-data file (that we created on our HTTP server)

The official Ubuntu documentation for this is at https://ubuntu.com/server/docs/install/autoinstall-reference The problem with the documentation is it’s incomplete. (And the occasional error in it too)

The docs say you can take the configuration file from a 20.04 install (the autoinstall-user-data file we copied off our 20.04 VM right at the start). The problem is that this file is the idealized file and doesn’t match what the installer actually wants.

Note that so far I’ve not seen mention of what needs to be in the meta-data file. It just has to exist.

5.1 – Network Configuration

There’s a bug in the installer. The documentation says the configuration file should look something like:

#cloud-config
autoinstall:
  verson: 1
  network:
    version: 2
    ethernets:
      ens160:
      critical: true
      dhcp-identifier: mac
      dhcp4: true

However, if you try this you’ll get an error from autoinstall. If you dig through the log file from the installer, you’ll find reference to a missing key “network”. This is due to the installer having a bug. To work around this, you have to add an extra “network” level/section to the configuration

#cloud-config
autoinstall:
  verson: 1
  network:
   network:
    version: 2
    ethernets:
      ens160:
      critical: true
      dhcp-identifier: mac
      dhcp4: true

5.2 – Disc Configuration

To quote Jonathan Corbett, this section is ruthlessly undocumented. If you look at the autoinstall-user-data file from your 20.04 VM, you’ll see that the configuration isn’t exactly user friendly. This is from my basic VM which had a 30GB HDD:

#cloud-config
autoinstall:
  verson: 1
  storage:
    config:
    - {ptable: gpt, path: /dev/sda, wipe: superblock-recursive, preserve: false, name: '', grub_device: true, type: disk, id: disk-sda}
    - {device: disk-sda, size: 1048576, flag: bios_grub, number: 1, preserve: false, type: partition, id: partition-0}
    - {device: disk-sda, size: 32209108992, wipe: superblock, flag: '', number: 2, preserve: false, type: partition, id: partition-1}
    - {fstype: ext4, volume: partition-1, preserve: false, type: format, id: format-0}
    - {device: format-0, path: /, type: mount, id: mount-0}

6 – Going Further

The observant will have noticed that our “default” PXE Boot configuration file referenced a specific directory for the installer. The way to specify a per machine configuration is to name the file with the machine’s booting MAC address. e.g.

# cd  /var/lib/tftpboot/ubuntu2004/pxelinux.cfg
# mv default xx-xx-xx-xx-xx-xx

Problems Upgrading Ubuntu Server LTS

I’ve been doing some housekeeping and updating some of my Ubuntu servers from 16.04 to 18.04. At first the upgrade progresses fine, but then near the end, it stops when it’s trying to clean up old packages:

Searching for obsolete software
reading state information... 54%
=== Command detached from window (Mon Jun 1 12:00:00 2020) ===
=== Command terminated with exit status 1 (Mon Jun 1 12:00:10 2020) ===

Looking at the logs in /var/log/dist-upgrade/apt.log there were all sorts of errors about broken packages.

I also noticed that during the earlier stages of the upgrade there were numerous messages saying:

No candidate ver: <perl-modules>

So I rolled back to my pre-upgrade snapshot and started googling for checking installed packages. All the answers seemed to revolve around running the command “apt check” which always said everything was OK.

The best I could find was the command:

# dpkg -l | grep -v '^ii

which revealed a list of packages (which looked similar to both the “No candidate ver” messages and the broken packages messages in /var/log/dist-upgrade/apt.log.

Asking a friendly unix guru suggested that these packages, whilst not being installed, were still in the configuration and hadn’t been purged.

The clue about purging packages led me to https://chrisjean.com/finding-and-purging-unpurged-packages-on-ubuntu/ and the command:

# apt purge `dpkg --list | grep ^rc | awk '{ print $2; }'`

Ansible, APT & Proxies

I’ve been working on a new Ansible setup which is behind a HTTP proxy and I’ve run into what seems to be a common problem: The Ansible modules apt_key and apt_repository don’t work behind proxies. There appear to be multiple bug reports in Ansible about these problems, but no resolution yet despite the bugs being several years old.

Adding a new repository to APT involves two steps: Adding the repository’s key to APT’s GPG keyring, then actually adding the repository.

Step 1 – Adding the key (Method 1)

The apt_key module in Ansible has the nice bug that it doesn’t honor any standard unix method for specifying use of a HTTP proxy.

Putting “Acquire::http::Proxy” in /etc/apt/apt.conf doesn’t work. Nor does setting the http_proxy environment variable.

Instead, you have to execute the apt-key command (not Ansible module) directly and supply a http-proxy parameter to it. This is from my playbook for installing Ansible:

- name: Add Ansible Repo Key
  command:
   argv:
    - /usr/bin/apt-key
    - adv
    - --keyserver-options
    - http-proxy=http://192.168.1.1:3128
    - --keyserver
    - hkp://keyserver.ubuntu.com:80
    - --recv-keys
    - 6125E2A8C77F2818FB7BD15B93C4A3FD7BB9C367

When adding other repositories, you may have to do some digging to get the ID of the key you wish to receive.

Step 1 – Adding the key (Method 2)

Method 1 only works if the repository key is in Ubuntu’s key server. If the key is in a third party site (such as what Gitlab do), things can get even more complicated. This is the best I’ve come up with.

The way I do it, is to create a temporary file, download the key into this temporary file and then pass the key to the apt_key module.

First off, create the temporary file:

- name: Create temp file
  tempfile:
   state: file
  register: tempfile_1

Now we download the key from the remote server and save it into our temporary file:

- name: Get Gitlab Key from remote server
  get_url:
   url: https://packages.gitlab.com/gpg.key
   dest: "{{ tempfile_1.path }}"
  environment:
   http_proxy: "http://192.168.1.1:3128"
   https_proxy: "http://192.168.1.1:3128"

Now we need to pass the file’s contents to apt_key. You may (as I was!) be tempted to use the lookup(‘file’, ‘/somewhere/filename’) helper function. This won’t work ☹

The lookup helper function runs on the local Ansible machine and not the remote machine we’re configuring. So lookup() fails to find the file because it doesn’t exist “here”, only “over there”.

Instead, we have to use the slurp module in Ansible to read the file contents into a variable.

- name: Read Gitlab Key into memory
  slurp:
   src: "{{ tempfile_1.path }}"
  register: key_data

And then we just pass this to apt_key:

- name: Add Gitlab Key
  apt_key:
   data: "{{ key_data['content'] }}"

Except this doesn’t work. This is because when the slurp module reads the file into memory, it Base64 encodes it. So when we read the data back out again, we need to Base64 decode it.

- name: Add Gitlab Key
  apt_key:
   data: "{{ key_data['content'] | b64decode }}"

Step 2 – Adding the repository

The apt_repository is slightly more helpful: It does honor the http_proxy variables. But you also need another variable: APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE This is needed as the apt_repository module still kicks off apt-key somewhere and apt-key really doesn’t want to be run from a script. My play for this step looks like:

- name: Add Ansible repo
  apt_repository: repo="ppa:ansible/ansible" state=present update_cache=yes
  environment:
   http_proxy: http://192.168.1.1:3128
   https_proxy: http://192.168.1.1:3128
   APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE: DontWarn

Note that you have to set both the http_proxy AND https_proxy variables.

Step 3 – Install Ansible

Now we can fall back to standard APT configurations! Create the file /etc/apt/apt.conf.d/10proxy.conf  and set its contents to be:

Acquire {
 HTTP::proxy "http://192.168.1.1:3128";
 HTTPS::proxy "http://192.168.1.1:3128";
}

Then in your Ansible play you can just say:

- name: Add Ansible
  apt:
   name: ansible
   state: present

Or, if you’re installing Gitlab’s Runner package:

- name: Install gitlab runner
  apt:
   name: gitlab-runner
   state: present

That was way more convoluted that it needed to be.

Appendix 1 – Proxy Variables

You will have noticed we replicated the proxy settings in a couple of places. Being the good programmers that we are, we see that this is a bad thing and we want to place these in a variable.

There are several places we can define the a variable. In my simple case, I want it accessible to all hosts, so I put it in the group_var/all.yaml file:

proxy_env:
  no_proxy: example.com
  http_proxy: "http://192.168.1.1:3128"
  https_proxy: "http://192.168.1.1:3128"

Then in our roles, we can just put:

 environment:
  - "{{ proxy_env }}"

And if we need to add any extra variables, we just add them to the bottom of the list:

environment:
 - "{{ proxy_env }}"
 - APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE: DontWarn

Appendix  2 – APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE

In my playing with these issues, I’ve discovered something I don’t understand. For some APT sources, I have to supply the APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE environment variable, yet for others, I don’t.

e.g. To install Ansible, I have to add the variable, but for Gitlab’s runner, I don’t.

I’ll leave it as an exercise for the reader to work out what’s going on.

Dummies Introduction to Django. Part 4 – Settings

We have covered the basics of creating a Django web application. However, before we can go ahead and deploy our application to a website, there’s an elephant in the room we have to acknowledge. Go and take a look at the project/settings.py file. Here are (some) of the settings in there that we need to take care of:

...
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = '...'

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True

ALLOWED_HOSTS = []
...
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'djangodb',
        'USER': 'djangouser',
        'PASSWORD': 'secret',
        'HOST': 'localhost',
    }
}
...

These are all either settings that need to vary between our development environment and a live environment, or settings that don’t belong in version control. We need to pull these from somewhere outside our application.

So how do we do this? Unfortunately, there does not appear to be any standard way to do this in Django. The Django website offers half a dozen different solutions. I asked a couple of Django developers in my building and they all had different ways to do this too. There is no “right” way to do it: Just a way that works for you.

My chosen method is to store the settings in an external YAML file. This is easier than you think because the settings.py file is just a plain Python file so you can embed whatever code you want in there. (This is probably also why there are so many ways to tackle this problem: You can code whatever solution you can dream of in settings.py)

First off, we need to add a YAML library to our project.

(Django_Project) $ pip install pyyaml

Now we have to create a YAML configuration file for our application to use. This is the core of what I use:

application:
 - secret_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
   debug: true
   allowed_hosts:
   database_password: secret

Yaml is quite a powerful and flexible file format, so you could structure the data in numerous ways. But this is the way I choose. Make sure to change the “secret_key” and “database_password” value to whatever is currently in your settings.py file.

I called this file “settings.yaml” and put in in the root of our virtual environment:

.
├── bin
├── include
├── lib
├── main
├── manage.py
├── project
└── settings.yaml

Now we have to tell our Django app to read and use this new file. First off, at the top of the project/settings.py file, add the line “import yaml” just below the existing “import os” line”

For the full list of settings and their values, see
https://docs.djangoproject.com/en/3.0/ref/settings/
"""

import os
import yaml   <==== Our new line

# Build paths inside the project like this: os.path.join(BASE_DIR, ...)

Next, at the bottom of the settings.py file, we need to add a chunk of code to read and use settings.yaml. First we need to open the file. But before we can do that, we need to know where to find the file. First, we’ll look for an environment variable called “APPLICATION_CONFIG_FILE”. If it exists, we’ll use it as the full path for our yaml file. If the environment variable is not set, we just assume the file is in the root of our virtual environment. This technique will be very useful later on – trust me!

configuration_file = os.getenv('APPLICATION_CONFIG_FILE')
if configuration_file is None:
  configuration_file = './settings.yaml'

Next we have to load our YAML data into memory. Pyyaml provides several methods for loading yaml files. We’ll start with using the safest one.

yaml_data = yaml.safe_load(open(configuration_file, 'r'))

You can read more about the different loader methods available at Pyyaml’s Github page.

Finally, we can take the parameters from the yaml file and apply them to Django

SECRET_KEY = yaml_data['application'][0]['secret_key']
DEBUG = yaml_data['application'][0]['debug']
DATABASES['default']['PASSWORD'] = yaml_data['application'][0]['database_password']
hosts = yaml_data['application'][0]['allowed_hosts']
if hosts is not None:
  for host in hosts:
    ALLOWED_HOSTS.append(host)

There is no error checking in this code. This is a deliberate decision: If we can’t load any of these settings, there’s no point in carrying on as our application won’t work.

Save all theses files and you should be able to run your Django application as before.

Dummies Introduction to Django. Part 3 – Databases

The next big step our application needs is to talk to a database. For the purposes of this, I’ll assume you’re using Postgres.

Step 1 – Installing Postgres.

If you’re using a Linux system, then you can install Postgres using your distribution’s package manager. e.g.

# apt-get install postgresql

If you’re using Mac there are multiple ways you can install Postgres. Homebrew is one (“brew install postgresql”) or download Postgres.App which is a macOS application which contains Postgres and all the normal command line tools)

Step 2 – Setting up Postgres

In this step I’ll assume we’re starting with a brand new setup. There are multiple ways you could configure this, but this is how I would configure the Postgres environment for Django.

To start with, connect to the Postgres database engine with some kind of administrative privileges. On Linux, you might do this by connecting to the Postgres engine as the unix user “postgres”.

$ sudo - U postgres psql

First, we create a user that Django can connect to Postgres with.

postgres=# CREATE USER djangouser LOGIN UNENCRYPTED PASSWORD 'secret';

Next, we create a database:

postgres=# CREATE DATABASE djangodb WITH OWNER = djangouser;

Then we create a database schema to hold our tables:

postgres=# CREATE SCHEMA djangoschema AUTHORIZATION djangouser;

We give our Django user rights to use this schema:

postgres=# GRANT USAGE ON SCHEMA djangoschema TO djangouser;

And finally we make this schema the default one for our Django user by setting its schema search path:

postgres=# ALTER ROLE djangouser SET search_path TO djangoschema;

We can now test this all works by connecting to Postgres as our Django user and performing a couple of basic operations:

$ psql -W djngodb djangouser
djangodb=> CREATE TABLE test1 (id SERIAL, msg VARCHAR);
djangodb=> INSERT INTO test1 (msg) VALUES ('Hello World');
djangodb=> SELECT * FROM test1;
djangodb=> DROP TABLE test1;

If you get an error message connecting to postgres and you’re on linux, try the following command instead:

$ sudo -U postgres psql djngodb djangouser

Step 3 – Setting up Django connection

Now Postgres is ready for us, we are ready to tell Django about Postgres. First, we need to install a Django driver for Postgres. Within our Python virtual environment:

(Django_Project) $ pip install psycopg2--binary

Now we’ve installed the driver, we need to get Django to use it.

Edit the file project/settings.py and search for the section “DATABASES”. It currently says:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}

We have to change this to use our Postgres database. Hopefully, all the settings we need should be self explanatory:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'djangodb',
        'USER': 'djangouser',
        'PASSWORD': 'secret',
        'HOST': 'localhost',
    }
}

Step 4 – Using the database

Now Django knows how to talk to our database, we need to actually use it.

We need to create a model – a mapping or representation of a database table into a python/Django data structure. These are stored in the application/models.py file. Edit the file main/models.py

from django.db import models


class Message(models.Model):
  msg = models.TextField()

  def __str__(self):
    return(msg)

This defines a database table called “message” with just one field/column called “msg” which is a text field. We’ve also defined the __str__ method to return the message field if we ever print out the class.

Note that we aren’t going to create any tables in Postgres ourselves: We’ll get Django to do that for us later on.

Next, we’ll need to create a new view for this feature. In views.py, add the following import line at the top:

from main.models import *

Then add the following function call at the bottom of the file:

def messages(request):
  messages = Message.objects.all()
  context = {
    'messages': messages,
  }
  return render(request, 'messages.html', context)

Add the following entry to the list of urlpatterns in the projects/urls.py file:

    path("messages", views.messages, name="messages"),

Finally, we create our view file in main/templates/messages.html

<!DOCTYPE html>
<html>
  <head>
    <title>List of messages</title>
  </head>
  <body>
    <h2>A list of messages</h2>
    <table border="1">
      <thead>
        <tr>
          <th>Message</th>
          <th>Action</td>
        </tr>
      </thead>
      <tbody>
       {% if messages is not None %}
         {% for message in messages %}
           <tr>
            <td>{{ message.msg }}</td>
            <td></td>
           </tr>
         {% endfor %}
       {% endif %}
      </tbody>
   </table>
  </body>
</html>

Now we’re almost ready to run our Django app. But before we can do that, we need to get Django to update our database schema. The following command should do the trick:

python (Django_Project)$ manage.py migrate

If you don’t get a series of “OK”s, check your database connections. Otherwise, you can go for “python manage.py runserver” and go to http://127.0.0.1:8000/messages and..Blank List of Messages

As we haven’t created any means (yet) to add messages, we’ll quickly add a couple direct into the database:

djangodb=> INSERT INTO main_message (msg) VALUES ('Message 1'), ('Message 2');

And reload the web page:

List of messages 1

Success!

Now we want to be able to add messages through the web page, rather than having to delve into SQL.

First, edit main/views.py and add “, redirect” to the initial import line so it now looks:

from django.shortcuts import render, redirect

Then add the following function to the bottom of the file:

def add_message(request):
   new_message = request.POST.get('newMessage')
   if new_message is not None:
     message = Message(msg=new_message)
     message.save()

   return redirect('messages')

Now, edit main/templates/messages.html and add the following section after the “</table>” tag (But before the “</body>” tag:

<h2>Add Message</h2>
    <form action="addMessage" method="post">
      {% csrf_token %}
      <input type="text" name="newMessage" />
      <input type="submit" value="Add Message" />
    </form>

Finally, edit the project/urls.py and add the following line to the urlpatterns section:

path("addMessage", views.add_message, name="add_message"),

*Phew*. Now, reload the page http://127.0.0.1:8000/messages and you should see the new add message table.

List of messages add button

Put some words of wisdom in the text box and click “Add Message” and you should now see your message in the table:

List of messages Hello World

Let’s add the final basic feature: Deleting a message.

First, edit the messages.html file, and in the blank table cell (which is in the column “Action”) add the following HTML code:

<form method="post" action="deleteMessage">
  {% csrf_token %}
  <input type="hidden" name="messageId" value="{{ message.id }}" />
  <input type="submit" value="Delete" />
</form>

Next, add the following method into views.py:

def delete_message(request):
   messageId = request.POST.get('messageId')
   try:
     msg =  Message.objects.get(**{'id': messageId})
     msg.delete()
   except Message.DoesNotExist:
     # Do something
     return redirect('messages')

   return redirect('messages')

Finally, add an entry to urls.py

 path("deleteMessage", views.delete_message, name="delete_message"),

Reload the /messages page in your browser and you should now see a Delete button next to each message:

List of messages with delete button

Hopefully, you should be able to click “Delete” on a message and it will disappear.

Alternative Python Model Organization

In the above example, we’d store all the database models in one file: models.py. But once you start getting a number of model classes, this file can start to get large and unwieldy. So there’s another way to organize the models.

In the application directory, create a directory called “models”. In that directory, create files containing your models. Finally, create a file called “__init__.py” that just has a series of import statements. e.g.

from .message import Message

Dummies Introduction to Django. Part 2 – Simple Web Forms

Now we’ve got a blank Django app to play with, we can start creating web pages.

Hello World

An introduction in computing wouldn’t be the same without the famous quote.

Open up the file project/urls.py. Ignoring the comment at the start of the file, at the bottom we have:

from django.contrib import admin
from django.urls import path

urlpatterns = [
    path('admin/', admin.site.urls),
]

We need to add one import and change that path(…) statement, so the file looks:

from django.contrib import admin
from django.urls import include, path

from main import views

urlpatterns = [
    path("",views.homepage, name="homepage"),
]

Next, we want to edit the file main/views.py. Currently, it looks like:

from django.shortcuts import render

# Create your views here.

Add the following two lines after the comment line:

def homepage(request):
  return render(request, 'index.html')

Next, make a directory called “templates” in the main application folder:

$ mkdir main/templates

And in that new directory, create the file called “index.html” and put some basic HTML in there:

<!DOCTYPE html>
<html>
  <head>
  <title>Hello World</title>
 </head>
 <body>
  <h1>Hello World</h1>
 </body>
</html>

Save this file and run the Django app (“python manage.py runserver”) and go to http://127.0.0.1:8000/ and instead of the pretty picture of a rocket, you should see the exciting output:Hello World

So what have we done here?

  1. We’ve imported the views file from the main application
  2. We’ve told Django that the URL “” (the root or “/” URL) should be passed to the view function “homepage” in the  views file. We’ve also called this mapping “homepage”. Naming mappings is optional but has some benefits further down the line.
  3. In the views.py file, we defined the function “homepage” and told it to render the file “index.html”
  4. In the index.html file we put our basic HTML.

We could have merged steps 3 & 4, but it’s probably more likely you’ll be wanting to render something more complex than “Hello World” which is easier to do in a HTML file than putting the HTML in the middle of python code.

Ask and ye shall…

A web application isn’t much good if it can’t take user input. First, edit the index.html file so that it looks like:

<!DOCTYPE html>
<html>
  <head>
  <title>Hello World</title>
 </head>
 <body>
  <h1>Hello World</h1>
  <form method="get">
    <input type="text" name="question" />
    <input type="submit" value="Speak" />
  </form>
  {%  if answer is not None %}
    <p>{{ answer }}</p>
  {%  endif %}
 </body>
</html>

Next, edit the homepage function in views.py so it looks like:

def homepage(request):
  queryString = request.GET.get('question')
  context = {
        'answer': queryString,
  }

  return render(request, 'index.html', context)

If you reload the page (note, I didn’t say you needed to stop and restart the Django runserver process!) You should see the following:Hello World Blank Form

In the text box, enter some text and then press “Speak”

Hello World Woof

So what have we achieved here?

First, our html file can take more than HTML: It can take code!

Second, we’ve shown the homepage function taking a request parameter and outputting a variable to the template.

These are the first steps to writing web applications in Django. The next stage is to get our application talking to a database.

Dummies Introduction to Django. Part 1 – Installing Django

Being new to Django & Python programming in general, I thought I’d write up some notes on how to get started with a Django web application. I have plans as to where I want to get to with this series – but I’m not going to make any promises yet as I don’t want to make a rod for my own back.

Being a dummy in this area, I do not promise everything I say here is the best/right way to do it. It is just what I’ve discovered. If you think there are better ways to do something, let me know.

Most of the time I’ll assume you’re using a unix-type environment.

Step 0.0 – Python

The first step in working with Python is to ensure you have a reasonably up-to-date version of python installed.

$ python -V
Python 2.7.16

Python 2 isn’t good enough nowadays. We want  python 3.x

Due to incompatibilities between python versions 2 and 3, you tend to find that both may be installed in your environment, with the command “python” linked to one particular version. If python v3 is installed, it may be available under the name “python3”

$ python3 -V
Python 3.7.5

MacOS & Python

If you’re a MacOS user, MacOS comes with python 2 but not version 3 (And Apple have said they’ll remove python altogether from future versions of MacOS). An easy way to install python 3 is to use the Homebrew system:

$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ brew install python

Step 0.1 – Virtualenv

Although python has lots of features out of the box, there is a lot that isn’t in the box. A web application framework isn’t one of them. When installing add-ons to Python, there are two options: The first is to install the add-on as part of the python installation on the machine. The advantage is that the package is available for all uses/users of the machine. The downside, is if you run an application on another machine, you have to remember to install the same packages on the other machine.

So python has the option of creating a virtual environment. This allows you to compartmentalize the requirements for a piece of software to make running it elsewhere much simpler. To do this, you need another package called “virtualenv”

$ which virtualenv
/usr/bin/virtualenv

If you get no reponse to the “which” command, you’ll need to install it. It may be available via your package manager. e.g.

$ sudo apt-get install virtualenv

or, you may need to install it via pip:

$ pip3 install virtualenv

Step 1 – Creating a virtual environment

Now we’ve got the core python system installed, we’re ready for our first steps.

First off is to create a virtual environment for our Django application. At a basic level, this is just a directory with some particular files in it.

$ virtualenv -p python3 virtualenv

The “-p python3” is needed if you have both python 2 & 3 installed to ensure your virtual environment gets created to use the correct python version. “Django_Project” is just a directory name.

Once the command has completed, if you look inside the directory, you’ll see three sub directories: bin, include & lib. When we come to commit our Django project to version control, we need to make sure to exclude these.

Now we have a virtual environment, we need to switch it on. Make sure your CWD is Django_Project (Or whatever you named it) and then execute “source bin/activate”

$ source bin/activate
(virtualenv) $

Notice the change to your command prompt.

To leave the virtual environment, you can either exit the shell or run the command “deactivate”

(virtualenv) $ deactivate
$

Notice the command prompt changes back.

We are now ready to install Django into our virtual environment. So, make sure your virtual environment is active, then issue the command “pip install django”

(virtualenv) $ pip install django

This will download and install the latest Django version into our environment.

Step 2 – Creating a Django Application

We’re now ready to create a blank Django project. Again, make sure you’ve activated your virtual environment:

(virtualenv) $ django-admin startproject project .

Note the space & full-stop at the end of that command.

This will create a folder called “project” in your current directory. You should now see four directories and a file:

...
├── manage.py
└── project
    ├── __init__.py
    ├── __pycache__
    │   ├── __init__.cpython-37.pyc
    │   └── settings.cpython-37.pyc
    ├── asgi.py
    ├── settings.py
    ├── urls.py
    └── wsgi.py

This has created a Django “project” into which we can create “applications”.

(virtualenv) $ django-admin startapp main

You should now see a directory called “main” alongside all the others.

...
├── main
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   ├── migrations
│   │   └── __init__.py
│   ├── models.py
│   ├── tests.py
│   └── views.py
├── manage.py
└── project
    ├── __init__.py
    ├── __pycache__
    │   ├── __init__.cpython-37.pyc
    │   └── settings.cpython-37.pyc
    ├── asgi.py
    ├── settings.py
    ├── urls.py
    └── wsgi.py

Step 3 – Configuring our Django Application

We’re close now to being able to run our Django application for the first time. Before we run it, we have to do one final step: We have to tell the Django “project” about the “application” that we created.

Edit the file projects/settings.py and look for the block:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

At the end of that list, we need to add in “main” (Or whatever you called your application), so it looks:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'main',
]

Now, we can finally run our Django application with the command:

(virtualenv) $ python manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).

You have 17 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.

January 08, 2020 - 19:39:55
Django version 3.0.2, using settings 'project.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

If you point your browser at http://127.0.0.1:8000/ you should see the following:Django First Launch