Controlling non-production Aiven Services with the CLI

Use your resources wisely

In their very nature, production services normally need to be available 24/7. But non-production services are often only needed during working hours.

Let's assume you have six engineers that are all working on different branches of your codebase, a fairly standard n-tier code stack in front of a relational database, for instance Aiven for PostgreSQL® or Aiven for MySQL. So now we have six different developer databases that need to be managed, plus an integration environment, staging and UAT (user acceptance testing) services.
Our engineering team prioritise a work-life balance, so they are pretty good at 9 to 5, Monday to Friday. It doesn't make sense to have the database services running after hours and on weekends.

Aiven managed databases can be powered off. Powered off services do not incur any billing. Let's assume the following pricing for Aiven for PostgreSQL as an example, based on a Business 64 service running in UAT to give a like for like environment with production (this pricing is historical - it's not meant to be accurate).

Given those pricing assumptions, we can save ourselves 70% per month on this service alone if we automate stopping and starting it. In a real scenario, the actual saving might be more or less, but it's still likely to be significant.

Let's jump in and have a look at how this might work.

Looking at the documentation I can see that Aiven provides a number of different tools for controlling the services. Of course you could assign a task to one of the team to open the console every day and stop the relevant services, then start their day tomorrow by restarting these services, but it would be more efficient to automate this. The Aiven REST API and the Aiven CLI (command line interface) are both excellent tools for this task.

In the examples below, we're going to use the Aiven CLI and the jq JSON processor. See the referenced pages for how to install them.

Authentication

We are going to need an authentication token. Read the documentation to see how to create an authentication token.

Be sure you understand the details of authentication tokens in Aiven, specifically how the token is associated with the user account that is used to create the token. Authentication tokens have the same access level as the user who created them. For non-production projects, consider creating a user that only has access to those non-production projects. This way you can ensure the script cannot shut down production services by accident.

A word of caution

Before you do anything, read the specifics for the Aiven service you're dealing with, because they behave differently in how their storage is managed, and whether data is lost when the service is powered off. Streaming services such as Aiven for Apache Kafka® will not preserve data when shut down. Some services, such as Aiven for OpenSearch®, will only preserve data since the last backup.

Have a look at the backup frequency per service in the Backups for Aiven documentation, check out the details in the Service power cycle documentation.

For services that should not be shut down, you can enable termination protection. See the flags for avn service create and avn service update.

If you have read replicas running for an Aiven for PostgreSQL service, you will want to destroy them before attempting to shutdown the main service, as they won't be shutdown automatically (and you would get errors). Just remember to rebuild when you start these up again.

Using the Aiven CLI to power cycle a service

One of the tools that I find myself using pretty much every day is the Aiven CLI.

First, ensure you have authenticated your CLI.
Then, switch the default project to the one where your services are running, the services you wish to shut down that is. As shutdown-demo is the name for the project in Aiven where the services I am controlling are found, I type

avn project switch shutdown-demo
Copy to clipboard

which confirms with

INFO	Set project 'shutdown-demo' as the default project
Copy to clipboard

Of course, you would set your own default project in the above command.

Then double check your work, list the services in the default project to see that they are the ones your are targeting. Use:

avn service list
Copy to clipboard

to get output of the form:

SERVICE_NAME        SERVICE_TYPE  STATE    CLOUD_NAME                   PLAN        CREATE_TIME           UPDATE_TIME           NOTIFICATIONS
==================  ============  =======  ===========================  ==========  ====================  ====================  =============
cassandra-14fc5dba  cassandra     RUNNING  google-australia-southeast1  startup-4   2022-05-18T06:56:47Z  2022-06-21T02:29:11Z
os-157f6082         opensearch    RUNNING  google-australia-southeast1  business-4  2022-05-18T06:55:55Z  2022-06-21T02:24:00Z
grafana-3f786321    grafana       RUNNING  google-australia-southeast1  startup-1   2022-06-03T00:03:15Z  2022-06-21T02:21:28Z
kafka-32e61bb7      kafka         RUNNING  google-australia-southeast1  business-4  2022-05-24T23:21:47Z  2022-06-21T02:23:20Z
m3db-228ac07        m3db          RUNNING  google-australia-southeast1  startup-8   2022-06-02T23:47:56Z  2022-06-21T02:21:29Z
pg-25d65efa         pg            RUNNING  google-australia-southeast1  premium-4   2022-05-19T23:13:12Z  2022-06-21T02:22:37Z
redis-2bb763ee      redis         RUNNING  google-australia-southeast1  business-4  2022-05-18T06:55:31Z  2022-06-21T02:23:13Z
Copy to clipboard

We can see that STATE here which tells me whether each service is currently running or not.

So, for the services I want to power cycle, I can then issue this command

avn service update --power-off <SERVICE_NAME>
Copy to clipboard

An example shell script

Now we know how to use power services on or off with the Aiven CLI, let's write a bash script that can be run to start and/or stop services as required.

In a script, it's better to use the JSON output from the avn command, which we can get with the --json flag, and then process using the jq JSON processor.

We can put this all together into a nice, simple to use (and perhaps crontab) script. But again, take care, you'll want something more sophisticated for use in real life, as it will stop or start all services in a project, whether they'll lose data or not!

#!/bin/bash
#
# Start or stop all services in a named project.
#
# Does NOT take any account of whether data will be lost by stopping a service.

# exit on error of any command
set -e

# function to print usage and exit
exit_on_error() {
    echo "$1"
    echo "Usage : bash-aiven-controller -p <project> -o <start|stop>"
    exit 1
}

# function that starts an Aiven service
start_service() {
    echo "Starting $1"
    avn service update --project "$project" --power-on "$1"
}

# function that stops an Aiven service
stop_service() {
    echo "Stopping $1"
    avn service update --project "$project" --power-off "$1"
}

# read commandline flags
# we want the project (-p) and the operation (-o [start|stop])
while getopts ":p:o:" flag
do
    case "$flag" in
        p) project=${OPTARG};;
        o) operation=${OPTARG};;
        \?) exit_on_error "Unrecognised option -${OPTARG}";;
        :) exit_on_error "Option -${OPTARG} needs an argument";;
    esac
done

# test if the project flag exists
if [ -z "$project" ]
then
    exit_on_error "No project name specified. Use '-p <project>'"
fi

# test if the operation is either start or stop
if [ "$operation" == "start" ] || [ "$operation" == "stop" ];
then
    echo "Getting ready to $operation all services"
else
    exit_on_error "Operation (-o $operation) is not recognised. Use '-o start' or '-o stop'"
fi

services=$(avn service list --project "$project" --json)

# Loop over all the services in this project, doing the operation.
# The `-raw-ouptut` switch to jq makes it write strings to stdout without extra quotes.
# The `@base64` encodes the data for each service as base64 to "hide" the newlines in
# the output.
for row in $(echo "$services" | jq --raw-output '.[] | @base64'); do

    name=$(echo "$row" | base64 --decode  | jq -r '.service_name')
    state=$(echo "$row" | base64 --decode | jq -r '.state')

    if [ "$operation" == "stop" ]
    then
        if [ "$state" == "RUNNING" ]
        then
            stop_service "$name"
        else
            echo "Not attempting to stop $name because it is currently $state"
        fi
    else
        if [ "$state" == "POWEROFF" ]
        then
            start_service "$name"
        else
            echo "Not attempting to start $name because it is currently $state"
        fi
    fi
done
Copy to clipboard

Things to remember

It is not immediately obvious that you can start and stop services when using a managed service provider like Aiven. However, you can take advantage of this to save yourself some spend on non-production resources.
It is also worth remembering that your engineers are human and humans forget things, so this kind of automation ensures that you get all the value possible from your Aiven investment.

Our key things to remember when doing this are :

• Not all services are created equal, some will destroy your data.
• Allow your engineers the ability to restart when they need to.
• Organising services using projects will make this easier.

Using a shell script, the Aiven CLI, and a small cron job on a server somewhere, you will be able to stretch your non-production resources and make sure that the TCO (total cost of ownership) of your Aiven investment is even better than what the team first calculated.