Locked database - DatabaseChangeLogLock

Introduction

The lock on the changelog table could not be removed. The Relution service is no longer starting or keeps rebooting (Docker). The log shows the following messages:

ion.boot.LiquibaseConfig: Relution build version is ... []
ion.boot.LiquibaseConfig: Database is ... (Data source 'dataSource') []
ion.boot.LiquibaseConfig: Detected database type '...', version '...' []
ion.boot.LiquibaseConfig: Registering Liquibase customizations []
iquibase.MariaDBDatabase: Connected to relution@... []
iquibase.MariaDBDatabase: Setting auto commit to false from true []
vice.StandardLockService: Waiting for changelog lock.... []
vice.StandardLockService: Waiting for changelog lock.... []

The last message repeats periodically until the service finally terminates with a timeout and restarts automatically if necessary (Docker).

Causes

At startup, the Relution service first checks if database updates need to be executed. To prevent two nodes from changing the database schema at the same time in cluster mode, a lock is always set in the DATABASECHANGELOGLOCK table for this purpose. This is released again after the check and/or the updates are completed.

If the Relution service is terminated in this state, this lock remains in place. The next time Relution is started, it waits for the lock to be released because the service assumes that another node is updating the database. If this is not the case after about 5 minutes, the service terminates with an appropriate error message.

This condition usually occurs when Relution terminates early during the execution of major database updates, as the startup process takes an unusually long time.

Changing the DatabaseChangeLogLock Value

Native

To fix the issue, you can either remove the (only) entry from the DATABASECHANGELOGLOCK table or change the value of the LOCKED column from 1 to 0. Depending on the database, 1 may also be displayed as True and should then be changed to False.

To update the database, you can use the database’s management tool or a free alternative (BeeKeeper, DBeaver, HeidiSQL, …). Depending on the database, the update may also be possible via the corresponding command-line tools. For example, in DBeaver:

TRUNCATE TABLE DATABASECHANGELOGLOCK;

The placeholders in angle brackets should be replaced with the values from your application.yml or docker-compose.yml.

Docker

For a Docker setup, the command to unlock the table can be executed from the server in the /opt/relution/ directory as follows:

Long example

docker exec -it <containername> \
  mariadb --user=<username> \
        --password=<password> \
        --database=<database> \
        --execute="TRUNCATE TABLE DATABASECHANGELOGLOCK;"

Long example filled in:

docker exec -it docker_mariadb \
  mariadb --user=relution \
        --password=admin123 \
        --database=relution \
        --execute="TRUNCATE TABLE DATABASECHANGELOGLOCK;"

Short example

docker exec -it docker_mariadb mariadb -u <username> -p<password> <database> -e "TRUNCATE TABLE DATABASECHANGELOGLOCK;"

Short example filled in

docker exec -it docker_mariadb mariadb -u relution -padmin123 relution -e "TRUNCATE TABLE DATABASECHANGELOGLOCK;"

Note To avoid storing the password in plain text in the Bash history, you can omit the -p or --password option when running the command without entering the password. > This way, the database password will be requested interactively after submitting the command.