Locked database - DatabaseChangeLogLock

Introduction #

The lock on the changelog table could not be removed. The Relution service is no longer starting or keeps rebooting (Docker). The log shows the following messages:

ion.boot.LiquibaseConfig: Relution build version is ... []
ion.boot.LiquibaseConfig: Database is ... (Data source 'dataSource') []
ion.boot.LiquibaseConfig: Detected database type '...', version '...' []
ion.boot.LiquibaseConfig: Registering Liquibase customizations []
iquibase.MariaDBDatabase: Connected to relution@... []
iquibase.MariaDBDatabase: Setting auto commit to false from true []
vice.StandardLockService: Waiting for changelog lock.... []
vice.StandardLockService: Waiting for changelog lock.... []

The last message repeats periodically until the service finally terminates with a timeout and restarts automatically if necessary (Docker).

Causes #

At startup, the Relution service first checks if database updates need to be executed.
To prevent two nodes from changing the database schema at the same time in cluster mode, a lock is always set in the DATABASECHANGELOGLOCK table for this purpose.
This is released again after the check and/or the updates are completed.

If the Relution service is terminated in this state, this lock remains in place.
The next time Relution is started, it waits for the lock to be released because the service assumes that another node is updating the database.
If this is not the case after about 5 minutes, the service terminates with an appropriate error message.

This condition usually occurs when Relution terminates early during the execution of major database updates, as the startup process takes an unusually long time.

Change DatabaseChangeLogLock value #

To fix the condition, either the (only) entry from the table DATABASECHANGELOGLOCK can be removed or the value of the column LOCKED can be changed from 1 to 0. Depending on the database, True may also be displayed instead of 1 and must be changed to False accordingly.

To update the database, the corresponding database administration tool or a free alternative can be used (BeeKeeper, DBeaver, HeidiSQL, …). Depending on the database, the update is also possible using appropriate command line tools. In case of MariaDB/MySQL for example:

mysql --host=<hostname> --user=<username> --password --database=<database> --execute="TRUNCATE TABLE DATABASECHANGELOGLOCK;"

The placeholders in angle brackets must be replaced accordingly with the values from application.yml or docker-compose.yml.