DatabaseChangeLogLock locked

The lock on the changelog table could not be removed. The Relution service is no longer starting or keeps rebooting (Docker). The log shows the following messages:

ion.boot.LiquibaseConfig: Relution build version is … []
ion.boot.LiquibaseConfig: Database is … (Data source 'dataSource') []
ion.boot.LiquibaseConfig: Detected database type '…', version '…' []
ion.boot.LiquibaseConfig: Registering Liquibase customizations []
liquibase.lockservice   : Waiting for changelog lock.... []
liquibase.lockservice   : Waiting for changelog lock.... []

The last message repeats periodically until the start aborts after the waiting time with a message such as Could not acquire change log lock. Currently locked by <host> since <timestamp> and the service restarts automatically if necessary (Docker).

Causes #

At startup, the Relution service first checks if database updates need to be executed. To prevent two nodes from changing the database schema at the same time in cluster mode, a lock is always set in the DATABASECHANGELOGLOCK table for this purpose. This is released again after the check and/or the updates are completed.

If the Relution service is terminated in this state, this lock remains in place. The next time Relution is started, it waits for the lock to be released because the service assumes that another node is updating the database. If this is not the case after about 5 minutes, the service terminates with an appropriate error message.

This condition usually occurs when Relution terminates early during the execution of major database updates, as the startup process takes an unusually long time.

The LOCKEDBY and LOCKGRANTED columns of the DATABASECHANGELOGLOCK table show which host has held the lock and since when. In cluster mode, it is ensured before unlocking that all Relution nodes are stopped and that no database migration is actually running anymore.

Native #

To fix the issue, either the (only) entry is removed from the DATABASECHANGELOGLOCK table or the value of the LOCKED column is changed from 1 to 0. Depending on the database, 1 may also be displayed as True and should then be changed to False.

To update the database, the database’s management tool or a free alternative (BeeKeeper, DBeaver, HeidiSQL, …) can be used. The following SQL statement can be executed in an SQL editor (e.g. DBeaver) or via the database’s command-line tool (e.g. mariadb, psql):

TRUNCATE TABLE DATABASECHANGELOGLOCK;

Docker #

For a Docker setup, the command to unlock the table can be executed from the server in the /opt/relution/ directory. The command depends on the database used.

The placeholders in angle brackets are replaced with the values from the application.yml or the compose.yaml (docker-compose.yml for older installations).

PostgreSQL #

With an installation according to the current guide Docker (Recommended) →, the database runs in the database container.

Example

docker exec -it <containername> psql -U <username> -d <database> -c "TRUNCATE TABLE DATABASECHANGELOGLOCK;"

Example filled in

docker exec -it database psql -U relution -d relution -c "TRUNCATE TABLE DATABASECHANGELOGLOCK;"

If psql asks for a password, it is stored in the file /opt/relution/secrets/postgres_password for an installation according to the current Docker guide.

MariaDB #

Older installations that have not yet been migrated to PostgreSQL use MariaDB.

Long example

docker exec -it <containername> \
  mariadb --user=<username> \
        --password=<password> \
        --database=<database> \
        --execute="TRUNCATE TABLE DATABASECHANGELOGLOCK;"

Long example filled in:

docker exec -it docker_mariadb \
  mariadb --user=relution \
        --password=admin123 \
        --database=relution \
        --execute="TRUNCATE TABLE DATABASECHANGELOGLOCK;"

Short example

docker exec -it <containername> mariadb -u <username> -p<password> <database> -e "TRUNCATE TABLE DATABASECHANGELOGLOCK;"

Short example filled in

docker exec -it docker_mariadb mariadb -u relution -padmin123 relution -e "TRUNCATE TABLE DATABASECHANGELOGLOCK;"