Migrate Docker DB to Amazon RDS

While Docker Compose is a fast and convenient way to deploy your Seqera instance, production deployments require a robust database solution to minimize the risk of data loss.

Points to consider before database migration

• Target database: You have options when selecting your new MySQL-compliant database. While the process is mostly the same, some of the commands will differ (example: MariaDB on RDS).

• Database size: The data in your database must be exported from the MySQL container and imported to the new instance. Depending on the amount of data in your database and the remaining EC2 EBS capacity, you can save your data directly to the instance, or use a service with more capacity (such as AWS S3).

• Testing: What level of testing is required to confirm data has been properly migrated?

• Maintenance window: It is easier to initiate a migration once all transactions to the database cease than to do so while jobs are still running. Perform the migration at a time when an outage may occur, and notify your users in advance.

• MySQL container volume retention: Seqera recommends retaining your original volume until you are 100% satisfied that the migration occurred without error. So long as the volume is kept, you can fall back to the MySQL container to ensure retention of the material generated thus far.

Prerequisites

Before starting your migration:

1. Create an RDS MySQL-compliant instance and populate it with a Seqera user and database.

2. Ensure both your database and EC2 instances' Security Group(s) have been configured to allow MySQL traffic (default: Port 3306).

Migration steps

These migration instructions assume:

• You have sufficient EC2 instance space to store your data.
• You have implemented a full maintenance outage.

1. Connect to the EC2 instance with SSH and navigate to your Seqera instance's docker compose folder.

2. Shut down the application:

   docker compose down

3. Mount a new folder into the MySQL container for migration activities:

   1. Create a new folder to hold migration artefacts:

      mkdir ~/tower_migration

   2. Back up the database:

      sudo tar -zcvf ~/tower_migration/tower_backup.tar.gz ~/.tower/db/mysql

4. Add a new volume entry for the db container to the Seqera docker-compose.yml:

   $HOME/tower_migration:/tower_migration

5. Restart the application to confirm your changes were successful:

   docker compose up

6. Stop all the non-MySQL containers:

   docker container stop <CONTAINER ID>

7. Exec onto the MySQL container:

   docker exec -it <CONTAINER ID> /bin/bash

   1. Dump your data to disk:

      mysqldump -u tower -p --databases tower --no-tablespaces --set-gtid-purged=OFF > /tower_migration/tower_dumpfile.sql

   2. Exit the container.

8. Stop the MySQL container:

   docker container stop <CONTAINER ID>

9. Import the dump file into your new RDS instance:

   mysql --host <DB-HOST> --port 3306 -u tower -p tower < ~/tower_migration/tower_dumpfile.sql

   If you encounter an Access denied; you need (at least one of) the SUPER or SET_USER_ID privilege(s) for this operation error, do the following:

   1. Create a backup of your MySQL dump file (tower_dumpfile.sql).

   2. Run the following command on the dump file:

   sed 's/\sDEFINER=`[^`]*`@`[^`]*`//g' -i tower_dumpfile.sql

10. Log in to the RDS instance and ensure that the Seqera database is populated with tables prefixed by tw_.

11. Modify the tower.env config file in the Seqera Docker folder:

    1. Comment out the existing TOWER_DB-* variables.
    2. Add new entries relevant to your database.
    3. Save and exit.

12. Restart the application:

    docker compose up

13. Confirm that Seqera Enterprise starts and that your data is available when you log in.

The migration is complete and testing can begin.