Ggtcf

I have a VPS (Ubuntu Server 18.04) hosted with OVH. They offer snapshots, that are supposed to be able to rollback a VPS to a previous state. I have never used this feature before. But I made a snapshot last night before I started messing with BTCpay. I royally borked that install so I decided to roll back to the snapshot.

Now my Mariadb install isn't working properly. The only thing hosted on the server is Wordpress multisite. If I reboot the server (or start Mariadb with systemctl) it loads and I can access all my wordpress sites, and admin panel. But after a couple of minutes, Mariadb crashes.

Running mysqld_safe --skip-grant-tables outputs:

190308 15:10:20 mysqld_safe Logging to syslog.
190308 15:10:20 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql

This gets Wordpress up and running, but obviously isn't a secure solution.

These are the only errors that show up in /var/log/mysql/error.log but there's only the one entry and they do not repeat each time Mariadb crashes:

2019-03-08 13:08:24 139897840925824 [ERROR] mysqld: Table './mysql/db' is marked as crashed and should be repaired
2019-03-08 13:08:24 139897840925824 [ERROR] mysql.db: 1 client is using or hasn't closed the table properly

and CHECK TABLE mysql.db; outputs:

+----------+-------+----------+----------+
| Table | Op | Msg_type | Msg_text |
+----------+-------+----------+----------+
| mysql.db | check | status | OK |
+----------+-------+----------+----------+
1 row in set (0.00 sec)

So far the steps I have taken are:

• To use mysqldump for the 'wordpress' database, which is the one I
  setup for my Wordpress multisite install.




• Run # mysqlcheck --all-databases which came back all with 'OK'




• Tried the fix listed here
  https://stackoverflow.com/a/3322930/8644833 after starting Mariadb with mysqld_safe --skip-grant-tables

I am not great with mysql, so I'm just looking for a solution to get me up and running again without losing too much data. I wouldn't mind performing a clean install of Mariadb and setting up a new Wordpress database, but I'm not exactly sure how to back up all of my data so I wouldn't have to rebuild all my sites. Seems like this should be possible as the sites all work fine until Mariadb crashes.

Here is all the other relevant info I have:

dmesg:

[ 108.430534] audit: type=1400 audit(1552073977.631:19): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=939 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=111 ouid=0
[ 108.534100] audit: type=1400 audit(1552073977.739:20): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=939 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=111 ouid=0
[ 108.634399] audit: type=1400 audit(1552073977.839:21): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=939 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=111 ouid=0
[ 108.734779] audit: type=1400 audit(1552073977.939:22): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=939 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=111 ouid=0
[ 108.835027] audit: type=1400 audit(1552073978.039:23): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=939 comm="mysqld" requested_mask="w"denied_mask="w" fsuid=111 ouid=0
[ 108.935311] audit: type=1400 audit(1552073978.139:24): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=939 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=111 ouid=0
[ 109.035562] audit: type=1400 audit(1552073978.235:25): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=939 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=111 ouid=0
[ 109.136162] audit: type=1400 audit(1552073978.339:26): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=939 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=111 ouid=0
[ 110.038191] audit: type=1400 audit(1552073979.243:27): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=939 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=111 ouid=0
[ 110.040919] audit: type=1400 audit(1552073979.243:28): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=939 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=111 ouid=0

systemctl status mariadb.service:

 ● mariadb.service - MariaDB 10.1.38 database server
 Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
 Active: failed (Result: timeout) since Fri 2019-03-08 14:39:39 EST; 14min ago
 Docs: man:mysqld(8)
 https://mariadb.com/kb/en/library/systemd/
 Process: 939 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=0/SUCCESS)
 Process: 839 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
 Process: 809 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
 Process: 770 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
 Main PID: 939 (code=exited, status=0/SUCCESS)

Mar 08 14:38:08 mydomain.com systemd[1]: Starting MariaDB 10.1.38 database server...
Mar 08 14:38:09 mydomain.com mysqld[939]: 2019-03-08 14:38:09 140251492867200 [Note] /usr/sbin/mysqld (mysqld 10.1.38-MariaDB-0ubuntu0.18.04.1) starting as process 939 ...
Mar 08 14:39:37 mydomain.com systemd[1]: mariadb.service: Start operation timed out. Terminating.
Mar 08 14:39:39 mydomain.com systemd[1]: mariadb.service: Failed with result 'timeout'.
Mar 08 14:39:39 mydomain.com systemd[1]: Failed to start MariaDB 10.1.38 database server.

mysql log:

2019-03-08 14:59:39 140597857991808 [Note] InnoDB: innodb_empty_free_list_algor
ithm has been changed to legacy because of small buffer pool size. In order to 
use backoff, increase buffer pool at least up to 20MB.

2019-03-08 14:59:39 140597857991808 [Note] InnoDB: Using mutexes to ref count b
uffer pool pages
2019-03-08 14:59:39 140597857991808 [Note] InnoDB: The InnoDB memory heap is disabled
2019-03-08 14:59:39 140597857991808 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-03-08 14:59:39 140597857991808 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-03-08 14:59:39 140597857991808 [Note] InnoDB: Compressed tables use zlib 1.2.11
2019-03-08 14:59:39 140597857991808 [Note] InnoDB: Using Linux native AIO
2019-03-08 14:59:39 140597857991808 [Note] InnoDB: Using SSE crc32 instructions
2019-03-08 14:59:39 140597857991808 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2019-03-08 14:59:39 140597857991808 [Note] InnoDB: Completed initialization of buffer pool
2019-03-08 14:59:39 140597857991808 [Note] InnoDB: Highest supported file format is Barracuda.
2019-03-08 14:59:39 140597857991808 [Note] InnoDB: 128 rollback segment(s) are active.
2019-03-08 14:59:39 140597857991808 [Note] InnoDB: Waiting for purge to start
2019-03-08 14:59:39 140597857991808 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 446057526
2019-03-08 14:59:39 140597857991808 [Note] Plugin 'FEEDBACK' is disabled.
2019-03-08 14:59:39 140597201463040 [Note] InnoDB: Dumping buffer pool(s) not yet started
2019-03-08 14:59:39 140597857991808 [Note] Server socket created on IP: '127.0.0.1'.
2019-03-08 14:59:39 140597857991808 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.1.38-MariaDB-0ubuntu0.18.04.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Ubuntu 18.04
2019-03-08 15:01:09 140597856737024 [Note] /usr/sbin/mysqld: Normal shutdown
2019-03-08 15:01:09 140597856737024 [Note] Event Scheduler: Purging the queue. 0 events
2019-03-08 15:01:09 140597251774208 [Note] InnoDB: FTS optimize thread exiting.
2019-03-08 15:01:09 140597856737024 [Note] InnoDB: Starting shutdown...
2019-03-08 15:01:09 140597856737024 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
2019-03-08 15:01:11 140597856737024 [Note] InnoDB: Shutdown completed; log sequence number 446281568
2019-03-08 15:01:11 140597856737024 [Note] /usr/sbin/mysqld: Shutdown complete

Would make a comment if I could, but I decided this is too important to just skip entirely.

Server snapshots and database backups are different things. The problem is that a snapshot might capture the database server in the middle of something; if you later re-start the system based on the snapshot the database server might be confused. Most likely it took a few minutes to trip on the lurking inconsistency and crash. Presumably the re-install indirectly started a more-aggressive-than-usual cleanup, which removed the inconsistency. For details, and to confirm my suppositions, you might try https://dba.stackexchange.com/.

Going forward, it might be better to do regular database backups in addition to the system snapshots. It might also work to put WordPress in read-only mode (which isn't simple, but there's a plugin) while the snapshot is being made. (Though it would be a reasonable to ask whether this would work.)

I still have no idea what the root cause of this problem was, but uninstalling and reinstalling Mariadb fixed the issue.

Specifically I did:

# apt-get remove --purge mariadb-server
# apt-get autoremove --purge
# apt-get autoclean

When prompted, I selected to keep the existing databases.

Then I reinstalled Mariadb

# apt-get install mariadb-server

And after this everything worked and I did not need to restore any databases.

The steps Above Did Not Work

After doing the above, everything would work fine until the system was rebooted or until mariadb-server was restarted. Then the original issue would be present again, and the mariadb server would crash after running normally for a minute or so.

I still don't know what the problem was. I know restoring from the snapshot is what caused the issue, but I don't know why or in what way Mariadb was breaking. I even tried removing mariadb-server and all the databases, but when it was reinstalled I'd have the same problem.

I got tired of trying to hunt down the problem so I dumped the wordpress database, removed mariadb-server and all databases, and installed mysql-server. This has fixed the issue.