- Innodb table can crash if there is power lose for your dedicated server.
- Force shutdown of MySQL service can end up in Innodb crash.
- Space issue on the serve can case Innodb crash, if the server space is 100% filled and mysql not able to write on the disk.
- High load on the server can cause Innodb crash.
- High I/O wait on the disk can also cause MySQL innodb crash.

How to determine if innodb is crashed.?
You can determine this by checking MySQL logs, On most of the servers location of the Log is /var/lib/mysql/HOSTNAME.err
You can also use commands:

systemctl status mariadb.service
journalctl -xe

Full MySQL backup
Backup MySQL users and permissions

pt-show-grants > /root/pt-show-grants.sql

Backup Full MySQL folder, Make sure MySQL is STOPPED for this step

rsync -a /var/lib/mysql /home/mysql_bkp/ 

Instructions for full mysql reinstall and recovery
If you can't start mysql, you can try by adding the following code in file /etc/my.cnf.d/server.cnf under [galera] settings

innodb_force_recovery = 1

Now try to start/restart MariaDB

systemctl restart mariadb.service

* If started successfully then you can start with backup, please note that MySQL is now in read-only mode.
* If not starting then try increasing recovery number for one number in each test

innodb_force_recovery = 2

innodb_force_recovery = 3

innodb_force_recovery = 4

innodb_force_recovery = 5

innodb_force_recovery = 6

When the MySQL is started in recovery mode we need to take backup of all the crashed databases and restore it

Create backup location folders

mkdir /home/mysql_innodb_recovery
mkdir /home/mysql_innodb_recovery/database_backup

Create the list of databases

mysql -e 'show databases;' | grep -v information_schema | grep -v Database  > /home/mysql_innodb_recovery/database_list.txt

* Please confirm that all databases are listed in the file /home/mysql_innodb_recovery/database_list.txt

Make backup of each databases including the database called mysql which is very important.

for db in `cat /home/mysql_innodb_recovery/database_list.txt`; do mysqldump $db > /home/mysql_innodb_recovery/database_backup/$db.sql;done

* this process can take a lot of time depending on your MySQL disk usage.
* Make sure you confirm that all .sql files in folder /home/mysql_innodb_recovery/database_backup/ are not empty.

Drop current databases (We can remove them if the backup was successful)

for db in `cat /home/mysql_innodb_recovery/database_list.txt`; do mysqladmin drop $db;done

Some database won’t get dropped which can be directly removed using the rm -Rf command

rm -Rf /var/lib/mysql/DATABASENAME

Next is to remove ibdata and ib_log files

rm -f ibdata* ib_logfile*

Restarting MariaDB
Remove recovery line innodb_force_recovery from /etc/my.cnf.d/server.cnf and restart MariaDB

systemctl restart mariadb.service

Creating Databases
If the MySQL now works we can start restoring databases, first create databases

for db in `cat /home/mysql_innodb_recovery/database_list.txt`; do mysqladmin create $db;done

Restoring Databases
Now we will restore the backups we have taken.

for db in `cat /home/mysql_innodb_recovery/database_list.txt`; do mysqldump $db < /home/mysql_innodb_recovery/database_backup/$db.sql;done

* This process will take some time, after it is done confirm that there are no any issues in the MySQL logs.

In case of some issues you still have the exact replica of databases in folder /home/mysql_bkp/
MySQL users and permissions are stored in file /root/pt-show-grants.sql
Individual databases backups are stored in .sql files in folder /home/mysql_innodb_recovery/database_backup/

* We suggest to keep all those files few days after the successful restore.

Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=stock error was
14: curl#6 - "Could not resolve host: mirrorlist.centos.org; Name or service not known"

On July 1, 2024, CentOS 7 reached end of life, and the CentOS team has moved its repositories to the archive at vault.centos.org. Without updating the repository URLs, packages cannot be updated or validated, resulting in these errors.

Solution, run script:

/scripts/centos7_fix_repository

[core:emerg] [pid 2013449:tid 140639997298560] (28)No space left on device: AH00023: Couldn't create the proxy mutex
[proxy:crit] [pid 2013449:tid 140639997298560] (28)No space left on device: AH02478: failed to create proxy mutex

In most cases this error message will not allow you to start apache, solution is to restart the server or use the command bellow.

Solution:

ipcs -s | awk -v user=nobody '$3==user {system("ipcrm -s "$2)}'

Example Error message:

PHP Fatal error: 
The encoded file /usr/local/cwpsrv/htdocs/resources/admin/include/cron.php has expired.
 in Unknown on line 0

The solution is to manually update CWP from the bash shell.

cd /usr/local/cwpsrv/htdocs
chattr -i -R /usr/local/cwpsrv/htdocs/admin
wget static.cdn-cwp.com/files/cwp/el7/cwp-el7-0.9.8.1160.zip
unzip -o -q cwp-el7-0.9.8.1160.zip
rm -f cwp-el7-0.9.8.1160.zip

cd /usr/local/cwpsrv/var/services/
wget static.cdn-cwp.com/files/cwp/el7/cwp-services.zip
unzip -o -q cwp-services.zip
rm -f cwp-services.zip

Don't forget to change the version from this example 0.9.8.1160 to the latest version which you can find on this link:

https://control-webpanel.com/changelog

Make sure that you check after this if your automatic updates work.

First stop the future updates via yum/dnf package manager, if you skip this step then after each update of curl from the base repo you need to rebuild curl

Centos 7 /EL7 :

cat /etc/yum.conf |grep "^exclude="|grep kernel 1> /dev/null 2> /dev/null || echo 'exclude=curl* libcurl*' >> /etc/yum.conf

Centos 8 stream /EL8 :

cat /etc/dnf.conf |grep "^exclude="|grep kernel 1> /dev/null 2> /dev/null || echo 'exclude=curl* libcurl*' >> /etc/dnf.conf

Now you need to install dependencies to build curl and libcurl:

Centos 7 /EL7 :

yum install libssh libssh-devel libnghttp2-devel libnghttp2 libgsasl libgsasl-devel zstd libzstd-devel libzstd brotli brotli-devel libbrotli 

Centos 8 stream /EL8 :

dnf install libssh libssh-devel libnghttp2-devel libnghttp2 libgsasl libgsasl-devel zstd libzstd-devel libzstd brotli brotli-devel libbrotli 

After you've installed the dependencies build CURL from the source :

cd /usr/local/src
rm -rf curl*
wget https://curl.se/download/curl-7.88.0.zip
unzip curl-7.88.0.zip
cd curl-7.88.0
./configure --with-ssl --with-zlib --with-gssapi --enable-ldap --enable-ldaps --with-libssh --with-nghttp2
make
make install

After the successful build check the cURL version :

curl -V

To update Netdata on EL 7/8 simply run these commands :

yum install lz4-devel lz4 json-c-devel libuv-devel libuv --nobest

First

Ensure and check that you've wildcard DNS added to your current DNS hosting provider or in CWP DNS if you're using your own nameservers :

example DNS entry:-

*.domain.tld. 14400 IN A 192.168.1.100

Second

Go to CWP-admin dashboard >> Webserver Settings >> Webservers Domain Settings

there you need to choose the user and then the domain name where you need to set the wildcard subdomain support.

Third

Click on the button "Create configuration" which is located next to the domain name for which you need to enable the wildcard subdomain

Fourth

Choose "Nginx + Php-fpm" or "apache + php-fpm" or "nginx + apache + php-fpm" or "nginx + apache + php-cgi" according to your choice of engines.

If you've chosen "nginx + php-fpm" select the "wildcard" template under "Nginx default vhost template" and save the changes by click on rebuild vhost option.

If "apache + php-fpm" is chosen select "wildcard" template under "Apache default vhost template" and save the changes by click on rebuild vhost option.

If "nginx + apache + php-fpm" or "nginx + apache + php-cgi" is chosen select the "wildcard" template under "Nginx default vhost template" also select "wildcard" template under "Apache default vhost template" and save the changes by click on rebuild vhost option.

That's it you've easily enabled wildcard subdomain support for your domain

**SAME CAN BE DONE FOR SUB-DOMAIN

Admin Panel
If your root password is not working then the only option is to reset it via shell using passwd command.

If you also don't have shell/ssh/console access then you would need to request your hosting provider to recover the root password or try to recover it by yourself using a rescue boot.

1st case :

you've low on RAM we recommend adding at least 1.5-2 Gb of available ram to compile PHP 7.xx and 8.xx as the compiler needs more memory for this version of PHP

you can check if you've sufficient RAM by this command :

free -h

example output :

root@server:~# free -h
               total        used        free      shared  buff/cache   available
Mem:           1.9Gi       328Mi        90Mi        56Mi       1.4Gi       1.5Gi
Swap:          2.0Gi        38Mi       1.9Gi

here you can see Mem which indicates the total RAM and you need to check the `available` memory which should be more than 1.5Gb listed

Also for low memory servers, we always recommend adding at least 2Gb of Swap memory to prevent OOM (Out of memory) incidents.

2nd Case :

You're using an outdated OS that is already ON EOL period

Like Centos 6 and Centos 8 which are already in the EOL period and many repositories are disabled in this OS and you may experience non-availability of packages and dependencies which are required to build the PHP.

We always recommend upgrading you to Following OS :

Centos 7
Centos 8 stream
AlmaLinux 8
Rocky Linux 8
Oracle Linux 7 and 8

3rd Case :

Internet issue/Resolver issue

We've seen some cases where the OS resolver is not correctly set and many packages are not downloading due to the resolver failing.
Ensure you have the correct resolver added in /etc/resolv.conf

you can add this lines at the end of the file in /etc/resolv.conf :

nameserver 1.1.1.1
nameserver 8.8.8.8

* Your server provider might also have some custom resolvers

Now lock the file :

chattr +i /etc/resolv.conf

The next step is to reboot the server :

reboot

Make sure you check the logs as it can be also any other reason

Verify if MariaDB 10.2 is installed

rpm -qa|grep -i maria|grep "\-10.2."

First you need to change the MariaDB repo and replace it with MariaDB 10.4 repo:

sed -i 's/10.2/10.4/g' /etc/yum.repos.d/mariadb.repo

Second remove MariaDB 10.2 :

systemctl stop mariadb mysql mysqld
systemctl disable mariadb
rpm --nodeps -ev MariaDB-server

Third Install MariaDB 10.4 :

yum clean all
yum -y update "MariaDB-*"
yum -y install MariaDB-server
systemctl enable mariadb
systemctl start mariadb

Fourth you need to upgrade your database tables to the latest version:

mysql_upgrade --force

After upgrade, you can use "mysql" command to verify the MariaDB version running on your server

mysql --version