Rivervo
← Previous November 2018 – October 2019 Next →
Incident History
September 2019 1 incident
▼
Scheduled maintenance — OPcache and Redis server-level optimisation
10 Sep 2019 hosting servers resolved 30m
▼
Server-wide OPcache tuning applied and Redis object cache daemon installed for shared hosting accounts. Service restart required. Maintenance at night (GMT+4).
22:00 UTC
Maintenance window opened. OPcache and Redis packages installing on server.
22:20 UTC
Services restarting. Brief interruption expected.
22:30 UTC
OPcache and Redis active server-wide. Server response times improved.
July 2019 1 incident
▼
Disk I/O saturation — runaway MySQL queries
17 Jul 2019 hosting servers server load resolved 1h 25m
▼
A runaway database process on one shared account generated excessive disk I/O, saturating the server's I/O queue and degrading all co-hosted accounts. Process terminated and account resource limits tightened.
09:30 UTC
Server disk I/O at 100%. Multiple accounts reporting 504 errors.
10:00 UTC
Offending MySQL process identified and killed. I/O pressure easing.
10:55 UTC
I/O normalised. All accounts responding. Per-account MySQL limits tightened.
June 2019 1 incident
▼
Scheduled maintenance — PHP 7.3 server upgrade
03 Jun 2019 hosting servers resolved 25m
▼
PHP runtime upgraded to 7.3 on the production server. PHP-FPM pools restarted. Maintenance performed at night (GMT+4).
21:30 UTC
PHP 7.3 installation started on server.
21:45 UTC
PHP-FPM pools reloading under new runtime.
21:55 UTC
All accounts confirmed running PHP 7.3. Server stable.
April 2019 1 incident
▼
Scheduled maintenance — Nginx reverse proxy layer added
08 Apr 2019 hosting servers resolved 28m
▼
Nginx installed in front of Apache on the web server to handle static asset serving and reduce PHP-FPM load. Stack restart required. Maintenance performed at night (GMT+4).
22:00 UTC
Maintenance window opened. Nginx packages installed and vhost configuration generated.
22:18 UTC
Nginx active. Static asset routing confirmed. Apache handling PHP requests only.
22:28 UTC
Dual-stack fully operational. Average server response time reduced.
February 2019 1 incident
▼
Apache worker pool exhaustion under load spike
22 Feb 2019 hosting servers server load resolved 1h 50m
▼
Concurrent connection surge exceeded the Apache MPM prefork MaxRequestWorkers ceiling. New requests queued causing elevated response times. MPM settings tuned live to increase worker capacity.
13:00 UTC
Response time degradation across shared hosting. Server load rising.
13:30 UTC
MaxRequestWorkers limit reached. Increasing worker ceiling and tuning KeepAlive settings.
14:50 UTC
Worker limits adjusted. Server load normalised. All accounts responding.
January 2019 1 incident
▼
Scheduled maintenance — PHP 7.2 server upgrade
15 Jan 2019 hosting servers control panel resolved 35m
▼
PHP runtime on the web server upgraded from 7.0 to 7.2. PHP-FPM pools restarted during the upgrade window causing a brief service interruption. Maintenance performed at night (GMT+4).
21:00 UTC
Maintenance window opened. PHP 7.2 packages being installed on server.
21:20 UTC
PHP-FPM pool restart in progress. Brief 502s expected.
21:35 UTC
All PHP-FPM pools confirmed running PHP 7.2. Server healthy.
December 2018 2 incidents
▼
mod_ssl AutoSSL job failure
19 Dec 2018 hosting servers control panel resolved 2h 15m
▼
Misconfigured Let's Encrypt plugin on the web server prevented the AutoSSL cron from issuing certificates for approximately 30 accounts. Each account required manual certificate reissuance via WHM.
14:00 UTC
AutoSSL cron failing silently. Certificate warnings reported.
15:10 UTC
Defective Let's Encrypt plugin identified. Reinstalled and reconfigured on server.
16:15 UTC
AutoSSL operational. All 30 affected accounts reissued certificates.
Nameserver zone propagation failure on new accounts
04 Dec 2018 domain service resolved 5h 20m
▼
Incorrect default TTL values in the server's zone template caused newly created DNS zones to propagate slowly. Existing zones were unaffected. Zone template corrected and affected records force-refreshed.
11:00 UTC
New domain zones not resolving for customers. Server DNS investigated.
12:30 UTC
TTL misconfiguration found in BIND zone template. Correcting all affected zones.
16:20 UTC
Zone template corrected. All newly created records propagating normally.
November 2018 1 incident
▼
Server launch — initial web server instability
26 Nov 2018 hosting servers control panel api resolved 3h 40m
▼
First day of production operation. Apache MaxRequestWorkers and PHP-FPM pool sizes were under-provisioned for live traffic, causing request queuing and intermittent 502 errors during the first hours. Configuration was tuned live.
08:00 UTC
Server goes live. First customer traffic received.
09:15 UTC
Apache worker pool exhausted under concurrent connections. 502 errors observed.
10:00 UTC
MaxRequestWorkers raised, PHP-FPM pool count increased. Services stabilising.
11:40 UTC
Server fully stable. Baseline configuration documented for future scaling.