Pirate Praveen Mon 23 May 2016
I'm adding more disk, but you have to bring down the server for adding new disks.
Also backup was not running :( (cron job was commented out). So I got to add disk to backup server first and initiate rsync again (just to be extra safe, as their shutdown process is not familiar).
Pirate Praveen Mon 23 May 2016
backup.poddery.com took about an hour to shutdown. Added 150GB disk and it is starting up. It will take another hour to come back.
Pirate Praveen Tue 24 May 2016
backup.poddery.com was live at night and enabled cron job for backing up uploaded files (cron job was running when I went to sleep).
All uploaded files are now backed up on backup.poddery.com. An hourly cron job will keep the uploads in sync. Now poddery.com is shut down so we can add 100GB disk to it.
Pirate Praveen Tue 24 May 2016
Additional 100GB disk is attached to poddery.com now. It will take about an hour for the server to come back online.
Pirate Praveen Tue 24 May 2016
Server is not starting, it just throws "starting server (Unable to allocate)". Opened a ticket, waiting for their response.
Pirate Praveen Tue 24 May 2016
The ticket is still not resolved, seems they allocated all their servers.
Pirate Praveen Tue 24 May 2016
Its 7.44 am in Paris, may be it will be looked at in their office hours.
Pirate Praveen Tue 24 May 2016
Frustrating support from scaleway.com. After 4 hours of shutting down, I still can’t boot the machine.
Pirate Praveen Tue 24 May 2016
@tvm @anisha how does http://www.kimsufi.com/en/vps-ssd.xml compare to scaleway? This level of support is frustrating and if we have to go through this again, I'd rather choose a better provider.
Pirate Praveen Tue 24 May 2016
Finally scaleway responded, but we are not able to resolve the issue yet. It seems they don’t have any server which can take two 100GB disks (I don’t know how that can be a problem as these disks are connected as network disks). I’m trying if I can attach these disks onto a new machine. But C2 (baremetal) is out of stock and VC1 (VPS) has limitation on how big the disks can be.
Pirate Praveen Tue 24 May 2016
I assigned poddery.com ip to backup.poddery.com and added a message about down time.
Pirate Praveen Tue 24 May 2016
I'm able to connect the disks to a C1 instance (2GB ARM). I will setup diaspora on this box, it will suffer from performance issues, but that is the only option at this point.
Pirate Praveen Wed 25 May 2016
poddery.com is back. prosody is not configured yet.
Pirate Praveen Wed 25 May 2016
I got C2S instance to start again by adding a single 150 GB disk. I hope it'll last enough till we find a new host. We need to sync backup.poddery.com with the new server. I'll be travelling tomorrow, anyone else wants to do it?
@anisha @tvm @balasankarchelamat @fayadfami ?
You'll need to update ssh known_hosts for the new machine for root (used for rsync of uploads) and sshtunnel user (tunnel established for mysql slave).
Docs are here http://fosscommunity.in/index.php?title=Poddery
I'm planning to add "a fund to support me or others who wants to step in to maintain" part of the next crowd funding. If that does not work out, the other option is to drop poddery.com (hopefully we'll have pod migration by then).
Pirate Praveen Fri 27 May 2016
poddery.com is fully functional again (after a crisis of last 3 days) https://www.loomio.org/d/KbATojC4/image-upload-is-broken
Just remember to update private ip of poddery.com in backup.poddery.com's /etc/hosts file if you reboot poddery.com We are using private ip for backup to save cost of public ip.
Pirate Praveen Sun 29 May 2016
12gb used up in just 2 days, now just 14gb remaining :(
We have to find which user is eating our disk so fast.
Pirate Praveen Sun 29 May 2016
mysql db grew from 21gb to 35gb :( so we need to scale storage upto 6gb per day.
Pirate Praveen Sun 29 May 2016
It was a hidden file left out by an interrupted rsync. We have 27 GB left. Hopefully it'll last for a few more months.
Pirate Praveen Mon 30 May 2016
It was a hidden file left out by an interrupted rsync. We have 27 GB left. Hopefully it'll last for a few more months.
Pirate Praveen Tue 31 May 2016
icu bot is uploading images like crazy, icu account is locked now
mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-30%' and diaspora_handle='
icu@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-----------------+-------+
| diaspora_handle | c |
+-----------------+-------+
| icu@poddery.com | 29618 |
+-----------------+-------+
1 row in set (1.50 sec)
mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-29%' and diaspora_handle='
icu@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-----------------+-------+
| diaspora_handle | c |
+-----------------+-------+
| icu@poddery.com | 28030 |
+-----------------+-------+
1 row in set (1.40 sec)
mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-15%' and diaspora_handle='
icu@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-----------------+------+
| diaspora_handle | c |
+-----------------+------+
| icu@poddery.com | 9017 |
+-----------------+------+
1 row in set (1.72 sec)
mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-01%' and diaspora_handle='
icu@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-----------------+------+
| diaspora_handle | c |
+-----------------+------+
| icu@poddery.com | 4133 |
+-----------------+------+
1 row in set (1.40 sec)
compare that with my account
mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-30%' and diaspora_handle='praveen@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
Empty set (1.30 sec)
mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-29%' and diaspora_handle='praveen@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+---------------------+---+
| diaspora_handle | c |
+---------------------+---+
| praveen@poddery.com | 3 |
+---------------------+---+
1 row in set (1.62 sec)
Pirate Praveen Tue 31 May 2016
To get a feel of how it compares with other users
mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-30%' and diaspora_handle li
ke '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+--------------------+-------+
| diaspora_handle | c |
+--------------------+-------+
| icu@poddery.com | 29618 |
| isabel@poddery.com | 4 |
| lise@poddery.com | 4 |
+--------------------+-------+
3 rows in set (1.88 sec)
mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-29%' and diaspora_handle li
ke '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+------------------------+-------+
| diaspora_handle | c |
+------------------------+-------+
| evilhippie@poddery.com | 3 |
| icu@poddery.com | 28030 |
| praveen@poddery.com | 3 |
| riderplus@poddery.com | 7 |
+------------------------+-------+
4 rows in set (1.79 sec)
mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-28%' and diaspora_handle li
ke '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-------------------------+-------+
| diaspora_handle | c |
+-------------------------+-------+
| anders@poddery.com | 32 |
| commitstrip@poddery.com | 5 |
| icu@poddery.com | 26431 |
| isabel@poddery.com | 3 |
| lise@poddery.com | 3 |
| praveen@poddery.com | 3 |
| the_gog@poddery.com | 3 |
+-------------------------+-------+
7 rows in set (2.03 sec)
mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-27%' and diaspora_handle like '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+----------------------------+------+
| diaspora_handle | c |
+----------------------------+------+
| aster@poddery.com | 2 |
| icu@poddery.com | 9802 |
| katharsisdrill@poddery.com | 2 |
| lise@poddery.com | 5 |
| modesty@poddery.com | 5 |
| neomatia@poddery.com | 2 |
+----------------------------+------+
6 rows in set (1.84 sec)
mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-26%' and diaspora_handle like '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+------------------------+------+
| diaspora_handle | c |
+------------------------+------+
| anders@poddery.com | 22 |
| icu@poddery.com | 5462 |
| lise@poddery.com | 6 |
| mataperros@poddery.com | 3 |
+------------------------+------+
4 rows in set (1.63 sec)
Pirate Praveen Wed 1 Jun 2016
Thanks to @mintojoseph for helping track this issue.
Pirate Praveen · Mon 23 May 2016
Disk is full :(