Image upload is broken

Disk is full :(

I'm adding more disk, but you have to bring down the server for adding new disks.

Also backup was not running :( (cron job was commented out). So I got to add disk to backup server first and initiate rsync again (just to be extra safe, as their shutdown process is not familiar).

backup.poddery.com took about an hour to shutdown. Added 150GB disk and it is starting up. It will take another hour to come back.

backup.poddery.com was live at night and enabled cron job for backing up uploaded files (cron job was running when I went to sleep).

All uploaded files are now backed up on backup.poddery.com. An hourly cron job will keep the uploads in sync. Now poddery.com is shut down so we can add 100GB disk to it.

Additional 100GB disk is attached to poddery.com now. It will take about an hour for the server to come back online.

Server is not starting, it just throws "starting server (Unable to allocate)". Opened a ticket, waiting for their response.

The ticket is still not resolved, seems they allocated all their servers.

Its 7.44 am in Paris, may be it will be looked at in their office hours.

Frustrating support from scaleway.com. After 4 hours of shutting down, I still can’t boot the machine.

@tvm @anisha how does http://www.kimsufi.com/en/vps-ssd.xml compare to scaleway? This level of support is frustrating and if we have to go through this again, I'd rather choose a better provider.

Finally scaleway responded, but we are not able to resolve the issue yet. It seems they don’t have any server which can take two 100GB disks (I don’t know how that can be a problem as these disks are connected as network disks). I’m trying if I can attach these disks onto a new machine. But C2 (baremetal) is out of stock and VC1 (VPS) has limitation on how big the disks can be.

I assigned poddery.com ip to backup.poddery.com and added a message about down time.

I'm able to connect the disks to a C1 instance (2GB ARM). I will setup diaspora on this box, it will suffer from performance issues, but that is the only option at this point.

poddery.com is back. prosody is not configured yet.

I got C2S instance to start again by adding a single 150 GB disk. I hope it'll last enough till we find a new host. We need to sync backup.poddery.com with the new server. I'll be travelling tomorrow, anyone else wants to do it?

@anisha @tvm @balasankarchelamat @fayadfami ?

You'll need to update ssh known_hosts for the new machine for root (used for rsync of uploads) and sshtunnel user (tunnel established for mysql slave).

Docs are here http://fosscommunity.in/index.php?title=Poddery

I'm planning to add "a fund to support me or others who wants to step in to maintain" part of the next crowd funding. If that does not work out, the other option is to drop poddery.com (hopefully we'll have pod migration by then).

poddery.com is fully functional again (after a crisis of last 3 days) https://www.loomio.org/d/KbATojC4/image-upload-is-broken

Just remember to update private ip of poddery.com in backup.poddery.com's /etc/hosts file if you reboot poddery.com We are using private ip for backup to save cost of public ip.

12gb used up in just 2 days, now just 14gb remaining :( ​

We have to find which user is eating our disk so fast.

mysql db grew from 21gb to 35gb :( so we need to scale storage upto 6gb per day.

It was a hidden file left out by an interrupted rsync. We have 27 GB left. Hopefully it'll last for a few more months.

It was a hidden file left out by an interrupted rsync. We have 27 GB left. Hopefully it'll last for a few more months.

icu bot is uploading images like crazy, icu account is locked now

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-30%' and  diaspora_handle='
icu@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-----------------+-------+
| diaspora_handle | c     |
+-----------------+-------+
| icu@poddery.com | 29618 |
+-----------------+-------+
1 row in set (1.50 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-29%' and  diaspora_handle='
icu@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-----------------+-------+
| diaspora_handle | c     |
+-----------------+-------+
| icu@poddery.com | 28030 |
+-----------------+-------+
1 row in set (1.40 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-15%' and  diaspora_handle='
icu@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-----------------+------+
| diaspora_handle | c    |
+-----------------+------+
| icu@poddery.com | 9017 |
+-----------------+------+
1 row in set (1.72 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-01%' and  diaspora_handle='
icu@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-----------------+------+
| diaspora_handle | c    |
+-----------------+------+
| icu@poddery.com | 4133 |
+-----------------+------+
1 row in set (1.40 sec)

compare that with my account

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-30%' and  diaspora_handle='praveen@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
Empty set (1.30 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-29%' and  diaspora_handle='praveen@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+---------------------+---+
| diaspora_handle     | c |
+---------------------+---+
| praveen@poddery.com | 3 |
+---------------------+---+
1 row in set (1.62 sec)

To get a feel of how it compares with other users

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-30%' and diaspora_handle li
ke '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+--------------------+-------+
| diaspora_handle    | c     |
+--------------------+-------+
| icu@poddery.com    | 29618 |
| isabel@poddery.com |     4 |
| lise@poddery.com   |     4 |
+--------------------+-------+
3 rows in set (1.88 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-29%' and diaspora_handle li
ke '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+------------------------+-------+
| diaspora_handle        | c     |
+------------------------+-------+
| evilhippie@poddery.com |     3 |
| icu@poddery.com        | 28030 |
| praveen@poddery.com    |     3 |
| riderplus@poddery.com  |     7 |
+------------------------+-------+
4 rows in set (1.79 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-28%' and diaspora_handle li
ke '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-------------------------+-------+
| diaspora_handle         | c     |
+-------------------------+-------+
| anders@poddery.com      |    32 |
| commitstrip@poddery.com |     5 |
| icu@poddery.com         | 26431 |
| isabel@poddery.com      |     3 |
| lise@poddery.com        |     3 |
| praveen@poddery.com     |     3 |
| the_gog@poddery.com     |     3 |
+-------------------------+-------+
7 rows in set (2.03 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-27%' and diaspora_handle like '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+----------------------------+------+
| diaspora_handle            | c    |
+----------------------------+------+
| aster@poddery.com          |    2 |
| icu@poddery.com            | 9802 |
| katharsisdrill@poddery.com |    2 |
| lise@poddery.com           |    5 |
| modesty@poddery.com        |    5 |
| neomatia@poddery.com       |    2 |
+----------------------------+------+
6 rows in set (1.84 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-26%' and diaspora_handle like '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+------------------------+------+
| diaspora_handle        | c    |
+------------------------+------+
| anders@poddery.com     |   22 |
| icu@poddery.com        | 5462 |
| lise@poddery.com       |    6 |
| mataperros@poddery.com |    3 |
+------------------------+------+
4 rows in set (1.63 sec)

Thanks to @mintojoseph for helping track this issue.