Loomio

Image upload is broken

PP
Pirate Praveen Public Seen by 379

Can anyone check?

PP

Pirate Praveen Mon 23 May 2016

Disk is full :(

PP

Pirate Praveen Mon 23 May 2016

I'm adding more disk, but you have to bring down the server for adding new disks.

Also backup was not running :( (cron job was commented out). So I got to add disk to backup server first and initiate rsync again (just to be extra safe, as their shutdown process is not familiar).

PP

Pirate Praveen Mon 23 May 2016

backup.poddery.com took about an hour to shutdown. Added 150GB disk and it is starting up. It will take another hour to come back.

PP

Pirate Praveen Tue 24 May 2016

backup.poddery.com was live at night and enabled cron job for backing up uploaded files (cron job was running when I went to sleep).

All uploaded files are now backed up on backup.poddery.com. An hourly cron job will keep the uploads in sync. Now poddery.com is shut down so we can add 100GB disk to it.

PP

Pirate Praveen Tue 24 May 2016

Additional 100GB disk is attached to poddery.com now. It will take about an hour for the server to come back online.

PP

Pirate Praveen Tue 24 May 2016

Server is not starting, it just throws "starting server (Unable to allocate)". Opened a ticket, waiting for their response.

PP

Pirate Praveen Tue 24 May 2016

The ticket is still not resolved, seems they allocated all their servers.

PP

Pirate Praveen Tue 24 May 2016

Its 7.44 am in Paris, may be it will be looked at in their office hours.

PP

Pirate Praveen Tue 24 May 2016

Frustrating support from scaleway.com. After 4 hours of shutting down, I still can’t boot the machine.

PP

Pirate Praveen Tue 24 May 2016

@tvm @anisha how does http://www.kimsufi.com/en/vps-ssd.xml compare to scaleway? This level of support is frustrating and if we have to go through this again, I'd rather choose a better provider.

PP

Pirate Praveen Tue 24 May 2016

Finally scaleway responded, but we are not able to resolve the issue yet. It seems they don’t have any server which can take two 100GB disks (I don’t know how that can be a problem as these disks are connected as network disks). I’m trying if I can attach these disks onto a new machine. But C2 (baremetal) is out of stock and VC1 (VPS) has limitation on how big the disks can be.

PP

Pirate Praveen Tue 24 May 2016

I assigned poddery.com ip to backup.poddery.com and added a message about down time.

PP

Pirate Praveen Tue 24 May 2016

I'm able to connect the disks to a C1 instance (2GB ARM). I will setup diaspora on this box, it will suffer from performance issues, but that is the only option at this point.

PP

Pirate Praveen Wed 25 May 2016

poddery.com is back. prosody is not configured yet.

PP

Pirate Praveen Wed 25 May 2016

I got C2S instance to start again by adding a single 150 GB disk. I hope it'll last enough till we find a new host. We need to sync backup.poddery.com with the new server. I'll be travelling tomorrow, anyone else wants to do it?

@anisha @tvm @balasankarchelamat @fayadfami ?

You'll need to update ssh known_hosts for the new machine for root (used for rsync of uploads) and sshtunnel user (tunnel established for mysql slave).

Docs are here http://fosscommunity.in/index.php?title=Poddery

I'm planning to add "a fund to support me or others who wants to step in to maintain" part of the next crowd funding. If that does not work out, the other option is to drop poddery.com (hopefully we'll have pod migration by then).

PP

Pirate Praveen Fri 27 May 2016

poddery.com is fully functional again (after a crisis of last 3 days) https://www.loomio.org/d/KbATojC4/image-upload-is-broken

Just remember to update private ip of poddery.com in backup.poddery.com's /etc/hosts file if you reboot poddery.com We are using private ip for backup to save cost of public ip.

PP

Pirate Praveen Sun 29 May 2016

12gb used up in just 2 days, now just 14gb remaining :( ​

We have to find which user is eating our disk so fast.

PP

Pirate Praveen Sun 29 May 2016

mysql db grew from 21gb to 35gb :( so we need to scale storage upto 6gb per day.

PP

Pirate Praveen Sun 29 May 2016

It was a hidden file left out by an interrupted rsync. We have 27 GB left. Hopefully it'll last for a few more months.

PP

Pirate Praveen Mon 30 May 2016

It was a hidden file left out by an interrupted rsync. We have 27 GB left. Hopefully it'll last for a few more months.

PP

Pirate Praveen Tue 31 May 2016

icu bot is uploading images like crazy, icu account is locked now

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-30%' and  diaspora_handle='
icu@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-----------------+-------+
| diaspora_handle | c     |
+-----------------+-------+
| icu@poddery.com | 29618 |
+-----------------+-------+
1 row in set (1.50 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-29%' and  diaspora_handle='
icu@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-----------------+-------+
| diaspora_handle | c     |
+-----------------+-------+
| icu@poddery.com | 28030 |
+-----------------+-------+
1 row in set (1.40 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-15%' and  diaspora_handle='
icu@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-----------------+------+
| diaspora_handle | c    |
+-----------------+------+
| icu@poddery.com | 9017 |
+-----------------+------+
1 row in set (1.72 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-01%' and  diaspora_handle='
icu@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-----------------+------+
| diaspora_handle | c    |
+-----------------+------+
| icu@poddery.com | 4133 |
+-----------------+------+
1 row in set (1.40 sec)


compare that with my account

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-30%' and  diaspora_handle='praveen@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
Empty set (1.30 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-29%' and  diaspora_handle='praveen@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+---------------------+---+
| diaspora_handle     | c |
+---------------------+---+
| praveen@poddery.com | 3 |
+---------------------+---+
1 row in set (1.62 sec)

PP

Pirate Praveen Tue 31 May 2016

To get a feel of how it compares with other users

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-30%' and diaspora_handle li
ke '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+--------------------+-------+
| diaspora_handle    | c     |
+--------------------+-------+
| icu@poddery.com    | 29618 |
| isabel@poddery.com |     4 |
| lise@poddery.com   |     4 |
+--------------------+-------+
3 rows in set (1.88 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-29%' and diaspora_handle li
ke '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+------------------------+-------+
| diaspora_handle        | c     |
+------------------------+-------+
| evilhippie@poddery.com |     3 |
| icu@poddery.com        | 28030 |
| praveen@poddery.com    |     3 |
| riderplus@poddery.com  |     7 |
+------------------------+-------+
4 rows in set (1.79 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-28%' and diaspora_handle li
ke '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-------------------------+-------+
| diaspora_handle         | c     |
+-------------------------+-------+
| anders@poddery.com      |    32 |
| commitstrip@poddery.com |     5 |
| icu@poddery.com         | 26431 |
| isabel@poddery.com      |     3 |
| lise@poddery.com        |     3 |
| praveen@poddery.com     |     3 |
| the_gog@poddery.com     |     3 |
+-------------------------+-------+
7 rows in set (2.03 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-27%' and diaspora_handle like '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+----------------------------+------+
| diaspora_handle            | c    |
+----------------------------+------+
| aster@poddery.com          |    2 |
| icu@poddery.com            | 9802 |
| katharsisdrill@poddery.com |    2 |
| lise@poddery.com           |    5 |
| modesty@poddery.com        |    5 |
| neomatia@poddery.com       |    2 |
+----------------------------+------+
6 rows in set (1.84 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-26%' and diaspora_handle like '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+------------------------+------+
| diaspora_handle        | c    |
+------------------------+------+
| anders@poddery.com     |   22 |
| icu@poddery.com        | 5462 |
| lise@poddery.com       |    6 |
| mataperros@poddery.com |    3 |
+------------------------+------+
4 rows in set (1.63 sec)

PP

Pirate Praveen Wed 1 Jun 2016

Thanks to @mintojoseph for helping track this issue.