Loomio

Image upload is broken

Pirate Praveen
Pirate Praveen Public Seen by 289

Can anyone check?

Pirate Praveen

Pirate Praveen May 23rd, 2016 14:17

Disk is full :(

Pirate Praveen

Pirate Praveen May 23rd, 2016 16:38

I'm adding more disk, but you have to bring down the server for adding new disks.

Also backup was not running :( (cron job was commented out). So I got to add disk to backup server first and initiate rsync again (just to be extra safe, as their shutdown process is not familiar).

Pirate Praveen

Pirate Praveen May 23rd, 2016 17:34

backup.poddery.com took about an hour to shutdown. Added 150GB disk and it is starting up. It will take another hour to come back.

Pirate Praveen

Pirate Praveen May 24th, 2016 03:35

backup.poddery.com was live at night and enabled cron job for backing up uploaded files (cron job was running when I went to sleep).

All uploaded files are now backed up on backup.poddery.com. An hourly cron job will keep the uploads in sync. Now poddery.com is shut down so we can add 100GB disk to it.

Pirate Praveen

Pirate Praveen May 24th, 2016 04:05

Additional 100GB disk is attached to poddery.com now. It will take about an hour for the server to come back online.

Pirate Praveen

Pirate Praveen May 24th, 2016 04:27

Server is not starting, it just throws "starting server (Unable to allocate)". Opened a ticket, waiting for their response.

Pirate Praveen

Pirate Praveen May 24th, 2016 05:18

The ticket is still not resolved, seems they allocated all their servers.

Pirate Praveen

Pirate Praveen May 24th, 2016 05:45

Its 7.44 am in Paris, may be it will be looked at in their office hours.

Pirate Praveen

Pirate Praveen May 24th, 2016 07:08

Frustrating support from scaleway.com. After 4 hours of shutting down, I still can’t boot the machine.

Pirate Praveen

Pirate Praveen May 24th, 2016 09:01

@tvm @anisha how does http://www.kimsufi.com/en/vps-ssd.xml compare to scaleway? This level of support is frustrating and if we have to go through this again, I'd rather choose a better provider.

Pirate Praveen

Pirate Praveen May 24th, 2016 12:02

Finally scaleway responded, but we are not able to resolve the issue yet. It seems they don’t have any server which can take two 100GB disks (I don’t know how that can be a problem as these disks are connected as network disks). I’m trying if I can attach these disks onto a new machine. But C2 (baremetal) is out of stock and VC1 (VPS) has limitation on how big the disks can be.

Pirate Praveen

Pirate Praveen May 24th, 2016 14:14

I assigned poddery.com ip to backup.poddery.com and added a message about down time.

Pirate Praveen

Pirate Praveen May 24th, 2016 17:16

I'm able to connect the disks to a C1 instance (2GB ARM). I will setup diaspora on this box, it will suffer from performance issues, but that is the only option at this point.

Pirate Praveen

Pirate Praveen May 25th, 2016 06:49

poddery.com is back. prosody is not configured yet.

Pirate Praveen

Pirate Praveen May 25th, 2016 14:43

I got C2S instance to start again by adding a single 150 GB disk. I hope it'll last enough till we find a new host. We need to sync backup.poddery.com with the new server. I'll be travelling tomorrow, anyone else wants to do it?

@anisha @tvm @balasankarchelamat @fayadfami ?

You'll need to update ssh known_hosts for the new machine for root (used for rsync of uploads) and sshtunnel user (tunnel established for mysql slave).

Docs are here http://fosscommunity.in/index.php?title=Poddery

I'm planning to add "a fund to support me or others who wants to step in to maintain" part of the next crowd funding. If that does not work out, the other option is to drop poddery.com (hopefully we'll have pod migration by then).

Pirate Praveen

Pirate Praveen May 27th, 2016 16:04

poddery.com is fully functional again (after a crisis of last 3 days) https://www.loomio.org/d/KbATojC4/image-upload-is-broken

Just remember to update private ip of poddery.com in backup.poddery.com's /etc/hosts file if you reboot poddery.com We are using private ip for backup to save cost of public ip.

Pirate Praveen

Pirate Praveen May 29th, 2016 12:08

12gb used up in just 2 days, now just 14gb remaining :( ​

We have to find which user is eating our disk so fast.

Pirate Praveen

Pirate Praveen May 29th, 2016 15:12

mysql db grew from 21gb to 35gb :( so we need to scale storage upto 6gb per day.

Pirate Praveen

Pirate Praveen May 29th, 2016 16:28

It was a hidden file left out by an interrupted rsync. We have 27 GB left. Hopefully it'll last for a few more months.

Pirate Praveen

Pirate Praveen May 30th, 2016 06:40

It was a hidden file left out by an interrupted rsync. We have 27 GB left. Hopefully it'll last for a few more months.

Pirate Praveen

Pirate Praveen May 31st, 2016 16:11

icu bot is uploading images like crazy, icu account is locked now

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-30%' and  diaspora_handle='
icu@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-----------------+-------+
| diaspora_handle | c     |
+-----------------+-------+
| icu@poddery.com | 29618 |
+-----------------+-------+
1 row in set (1.50 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-29%' and  diaspora_handle='
icu@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-----------------+-------+
| diaspora_handle | c     |
+-----------------+-------+
| icu@poddery.com | 28030 |
+-----------------+-------+
1 row in set (1.40 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-15%' and  diaspora_handle='
icu@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-----------------+------+
| diaspora_handle | c    |
+-----------------+------+
| icu@poddery.com | 9017 |
+-----------------+------+
1 row in set (1.72 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-01%' and  diaspora_handle='
icu@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-----------------+------+
| diaspora_handle | c    |
+-----------------+------+
| icu@poddery.com | 4133 |
+-----------------+------+
1 row in set (1.40 sec)


compare that with my account

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-30%' and  diaspora_handle='praveen@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
Empty set (1.30 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-29%' and  diaspora_handle='praveen@poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+---------------------+---+
| diaspora_handle     | c |
+---------------------+---+
| praveen@poddery.com | 3 |
+---------------------+---+
1 row in set (1.62 sec)

Pirate Praveen

Pirate Praveen May 31st, 2016 16:21

To get a feel of how it compares with other users

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-30%' and diaspora_handle li
ke '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+--------------------+-------+
| diaspora_handle    | c     |
+--------------------+-------+
| icu@poddery.com    | 29618 |
| isabel@poddery.com |     4 |
| lise@poddery.com   |     4 |
+--------------------+-------+
3 rows in set (1.88 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-29%' and diaspora_handle li
ke '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+------------------------+-------+
| diaspora_handle        | c     |
+------------------------+-------+
| evilhippie@poddery.com |     3 |
| icu@poddery.com        | 28030 |
| praveen@poddery.com    |     3 |
| riderplus@poddery.com  |     7 |
+------------------------+-------+
4 rows in set (1.79 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-28%' and diaspora_handle li
ke '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+-------------------------+-------+
| diaspora_handle         | c     |
+-------------------------+-------+
| anders@poddery.com      |    32 |
| commitstrip@poddery.com |     5 |
| icu@poddery.com         | 26431 |
| isabel@poddery.com      |     3 |
| lise@poddery.com        |     3 |
| praveen@poddery.com     |     3 |
| the_gog@poddery.com     |     3 |
+-------------------------+-------+
7 rows in set (2.03 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-27%' and diaspora_handle like '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+----------------------------+------+
| diaspora_handle            | c    |
+----------------------------+------+
| aster@poddery.com          |    2 |
| icu@poddery.com            | 9802 |
| katharsisdrill@poddery.com |    2 |
| lise@poddery.com           |    5 |
| modesty@poddery.com        |    5 |
| neomatia@poddery.com       |    2 |
+----------------------------+------+
6 rows in set (1.84 sec)

mysql> SELECT diaspora_handle, COUNT(*) c FROM photos where created_at like '2016-05-26%' and diaspora_handle like '%poddery.com' GROUP BY diaspora_handle HAVING c > 1;
+------------------------+------+
| diaspora_handle        | c    |
+------------------------+------+
| anders@poddery.com     |   22 |
| icu@poddery.com        | 5462 |
| lise@poddery.com       |    6 |
| mataperros@poddery.com |    3 |
+------------------------+------+
4 rows in set (1.63 sec)

Pirate Praveen

Pirate Praveen June 1st, 2016 08:41

Thanks to @mintojoseph for helping track this issue.