Loomio
Thu 23 Nov 2017

Comment/Feedback section in article page

D
David Public Seen by 456

One of the main advantages of posting a preprint is the possibility of receiving early feedback. Depending on readers interest/expertise on the subject matter, this could be as relevant as a formal peer-review. Of course, this could be made by personal email, but I think I'll be more useful if this could happen as well in a Comment/Feedback section associated with each pre- (or even post-) print.

Does this make sense for you as well?

JM

Jon Mound Mon 27 Nov 2017

Stéphanie has pretty well summarised my concerns with allowing comments. In principle, it sounds good, but in practice there are a lot of potential pitfalls. I think there is a big difference between a pre-print server and an open review / open discussion journal. Such journals are definitely valuable, but is that what EarthArXiv wants to be? If so, that is a whole other undertaking and not a trivial one.

My inclination is to stay centred on the core idea of being a pre-print server.

A

Anson Thu 23 Nov 2017

Possibly a good idea, but would it mean then that comments would have to be moderated? I wonder what more established pre-print servers do?

D

David Thu 23 Nov 2017

Good point. I don't know what other preprint means do, but I think one can avoid the need to moderate comments if only registered users can post. Everyone is entitled to their opinions and if they are unfounded or aggressive, they will be bounced off by other users. Personally, I think the community should move to an entirely open and accessible review-rebuttal process, where hidden interests cannot be hidden anymore - this might be a step in that direction!

HG

Han Geurdes Thu 23 Nov 2017

David, I agree. From my experience with hot headed debates on pubpeer, moderation (of the discussion tone of voice helpfulness of the remarks etc etc) would be a very good idea. A shameful example of a derailed discussion can be found on FQXi between Joy Christian and Richard Gill.

D

David Thu 23 Nov 2017

Ohp, I see... Flying knives over there! Well, it seems like moderation is the way to go if comments are allowed - still I think comments should be implemented

HG

Han Geurdes Thu 23 Nov 2017

Indeed, discussion moderation & actual removal from discussion or cooling down period when the (neutral ... is difficult) moderator thinks fit.

B

brandon Thu 23 Nov 2017

This is an interesting idea. I don't think this is a trivial thing. Who is going to moderate? Are they volunteers, or paid site moderators...or primarily bots?

It may make more sense, at least for the time being, to use existing infrastructure for reviews and comments. For example, I could post a summary of an EarthArXiv article on Medium (or any blogging type site) where comment functionality is already integrated. In this scenario, what is important are the links. Obviously a link from the Medium article/post to the EarthArXiv preprint (I assume this would use the DOI) would be useful. However, a link from EarthArXiv to the discussion area (i.e. further discussion here: --link to Medium post--) would also seem appropriate.

Does this make sense to anyone else?

CJ

Christopher Jackson Thu 23 Nov 2017

Useful indeed. And I'm fully supportive. @leonardouieda was looking into Authorea(?), which I think could provide this functionality. Howeverm, I recall that Hypothes.is may be another option, with someone called Heather Staines telling me they had had some discussions with OSF about this. Maybe you can confirm @mattspitzer?

MS

Matt Spitzer Thu 23 Nov 2017

@christopherjackson3 Yes, we are actively exploring a hypothes.is integration right now. we should have some estimates of when this would be available soon. You are also correct that moderation of comments is not insignificant and this is something that I know bioarxiv spends ample time on. Alternatively, arvix.org does not have commenting on preprints. So I envision that any integrated commenting engine will be optional for OSF hosted services, for as others point out, commenting through an external service with two-way links could be ideal for groups not interested in providing moderation on comments directly.

LU

Leonardo Uieda Tue 5 Dec 2017

I thought Jon had taken that over. If not, I can contact them.

CJ

Christopher Jackson Thu 23 Nov 2017

I stumbled on this (https://osf.io/mk3ny/) whilst looking for the OSF roadmap to see what's in the pipeline, but it doesn't seem to be what we need...

B

brandon Thu 23 Nov 2017

Yeah, Hypothes.is would be DOPE AS!!

SG

Stéphanie Girardclos Thu 23 Nov 2017

I think comment/feedback on preprints deposited on EarthArXiv would be indeed very interesting.
However I'm not sure if comment/feedback should happen/be hosted within EarthArXiv itself. The best way might indeed be a linking of EarthArXiv DOI to another community-based / open access / non-profit structure that has all the tech frame AND the staff support for moderation. Because we all know that inapropriate comments by trolls always happen online! Such inapropriate public comments linked to our preprints could act as a strong rebutal for authors to actually deposit their preprints. Potential inaproriate comments could also act badly on the 'seriousness' of our platform. Inaproriate comments could even be used by hostile actors to have EarthArXiv 'looking' bad. Thus I feel we need to fully think/prepare a policy and a moderated structure before we enter in this process.

More fundamentally, we also need to think deeper about what comment/feedback on our preprints would mean. To my opinion, it would mean that EarthArXiv would become closer to the function of OpenAccess/OpenDiscussion journals.

I strongly believe we authors-scientists should get the control of our publications meaning on the long term on the entire publication process (because we are already doing all the jobs of writing-reviewing-editing). And building a strong EarthArXiv platform is a key part of this idea! But in order to become strong, I believe we need to first focuss on preprints and convince our community to broadly use it.

Thus I think we shouldn't rush on adding comments/feedbacks, in order not to dilute our main goal, particularly when everything still is at the beginning of the process.

Sorry for the long text.

CJ

Christopher Jackson Fri 24 Nov 2017

Thanks @mattspitzer for the info. This is helpful to see, and I'll bookmark this page and share it via Twitter if that's OK. I see it's view-only. So, as Stephanie points out, this could be a bit of hornet's nest. But I realy think that such negative exchanges are very rare, and that the positives massively outweigh the negatives. Also, I'd hate to thin a (the?) key piece of functionality wasn't implemented because of the bad behaviour of a few scholars. In situtations like this, I firmly believe that people who behave well shouldn't be puninished by, in this case, taking away key functionality. Instead, we should tackle the aggressive bullies, pointing out that their behaviour is absolutely not acceptable. Placing the burden on the well-behaved people seems unfair to me, and not a way to change behaviours in the long-term.

SG

Stéphanie Girardclos Fri 24 Nov 2017

Dear Chris, I fully agree with you. We don't want to loose an opportunity because of marginal behavior! But the comments/feedbacks function should come with a very short statement that describes what we expect and what we forbid.

VV

Victor Venema Fri 24 Nov 2017

My immediate response was: yes, wonderful.

However, to keep the discussion at a scientific level, we would really need moderation and if EarthArXiv becomes a big success, this will be a major task. It is a task that can be split up. I could do the climate part and once that gets to big the climate observations part. With EarthArXiv getting bigger, we would hopefully also have more people to do the moderation.

My experience as blogger suggest that just moderation is not enough. To keep the level high, pre-moderation is the best. If comments are published first, it is harder to remove them again, people may already have responded and all that makes the level go down. Pre.moderation is also less work. My blog is pre-moderated and I do not often have to moderate; people know their comment will not appear if below the standard. On blogs with moderation after the fact the moderation is more work.

Should also be a system that can handle spam.

I wanted to write Hypothes.is (web annotation) can always be used, but it looks as if the OSF somehow made that hard. This link somehow does no work:
https://via.hypothes.is/https://eartharxiv.org/pfb7u

Climate Feedback is using Hypothes.is to review press articles. Here is an example annotations by Climate Feedback to have an idea how such comments would look like.

At the moment Hypothes.is or any other web annotation software does not have moderation functions yet. They are working on it.

Should comments be op-in? Would otherwise comments scare some people away?

While I write much about science and at conferences colleagues tell me they read my blog, getting comments from scientists is quite rare. Those that do are typically active on social media. This group is also likely nor representative for the scientific community in such matters. I can imagine that other scientists have much less appetite for public bar fights.

AK

Aidan Karley Sat 25 Nov 2017

Moderation is probably a necessary thing - on my very little blogging I get occasional rashes of spam bots advertising cheap imitation Nike shoes, for example. But whether EarthArxiv would want to get involved in hosting it, maintaining links to deleted or disputed posts etc is a separate question.
One option for consideration might be for the (corresponding) author of a submission to include a link to their choice of site, or the statement that "Comments are not accepted." And they - the corresponding author(s) - take on the task of moderating. There's an element of turning poaches into gamekeepers here, but I doubt that sincere researchers are going to go around deleting all comments of even the slightest degree of criticism. Why in that case, publish at all?
If particular sites show themselves to be particularly popular ... well integrating them into a later version might then at least b based on statistics, not a seemingly arbitrary choice.
Down-side : to comment, some people may need to establish accounts on NameOfDiscussionService. Irritating, but many of us have many of these accounts already, and I'm pretty sure that a small number of most-popular targets would soon become clear leaders.
Two cents worth - if that.

HG

Han Geurdes Sat 25 Nov 2017

Corresponding authors as guardians of the discussion looks like a bad idea to me. Suppose, the claim is that one can safely use models to fill in the gaps of measurements. Now, a mathematician discovers that the basic differential equation has not a unique solution. So, care must be taken using the model that is based on the differential equation.

Now, if the corresponding author of the model paper does not want to handle this particular methodological criticism, he or she can exclude the difficult explanation from the discussion, right? Not every Geochemist is well versed in nonlinear differential equations, linearized with measured integral equations. Moreover, the criticism is theoretical. The question can be raised... how bad is that methodological flaw in numerical terms. The mathematical researcher cannot answer that. He just knows that there are, in principle, countable infinite not mutually matching solutions.

Result is most likely that the model is widely used without checking if the interpolated values belong to the same branch of solution. Perhaps political measures are taken based on not properly checked model conclusions. Enemies of those measures will find the methodological criticism one day and raise counter political measures.

Is that bad... perhaps. Is it unscientific... yes, I believe it is.

A long and winding story just to show how extremely complicated a fair judgement is in a controversial discussion.

VV

Victor Venema Fri 15 Dec 2017

An alternative could be to have comments that are send to the authors and not published below the preprints. That would be sufficient for feedback to the authors. Publishing the comments would make the preprint server more into a discussion journal, a place to review manuscripts.

CJ

Christopher Jackson Fri 15 Dec 2017

I agree we should perhaps vote.

Personally, I think we strongly neuter the power of preprints by not hosting our own commenting; it seems to be one of the key things the community wants and needs, and it's something we've sort of been selling. And you
know my feelings about anonymity...😉


Professor Christopher Aiden-Lee Jackson
Statoil Professor of Basin Analysis

Basins Research Group (BRG ( http://www.basinsresearchgroup.com/index.html ))

Department of Earth Science & Engineering
Imperial College
Prince Consort Road
London
SW7 2BP

England


Email: c.jackson@imperial.ac.uk
Phone: +44(0)207 59 47450
Webpage: www.imperial.ac.uk/people/c.jackson ( http://www.imperial.ac.uk/people/c.jackson )

Twitter: @seis_matters

Co-founder of EarthArXiv ( https://eartharxiv.org/ ), a preprint server for the Earth Sciences

HG

Han Geurdes Fri 15 Dec 2017

Indeed, I tend to agree. arXiv is an example.

But then.... no biased moderator blocking response.. "because .. the moderator feels the paper must be improved by review". This, in turn, of course implies other mischief is possible...

So,... keeping an open eye and mind. Kicking out biased & misbehaving participants could be the remedy. E.g.:

Misbehaving as in... personal attacks frustrating scientific careers of the disputed proponents.
Biased as in .. unable to give reasonable scientific counter argument. But assembling "experts" who all say no.

Then, in addition, always keep the door open for rehabilitation..! We are but human. Trolling out of anger and frustration yesterday is not by definition being a Troll once and for all... or is it ?

SG

Stéphanie Girardclos Fri 15 Dec 2017

Linking with the other line "How should people identify themselves when giving feeback/review to EarthArXiv preprints?" I agree with @victorvenema that we should now go step-by-step to find a consensus to the contrasted views about online comments. I propose a possible step-by-step voting process to solve the case (see attached .pdf). Questions are in black and decisions in blue. Please note that if comments are allowed we will in any case need to decide what we do when comments are inappropriate and what is 'inappropriate' (dashed arrows in red). Does this chain of question also apply to postprints ? What majority do we need to decide ? What proportion of the community needs to vote ? Is it solved by a timely deadline ? I guess we need to open a new Loomio line about how do we vote ... Cheers to all. Stéphanie

VV

Victor Venema started a poll Fri 15 Dec 2017

Comments below EarthArxiv manuscripts Closed Sun 31 Dec 2017

Outcome
by Victor Venema Mon 1 Jan 2018

There is a clear majority for adding comments to the manuscripts we host: 19 people thought this was a good idea, only 2 thought we should just host manuscripts. The latter includes me, so I would suggest that someone else guides us through the discussion on how these comments should be implemented.

Do we want to have public comments below the manuscripts we publish on EarthArxiv? It sounds like a good step towards open science, but would also change the nature of what we do and would be a lot of work to do well. There were two good discussion threads on this topic. It is appreciated to read them before voting.

https://www.loomio.org/d/cfyFQU3X/comment-feedback-section-in-article-page
https://www.loomio.org/d/pFn5clCz/how-should-people-identify-themselves-when-giving-feeback-review-to-eartharxiv-preprints-

If we decide that comments may be a good idea, there will be other discussions later how to do this. There could be a compromise where we allow for comments that are not published, but forwarded to the authors. For this poll I would see that option as no published comments.

19 - Comments are a good idea (if done well)
2 - EarthArxiv should just host manuscripts
HG

Han Geurdes Fri 15 Dec 2017

Comments are a good idea (if done well)
VV

Victor Venema Fri 15 Dec 2017

EarthArxiv should just host manuscripts
SL

Sabine Lengger Fri 15 Dec 2017

Comments are a good idea (if done well)
D

David Fri 15 Dec 2017

Comments are a good idea (if done well)
DU

[deactivated account] Fri 15 Dec 2017

Comments are a good idea (if done well)
JH

Jon Hill Fri 15 Dec 2017

Comments are a good idea (if done well)
SG

Stéphanie Girardclos Fri 15 Dec 2017

Comments are a good idea (if done well)
RW

Rebecca Williams Fri 15 Dec 2017

Comments are a good idea (if done well)
DI

Daniel Ibarra Sat 16 Dec 2017

Comments are a good idea (if done well)
JB

Jeroen Bosman Sat 16 Dec 2017

Comments are a good idea (if done well)
LB

Latisha Brengman Mon 18 Dec 2017

Comments are a good idea (if done well)
PA

Pablo Ampuero Mon 18 Dec 2017

Comments are a good idea (if done well)
EG

Evan Goldstein Mon 18 Dec 2017

Comments are a good idea (if done well)
TN

Tom Narock Mon 18 Dec 2017

Comments are a good idea (if done well)
A

Alodie Tue 19 Dec 2017

Comments are a good idea (if done well)
JF

Jamie Farquharson Tue 19 Dec 2017

Comments are a good idea (if done well)
AE

Allison Enright Tue 19 Dec 2017

Comments are a good idea (if done well)
CJ

Christopher Jackson Fri 22 Dec 2017

Comments are a good idea (if done well)
DC

Domenico Chiarella Sat 23 Dec 2017

Comments are a good idea (if done well)
CS

Colin Sayers Sat 30 Dec 2017

EarthArxiv should just host manuscripts
AK

Aidan Karley Sun 31 Dec 2017

Comments are a good idea (if done well)
VV

Victor Venema Fri 15 Dec 2017

Agree with @sgirardclos that those are the questions.

Would suggest to start with the first question. I made a poll on the question whether we want to have comments. https://www.loomio.org/p/7m2KbkCB/comments-below-eartharxiv-manuscripts

VV

Victor Venema Mon 18 Dec 2017

13 to 1, even if it is Christmas time, it looks like we do not have to wait until the end of the poll. On to the discussion on how to do comments well.

VV

Victor Venema Mon 1 Jan 2018

Important would be to be able to give feedback to the authors. Currently there is no option to contact the author who uploaded the manuscript. Thus if someone forgot to mention their email in the manuscript it would currently be hard to contact the author.

HG

Han Geurdes Tue 2 Jan 2018

Best wishes for the New Year. Thanks Victor for the discussion and presentation of the results.

Let's ask ourselves: what's "done well".

Let's also set up a schedule of moderators. We have 19 Pro. So if each Pro does a month of moderation we will be a year and half wiser. ..

I propose: if there are comments on paper X run them past a moderator. Find IT ways to get that going. The moderator can reject or accept the comment via a button.

If I would have my turn, i.e. my month, then I would only look at the way "things are said". E.g. one time a ... "this is wrong" is ok but in need of an explanation. If not given... then reject. If second time only ...a merely "this is wrong"... the guy is blocked on my watch. Abusive language ... blocked, etc.

Learning by doing.

Point 1 IT necessities: a bit of text run through the moderator. Text size of comments is limited. Three buttons, accept, reject, block. Only once a day please (batch processing).

Point 2 procedure necessity: a moderator "discussion" at each change of the watch. Biasses are signalled, changes suggested..... etc

Possible changes in moderator schedule because of illness. That kind of stuff.

Well. Why is all that, or similar, an extremely bad idea ?

AK

Aidan Karley Mon 8 Jan 2018

This needs a bit of clarification : "Text size of comments is limited. Three buttons, accept, reject, block. Only once a day please (batch processing)."
Are you saying that commentators should only be commenting once per day, or that the IT "black box" should only send one email per day to a moderator? The latter would be achievable, but fairly difficult, particularly if a moderator is covering several "sections" of the service (I see some murky database shenanigans to keep track of which comments have been moderated). There would also, predictably, be a case of someone submitting a comment 3 minutes after the daily mail gets sent out, then 23 hours later complaining that their comment was being "suppressed".
Possible changes in moderator schedule because of illness. That kind of stuff.
TBH, we've dealt with that at work for years by having "functional" email accounts. In this context, for example "comment-mod-section-tectonics@geoarxiv.org" , "comment-mod-section-mineralogy@geoarxiv.org" ...
The outgoing moderator passes the login details for the account to the incoming moderator. Someone has keys to the login credentials to cover the event of sudden illnesses, assassination by bus, etc. as a back up. The contents of the email account then provide the context information for the incoming moderator.

Time for me to head to the station!

CJ

Christopher Jackson Wed 10 Jan 2018

Now we are getting to the details of commenting, might I suggest that one or two of the main protagonosts with a passion for optimising this for EarthArXiv, consider remotely(?) attending the Hypothesis meeting taking place on January 25th in NYC. They have invited several preprint services to take part, to learn from one another about how commenting might work. If you're interested, please let us/me know and we can forward on the details!

JB

Jeroen Bosman Wed 10 Jan 2018

Hi all,

Yes I would be very much interested in attending. I have been using Hypothesis for some years now (sometimes to the effect of articles in Nature and the Guardian being changed) and know about the technical discussion about making commenting with Hypothsis on PDFs in iframes possible. I have corresponded with the people at COS and Hypothesis about this and know they are working on it. Extending the functionality of Hypothesis for moderation is one step further of course and is an interesting topic that I would be prepared to discuss in that meeting (from a distance, that is).

VV

Victor Venema Wed 10 Jan 2018

Chris, I would be interested in principle. What exactly would one commit to? I would only be able to join remotely. How long would the meeting be, what do we have to prepare, are there things we as ArXiv would have to discus in advance?

At the moment Hypothesis has no moderation tools.

It does know groups, but someone making a comment would first have to become member of the group, only group members would be able to read the comments/annotations and I do not see any way for the owner of the group to remove annotations. So they way groups currently work would not help us moderating comments, it is more for a lecture jointly working on a document in private.

Annotations in EarthArXiv are also currently not technically possible because the PDFs are embedded in a webpage. Hypothesis only works on clean PDFs or webpages. This is also a problem for the PDFs of AGU journals, but fortunately they also have HTML version.

For people who do not know Hypothesis yet, here is a review of a published paper I made with web annotations. That probably looks a lot like how annotations would be used to give feedback on our manuscripts.

D

David Wed 10 Jan 2018

I would be interested as well if not much duty is involved. I can only do it remotely and I essentially have the same questions than Victor.

I am not aware of Hipothesis yet, so thank you Victor for the share.

CJ

Christopher Jackson Fri 12 Jan 2018

Hi @geoda, @victorvenema and @jeroenbosman! Thanks for your positive responses. I would therefore be great if you could attend the meeting via video/telephone. If you're willing to do this, please send me your email details and I'll pass them on to the meeting organiser, Heather Staines.

JB

Jeroen Bosman Fri 12 Jan 2018

Hi Chris,

Mine is j.bosman@uu.nl

Thanks!

Jeroen

D

David Sat 13 Jan 2018

Hiya!

geo.david.fernandez@gmail.com is mine - thanks!

Shall the three of us meet before the meeting? @jeroenbosman - knowing of your experience, I'm sure @victorvenema and I could listen to what you have to say on the matter for hours... (potentially making -at least myself- more useful for the meeting)

CJ

Christopher Jackson Sat 13 Jan 2018

Thanks all for the details! I have forwarded them on to Heather (Staines) at Hypothesis. If might be useful for you three to chat ahead of time, but not essential. It is clear there are different views about how to implement commenting, so I don't think we need an EarthArXiv party line.

VV

Victor Venema Sat 13 Jan 2018

It might not be a good idea to try to understand the position of the community on the commenting system we would like or at least to know the range of opinions the system would need to be able to handle.

@sgirardclos, made a tree of questions, the next question would be whether comments should be premoderated, that is whether one of us should have a look before they are posted to the web or forwarded to the authors. That would give us the option of deleting to editing a comment in case it is not worthwhile (including spam), defamatory, trivially wrong, etc. Whatever we decide on next to use as moderation criteria and what we decide on next to organise this.

Or whether we want to do the moderation afterwards. My experience is that moderating afterwards is harder. If it is already published and people may also already have responded, you tend to moderate less, which means that the quality of the comments go down. Furthermore, this encourages bad quality comments and trolling. If comments are not published at all there is not much satisfaction in trolling.

I think the case of forwarding comments to the authors is different from publishing the comments in public.In the former case I see less need for moderation, although spam filtering would still be important.

So the suggestion would be for the next poll: Do you want comments to be pre-moderated?
Yes always (also emails to authors),
Yes for public comments,
No.

If people agree with that formulation, I would suggest putting that poll up tomorrow.

VV

Victor Venema started a poll Sun 14 Jan 2018

Comments pre-moderated? Closed Fri 19 Jan 2018

Do you want comments to EarthArXiv to be moderated in advance, i.e. before they are published or forwarded to the authors?

A "No" would mean that comments are published immediately, but we could still moderate inappropriate comments afterwards, to be "decided" in a later poll.

5 - Yes, public comments should be pre-moderated
0 - Yes, all comments should be pre-moderated (also private messages to authors)
7 - No pre-moderation of comments (to be determined what we want post-moderation)
6 - No premoderation if you reach a certain threshold (e.g. 10 accepted moderated comments)
VV

Victor Venema Sun 14 Jan 2018

Yes, public comments should be pre-moderated
JB

Jeroen Bosman Sun 14 Jan 2018

No premoderation if you reach a certain threshold (e.g. 10 accepted moderated comments)
HG

Han Geurdes Mon 15 Jan 2018

No pre-moderation of comments (to be determined what we want post-moderation)
D

David Mon 15 Jan 2018

No premoderation if you reach a certain threshold (e.g. 10 accepted moderated comments)
RW

Rebecca Williams Mon 15 Jan 2018

Yes, public comments should be pre-moderated
S

Sara Mon 15 Jan 2018

No premoderation if you reach a certain threshold (e.g. 10 accepted moderated comments)
JF

Jamie Farquharson Mon 15 Jan 2018

Yes, public comments should be pre-moderated
PA

Pablo Ampuero Mon 15 Jan 2018

No premoderation if you reach a certain threshold (e.g. 10 accepted moderated comments)
JK

James King Mon 15 Jan 2018

Yes, public comments should be pre-moderated
EJF

Eric Jameson Fielding Mon 15 Jan 2018

No premoderation if you reach a certain threshold (e.g. 10 accepted moderated comments)
CS

Colin Sayers Mon 15 Jan 2018

No pre-moderation of comments (to be determined what we want post-moderation)
DI

Daniel Ibarra Tue 16 Jan 2018

No premoderation if you reach a certain threshold (e.g. 10 accepted moderated comments)
SG

Stéphanie Girardclos Wed 17 Jan 2018

Yes, public comments should be pre-moderated
SL

Sabine Lengger Wed 17 Jan 2018

No pre-moderation of comments (to be determined what we want post-moderation)
SL

Sabine Lengger Wed 17 Jan 2018

No pre-moderation of comments (to be determined what we want post-moderation)

I'm just a little bit worried about the workload for us with the 10 accepted moderated comments and think that the fact that you have to give your Orcid is enough to prevent problems.

LB

Latisha Brengman Thu 18 Jan 2018

No pre-moderation of comments (to be determined what we want post-moderation)
CJ

Christopher Jackson Thu 18 Jan 2018

No pre-moderation of comments (to be determined what we want post-moderation)
DC

Domenico Chiarella Fri 19 Jan 2018

No pre-moderation of comments (to be determined what we want post-moderation)
A

Alodie Fri 19 Jan 2018

No pre-moderation of comments (to be determined what we want post-moderation)
VV

Victor Venema Wed 17 Jan 2018

I voted against comments because I feel that if we do this we should do this well and this will indeed be a lot of work. I expect the comments will be much more work than just supervising a preprint server.

In the Brexit referendum it was not known what Brexit was. Similarly in our first vote on comments everyone likely had a different idea of what that would entail. (The difference was that the referendum was close, our vote was not.)

Just like in case of Brexit I think it is reasonable to ask the population at the end do you want this Brexit or keep things like they are, it may be a good idea at the end of this process to ask everyone do you want this comment system or do you want to stay a preprint server?

HG

Han Geurdes Thu 18 Jan 2018

Victor you may have a good point there. At atXiv one submits a paper for comment. Each first paper needs endorsement (pre moderation). In some sections subsequent papers are moderated. They do not allow comments.

VV

Victor Venema Thu 18 Jan 2018

One more day to vote on whether comments should be pre-moderated. We have less votes than last time. Are these people undecided or busy?

SG

Stéphanie Girardclos Fri 19 Jan 2018

I somehow agree with @victorvenema on comments and the comparison with Brexit.
This is why I proposed a tree of successive question/votes with their end consequences that could have been used for vote steps (people thus understanding toward which goals they go when they vote).
Now with we have a vote with choices (instead of simple yes and no questions). This is fully ok. But what happens if we have a 33%, 33%, 33% picture as if seems like happening at this point. In democracy this is the typical 'nobody-is-happy' situation because if you pick one choice, the left majority (i.e. 66%) is not happy. If this voting tendency proves to stabilize. I definitively think we should proceed differently (either successive yes - no questions in a tree of questions, or with ranking of all choices not just one).

SG

Stéphanie Girardclos Fri 19 Jan 2018

In relation to my comment 15 minutes ago.
For future votes presenting several alternatives I suggest to use the 'ranked choices' vote option. This will give us a better chance to make a sound and fair decision.

CJ

Christopher Jackson Fri 19 Jan 2018

Indeed. The precise problem with democracy…

I suggest we wait until next week’s meeting with Hypothesis to take this further. Jeroen, David, and Victor will attend, so they will be able to get feeling as to how the broader preprint community
are handling this key issue.

Chris


Professor Christopher Aiden-Lee Jackson

Basins Research Group (BRG ( http://www.basinsresearchgroup.com/ ))

Department of Earth Science & Engineering

Imperial College

Prince Consort Road

LONDON

SW7 2BP

UK

Email: c.jackson@imperial.ac.uk ( c.jackson@imperial.ac.uk )

Web: www.imperial.ac.uk/c.jackson ( http://www.imperial.ac.uk/c.jackson )

Twitter: @seis_matters


Co-founder of EarthArXiv ( https://eartharxiv.org/ ), a preprint server and postprint archive for the Earth Sciences

SG

Stéphanie Girardclos Fri 19 Jan 2018

Yes. Best solution is waiting as @christopherjackson3 suggests. (In Switzerland we faced once this type of multiple-choice voting problem that was not well formulated. And they had to entirely redo the voting ... imagine ! :laughing: )

JF

Jamie Farquharson Fri 19 Jan 2018

Forget votes. We need a benevolent dictator.

CJ

Christopher Jackson Fri 19 Jan 2018

lol

pages Tom Narock


Professor Christopher Aiden-Lee Jackson

Basins Research Group (BRG ( http://www.basinsresearchgroup.com/ ))

Department of Earth Science & Engineering

Imperial College

Prince Consort Road

LONDON

SW7 2BP

UK

Email: c.jackson@imperial.ac.uk ( c.jackson@imperial.ac.uk )

Web: www.imperial.ac.uk/c.jackson ( http://www.imperial.ac.uk/c.jackson )

Twitter: @seis_matters


Co-founder of EarthArXiv ( https://eartharxiv.org/ ), a preprint server and postprint archive for the Earth Sciences

TN

Tom Narock Sun 21 Jan 2018

Where do I sign up? "Benevolent Dictator" would look lovely on my tenure application...

VV

Victor Venema Fri 19 Jan 2018

I am more a we-need-to-talk kind of guy.

Given that I prefer not to have comments, I should probably say "great that we are blocked 33%, 33%, 33%", but that is not how I see the situation. No premoderation after a high threshold is not that different from having premoderation. At least I could live with that variation well. To put it in a ranked vote

1 Premoderation

2 Threshold

20 Postmoderation

1000 No moderation

(No, Dr. Venema that is not how ranked voting works.)

But even in that case there would be a large minority against premoderation and democracy should not be a dictatorship of the majority. If only because everyone is a minority sometimes. I would see the polls as guiding us what we need to talk about.

To understand my preferences people may have to know that I am a climate scientist and a blogger. I thus have experience with a large number of unreasonable people and with the moderation of blog comments. For me the experience of the reader, and thus the signal to noise ratio (SNR) of the comments, is important. Our situation on EarthArXiv is different and other people may have other experiences they are thinking of when it comes to moderation such as power abuses during closed peer review.

The kind of moderation has four aspects (did I miss any?)
1. Quality of service for the reader.
2. Quality of service for the authors.
3. How much work it is.
4. Abuse of power/censorship.

For the reader a high SNR is important. The authors are likely willing to do more work to see whether lower SNR comments contain aspects that could improve their paper.

Someone worried that premoderating 10 comments before we switch to postmoderation would be too much work. My experience with blogging is the opposite. In case of postmoderation people write more comments and try to game the system and look how far they can go. Blogs with postmoderation typically put quite a lot of work into moderation. My blog has premoderation, it is rare I even have to think whether a comment is on the edge. Nearly all comments are high quality, maybe as trolls do not get the pleasure of seeing their comments on screen.

Because comments are immediately visible with postmoderation the discussions go faster. That is nice to build a large audience, but a slower debate where people think longer may be better for science. The delay with premoderation means that sometimes multiple people write the same comment, which is inefficient.

If we use web annotation for the comments everyone can write annotations in the public stream. They only thing we could control is what appears in our own group/stream. Thus if we reject an annotation unfairly, everyone would be able to see that comment in the public stream and lower their opinion of EarthArXiv accordingly. Thus I do not fear unfair censorship that much. (And authors could look in the public stream whether the low SNR annotations are helpful.)

A long post, sorry. One of the advantages of a parliamentary democracy over direct democracy is that you can talk with each other and come up with better solutions. How do you see moderation and why are what features important to you? (Especially for the variety of setting a threshold, which we did not discuss before the poll.)

CJ

Christopher Jackson Sun 21 Jan 2018

Hi @victorvenema. Thanks for your ongoing engagement and discussion in this key matter. I maintain we wait until this week's meeting's with Hypothesis (Thursday) and the other preprint services (Friday) before taking this further. At that point, with a lot more information and guidance in-hand, you, @jeroenbosman, and @geoda can report back to us, such that the Advisory Board can ultimately decide on the proposal we formally take to COS. COS (and any nominated commenting service provider, such as Hypothesis) will decide what is practicable and what is not. I hope that makes sense. Chris

VV

Victor Venema Sun 21 Jan 2018

That is fine. I am less optimistic than you to hear at the meeting how other preprint servers do this, because this is not a task preprint servers normally do.

I expect that we are entering new territory and that Hypothesis and COS would like to know from us what we want/need.

We can wait, especially together with Jeroen and David I expect that we can already provide Hypothesis with some useful feedback.

Given the complexities of the topic, we may also want to consider a teleconference or to have a small group hack out a plan and ask for comments on such a concrete plan here.

CJ

Christopher Jackson Mon 22 Jan 2018

Sounds like a plan! Let’s reconvene early next week. Chris

Professor Christopher A-L Jackson
Professor of Basin Analysis
Basins Research Group (BRG)
Department of Earth Science & Engineering
Imperial College
LONDON
SW7 2BP

UK

Co-founder of EarthArXiv (🌎), a preprint server for the Earth Sciences (www.eartharxiv.org ( http://www.eartharxiv.org ))

VV

Victor Venema Fri 2 Feb 2018

The meeting did not report much on how other archives handle comments because this is rare. Hypothesis mainly wanted to know from us what we would like.

That being say, BioArXiv claimed to have comments for about 10% of their manuscripts. I looked through 10 older preprints, but did not find any with comments I could present as an example here.

Independent of this meeting, I just came across the news that PubMed will close its comments due to lack of participation. http://retractionwatch.com/2018/02/02/pubmed-shuts-comments-feature-pubmed-commons/

PubMed wrote:

The service was first introduced as a pilot project in the fall of 2013 and was reviewed in 2015. Despite low levels of use at that time, NIH decided to extend the effort for another year or two in hopes that participation would increase. Unfortunately, usage has remained minimal, with comments submitted on only 6,000 of the 28 million articles indexed in PubMed.
While many worthwhile comments were made through the service during its 4 years of operation, NIH has decided that the low level of participation does not warrant continued investment in the project, particularly given the availability of other commenting venues.

The post publication peer review site PubPeer responded:

We believe that every obstacle to commenting reduces the volume of comments. Compared to PubPeer, PubMed Commons forbade anonymity. We have always felt that many users, particularly those with the most significant criticisms, prefer to comment anonymously in order to avoid any risk of reprisals. Irrespective of the true risk of such reprisals (and we have seen several legal threats in addition to the very public Sarkar suit), users do perceive a risk and alter their behavior in consequence. Anonymous commenting remains controversial, but we remain convinced that on balance it is beneficial.

I am biased, but maybe our priority should be with designing a system for the moderation of the preprints themselves, which at the moment is done by @christopherjackson3, but would be better done by groups of experts according to clear published criteria.

CJ

Christopher Jackson Sat 3 Feb 2018

Thanks Victor,

I’ll reply to this fully after the weekend, but at this point I’ll add that, as I understood it, PubMed’s issue may at least partly relate to that it dealt with post- rather than pre-publication commenting. We are really advocating the latter, which can assist development of ideas, papers, etc, ahead of rather than after review. I too can see why so few people comment afterwards; maybe the value of the comment is reduced.

In any case, your point about a more structured moderation process, as discussed before already, is well taken and clearly a priority item.

Chris

VV

Victor Venema Sat 3 Feb 2018

Yes, it could work. I just took away from the meeting/telecon that it will be hard. I forgot to mention that the representative of SocArXiv https://socopen.org/ told us that they allow for comments and that no one used them.

Naturally the Open Review of the EGU journals allow for comments. Apart from the peer reviewers solicited by the journal, it is rare to see any additional comments (which are not allowed to be anonymous).

We could look for inspiration to the ArXiv overlay journals. For example the journal Discrete Analysis. http://blog.scholasticahq.com/post/130145117128/introducing-discrete-analysis-an-arxiv-overlay

And Science Open has collections that also allow for peer review. http://blog.scienceopen.com/2016/03/collections-as-the-future-of-academic-led-journals/

VV

Victor Venema Sat 3 Feb 2018

Something else from the Hypothesis meeting. My impression was that the hardest organisational problem to solve with comments would be the question whether we allow comments to be deleted.

Not being able to delete/modify comments may make people more hesitant to participate, ask (dumb) questions and stifle the discussion. On the other hand, the comments are part of the scientific literature and in that respect should be preserved.

A compromise could be to have someone responsible for a manuscript (an editor) and have this person decide that the discussion has run its course for now and will be added to the literature. Then the participants would still have some time to delete/modify their comments before they are committed to the scientific literature. In case of GitHub that would be equivalent to pushing your code to GitHub.

EG

Evan Goldstein Sat 3 Feb 2018

I have scraped some of the EGU journals (to look at blind vs signed reviews, and if signed R1 begets signed R2). And I agree with @victorvenema — interactive comments from non-reviewers are not common.

for completness sake, the scraper is on github here and the signed vs blind results are here

VV

Victor Venema Sat 3 Feb 2018

BioArXiv uses Disqus for their comments. Disqus has a list with comments. https://disqus.com/home/forums/biorxivstage/

I have no idea about the percentage with comments, but at least here we can find some comments. Looks like they are not strict in moderating their comments. The first ones I found were:

We look forward to your comments, critiques, and other feedback on our study and paper!

Share it..

This package seems to require R >= 3.5.0 which is not available yet

For your information, version 2 corrects a typo of an authors email
(!!!), corrects a reference, and corrects a mistaken omission in the
acknowledgements section.

Great to see this finally out! Looking forward to reading it more thoroughly. Couple of quick comments though:
1. I know that you don't believe that it will make any difference, but it would be good to show results using ANTs in addition to FNIRT ...
2. Also I would consider using a different colormap. ...

Seems mostly useful, but I would have moderated a bit stronger and have removed the first two comments, which do not add value.