Loomio
Thu 7 Dec 2017

How should people identify themselves when giving feeback/review to EarthArXiv preprints?

SG
Stéphanie Girardclos Public Seen by 394

When giving online feedback to a paper, there is a risk that people hide behind a fake identity. Should an Orcid author identification be compulsory for feeback? or a verified institutional email? any other idea?

MDF

Matteo De Felice Thu 7 Dec 2017

Totally agree with you, however not sure about the options here...a "verified institutional email" does not sound very good and very restrictive, Orcid would be ok for me but don't know how much is used around the world...

HG

Han Geurdes Thu 7 Dec 2017

Feedback and ORCID. This sounds ok to me.

However, what about clever remarks of people without ORCID?

E.g. I live in an agricultural area. Suppose, there is a paper that claims that certain levels of ground water are not affecting natural habitat.

My neighbor doesn't have an ORCID. He is a notorious bird watcher. One day I tell him... listen Jim , the ground water, according a paper on EarthArxiv .....

He notes that something is wrong with the claim. After the change in ground water level around here, e.g. the red-yellow-and-blue merula isn't nesting in the neighborhood. Then he will be in need of my ORCID friendliness to deliver his message to the discussion of the paper.

He would say something like .. Wait a minute y'all professors. I noticed....

So ... no ORCID no "wait a minute .... " from people on the ground.

I made it all up of course. .... This neighbor's name is Hank. :-)

Cheers

A

Andy Thu 7 Dec 2017

ORCiD sounds like a good idea. My impression is that it is fast becoming a fairly standard way of uniquely identifying researchers when publishing. There is ResearcherID as well, but I think ORCiD has gained more traction with journals.

CJ

Christopher Jackson Thu 7 Dec 2017

I think you all know my opinion about this; all reviewers should disclose their names, and all reviews should be open and published with manuscripts. HOWEVER, I know others have competing views, which often seem quite subject specific (i.e. the geologists I roll/fight with are only very rarely anonymous when reviewing papers). If we require disclosure of ORCID, which I agree would be good to use, then, essentially, there is no choice of anonymity if you wish to comment on a paper. The people who want to be anonymous will thus not comment on papers, which I don't massively have an issue with...Thoughts?

HG

Han Geurdes Thu 7 Dec 2017

ORCID ok. However, I think it is the vulnerability of the proponent / the proposed at issue. Therefore, I would like to propose that a comment can't be acceptable without giving full disclosure of why the comment is made.

So, .... it can't be. Agreed Christopher! But it must be.. Agreed, because the use of ORCID will always disclose what assumptions are behind the comments.

The basic problem is an open discussion and not me taking my lawnmower to a tank battle.

LU

Leonardo Uieda Fri 8 Dec 2017

I agree with @christopherjackson3 (3?). Having the ORCID seems good and removes the barrier of people having to create yet another login. Even if we decide on anonymity, having the ORCID be mandatory for logging might be a good idea. The author might not know who the reviewer is but the admins should. That way, they can block abusers who might otherwise just make a new login. That might help deter trolls.

Institutional emails might not be a good idea because it would block any scientist with a unconventional career path or who is taking a break.

CJ

Christopher Jackson Fri 8 Dec 2017

Yes. That might work @leonardouieda. Although some people may still be hesitant if they know anyone knows who they are. But it could be worth trialing it...

SL

Sabine Lengger Fri 8 Dec 2017

Hi, I agree with a mandatory Orcid for login. I think the identity of the reviewer though should only be available to admins if moderation is necessary. I think peer review is more rigorous when the identity of the reviewer is not known. It could potentially descend into an "I scratch your back, you scratch mine" system otherwise, and also, the opinion of more junior or female colleagues might not be taken as seriously as the opinion of senior colleagues.

HG

Han Geurdes Fri 8 Dec 2017

Correct. So, we have:

  1. ORCID

  2. Email

  3. Other id

All known to the admin. What about..

  1. only severe criticism when counter paper is produced as a requirement.

My experience is that the opponent acted upon the believes of the standers by with verbalisation such as... it is clear that. But the person didn't deliver any clear proper mathematical support.

E.g. reiteration of the theorem that I attacked without mathematical support. The ...this is impossible ... objection. The "experts claim that it cannot be done". Look at arXiv 1704.00005 to see what was needed to get through. It was possible.

So, ... framing the proponent as a fool can only be avoided when, in my case, real proper maths is provided. That is with a paper. Not with blabla to convince important bystanders.

Sorry for the long post. However, everybody can imagine how an opponent can get away with bull shit argument.

Han Geurdes,

Geurdes data science kvk64522202

Member of the UNGGIM Private Sector Network.

..............

Read my solution of the Clay millennium Navier-Stokes problem at:

http ( http://www.tandfonline.com/doi/abs/10.1080/23311835.2017.1284293 ):// ( http://www.tandfonline.com/doi/abs/10.1080/23311835.2017.1284293 )www.tandfonline.com ( http://www.tandfonline.com/doi/abs/10.1080/23311835.2017.1284293 )/ ( http://www.tandfonline.com/doi/abs/10.1080/23311835.2017.1284293 )doi ( http://www.tandfonline.com/doi/abs/10.1080/23311835.2017.1284293 )/abs/10.1080/23311835.2017.1284293 ( http://www.tandfonline.com/doi/abs/10.1080/23311835.2017.1284293 )

Read our solution to Bells theorem at:

https ( https://arxiv.org/abs/1704.00005 ):// ( https://arxiv.org/abs/1704.00005 )arxiv.org ( https://arxiv.org/abs/1704.00005 )/abs/1704.00005 ( https://arxiv.org/abs/1704.00005 )

Van: Sabine Lengger (Loomio)

Verstuurd: vrijdag 8 december 17:21

Onderwerp: [EarthArXiv] How should people identify themselves when giving feeback/review to EarthArXiv preprints?

Aan: han.geurdes@gmail.com

TN

Tom Narock Fri 8 Dec 2017

The Center for Open Science (COS) has just released it's latest roadmap: https://docs.google.com/presentation/d/1rE9kfMIYhGwFqvzPC5_iM73dry53jJfbgtFvqVkATXg/edit?ts=5a289939#slide=id.g2bf07f93ed_0_0 In the roadmap they indicate they are looking at https://web.hypothes.is/ as a means of preprint annotation. COS doesn't have a delivery date yet for the annotation service, though. I saw somewhere else that hypothes.is was working with ORCID to integrate services such that annotations would include the user's ORCID ID. Although, I'm not sure of where they are in terms of development.

VV

Victor Venema Fri 8 Dec 2017

My impression from the previous discussion Comment/Feedback section in article page was that while people were sympathetic to the idea of adding comments, there were good arguments not to do this ourselves because it is a difficult task (to guard against spam, etc., tools for moderation) and would time consuming (moderation). In this new thread people implicitly seem to accept the idea of comments. Would it be an idea to put up a poll or something like that to test the waters?

Personally I would argue that if we do this, also anonymous comments should be possible. In science the arguments count, not who makes them. Someone accusing a senior researcher of making a clear mistake or worse in public can reasonably want to have some protection in the form of anonymity. If any comments are not appropriate that should be prevented by pre-moderating the comments, in my view.

Hypothes.is would allow for comments. At the moment everyone could do so; it is beyond our control; there are no moderation tools, that does not fit to the philosophy of web annotation. There are also no moderated channels for now. At least in theory everyone could because somehow the EarthArxiv does not work together with Hypothes.is. Maybe because of the fancy PDF embedding in a web page. See previous thread.

HG

Han Geurdes Fri 8 Dec 2017

Let's try this and see how it develops.

Han Geurdes,

Geurdes data science kvk64522202

Member of the UNGGIM Private Sector Network.

..............

Read my solution of the Clay millennium Navier-Stokes problem at:

http://www.tandfonline.com/doi/abs/10.1080/23311835.2017.1284293

Read our solution to Bells theorem at:

https://arxiv.org/abs/1704.00005

A

Alfonso Fri 8 Dec 2017

Just to be clear, the sign up section has already an option to use ORCID, doesn't it? So if comments are allowed, wouldn't they have the option to sing up using their ORCID account like authors have when they upload a MS? Will this suffice? If this is the case, we perhaps should encourage everyone to sign up in ORCID before signing up in EarthArXiv

D

David Wed 13 Dec 2017

While I acknowledge the value of the comments made above about anonymity, I beg to disagree. If there were comments (see Victor Venema's first paragraph above), my take is that full disclosure is the way to go (to administrators and rest of users alike). To me, the main problem with reviewers is that they may hide behind anonymity to unfairly discredit the research they review (especially when it challenges the "old" -their- view). I agree that the arguments are what counts not who make them and that a dangerous favour-by-favour approach could potentially settle among few, but I'd argue that those are minor in comparison with the research blockage that anonymity tends to produce. Anyone fairly pointing out somebody else's mistake improves the outcome, and should be the norm and not something to fear. If we don't promote the proper actions, we succumb to the improper ones.

VV

Victor Venema Wed 13 Dec 2017

To me, the main problem with reviewers is that they may hide behind anonymity to unfairly discredit the research they review (especially when it challenges the "old" -their- view).

To me the main problem with forcing identity is that it becomes risky to challenge the old view, which is typically held by the old alpha males. The large number of women who do not report sexual harassment, #MeToo, indicates how much power they have to make and break your career.

In the reviewing system if Grassroots scientific publishing I was thinking of three levels:

  1. A synthesis written by a named editor.

  2. Peer reviews of the full paper including a numerical assessment of the quality, where the editor should at least know the names to assess the expertise and reliability of the reviewer.

  3. Comments, which can be made by anyone, also anonymously.

Comments and peer reviews would be pre-moderated. For me moderation is the best tool against the fear that studies are "unfairly discredited".

VV

Victor Venema Thu 14 Dec 2017

By the way, @geoda, what do you mean with "research blockage"? Do you mean that a paper could be blocked by a comment on a preprint server? We are not in the scenario here of a peer reviewer who can advice the editor to reject a manuscript.

A powerful senior researcher may have the ability to write a comment that influences the reviewers and thus have some indirect ability to block a paper, but he will only have that kind of power if he uses his name. An anonymous comment can only convince by the power of its arguments, not by the authority and the power of the writer.

P.S. It has a certain irony that @geoda is commenting pseudonymously here. It makes it easier to write something. Thus having the option of anonymous comments will also increase the feedback the authors will get, which would be my main (selfish) motivation to send my manuscripts to a preprint server.

HG

Han Geurdes Wed 13 Dec 2017

Agreed @geoda. But here we also have a case of the test of the pudding is in the eating. I do not want to go on and on about my "it was mathematically possible" despite the many impossibles of stakeholders who were biased. If a full paper is given next to the ridicule and the you are a crackpot, then the attacked authors can try to turn the table.

What I would definitely like to be cautious about us the arXiv gen-phys verbal trick of moderators to suppress a view "because the moderator feels that the paper can be improved by peer review". Then the "peers" dishonestly review. Then the whole proposition is off the table without a fair discussion of really impartial scientists.

HG

Han Geurdes Wed 13 Dec 2017

In general I would also like to warn against the (hidden from the public) black listing. If you put a researcher on such a list and think it is unavoidable, then make the list public. Let everybody know what the criteria are to be on that list and what to do to be off the list. Blacklists, framing etcetera are unscientific ways to solve a scientific dispute. Even Phys.Rev.A did not escape from using blacklists to suppress authors from rehabilitation. Essentially it is a wicked practice. Einstein fell victim to it.

CJ

Christopher Jackson Thu 14 Dec 2017

"If we don't promote the proper actions, we succumb to the improper ones.". This is absolute FIRE from @geoda, and something I absolutely, 100% agree with. Everything we've done for EarthArXiv has been in the interest of fairness and openness, with constructive discussion being a key part of that. I here some of the arguments about anonymity, some of which are good, but I think the main reason people deploy it is to be destructive. That's not what we're about.

VV

Victor Venema Thu 14 Dec 2017

Some senior researchers have a lot of power due to their roles in funding agencies, writing reviews for manuscript and research proposal, and through hiring decisions they make themselves and those made by their friends and dependants. I do not think it is fair that it becomes risky to point our problems with their work by forcing people to sign comments, while it is easy to bash a PhD student.

Where is the openness when people low on the food chain have to fear writing comments and will not write them? I think the main purpose of anonymity is to facilitate speaking truth to power. The comments here are naturally short, on my blog I write more often about the destructive power of badly implemented openness, for example in my last post on the Scholarly Communication conference FORCE17.

On my blog I allow for anonymous comments. This has never been destructive because I moderate the comments. Someone needs to feel responsible for the quality of the conversation. The lack of moderation is what makes Twitter unpleasant and newspaper and YouTube comments nearly useless and sometimes destructive.

D

David Thu 14 Dec 2017

A powerful argument can be equally made anonymously or in the open - if it's a fair point to it out, one points it out either way.

Unfair arguments can only be made anonymously - doing it with your name attached to it makes it easier for third parties to understand potential biases.

I consider research blockage anything that delays the creation and dissemination of knowledge. Bad comments on a preprint would probably influence the peer-reviewed process, and as said above, I think bad comments are much easier to do in the hidden.

P.S. My pseudonymous is just my "internet name", btw - kinda like a street name, but less cool XD
David Fernández-Blanco -- orcid.org/0000-0002-5326-9164

HG

Han Geurdes Thu 14 Dec 2017

Dear Victor & David,

Sure we all know how it works. So, the owners, administrators, moderators and plain readers (such as myself), know what an anonymous comment can imply.

If we disallow EarthArxiv the biases and the favoritism that really & truly exist in arxiv (believe me it is so), then we are doing very well indeed.

If EarthArxiv derails into all the things you talked about, then I am out. I will protest but if bullshitted, it's bye bye.

I am running a business. Any misbehaving senior can get the ass kicked if and when necessary.

Simple as that.

Start the thing up and let's get rolling.

SL

Sabine Lengger Fri 15 Dec 2017

Great discussion! I presented EarthArXiv & preprint servers in general yesterday at our research group to researchers from UG students up to emeritus profs, it was very well received across the board, and we had a great discussion which for a large part focused on this anonymity issue. I think both sides of the argument are valid, and there were differing views pretty much in line with the discussion we are having on here as well.

I think a lot of how people feel about this is centred around personality and confidence. I personally support the anonymity of reviewers, because I feel that it protects and encourages people with less confidence or a lower standing in terms of where they are in their career. In my opinion, part of this issue has to do with gender, as I see a lot of men getting out there and voicing their opinions very loudly, while less women feel confident enough to do it. I certainly would be less encouraged to point out, even politely, what the flaws in a methodology are, particularly to senior profs, or friends, who I know will take criticism personally.

So, in an ideal world, peer review should be double open, but in this less than ideal system, I think we should consider how we can encourage equality rather than squashing the quiet people.

VV

Victor Venema Fri 15 Dec 2017

I guess everyone has a different film playing in their head when they think of anonymous comments.

Maybe more importantly, I am not sure we need to have this debate on anonymity. My impression from the previous thread on whether we want to have comments in the first place was that people mostly thought we should concentrate on our core competence, hosting preprints, and do the reviewing/commenting elsewhere.

Shall we make a poll on the question whether we want to have comments on EarthArxiv?

VV

Victor Venema Fri 15 Dec 2017

I made a poll on the question whether we want to have comments. https://www.loomio.org/p/7m2KbkCB/comments-below-eartharxiv-manuscripts