PWRD RESOURCE SITE: Improving Current Practice – IEEE Transactions on Power Delivery

Discussions on Why a New Review Scheme is Needed

PWRD editorial board welcomes the suggestions, comments, and feedback from the reader, author and reviewer community of PWRD. Please send your comments to EIC or editors. Comments of general interest to the community will be posted here (with the agreement of the writer and with minor editing to hide information that may identify the writer). If applicable, responses will be included as well.

===============

I would like to give a suggestion: clarify accurately (for yourself) the objective of a change, before to implement it.

For example, in the recent changes, the need of them does not seem to be very clear. From my point of view, the system was working very well, and it was simpler. Maybe, for this type of systems, where many people is involved, it is advisable to follow the well-known phrase: “if something works, don’t change it”.

===============

EIC Email #1 on the need for improving paper review

Some reviewers/authors have asked me to clarify why PWRD makes change to its paper review scheme. Since my answer may be of general interest, I am sharing it with you.

In recent years, IEEE transactions including PWRD have experienced increased authors and reviewers from various countries and diverse research cultures. There is also a new generation of authors and reviewers. Many of them have different opinions on what constitute a high quality paper. The lack of a common understanding has resulted in cases like the following:

* A innovative paper was rejected due to lack of mathematical sophistication;
* A paper using latest fancy signal processing method to solve a non-existent problem received favourable reviews;
* A paper that presents field results useful to industry was labelled as lack of novelty by more reviewers;

If we don’t do something about this situation, the nature of PWRD will change overtime. Its usefulness to readers might eventually disappear. As a result, a rubrics-based review scheme is adopted as one of the solutions to address the situation. We hope that the rubrics will clarify the characteristics of an acceptable paper and the focus of paper review. It is also intended to enhance the consistency and transparency of the paper review process. Furthermore, the rubrics can be updated gradually to reflect the latest interest of a journal as it evolves.

Also due to the above changing landscape, PWRD editorial board recommends prolific authors to contribute more to paper review. It typically takes 3 to 4 reviewers to process a paper. If such authors don’t contribute to the review, their papers may end up with less experienced reviewers who may not appreciate their work. We also appreciate frequent readers to contribute to the review process since it will help to ensure PWRD continues to publish papers of interest to such readers. We need to work together to make PWRD useful to authors, reviewers and readers.

EIC Email #2 on the need for improving paper review

My previous email of clarifying the new paper review scheme has received many enthusiastic comments and a few additional questions. I feel it is useful to share the following information with you:

1. The new review scheme is implemented as a pilot project for PES transactions. It was supported by PES and PES publication board. Prof. Marisa Crow, VP Publication of PES, is instrumental in making this change possible. So some changes may also happen to other PES transactions down the road based on PWRD experiences.

2. The scheme was shown to the IEEE publication board. Due to strong interests of some board members, it was discussed at one of its meetings last year. I don’t know what came out from that meeting. I guess the board may consider review scheme a society/journal level subject, so it may be up to a society/journal to decide what to do.

3. In addition to using rubrics, other changes adopted by PWRD to address the changing landscape include:

(a) Classifying papers into three types (research, application and review). Each type has a different set of evaluation rubrics. Authors decide their paper type when the paper is submitted;
(b) Authors do self-evaluation during paper submission. They are recommended to provide support comments for their paper with respect to the rubrics;
(c) Designation of “editor’s choice” paper. This is a form of recognizing high quality papers. The papers also serve as examples of the type of papers encouraged by PWRD;
(d) Editors with industry background are relied upon as much as possible to process papers of application type.

4. The rubrics based evaluation method is not new. It has been used successfully for proposal assessment, scholarship competition, and student achievement quantification etc. PWRD plans to review (and update if needed) the rubrics about every two years. So you are welcome to email your suggested improvements to me any time.

===============

Your message came as a pleasant surprise to me, being a humble individual member of the IEEE TPWRD community. I can only give you my sincere support.

I myself have noticed through last few years, not only in TPWRD, but in other journals of other publishers, too, that the review process grew more and more volatile, as you described on several examples.

There’s yet another aspect of the review process I would like to point to, given the fact that IEEE publications are of importance to global community. I think it would be worth to think about double blind review, for quite a few reasons. Here’s one:

As a TPWRD reviewer, for quite a few years I had a feeling was systematically being assigned with reviews of papers authored by nationals of countries from former USSR, as well as authors of Slavic ethnicity. I guess that was because I come from Eastern Europe. But, what kind of criteria was that?? I’m sure most people would consider this kind of practice almost a bit rasistic. I think the idea of “peer review” was somewhat distorted here.

Nevertheless, I just wanted to share a few thoughts on the subject, motivated by your efforts to improve review process, which I regard very important and, above all, needed.

EIC Response: Double-blind review scheme has been discussed by PES publication board a few times and was a hot topic of discussion during the EIC-Reviewer-Author meeting at the 2015 PES GM. The arguments for not using the scheme are (a) Reviewers may figure out authors names anyway, based on writing style, references and research topic. There is a risk that they may misidentify the authors. (b) If reviewers know the authors, it becomes easier to search papers previously published by the authors and check if a substantial advance is being reported in the paper under review, (c) Assuming reviewers are fair, knowing author names (rather than guessing author names) will help to avoid potential conflicts of interest. (d) It is more difficult to detect cases where papers are submitted to two journals simultaneously. Due to such considerations, double-blind review has not been considered for implementation at present. Note that IET had double-blind review more than 15 years ago. Now it has adopted single-blind review scheme. However, we are open to making changes if reasonable solutions to the above concerns can be found.

===============

Thank you for providing excellent reasons. I have seen many useful and very practical papers in which they are not trying to show-off mathematical skills / fancy signal processing because they focus more on the usefulness of the papers to the industry and the readers.

===============

Well put. As a long-time associate editor for TIE I can assert to these problems. Particularly it is frustrating how often people who are asked to review do not even bother to click “no” – they seem just to delete the email. That’s why I almost never say no when asked.

===============

I appreciate that you are trying to improve the review process. I read your comments and I agree with them. It would a please that I can help you to improve quality of Power Delivery papers.
My paper was rejected by Power Delivery community recently. In my opinion the paper was novel and explained new concept, but the reviewers comments were really strange and far from the topic.

===============

It is very important to clarify the “parameters” of papers acceptable for PWRD. A presentation of an IEEE Transaction (not in power area) suggests “Papers that are mathematically difficult to understand get accepted more easily”. Authors and reviewers of PWRD may think this “insight” also applicable to PWRD. I hope this will not be the case for PWRD and other PES transactions.

===============

Thanks for asking my opinions.
1. Straight forward I wish to mention that the review committee of IEEE trans. PWRD papers must be decided based on the reviewer’s experience in that field, but most of the experienced persons are either not free to do review due to their heavy work load or not willing.
2. The guide lines may be issued to reviewer as well as authors to improve the quality of papers published in IEEE trans. Some webinar may be organized for new reviewers.
3. If the paper have some innovative work or idea and some problem in writing or lack of mathematical sophistication, the author must be motivated and helped to change the paper based on comment received from reviewer, rather directly rejecting the paper.
4. Now IEEE is opened their transactions for an option of open access of Trans. papers, so some honorarium may also be given to reviewer’s to improve the quality of review work.
5. A Review certificate may also be issued to reviewer as recognition.
6. The maximum number of papers to be reviewed by a reviewer may be limited.
These are the few ideas in my mind to improve the quality of review process.

===============

Many thanks for this. The PWRD will be interesting soon only researchers not industry if you do nothing. Please consider this as an anonymous point of view.

I remember 10 years ago a speech in a Plenary Session at the General Meeting, an IEEE speaker ask to the audience writing more “application” papers related with industrial problems. Personally I had submitted paper two years later (and recently last year) rejected exactly for the raison in your first sentence. The problem is IEEE forgot ask reviewers to follow this! Another problem is 100% of the 3-4 reviewers have to agree the paper; and if a reviewer comes from Academic world, PhD, these guys lives with mathematics and focus on them to the detriment of industrial problems. The solution of this problem is a question of balance.

I am an engineer not a researcher, but I work with them in RD project 20% of my work time.

===============

Surely all the authors and reviewers worldwide have different experiences, different professional background and different interests. Nevertheless, I also feel there is a problem worth thinking about by both reviewers and authors.
For example, some reviewers strongly require authors papers something “novel”‘, something “mathematic”, but do not like something “classical”, do not like strict validation.
Personally, I think something novel but, meanwhile, must be validated and shown its usefulness, effectiveness, and current or potential applicability.
On the other hand, the well-established “classical” experimental and numerical methodologies are now and will be used in the future.

===============

I agree that the process has problems. This goes far beyond the PWRD – it applies to PES General Meeting and T&D paper reviews too. It is a major challenge.

Fundamentally, many – perhaps a large majority – of the papers are written at a level that many of us can’t be expected to reasonably comprehend but we are still mandated to review. We are seeing fewer and fewer Academia folks attending PSRC meetings. As a TPSP for the PSRC I often needed to have non-technical reviews made because the vast majority of papers were purely Academia, mathematical or basically PHD level research and there are far too many submittals for the few truly qualified people to review.

I agree paper submitters should be encouraged/required to review similar work of others – how that can be done is a challenge.

There should be totally different paper review request levels based on application or theory.

===============

Excellent summary, I could not agree more with you on the shift of the attitude of the reviewers to what we call innovative work. I think our research should be innovative and useful to the power engineering community at the same time. I would like to commend you on your effort and your leadership to make the IEEE Trans. PWRD one of the best journals in the field.

===============

In general it is not my use to discuss editorial polices. However, I think I should address some points on your reasonable comments on PWRD issues.
1- Recently, I was discussing with one of my PhD students a proper Magazine to address his publication on VFTs. If I am not completely wrong, one of the first results obtained in a full complete 550 kV transformer bay. Due to the full absence of a nice set of equations, it is a full experimental paper, showing the set up implementation, for sure, able of critics, and also the results, that in part disagree with CIGRE recommendations, I suggested him to send the paper to the Insulation Magazine Review not to PWRD.
I believe that the results are really interesting to the industry. However, I was not encouraged, if you wish, got the necessary guts, to submit it to the PWRD.
2- A little bit early, also in this year, I suggested to one of my partners at the XXXX – High Voltage Laboratory to send to the PWRD a paper addressing the aging phenomena of metal oxide resistors (Varistors) used in the manufacturing high voltage surge arresters. The idea is to verify the fingerprints of high current short duration pulses on the microstructure of these constructing parts. This can be useful in designing new tests or even to address better evaluation procedures in standards. The paper was not accepted because the subject appears to be related to material sciences, even the set of subjects of PWRD include Metal Oxide Varistors.
Resuming, or I am becoming old, what seems reasonable, or there is some lack of understanding on engineering matters, may be both.
Anyway, some points must be made very clear in order to keep the business running. I am in deep accord with your considering the set of points and I would like to get a better insight to be a better adviser to my colleagues in publishing matters.
Greetings and hoping that you succeed in your “Crusade”

===============

We had a reviews system that worked very well when the reviews were handles by the technical committees.

===============

Since our Transactions is on engineering matters related to Power Delivery, it is good to see that you are proposing a way to have engineers and scientists publishing together in the same transactions. The authos self-evaluation will help the reviewers to understand the real contribution. My suggestion is that this
self-evaluation should be limited to one page (A4).
I am sure all these measures will make the Transactions more valuable.

===============

A coherent review process as suggested by your emails is a very important step towards the sustainable success of PWRD as well as other PES journals. In your previous email, you mentioned that “The lack of a common understanding has resulted in cases like the following:

I would like to add a few:
* A innovative paper was rejected due to extremely demanding comments;
* A innovative paper was rejected due to the perception of lack of practical applications;
* ….

Our power industry is a very old and conservative industry. Such a conservative culture is reflected in paper review, and this will need to be changed over time. We could rely on editors from Industry for application type papers but we should be aware of their limitation and conservativeness.

===============

The new paper review scheme you are suggesting is a good evaluation scheme. Other than that I want to suggest that if you can incorporate blind review process in this journal. As authors cannot see which expert review their paper, experts should not see which author suggested the idea. Only thing you have to do to implement the scheme is not to include authors information in manuscript and provide authors information in separate sheet. Please consider this aspect of reviewing process.
EIC Response: See the response to a similar suggestion.

===============

Considering that you have mentioned the comments that you have received I would kindly ask you please to have some thoughts of the following change.
The proposed change is to make the reviewers unaware of the authors of the paper. The authors in my view should not be known to the review process till the completion of the same.
In my view this will strengthen the credibility of the entire reviewing process and the fact that both papers and authors should be treated fairly and equally in it.

EIC Response: See the response to a similar suggestion.

===============

I greatly appreciate your efforts for these changes. In recent years, I have been losing faith in PWRD quality paper publication due to more volume of impractical with complex theoretical papers, as compared to application or field deployment based papers. From Industrial application perspective, I am looking forward to this new rubric into PWRD, and anticipate this would encourage more industrial participations, esp. with “Application” classification. I will be happy to help PWRD for Application type paper from industrial implementation perspective…

===============

I think the reviewer should bear in mind the audience of the paper, and for this it would be helpful to know a little more about the background of the readers.
For my part, I add a criteria: “is the publication of this idea useful to the community?”
In one case, I rejected a paper since it would have stirred controversy on a topic that has been painfully negotiated in a standard group. The group was aware that the solution was suboptimal, it was a compromise.

===============

Thanks for trying to change this. I agree with the approach. Another issue (but that may be harder to say openly) is that I found reviewers having sometimes no insight in the subject at all. A few years ago a reviewer rejected a paper because “lack of novelty” whereas it was clear from the comments that he didn’t have a clue what harmonics or interharmonics are. The most embarrassing was in that case that the editor took over the recommendation.

===============

It is really a good try to introduce the self-assessment system, and I found it can not only help to improve the effectiveness of revision process, but also help the authors themselves to rethink the significance of their work.

But also, I found it may be not easy for the authors to write to much words for the evaluation of their work, so I hope the evaluation process can be simplified a little. And also, it may be not so suitable to ask the authors themselves to rank their work because the scores may be not so objective.

===============

As frequent reviewer for power delivery and power electronics transactions, I believe there is a need for the reviewers to provide clear reasons for their scores. I think this may help associate editors to judge the authenticity and soundness of individual reviewer’s comments based on the level of his/her expertise in the area. Although it takes effort from reviewers, as an active author I will see it as an important safeguard against one line dismissive review on several months of intensive work from authors.
I hope that these new improvements will take into account some of the aforesaid points.

===============

I just finished reviewing a paper at Editor XXX’s request – I assume you can look up the review. It’s another example of a student project being blown into something of interest and in this case is a third paper based on the same material, although the authors have done a good job to avoid accusations of overlapping by changing content details. I read the original Cigre paper in 2014 and remember thinking then, who accepted this paper???

The best IEEE papers used to come from utilities but unfortunately no more unless from China. IEEE PD seems to be going the same direction as the IET (former IEE) Generation, Transmission and Distribution Journal where the papers are virtually all academic and unread outside of academia, if then. IEEE is inadvertently promoting this outcome. I attended the last APPEEC conference in Hong Kong where the papers, with a few exceptions, were from universities. Most of the papers were not good (aka bad) and yet get referenced in Xplore but wouldn’t have a prayer of getting into INSPEC. Somehow IEEE management and IEEE PES has to address this, get the message – it’s not about making money!

EIC Response: Thanks for reviewing the paper. I agree with you that there are more and more low quality papers submitted these days – transactions are being hijacked by academic papers. This large trend is hard to fight since some journals thriving on the trend. At PWRD, we try to do our best to maintain its original characteristic. The rubrics based review scheme is developed precisely for this purpose. PES has a journal completely designed for industry papers/readers. However, authors have to pay for publishing. Perhaps, we shall have an author-pay journal just publishing highly academic papers.

Reviewer’s follow up email: That [industry] author-pay approach is a sick joke which essentially opens the door to vendors trying to push less than desirable products. I agree with your suggestion for an academic-pay journal. I really like the rubrics review scheme and it should (hopefully) promote better consistency between reviewers.