The Quality Of Pakistani Research

Scholarship funding from both government and foreign agencies for domestic and foreign studies is at an unprecedented level. Money available for graduate studies has never been available in Pakistan so plentifully. The state of affairs is in sharp contrast to the 80s and 90s when all that was available were a few dozen scholarships a year through the Science & Technology Scholarships Scheme. The Higher Education Commission (HEC) deserves to be commended for removing the first major hurdle in improving the state of higher education by making funds available for direct use by scholars.

Over time the HEC added several more funds to its portfolio for purposes ranging from travel to conferences, research projects, post-doctoral studies etc. Most of these opportunities are only open to aspiring Ph.D.s, with a heavy bias in favor of science and engineering students. ‘Do research’ is the ongoing mantra at the HEC. Every year the glut of scholarships is bringing more graduates into the fold of researchers-in-the-making. The Pakistani higher education sector’s government sponsored dash to catch up with the rest of the world has time and again been the scrutiny of the press. Issues such as appointment of ex-military to academic positions, the tenure track system and allocation of funds for research projects have frequently been the topic of debate in this context. And while there is frequent talk about the increase in the volume of research publications now being produced by our researchers I have so far not seen anyone touch upon the issue of quality of their accomplishments.

With the rush to ‘do research’ comes the inevitable rush to publish. Recent years have seen an increase in the number of publications from within Pakistan and Pakistanis studying abroad. A first glance at the publication lists that are now popping up will lead anyone who is not a domain expert to believe that the HEC’s strategy of throwing money at the dismal state of higher education in our country is bearing fruit. However, a closer look reveals that all is not well. A while back the Wireless and Video Communications Lab at Michigan State University, with which the author is affiliated with, was looking for potential international collaborators. We finally found the right person, but what I came to observe during that search process has been deeply disturbing and raises serious questions. Is the cost incurred by the HEC in training people to become researchers, producers of knowledge, worth the expenditure? The answer, it now appears, is not as straightforward or positive as one might like to hope. Since my own domain centers around electrical engineering focusing on wireless communication & networking my own evaluation of the quality of publications was limited to this area.

Among the forums available to researchers to circulate and publicize their work are journals and conferences. Articles published in journals enjoy a wider readership and go through a more rigorous and lengthy review process than conference publications and are considered more prestigious. Recent years have seen explosive growth in the number of conferences around the world. Researchers can often find themselves invited to submit their work as frequently as a few times a week by one conference or another. But not all conferences/ journals are alike. In practice, the reputation of the venue in which a paper is published is the measure by which its quality and significance are judged. Unfortunately, there are no reliable statistics by which a venue can be ranked. Merely ‘having published’ is not a sufficient indicator of a researcher’s worth and abilities.

A measure frequently quoted as an indicator of a conference’s quality is its acceptance rate, i.e. the percentage of submitted papers that are accepted for publication. However, there are plenty of unknown conferences that feature acceptance rates close to those of the most recognized conferences in the same area. Some international conferences held in Pakistan enjoy acceptance rates in the 30%+ range which is about the same as the International Conference on Communications, the IEEE’s largest conference. But as a review of their proceedings will verify, many accepted papers are nothing more than reformatted versions of undergraduate project reports.

So what is the solution? Should Pakistani conferences simply be boycotted and all work sent to conferences held abroad? By no means am I advocating that any conference held outside Pakistan’s borders automatically has a higher standard. A few years back the Massachusetts Institute of Technology’s (MIT) Computer Science and Artificial Intelligence Lab (CSAIL) developed SCIgen, a program generates meaningless yet properly formatted and grammatically correct research papers containing text generated from random snippets of text from actual papers. One of those non-sense papers was sent to an obscure and unknown conference, the World Multi-Conference on Systemics, Cybernetics and Informatics (WMSCI’05) held in Orlando, Florida in July of 2005. The paper was accepted.

Needless to say that once the origins of the accepted paper were revealed this incident thoroughly discredited the conference. A mere publication count is not a fair basis of assessing the quality of research at an individual or institutional level or measure the success of a set of policies at an even wider, national level.

To give an idea, even if specialized conferences in my own area are included that does not make for more than ten conferences a year that are worth publishing in. As a matter of fact, a few obscure conference publications on a CV’s publication list can end up discrediting, or at least casting doubt about, an author’s other works. The current state of affairs is that the vast majority of papers from Pakistan and Pakistanis are being published in low-quality, obscure venues. Not only does each of these conferences represent time and money wasted on travel expenses, but they also end up discrediting the author’s ability to produce work of quality.

Unfortunately, the current yardstick used for judging the quality of a publication by many institutions is whether a paper was published in an international or domestic conference/ journal. This is not to say that there is any conference in Pakistan that is treated unfairly by this rule. But if a conference is being held outside Pakistan that alone does not guarantee its quality and standing, as the example given earlier demonstrated. An individual paper’s true worth can be judged by the number of citations it has received. A review of the citation counts of papers published in even the best electrical engineering conferences in Pakistan will quickly reveal that excluding self-citations, i.e. citing one’s own work, those papers receive virtually no citations at all. A better statistical measure of a conference’s standing might be the average citation count of papers published in it. However, citations take time in coming and hence such a measure will have to be computed over an extended time window. Keeping track of citation counts for a large number of papers is not an easy task either. A better evaluation criteria may end up being a subjective one.

What is needed is a ranking of venues, both conferences and journals, prepared by experts from each domain. Decisions such as the award of travel funding and the evaluation of candidates for new positions should be based on this information. Granted this solution is subjective and will require a continuous maintenance effort, but it can go a long way towards stopping the hemorrhage of finances and raising the standard of research work. At the same time, there is a need to raise the standards of domestic conferences. There certainly are a sufficient number of qualified Ph.D. holders available in Pakistan to serve as reviewers who are aware of the standards I am talking about. Should those prove to be insufficient in number there is always the possibility of requesting contacts outside Pakistan to serve as reviewers.

Instead of spinning conspiracy theories and blaming ‘foreigners’ for actively undermining our research efforts by not recognizing the work we produce we should accept that the guidance and research culture that gives junior researchers a sense for these things is missing from our institutions. As a quick glance at websites of most graduate schools now offering Ph.D. programs will verify, the vast majority of faculty teaching there do not have any experience of doctoral studies themselves. Many departments are lucky just to have a Ph.D. as their chairperson. Granted, some of our country’s top schools are exceptions to such a generalization, but for some reason even their research outputs fall short of expectations. The things I have talked about so far are no secrets. Everybody, every single person who went through a decent doctoral program, and there are many in Pakistan who did, is fully aware about each and every one of these issues. Then why are so few grad students back home aware of these basics? There seems to be an apparent communication gap between faculty and students which leaves students drifting, unaware of what is expected of them?

If initially a relaxed attitude towards maintaining quality of research output was necessary to jumpstart a culture of publishing then now it is certainly high-time for the HEC to tighten the screws. The government is pouring extraordinary sums of money into this effort, too much not to get some kind of tangible, positive results. Depending on the country of study, some HEC scholarships are valued so much money as many government employees will not receive as salary in their entire lifetimes. Taxes from hundreds of citizens are funneled into one scholar’s studies. Ignoring the quality of research will make the rush for those fabled “1000 Ph.D.s every year” that Dr. Atta has been promising in the press time and again an exercise in futility.

The HEC as well as universities themselves spend significant funds on people to present papers at far-off conferences. There is a need to ensure that this investment is put to good use and travel funds become more than slush funds to finance vacations to exotic locations. Some conferences simply are not worth the money spent on attending them.

Pakistan had to wait nearly 60 years for a government that would take the challenge presented by the state of higher education seriously and have the political will to free up necessary resources to address the issue. We cannot afford to fail now. There is no telling if there will be a Pakistan left in another 60 years.