Mahala Nobis in the era of Big Data and AI

General Studies  3

Introduction

• Professor P.C. Mahalanobis, who introduced statistics to India, is a scientist whose absence is felt dearly even today. Mahalanobis’s lifelong courtship with statistics, his unwavering and fearless leadership to advance a statistics and survey culture in India, the founding of the Indian Statistical Institute — “a mighty monument of his handicraft” — and his nurturing of a generation of outstanding academicians have all left behind an enduring legacy.

The age of Big Data

• Over the past 20 years, there has been a global shift in both the nature of data and statistics.
• With the advent of the Internet and virtually everything confined to the Internet of Things, there has been a flood of data, most of it junk.
• We now have much more data than what available technology can leverage. This is widely perceived as the era of Big Data.
• Another significant yet related issue is how artificial intelligence (AI) is transforming our lives and lifestyles.
• The state of society is precarious. One can wonder how Mahalanobis, a statistical doyen and a key figure in the early development of Indian democracy, would have responded to the Big Data-related craziness and the AI-driven revolution.
• Historically, data often appears to be Big when the available technology at that time fails to analyse it.
• Mahalanobis also encountered a Big Data problem when his large-scale surveys yielded lots of data that needed to be looked into for effective planning.
• To handle tons of data and tackle the complex mathematical calculations, Mahalanobis persuaded the government and succeeded in procuring the first two digital computers of the country (and South Asia, too) at his Indian Statistical Institute, in 1956 and 1958, and thus ushered in the age of computers in India.

Problems during COVID-19

• Mahalanobis was “a physicist by training, a statistician by instinct and an economist by conviction”. He had an uncanny knack for embracing technology for human welfare, perhaps as a result of his background in physics.
• Thus, one may safely perceive that Mahalanobis would have embraced the power of AI in enhancing human productivity, such as Big Data analyses, and perhaps in a way that is far more effective than how AI is currently applied to that goal.
• One recent Big Data foible, for instance, involved numerous contradictory projections during the COVID-19 era.
• One could argue that if Mahalanobis was alive today, the country’s COVID-19 response could have been much stronger.
• If he was in the lead, our data might be beyond question, and the analyses might be far more accurate.
• And India’s “Plan Man” could be the best person for planning to build optimal health-related infrastructures for combating future disasters.

AI regulation

• Around seven decades ago, from the perspective of the newly independent nation, planning — with the aid of extensive technocratic exercises with democratic participation — moved from the realm of politics, primarily due to Mahalanobis. Now, we are at the crossroads.
• India’s upcoming census will be a digital exercise. The dynamics of other surveys are also bound to change in the new normal setup. That is how statistics is evolving. We would miss the leadership of an expert such as Mahalanobis from this changed statistical perspective.

Conclusion

• As AI is threatening to replace millions of jobs without creating alternatives and is also aiding in spreading disinformation, there is a substantial global attempt to clip its wings.