Using LLMs for Research Purposes – Anecdotal Insights

Introduction

Large language models (LLM) like chatGPT or you.com are fascinating in their capabilities, in particular with internet resource access. So, how useful are these models for research purposes?

In this blog, inspired by a nice blog post by Aaron Tay in Medium, I focus on a challenge we often face in research: understanding fundamental concepts and developing a head start on the topic with a set of relevant, foundational academic papers on the topic. Initial trials of LLMs a few weeks ago have been disappointing in this regard due to the well-known problem of model hallucinations. But LLMs have rapidly advanced, and it seems that now there is hope.

Although this blog is far from trying a systematic evaluation of the quality of LLMs for academic literature search, my recent trials asking you.com, perplexity.ai and chatGPT (using gpt-4) have been interesting.

I just tested some queries regarding research areas I’m familiar with due to my own research interests. Thus, this is at best anecdotal evidence. In this blog, I’m going to focus on „capital structure determinants of firms“ – one of the fundamental questions in Corporate Finance, ever since the seminal paper by the Nobel laureates Modigliani and Miller. So let’s dive into it.

Capital Structure Determinants

Query: What are important academic papers on firms‘ capital structure determinants? Please provide a list of citeable references for these papers, including the journal where it is published etc.

You.com: Here are the academic papers related to capital structure determinants that I could gather: 

  • Myers, S. C. (1984). The capital structure puzzle. The Journal of Finance, 39(3), 575-592.
  • Taggart, R. A. (1985). Determinants of corporate borrowing. Journal of Political Economy, 93(2), 304-323.
  • Fama, E. F., & French, K. R. (2002). Capital structure decisions: The role of financial assets. Journal of Financial Economics, 65(3), 271-292.
  • Lemmon, M. L., Roberts, J. M., & Zender, J. F. (2008). Testing trade-off and pecking order predictions about dividends and debt. Review of Financial Studies, 21(5), 1895-1929.
  • Coates, J. C. (2000). Capital structure and corporate control: The effect of antitakeover statutes on firm leverage. The Journal of Finance, 55(3), 1075-1091.
  • Jensen, M. C. & Meckling, W. H. (1976). Corporate financial structure and managerial incentives. Journal of Financial Economics, 3(4), 305-360.
  • Fan, J. P. H., & Wong, T. J. (2001). The determinants of capital structure: Evidence from China. Journal of Financial Economics, 58(1-2), 353-384.
  • Lee, B. S., & Megginson, W. L. (1997). Capital structure and its determinants in the United States and Japan: A comparative study. Journal of Finance, 52(3),

perplexity.ai: What are the most cited academic papers on capital structure determinants?

„There is no clear consensus on the most cited academic papers on capital structure determinants. However, there are several papers that are frequently cited in the literature. These include:

  1. „The Determinants of Capital Structure Choice“ by Stewart C. Myers (1984), published in the Journal of Finance.
  2. „Pecking Order Theory and the Gibrat’s Law“ by Michael Faulkender and Rong Wang (2006), published in the Journal of Finance.
  3. „Trade-Off Theory and the Pecking Order Theory“ by Eugene F. Fama and Kenneth R. French (2002), published in the Financial Management.
  4. „Capital Structure Decisions: The Role of Financial Assets“ by Michael L. Lemmon, Michael R. Roberts, and Jaime F. Zender (2008), published in the Journal of Finance.
  5. „Determinants of Capital Structure: Empirical Evidence from Pakistan“ by Muhammad Arslan, Muhammad Asif, and Muhammad Irfan (2021), published in the Journal of Risk and Financial Management.

These papers have been cited hundreds or even thousands of times in subsequent research on capital“

chatGPTplus: „Here is a list of important academic papers on firms‘ capital structure determinants. These papers have made significant contributions to the understanding of the factors that influence firms‘ capital structure decisions:

  • Modigliani, F., & Miller, M. H. (1958). The cost of capital, corporation finance and the theory of investment. The American Economic Review, 48(3), 261-297.
  • Myers, S. C. (1984). The capital structure puzzle. The Journal of Finance, 39(3), 575-592.
  • Jensen, M. C., & Meckling, W. H. (1976). Theory of the firm: Managerial behavior, agency costs and ownership structure. Journal of Financial Economics, 3(4), 305-360.
  • Myers, S. C. (1977). Determinants of corporate borrowing. Journal of Financial Economics, 5(2), 147-175.
  • Myers, S. C., & Majluf, N. S. (1984). Corporate financing and investment decisions when firms have information that investors do not have. Journal of Financial Economics, 13(2), 187-221.
  • Rajan, R. G., & Zingales, L. (1995). What do we know about capital structure? Some evidence from international data. The Journal of Finance, 50(5), 1421-1460.
  • Titman, S., & Wessels, R. (1988). The determinants of capital structure choice. The Journal of Finance, 43(1), 1-19.
  • Fama, E. F., & French, K. R. (2002). Testing trade-off and pecking order predictions about dividends and debt. The Review of Financial Studies, 15(1), 1-33.
  • Frank, M. Z., & Goyal, V. K. (2009). Capital structure decisions: Which factors are reliably important? Financial Management, 38(1), 1-37.
  • Lemmon, M. L., Roberts, M. R., & Zender, J. F. (2008). Back to the beginning: Persistence and the cross-section of corporate capital structure. The Journal of Finance, 63(4), 1575-1608.
  • DeAngelo, H., & Masulis, R. W. (1980). Optimal capital structure under corporate and personal taxation. Journal of Financial Economics, 8(1), 3-29.“

The results are interesting. At first glance, results by you.com seem promising, as many of the mentioned authors indeed have published high impact papers on capital structure issues. But a closer look reveals the tricky part of hallucinations by LLMs – except for the first paper by Stewart Myers, none of the other papers exist and doi-numbers proivded are either non-existent or linking to completely different papers.

Results by perplexity.ai are plagued by similar problems…

Notably, however, the answers by chatGPTplus are spot on. All papers can be viewed as having high-impact (although it is not obvious how to define „important papers“ – by citation? by journal? or other criteria?), and all of these papers exist. No hallucinations, perfect. Even Modigliani/Miller included!

Alternatives?

Google Scholar: Google Scholar is sort of disappointing – for sure, somewhere it’ll catch the important papers (Titman/Wessels 1988 is at least in the top hits), but this looks more like results from a key-word match rather than semantic search.

semanticscholar.org: This is a wonderful project by AllenAI, providing in-depth information about academic papers for free, with nice utiltities like full author information, all citations of papers /authors etc. Unfortunately, neither the option „sort by most influential“ nor „sort by citation“ yielded better results as the following snapshot. Again, this seems due to the search being based only on keywords. But one wonders about the sort on citations and the source for these citations.

Conclusions

Identifying and sharing foundational papers is crucial for making scientific knowledge available to society. While LLMs show promise to this end, and are well suited to simplify complex topics to make them accessible to everyone, researchers (and students) must still perform their own groundwork.

However, ChatGPT’s results and its improvements over just a few weeks are already impressive. With OpenAI’s upcoming apps and advances in open-source frameworks like Haystack, semantic search using reliable knowledge bases will become increasingly powerful. I’m looking forward to this.

Schlagwörter: , ,