Completing Arbitrary Paths for Non-user-interactive Retreival

[psionic-k]

I prototyped some tools for Elisp to use an LLM to recursively look up information, basically using Emacs introspection features as a RAG source. The results are extremely good.

I would like to do the same with the language server integration, but the LSP protocol seems highly focused around user-interactive queries.

Endpoints that would be useful instead would look up the packages that can be imported via the Cargo.toml and provide human-readable index data such as a list of modules in a package. Packages of interest would be queried for a list of public symbols. Once inside modules to pull source code to present to the LLM, the list of symbols found within the function (LLM can infer from source) would be used to perform recursive lookup to analyze implementation behavior.

Are there any endpoints I missed where I can already do this or does the language server protocol and API need to be expanded to support these new kinds of queries?

[Veykril]

So, you want to be able to query the packages of a cargo workspace, and then the resulting module tree for each of those packages to fetch all relevant source files? That is not something that is accessible via the LSP by default no. It also sounds unlikely that we would expose something like that (though you could use r-a as a library to build a tool that does that for you)

[psionic-k]

It also sounds unlikely that we would expose something like that

Is there technical reasoning?

[flodiebold]

Is there technical reasoning?

LSP is focused on user interaction, as you have noticed, so something like that would be out of scope for it. And implementing a whole different protocol would be out of scope for us. But as mentioned you could use r-a as a library. (Or maybe rustdoc json would be a better fit for you.)

[psionic-k]

I'm open to using RA as a library, especially to demonstrate the use case and guide this towards the usage pattern I'm highlighting. The limitation right now is not that RA doesn't have the data, it's that it wants to complete from a concrete point in a concrete file.

What I've been working on in Elisp is the re-use of indexes that were made for humans, such as completion and narrowing of available functions, to then employ exact-match source code navigation tools. The LLMs readily consume any index made for humans. Since LLMs are capable of recognizing which parts of a function implement some behavior of interest, they can then recursively use the exact match navigation through several layers of functions to discover the implementation. This is very powerful at drilling into an unfamiliar library. If I had this for Rust, I might be coding right now instead of talking. Diving into RA will take me some time.

How this will look with RA:

1. Anything re-usable as a semantic, human-readable index, such as a list of modules or a list of functions within a module or trait, are available for fetch by the LLM via tool-use integration in the Editor.
2. The LLM decides which of the functions appear semantically related to the user's query, a problem for which the heuristic capability of LLMs is a great application.
3. The LLM recursively makes additional tool calls through the editor integration to RA to complete module paths, fetch documentation, and navigate to source
4. Once the editor / LLM integration are brought to sources, on their own they assemble the relevant expressions into a context and reduce it to possible answers, summarizing and deducing toward useful conclusions.

The issue is really only 1. I have to give the LLM starting points that it can semantically (and based on embedded expectations) associate with a user query in order to later employ exact-match based source navigation. Once the Editor / LLM are brought to source, RA's job has been done. There's really no need for protocol or RA doing almost anything except completing strings without file positions.

To pump this conversation, we can presume a library is the better choice. What parts of RA do I want to be familiar with to obtain the starting points to prepare for exact-match based source navigation?

[flodiebold]

Have you looked at rustdoc json? It should give you the "semantic, human-readable index" of a crate's API that you want. If you do want to use RA though, take a look at analysis-stats as a starting point.

There's really no need for protocol or RA doing almost anything except completing strings without file positions.

That's not an operation that makes sense; the set of available crates, and even their names, is heavily dependent on where you are in the crate graph.