Autocompletion for `Path`

[tiangolo]

Privileged issue

• I'm @tiangolo or he asked me directly to create an issue here.

Issue Content

It seems autocompletion for Path types is not working correctly.

from pathlib import Path

import typer

app = typer.Typer()


@app.command()
def f(p: Path):
    print(p)


if __name__ == "__main__":
    app()

Running that, for example with Typer CLI:

$ typer demo.py run <TAB>

that should show the available files in the current directory. In Zsh it doesn't show the files, in Bash it shows the files.

Then, let's say it's run from the same Typer directory, where the LICENSE file is located:

$ typer demo.py run L<TAB>

That should autocomplete LICENSE, but it only adds TAB/spaces characters.

[svlandeg]

Related reports: #682, #625

[svlandeg]

I've been looking into this for a bit (these are quite technical internals 🤓)

So one route we could consider is something that's already marked in the code in _completion_classes.py, is to have the format_completion functions return CompletionItem.type alongside CompletionItem.value, cf. the commented lines for BashComplete:

    def format_completion(self, item: click.shell_completion.CompletionItem) -> str:
        # TODO: Explore replicating the new behavior from Click, with item types and
        # triggering completion for files and directories
        # return f"{item.type},{item.value}"
        return f"{item.value}"

Once we have access to the type, we can do something like Click does, e.g. their Bash script:

_SOURCE_BASH = """
(...)
    response=$(env COMP_WORDS="${COMP_WORDS[*]}" COMP_CWORD=$COMP_CWORD \
%(complete_var)s=bash_complete $1)

    for completion in $response; do
        IFS=',' read type value <<< "$completion"

        if [[ $type == 'dir' ]]; then
            COMPREPLY=()
            compopt -o dirnames
        elif [[ $type == 'file' ]]; then
            COMPREPLY=()
            compopt -o default
        elif [[ $type == 'plain' ]]; then
            COMPREPLY+=($value)
        fi
    done

    return 0
}
(...)
"""

which has specific support for different types. For plain types, the behaviour would stay as before, but for dir and file we can trigger autocompletion here. A quick check on Bash on my system confirms that this should work - but it will require proper implementation & testing for all the shells (I'm sure the docker container will be super useful for this!)

@tiangolo: is this the direction we want to go in? Or did you have a different type of solution in mind?

[helmut-hoffer-von-ankershoffen]

It's a step down from Click that path autocompletion does not work.

On mac with oh my zsh I get (eval):1: command not found: WARNING

[tiangolo]

I was checking these internals for a bit, now I remember, yep, these are very technical details. 😅

I have been writing this reply below for the past few hours, so it's super long. 😅

I'm adding a lot of context and info here, maybe too much. And I might be wrong in some of my assumptions. But better to state the obvious than to hide the unknown.

Also, I should probably clean up a bit of this text and put it in the docs somewhere. 😅 I think there's no central place on the internet explaining how to make these shell completion systems, at least not at this level of abstraction (only much lighter or much more low-level).

So, sorry in advance for so much text. ☕

How shell completion works

I think this is an important detail that I have never documented anywhere, and I don't think it is documented anywhere in general, it's just discovered by anyone who dares to try to build or work on these things. 😅

The way these shell completion systems work, at least from what I have seen, is this, and it applies to all, Bash, Fish, Zsh, PowerShell (and I suspect any other):

• There's some place to run startup commands and scripts, in Bash it would be ~/.bashrc (and others, but that's the main one). Each shell has some variant of this.

• In that config file, something will install the completion for the CLI program. It could be a command that loads and evals another script, loads another file from the filesystem, or maybe right there inline in that file there's a long shell script that does all the logic to install the completion. The point is that it's some code/script in the language of the shell (Bash, Zsh, Fish, PowerShell).

• That script or command (or whatever it loads) will at some point call some function/command that is part of the shell, with a couple of parameters, one telling the name of the program, (the CLI program) and the other with the thing to be called to get completions for that program. It can have more parameters but those are the main ones. That thing is normally a function created (or loaded from another file) in the language of the shell.

  For example, in Bash, the specific command to define completion for a program is complete, so the script line could look like complete -o nosort -F some_custom_bash_func myawesomecli.

  Another example, in Zsh, it would use #compdef myawesomecli and then later compdef some_custom_zsh_function_name myawesomecli.

• After that, whenever the user types in the terminal the name of the CLI program (e.g. myawesomecli), a space, and then TAB, the shell will right there in that instant call that thing (function) we declared above, and will pass it some environment variables with the current command and any arguments, etc. The specific environment variables passed change from shell to shell.

• That thing that is called (normally a function) can, in turn, call other things, for example, it could call another script, another binary, or even the same CLI program for which we are trying to get completion. And then it has to tell the shell what are the available options for completion.

• The way the thing (function) tells the shell what are the options for completion is:

  • For Bash, Zsh, Fish, it has to print stuff in some format. The communication back to the shell is via printing (writing to standard output).
  • For PowerShell, because it likes to be weird, the communication back to the shell is not printing (writing to stdout) but calling an internal PowerShell function.

• The shell will then take this output from that thing, which should be in the format the shell expects, will parse it, and will use the options received to render the completion options in the terminal.

The important points are:

• The installation of completion is by calling some shell-specific thing at startup.

• That will configure some thing to be called when completion is needed, that is normally a function in the shell language.

• This thing is the entry point, from the shell, to our logic. It will receive whatever arguments (sub-commands, CLI parameters, etc) are currently passed.

• This thing can, in turn, do whatever it needs, it could call another program (including our own CLI program in some way).

• This thing will return in some way (in most cases by printing/writing to stdout) the options to complete in the shell. And this output has to be in the format expected by the shell. This is our output passing control back to the shell.

• Then the shell renders the completion options.

• Then once the user finishes setting any sub-commands or parameters, and hits enter, the CLI program will run as normally.

How completion works in Click and Typer

In both Click and Typer, when you install completion, what it will do is add a script in the language of the specific shell to the startup config file for that shell.

This script is a function/script, it takes the input of the current arguments passed to the program, and then it calls the same CLI program with some extra env vars that tell Click or Typer that completion is being requested and the current arguments.

Then the CLI program is run with those extra env vars, Click or Typer detect them, and generate the completion for the existing commands or arguments. If there's a Python function to handle autocompletion for some CLI Parameter (so, a function written by the developer building an app with Typer), it is run to get the possible values. The result of that is printed (written to standard output).

And here's a funny detail, all that stuff that is "printed", doesn't show up in the terminal, that output is parsed by the shell asking for completion (or in PowerShell, by the function calling the program to then call the PowerShell-specific thing to set completion options).

And then those parsed options are shown in the completion in the shell.

All this happens in an instant, when the user hits TAB, before executing the command that the user intends to execute.

This shows some interesting details:

• The CLI program is called during completion (with some extra env vars) to provide the completion options for itself. Before the user actually hits enter to call the program.

• The CLI program (Click or Typer) will print stuff to communicate the completion options.

A couple of interesting conclusions:

• The top-level modules should not have expensive operations at load/import time (e.g. at the top level of the module, outside of a function). For example, loading an ML model at the top level. This would be executed during completion, after a TAB, so it could make completion very slow and waste resources loading something that si not needed yet.

• The top-level modules should not print stuff (write to stdout). For example, creating a top level config object and printing "config loaded". Because this will be printed during completion, and that would break the expected format for the completion.

Click 7, click-completion, Typer

On Click 7, it only had integrated support for simple Bash completions. There was a package click-completion that extended it by modifying stuff to support other shells: Fish, Zsh, and PowerShell.

If I remember correctly, some of those completions were not working correctly, I think PowerShell was not working at all.

I copied the shell scripts from there into Typer and tweaked them (e.g. fixing PowerShell). I think there were some issues with the others too (Zsh, Fish) but I don't remember exactly.

Click 8

Click 8 re-built the shell completion system. It added support for Fish and Zsh (but no PowerShell).

One important but non-obvious difference from Click 7 to Click 8 is: the keyword to trigger completion passed in an env var from the shell (e.g. Bash) to Python was reversed from complete_bash to bash_complete. I'm adding this note here because I remember spending a long time debugging things and it was just that little change.

Click - Typer - completion at runtime

There's another important and non-obvious difference between how Click expects things to happen vs Typer.

Click

Click has docs to install the completion that assume that the program is available at shell startup time (or however that is called). So, the program has to be installed globally before the shell completion config is run. So, before the ~/.bashrc is run (or ~/.zshrc, etc.), the executable program needs to be available, in the PATH.

This means that if a Click program is installed in a venv, then shell completion is installed, the next time a new shell is started, it will show an error saying that the program was not found. And loading the venv would not solve the problem, it's too late by that point. So, Click programs don't support completion if they are installed in a venv, and installing completion for Click programs in venvs would break (at least a bit) part of the shell startup.

Typer

Typer's completion stuff only runs the CLI program once some completion is requested, not at shell startup time. This means that, it's possible to install some CLI app inside of a venv, install its completion, and it will work after the venv is activated. And it won't explode at shell startup time if the venv is not activated and the CLI program is not yet in the PATH. Once the venv is activated, completion will work.

Click - Typer - completion source

Click

Another difference is that, Click has two completion installation modes, one is to eval a command that would call the program and load the shell script in the ~/.bashrc (or ~/.zshrc, etc). Again, this needs the program to be on the PATH at shell startup time.

The other mode is to generate the completion shell script and save it to a file, and then source that in the ~/.bashrc. This has the inconvenience that now there's a big file stored in the file system with a fixed script for the completion. If the program is updated and the commands change, the completion script will be obsolete giving wrong completions until that script generation is run again. But because it is already providing some completions, it's probably not obvious that the completion needs to be installed/updated. So I suspect this could cause frustration to users.

Typer

The way Typer works, taken from how click-completion used to work, is that the completion shell script that is installed is very minimal, and it doesn't call the CLI program on shell startup. Instead, it declares a plain tiny function that will be in charge of handling completion, and configures the shell to use this function as the completion thing for the CLI program.

When the shell needs completion and calls the function, that function will then (and only then) call the CLI program with the extra env vars. And the CLI program is the one in charge of printing things in the right format... except for PowerShell that needs to call a PowerShell function with the completion options.

So, if a CLI program is installed in a venv, and completion is installed, it won't explode at shell startup time, and once the venv is loaded, it will work as normally.

In fact, if two venvs have different versions of the same program, as the completion script is so minimal, it will work for both, and both could have different completions depending on the venv activated. This is an exotic case, but it's possible.

Click 8 shell scripts

In Click 8, there's support for setting a completion type, between dir, file, and plain (with plain being the regular completion, e.g. sub-commands, CLI options, values, etc.).

The completion systems for the shells have a way to say "hey, do whatever completions apply to file paths", or the same for dirs. This is the new feature feature in Click, supporting setting those configs.

But the way it is done in Click handles a lot of the logic in the shell-specific script. So, there's a fair amount of logic in each shell scripting language inside of strings in a Python file. I would prefer to avoid that or reduce it to the minimum in Typer.

My reasons to avoid script shell syntax in Python strings:

• It requires more knowledge of those shell script languages. It is needed to understand and maintain those scripts.
• There is no editor completion, inline errors, coloring for shell scripts inside of Python strings, so it's harder to write and maintain them and detect errors.
• Most of that logic can actually be done in Python, and we are already calling Python anyway, so we don't lose anything if we do it in Python.

I would prefer to handle the logic that is currently in those shell scripts in Click's source code in pure Python, and pass the information back to the shell script as ready as possible, to minimize the amount of shell-specific code.

For example, I think the shell function that calls the CLI program can keep calling eval as currently, and then the Python side would be in charge of printing any extra commands to tell the shell if it should complete files, dirs, or a set of values.

...there's still the amount of logic in the current PowerShell script, which is similar to the new flavor of Click, with a fair amount of logic in the shell scripting side. But I couldn't find a way to make it just eval some string or have a plain function without so much logic. It would be great to be able to simplify this one, but that's harder, because PowerShell is quite weird. So maybe for later.

---

So, yes, I would like us to be able to handle types and have completion specific for files and dirs as well, but I wouldn't like to just copy Click's code, I would want to do as much of the logic as possible in Python.

[bckohan]

I wrestled with these same issues on django-typer. I think the direction of keeping the installed completer scripts as simple as possible is the right way to go if you can make it work.

I ended up going the click route instead, because the django use case was a little more forgiving of the downsides of using more complicated shell scripts and had requirements that necessitated more complex shell scripts.

One suggestion that might help make the code more readable: instead of including the scripts as strings in the python you could save them as individual files and load them as resources. I do this here.