lfcd improvement proposal

[nicolasdumitru]

Goal

Simplifying lfcd and making it more POSIX compliant.

Issue

Currently, lfcd uses temporary files that it creates with mktemp in order to store the path of the directory that was last open in lf and into it. This poses multiple problems:

1. mktemp is not POSIX. This means that lfcd is also not POSIX (here, by POSIX I mean it only requires utilities/commands found in the POSIX specification and the obvious lf). While mktemp is pretty widely available as a GNU and BSD utility, it could be a problem on some POSIX compliant operating systems (e.g. non-GNU Linux distros).
2. Usage of temporary files is messy, overcomplicated and potentially unsafe (in some cases, not necessarily here).

Solution

Add a flag/option to lf that is similar to -last-dir-path, but instead of writing to a file, outputs to stdout. Why?

1. Lf can already write to a file, it would be trivial to instead print something to stdout.
2. the lfcd function body becomes a trivial, much simpler, more POSIX-compliant and less prone to breakage one-liner: \cd $(\lf -new-flag-to-print-path-to-stdout "$@")
3. lfcd doesn't have to check for the existence of a temporary file and then make sure it gets deleted.

The inspiration for this is fzf. By default, fzf prints the path to what you search for to stdout. Consequently, a user can do something like vim $(fzf) to edit a file that they don't know the exact path to or cd $(dirname $(fzf)) to do some fuzzy searching for a file and cd into the directory of said file. If you are interested and would like to try something similar without having to write a single line of code, try the above mentioned cd $(dirname $(fzf)) and search for a file or a directory where you have files.

[ilyagr]

I'm worried this would get messed up if the user calls any program that prints to stdout before quitting lf. This could be explicit (!cat $f) or a side effect of some previewer, editor, etc.

[gokcehan]

@nicolasdumitru Thank you for the proposal. @ilyagr is right in that it can get messy when shell commands are used. Infact, this idea has been discussed before in #140 and #472. If you want to see this in action, you can try one of the workarounds mentioned in these discussions (e.g. lf -last-dir-path /dev/stdout).

[nicolasdumitru]

@ilyagr valid concern. Do you think that forcing \lf -new-flag-to-print-path-to-stdout to print on a new line and using \cd $(\lf -new-flag-to-print-path-to-stdout "$@" | tail -n -1) could solve the problem?

[ilyagr]

No, it wouldn't. The user might need the output from the commands run from inside lf, and it'd be lost in your setup.

Try running lf|tail -n 1 and then !cat $f inside lf; it's not a great experience.

[joelim-work]

This is probably a crazy idea, but would this be possible if the shell commands were connected directly to the TTY instead of stdout?

diff --git a/app.go b/app.go
index fc9bd11..3057c00 100644
--- a/app.go
+++ b/app.go
@@ -327,20 +327,6 @@ func (app *app) loop() {
 			app.nav.dirPreviewChan <- nil
 
 			log.Print("bye!")
-
-			if gLastDirPath != "" {
-				f, err := os.Create(gLastDirPath)
-				if err != nil {
-					log.Printf("opening last dir file: %s", err)
-				}
-				defer f.Close()
-
-				_, err = f.WriteString(app.nav.currDir().path)
-				if err != nil {
-					log.Printf("writing last dir file: %s", err)
-				}
-			}
-
 			return
 		case n := <-app.nav.copyBytesChan:
 			app.nav.copyBytes += n
@@ -518,9 +504,15 @@ func (app *app) runShell(s string, args []string, prefix string) {
 	var err error
 	switch prefix {
 	case "$", "!":
-		cmd.Stdin = os.Stdin
-		cmd.Stdout = os.Stdout
-		cmd.Stderr = os.Stderr
+		tty, err := os.OpenFile("/dev/tty", os.O_RDWR, 0)
+		if err != nil {
+			log.Printf("getting tty: %s", err)
+			return
+		}
+
+		cmd.Stdin = tty
+		cmd.Stdout = tty
+		cmd.Stderr = tty
 
 		app.runCmdSync(cmd, prefix == "!")
 		return
diff --git a/client.go b/client.go
index 0fab7d5..7fbe32f 100644
--- a/client.go
+++ b/client.go
@@ -61,6 +61,10 @@ func run() {
 	app.loop()
 
 	app.ui.screen.Fini()
+
+	if gLastDirPath != "" {
+		fmt.Println(app.nav.currDir().path)
+	}
 }
 
 func readExpr() <-chan expr {

Even if it does work, I still have some concerns about this idea in general, as it involves breaking changes and involves a fair amount of risk when considering other platforms like Windows (which will require a different implementation).

I imagine that @gokcehan might have tried this already, perhaps there's a good reason this wasn't done.

[nicolasdumitru]

@ilyagr indeed, not a nice experience.
@joelim-work very interesting idea. Provided that it works as far as lf goes (at least on Linux), how would lfcd look after leveraging this change?
in the meantime, I have changed my lfcd to:

lfcd () {
	\umask 077
	tmp="$(command mktemp)"
	command lf -last-dir-path="$tmp" "$@"
	if [ -f "$tmp" ]; then
		dir="$(command cat "$tmp")"
		command rm -f "$tmp"
	else
		\return 1
	fi
	if [ -d "$dir" ] && [ "$dir" != "$(pwd)" ]; then
		\cd "$dir" || \return 1
	fi
}
```.
It has some additional safety measures and, in my opinion, it's slightly more readable.

[joelim-work]

Pretty much how you described above: cd $(lf ...).

That being said, the patch above is just a quick and dirty POC and takes a number of shortcuts. I haven't looked into it much further as I still have reservations about changing the design (i.e. having the shell commands bypass stdout and write to the console directly, just so it won't mess with last directory output).

[ilyagr]

I don't think @joelim-work 's idea is crazy, but I'm not sure what bugs might lurk if we do that.

I'm especially wary of issues on unusual OSes. We'd at least need a separate codepath for Windows, unless Go stdlib somehow provides a tty file object.

[gokcehan]

@joelim-work Thank you for coming up with a solution. I have not experimented with using /dev/tty for shell commands before. It seems like an interesting idea. However, I'm not yet sure about this change in general. I don't think it is a good idea to assume that /dev/tty is a portable interface. Windows is the obvious outlier, but even besides Windows I think there are slight differences among different unix systems (e.g. tty file should be opened as read/write/readwrite, tty file reads are blocking/nonblocking, tty file should be dup/open, tty file should be closed after use or not). As far as I'm aware, there is not a standard /dev/tty behavior described in the POSIX standard. Besides, isn't that the case that there are also different alternatives for tty nowadays (e.g. pty, pts, ptmx)? Personally, I still assume these things are magic devices, so I often prefer to stay at an abstraction level above these things.

Historically, we have had issues before when there was a change in tcell from /dev/tty to stdin/stdout (see #480 on our side and gdamore/tcell#394 on tcell side). It seems that tcell has changed back to /dev/tty since then. There seems to be a stdin/stdout implementation (i.e. stdIoTty) but it does not seem to be actively used currently. To be honest, I don't know the rationale for this choice. I'm aware these issues are not directly related to this discussion, but they can give an idea about the kind of problems that can result with the change discussed here. To be clear, I would be equally worried if we currently had used /dev/tty and someone had suggested changing it to stdin/stdout/stderr.

A slightly safer alternative might be to direct all command output to stderr with something like following:

cmd.Stdin = os.Stdin
cmd.Stdout = os.Stderr
cmd.Stderr = os.Stderr

Although, I'm not sure about this change either. There are just too many different kinds of commands that users can come up with, so it is difficult to tell if such a change would break anything.

All in all, I'm still not convinced that this change is worth the risks involved. I don't think we get any new functionality with this change. I can see the appeal of a simple command like cd $(lf ...), but I don't think there would be many users manually typing such commands, and if someone defines a custom alias or function for lfcd, it should not matter much how complicated the command looks like. If there are issues with the lfcd commands (e.g. leftover temporary files, security considerations), then it is worth a separate discussion.

I don't want to disregard this suggestion altogether, but if we go down this route, I would personally like to see this change tested extensively, preferably on all platforms that we support.

[joelim-work]

@gokcehan I had not thought of piping the shell output to stderr, although I'm not sure if it is much better than the current behavior, for example if the user wants to redirect stderr to a file to analyze stack traces during crashes.

The problem of obtaining a handle to the TTY for reading/writing in a portable manner is a serious concern to me. I would prefer to not handle it directly inside the lf (e.g. tty.go, tty_windows.go, etc.), and instead import such functionality from a library, although I am not sure that one exists because in most cases programs will just use the standard stdin/stdout/stderr streams and never need to have direct access to the TTY.

I agree that the benefit this feature provides is not really enough to warrant the risks involved. The problems described above seem to be theoretical in nature as opposed to actual issues that a user might encounter in practice. From my opinion, the sample lfcd scripts are just provided for convenience - there is no obligation to use them at all, and in the worst case scenario a user can just write their own lfcd script to suit their requirements.

That being said, I can see the appeal of using lf in a pipeline. For instance, the entire discussion in here can equally be applied to the -selection-path option. I think it's worth keeping this issue open just in case the situation changes and this feature ends up being easier to implement.

[nicolasdumitru]

@gokcehan @ilyagr @joelim-work what about setting a shell/enviroment variable (with a unique name that can be configured in lfrc) that holds the path to the last directory that was open in lf? I don't know how Window's shell works (and consequently if it could work on Windows), but lfcd is already different for Windows and this solution would work on every POSIX-compliant system and would circumvent stdout and the problems it poses. lfcd just becomes:
lfcd -new-flag && cd "$NEW_VARIABLE".

[joelim-work]

@nicolasdumitru Using a pre-defined environment variable as the argument for -last-dir-path would certainly be possible, but the point of the sample lfcd script is that the user can simply use it as is without any prior configuration.

That being said, after looking into this problem some more, I have found a library mattn/go-tty that can handle the task of portably obtaining a handle to the TTY. The usage would look like this:

tty, err := tty.Open()
if err != nil {
	log.Printf("getting tty: %s", err)
}

cmd.Stdin = tty.Input()
cmd.Stdout = tty.Output()
cmd.Stderr = tty.Output()

I have done some very brief testing on both Linux and Windows and it appears to work fine.

There is still some risk involved given that running shell commands is a core feature of lf which is widely used, and it is very unlikely that I am going to be able to test every single platform and usage scenario on my own. Perhaps it might be safer if this was hidden behind some option, or maybe this change could be added right after a new release to allow plenty of time for other users to test.

[nicolasdumitru]

@joelim-work it doesn't necessarily require prior configuration. The shell variable can have a default name (a very descriptive one that is very unlikely to coincide with what is already on the system). And the shell variable's name could just be a one-liner in (default) lfrc (basically set a particular string and then lf names the shell variable like that).

[joelim-work]

@nicolasdumitru I still have some trouble understanding, could you please provide a more detailed description about how this is supposed to work? If the user doesn't specify a path for this file, then we would have to provide one by default, and there isn't really a sane value for that, which is one of the reasons why mktemp is used in the first place.

Also, I'm not sure if this is relevant to the discussion, but AFAIK any environment variables set by lf (or any other process) aren't made available to the parent shell. This is why lf has to write the information to a file (or alternatively stdout) in order to communicate back to the parent shell.

[nicolasdumitru]

@joelim-work let me put it this way:
Say the new flag that I am talking about was "-lfcd". When a user runs lf -lfcd, when hitting the key that quits lf (default q), before lf quits, it sets a shell variable. By default, the shell variable could be called something like "lf_last_open_directory" or some other very descriptive name (so that it is very unlikely that another variable with that name already exists). This default name should be set in lf's source code. There should also be a line in the default lfrc that reflects the default (so that it is very simple to change if for whoever wants/needs to). The "-lfcd" flag would also take an optional argument which would set the name of the specific variable, but only for that specific invocation of lf. This is what I was thinking about in more detail. After this change, a simple version of lfcd could look something like:

lfcd () {
	lf -lfcd "$@" && cd "$lf_last_open_directory"
}

This does not take into account providing the optional argument, but you get the idea.