io/fs: add file system interfaces

[rsc]

In July, @robpike and I posted a draft design for file system interfaces. That doc links to a video, prototype code, and a Reddit discussion.

The feedback on that design has been almost entirely positive.

A few people raised concerns about the use of optional interfaces, but those are an established pattern in Go that we understand how to use well (informed in part by some earlier mistakes, such as optional interface like http.Hijacker with methods that cannot return an error to signal failure/unavailability).

A few people suggested radical redesigns of the os.File interface itself, but for better or worse installed base and the weight of history cautions against such drastic changes.

I propose to adopt the file system interfaces draft design for Go 1.16.

[rsc]

Accepting this proposal would also let us land the embedded files draft design in Go 1.16, which I've proposed in #41191.

[tooolbox]

A few people raised concerns about the use of optional interfaces

This is my concern.

It's not that the alternative (attempting to define all FS methods in one huge interface which packages may then implement as no-op) is better. Rather, my perception is that the ergonomics of this approach are poor enough that it won't achieve broad community adoption and the level of composability and general success that interfaces like io.Reader and io.Writer have.

For example, it's clear that I will be able to pipe a zip file to text/template, and that's good, but I'm concerned about more general composability of filesystems and files. I can wrap a stack of io.Reader with confidence, but with io/fs it seems like some middle layer may not have the right optional interfaces and I will lose access to functionality.

In spite of my concerns, it seems like the best approach available to Go at this time, and I anticipate it will be accepted given that the very exciting #41191 depends upon it.

However, I have this inkling that the advent of generics may allow a more powerful/robust/safe abstraction. Has any thought been given to this, or to how io/fs could evolve in a backwards-compatible fashion if/when that occurs? Again, not to hold up this proposal, but I think I would be more excited if I knew what the future held.

[networkimprov]

The feedback page:
https://www.reddit.com/r/golang/comments/hv976o/qa_iofs_draft_design/?sort=new

I think this API looks promising... and would benefit from a prototype phase.

A lot of feedback was posted, but there's been rather light discussion of the comments, presumably because you can't subscribe to a Reddit thread, and/or many in Go's github-centered community don't frequent Reddit. It would help to see a review and analysis of feedback here, and perhaps a roadmap to likely future features.

Problems were identified with the FileInfo interface, but not discussed and are in discussion #41188. Timeouts and/or interrupts bear consideration.

Landing a prototype in x/ seems like a logical step before stdlib. Go has long been deliberative and conservative about new features. Is this urgent somehow?

FWIW, my Go apps make heavy use of the filesystem, on Windows, MacOS, and Linux.

[earthboundkid]

I think optional interfaces can work if there is a way to indicate that even though a method exists on a wrapper, it hasn't been implemented by the underlying wrapped type. Something like ReadFile(name string) ([]byte, error) needs to be able to return ErrNotImplemented so that the function calling it can say "Oh, well, then let me fallback to Open() + Read()." The main sin of the existing optional interfaces is that there's no way to signal "I have this method just in case the thing I'm wrapping implements it." This shortcoming really needs to be addressed in the io/fs optional interfaces.

[earthboundkid]

So, for the ReadFile top level func, I am proposing this implementation:

func ReadFile(fsys FS, name string) ([]byte, error) {
	if fsys, ok := fsys.(ReadFileFS); ok {
		b, err := fsys.ReadFile(name)
		if err != ErrNotImplemented { // Or errors.Is?
			return b, err
		}
	}

	file, err := fsys.Open(name)
	if err != nil {
		return nil, err
	}
	defer file.Close()
	return io.ReadAll(file)
}

[rsc]

Discussion of ErrNotImplemented has moved to #41198.
I've marked @carlmjohnson's two comments above this one
as well as @randall77's comment below this one
as "off-topic" to try to funnel discussion over there.

[randall77]

On Thu, Sep 3, 2020 at 10:15 AM Carl Johnson ***@***.***> wrote: I think optional interfaces can work *if* there is a way to indicate that even though a method exists on a wrapper, it hasn't been implemented by the underlying wrapped type. Something like ReadFile(name string) ([]byte, error) needs to be able to return ErrNotImplemented so that the function calling it can say "Oh, well, then let me fallback to Open() + Read()." The main sin of the existing optional interfaces is that there's no way to signal "I have this method just in case the thing I'm wrapping implements it." This shortcoming really needs to be addressed in the io/fs optional interfaces.

The other way to handle this is to have a factory for the wrapper that returns a type with the correct methods on it. type I interface { Foo() } type Optional interface { Bar() } func NewWrapper(i I) I { if _, ok := i.(Optional); ok { return &wrapperWithBar{i: i} } return &wrapperWithoutBar{i:i} } type wrapperWithoutBar struct { i I } type wrapperWithBar struct { i I } func (w *wrapperWithoutBar) Foo() { w.i.Foo() } func (w *wrapperWithBar) Foo() { w.i.Foo() } func (w *wrapperWithBar) Bar() { w.i.(Optional).Bar() } —

…

You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#41190 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABUSAIFGSNLHB456YB5L6C3SD7FKNANCNFSM4QTHTZEA> .