Change in import behavior for already existing spaces

[Necr0x0Der]

With the new module system, I got the following error:

(Error (import! GroundingSpace-0x12a7846d8 space.metta) import! destination argument must be a symbol atom naming a new space, or &self.  Found: GroundingSpace-0x12a7846d8)

Why can't we import into the already existing space? It is inconvenient.

[luketpeterson]

The conceptual reasons are below, but this issue seems to be about providing a convenience. So I have two ideas:

1.) The inline! operation. I could add a destination space argument to it. So any atoms created by the inlined code would go into the specified space. inline! seems to be what a lot of people want for a lot of reasons.
2.) An automatic space-merge if the specified space already exists. I don't like this because it further muddies up what import! is conceptually supposed to do. But it would bring back the old behavior.

What do you think?

Conceptual Disucssion

The current behavior of the import! operation is a bad conceptual fit for the abstractions in the new MeTTa (or any language's) module system. For example, it wouldn't make any sense in Python to say from numpy import * into torch. If torch needed something from numpy it would do the import itself.

In MeTTa there is also a question about what to do about tokens from the modules, although that question also exists for just importing into a new space. https://github.com/trueagi-io/hyperon-experimental/blob/8540b90d8669484e945fb78d0d11135b68de9233/docs/modules_internal_discussion.md?plain=1#L41C1-L42C1

The arguments to import! need to be redone to better reflect the underlying semantics of the operation as discussed here: #509 and also #580 (comment)

[Necr0x0Der]

The current behavior of the import! operation is a bad conceptual fit for the abstractions in the new MeTTa (or any language's) module system. For example, it wouldn't make any sense in Python to say from numpy import * into torch

I would repeat that the initial proposal way to consider importing modules and loading spaces as two different use cases. And initial implementation of import! was more about loading spaces than about importing modules. But import! and extend-py! was merged into one, instead of separating them further. However, I disagree with from numpy import * into torch analogy. If you keep importing modules as spaces and since MeTTa does have a good and clear conceptual explanation what it means to load atoms to spaces, and in contrast to many languages it allows working directly with code and it is extremely useful to work (add / remove / etc.) with code in separate spaces, removing the possibility to load something into a space, was definitely a bad idea. Even if we restore this functionality, saying that it's a bad conceptual fit to have the possibility to load additional content into modules at runtime is debatable. Why a human can add something to a module manually, but MeTTa cannot? I believe not all the design patterns of "all other languages" are applicable to MeTTa because of its self-reflective nature.

I don't like inline! name too much. In any case, I wonder, if loading another space into an imported module space will work.

[Necr0x0Der]

All in all, I disagree with the "conceptual" argument. If we can import something into &self space, we should be able to import something in another space, e.g. via executing import! to &self in the context of this space. Making artificial restriction to this is just wrong. If the restriction is made not because of conceptual reasons, but because of natural implementation, this is another argument, which has more weight.

this issue seems to be about providing a convenience

Programming language design is mostly about convenience. Otherwise, everyone would program in assembler.

[luketpeterson]

All in all, I disagree with the "conceptual" argument. If we can import something into &self space, we should be able to import something in another space

That's precisely the issue. import! &self is just a legacy way to trigger "import *" behavior, but it doesn't make sense to think of import! as having a destination. All import behaviors ("import module as name", "import * from module", "import foo from module [as name]") effectively import into the currently running module. ie. self. For example !(import! &new-space my_mod) imports a new token into the self module's Tokenizer, to reference the sub-module's space.

Programming language design is mostly about convenience.

100% agreed on that point. So we (our team) need to do the hard work of thinking through all the edge cases, so we can come up with an API and a language that presents the best balance of an ergonomic, intuitive, and coherent interface with all the power and expressiveness people will want.

So along those lines, what convenience do you think makes sense to offer?

One option is a merge-space operator, so you could run

!(import! &new_space the_mod)
!(merge-space &new_space &destination)

This is more verbose than !(import! &destination the_mod), but merge-space will certainly have uses beyond just importing. Also another downside of this pattern is that &new_space will be unnecessary leftover trash.

An alternative idea might be to not use import! here, and instead add something to provide an accessor for a module's space. Like this:

!(merge-space (mod-space! the_mod) &destination)

What do you think?

I don't like inline! name too much.

Is it just the name you don't like, or the semantic as well?

[Necr0x0Der]

One option is a merge-space operator, so you could run...

I'd propose to start with use-cases, and make it convinient for them. My current use case was that there is a bunch of metta-files constituting bio-atomspace, and all of them are supposed to be loaded into the same space. It will be inconvenient to load them first in their own spaces (and assigning tokens to them), and then to merge them. I was also thinking about the function which would return the space atom, into which the content is loaded (unlike import!). However, it's still not precisely what needed, because loading additional content to the existing space might be more efficient than loading it into a separate space and then merging them (it, of course, depends on how spaces are implemented; if we have an algebra over spaces, then merge of spaces should not necesserily be a "physical" merge; OTOH such algebra poses some challenging for indexing mechanisms, so I wouldn't make it obligatory for any kind of spaces). In any case, while merging spaces can be useful by itself for other cases, I wouldn't introduce it for this case.
Thus, I think we need to restore the former functionality of loading additional content into the existing spaces. Obviously, it should not necessarily be import!. Although if we go away from importing to anywhere but &self, then we need to make it less misleading, because &self is a space. And it is definitely an inconsistent notation that we can pass &self space to import! but we cannot pass another space, and instead we need to specify a token for a new space. This is a separate discussion, however, and I wouldn't like to introduce rush changes to import! syntax without really thinking through it carefully and coming up with the version, which will be stable. Right now, it's definitely not a blocking issue. What is more urgent is the now missing functionality.
OK, let it not be import!. We need to load the content of a file to the already existing space. But there is still more that one sub-cases. One may want to load a pure space (but it can possibly contain numbers and strings as grounded atoms), or one can want to interpret the code of another script in the content of the existing space. This situation appeared in metta-motto, and after new modules were introduced I had to emulate this behavior in the metta-motto itself (which is ok for better controllability). As for bio-atomspace case, it will work for me with this PR #593 (which is needed not only to workaround the changed behavior, but for performance reasons as well).
Thus, I'd wait for other use cases to understand if we need first loading a pure spac, or running a script in context of existing module/space. Meanwhile, we have other useful features to implement like module downloading and installation.

Is it just the name you don't like, or the semantic as well?

It's the name. Although the details of semantics are to be discussed.

[vsbogd]

Can we say that we have "importing" and "loading" operations. When module is imported then atoms of the module become a part of a destination module and tokens of the module are used to parse following atoms of the destination module. (Strictly speaking import! not necessary loads atoms into the destination module but it makes them available for the destination module. But at the moment it is implemented by adding atoms into the destination module.) When module is loaded then atoms of the module also become a part of destination space but tokens of the module are used only to parse the modules itself. Thus from the atoms perspective importing and loading are not different at least before we have "import as" which will extend the semantics of the import. From this perspective inline! not so bad name but load! could be more clear. And import! is different because it treats tokens (and potentially atoms) differently.

[Necr0x0Der]

For me, inline! is associated with inline functions in C++. include! might be used, or insert!, but yeah, why not use load! if we do think about this operation as loading spaces from files. Following file operations analogy, open-space or read-space may also make some sense. But inline! is the most programmatically-sounding moving us away from spaces as storages. If we want to distinguish it from importing modules/programs, we should name it differently.

[Necr0x0Der]

@luketpeterson , it should be prioritized as many metta coders complain about this

[TeamSPoon]

I like the name include! because I used import! for swapping file names in order to swap out functions defined in .metta files. Even more I was using import! when I am including the same file with directives that I need into multiple metta files. Old import! was preventing me from including files more than once.

[luketpeterson]

I realized (after implementing it) that a merge-atoms operation to copy all atoms from a src space to a dst space is unneeded on account of the nested-space behavior. Therefore: !(add-atom &dst_space (mod-space! mod_name)) should be enough.