Zero (phantom, unvoiced) word support.

[linas]

Zero/phantom words: Expressions such as "Looks good" have an implicit "it" (also called a zero-it or phantom-it) in them; that is, the sentence should really parse as "(it) looks good". The dictionary could be simplified by admitting such phantom words explicitly, rather than modifying the grammar rules to allow such constructions.

Other examples, with the phantom word in parenthesis, include:

• I ate all (of) the cookies.
• I taught him (how) to swim.
• I told him (that) it was gone.
• (It) looks good.
• (You) go home!
• You know you can (do something).
• I wish I had (done something).
• I will (do something), if you do.
• How often (does it happen)?
• How big (is it)?
• Room w/sea view (is) available. -- zero copula
• (some) thieves rob(-bed a) bank! -- newspaper headline

One possible solution to the unvoiced-word problem might be to allow the LG rules to insert alternatives during the early culling stages. This avoids the need to pre-insert all possible alternatives during tokenization...

[linas]

The "(You) go home" variant is particularly important, as it provides a subject for directives/imperatives.

See opencog/relex#231

[linas]

One way to implement this would be with certain special link types. For example:

    +---Wi--+-MVp-+
    |       |     |
LEFT-WALL go.v home.r 

The Wi link could be interpreted as a phantom-expanding link, so that, after parsing, there would be a phantom-expansion stage, which would convert the Wi link to this:

     +--Wd---+-Sp-+-MVp-+
     |       |    |     |
LEFT-WALL (you) go.v home.r 

that is, the Wi link gets split in two, into Wd-& Sp+, and the phantom (you) gets dropped into place. However, for proper compatibility, we also want the WV link:

     +---->WV---->+      
     +--Wd---+-Sp-+-MVp-+
     |       |    |     |
LEFT-WALL (you) go.v home.

so it looks as if Wi gets renamed to WV. There are several practical difficulties with this proposal in LG itself. Although "trivial" if done in opencog, where graph-reiwriting is a built-in feature, it would be hard to do in LG itself, which does not support generic graph re-writing. That is,

• it requires a non-trivial post-parsing stage,
• it fails to actually simplify the dictionaries.

The second point being the more serious: one reason to entertain the idea of phantom words is the hope that the grammar would be simplified. However, in this proposal, the dicts would need to contain rules for both Wi as well as Wd, Sp, WV and so the post-parsing conversion does not simplify the grammar.

The post-parsing stage could be carried out using some graph re-write rules, e.g. in relex or with the opencog pattern matcher. Since this happens post-parsing, there is no particular point of putting it "inside" of LG itself.

[linas]

One possible solution is to perform a one-point compactification. The dictionary contains the phantom words, and thier connectors. Ordinary disjuncts can link to these, but should do so using a special initial lower-case letter (say, 'z', in addition to 'h' and 't' as is currently implemented). The parser, as it works, examines the initial letter of each connector: if it is 'z', then the usual pruning rules no longer apply, and one or more phantom words are selected out of the bucket of phantom words. (This bucket is kept out-of-line, it is not yet placed into sentence word sequence order, which is why the usual pruning rules get modified.) Otherwise, parsing continues as normal. At the end of parsing, if there are any phantom words that are linked, then all of the connectors on the disjunct must be satisfied (of course!) else the linkage is invalid. After parsing, the phantom words can be inserted into the sentence, with the location deduced from link lengths.

[linas]

Issue #1234 describes a special case.

[ampli]

One possible solution is to perform a one-point compactification.

@linas,
I didn't understand this method.
Can you construct an example?
Also, please consider that it is not a problem to insert phantom words/alternatives as needed during/after tokenization. Moreover, inserted phantom words can be marked as "optional", which means their slot would be automatically removed from the linkage if none of its words get fully linked.

[ampli]

Implementing a concept of "optional alternatives" can maybe help to handle phantom words.
The idea is that "optional alternatives" are to be used only if the sentence cannot be fully parsed using another alternative.
An extension of this idea is assigning a cost to optional alternatives so only the lowest cost one is actually used. This can be another kind of cost than disjunct cost. You once mentioned a similar idea - I will try to find its location. What I suggest here is to not use at all the higher cost alternatives, to prevent zillions of bogus parses in addition to the good ones.

This idea can also solve issue #224 "spell-guessing mis-handles capitalized words".

One obstacle in this idea is the case of more than one location of an "optional alternative" in a sentence when the lower-cost alternatives in different locations are mutually exclusive.
Another obstacle is how to handle "optional alternatives" if there is no full parse. A possible solution is just to omit them.

[ampli]

At the end of parsing, if any phantom words are linked, then all of the connectors on the disjunct must be satisfied (of course!) else the linkage is invalid. After parsing, the phantom words can be inserted into the sentence, with the location deduced from link lengths.

The phantom word addition may be redundant, i.e. that the sentence will get parsed w/o it. Of course, this cannot be found in advance (before parsing).
My updated algo is:

1. In determin_word_expressions(), examine the expression of the candidate word to find out which phantom word (if any) is needed to be inserted after it (or maybe before it for some languages or situations, but I would initially implement only after it).
2. For now, support only a single word-slot insertion instead of an insertion of a whole alternative (can be extended to whole alternative insertion if needed). Mark it with a boolean "phantom" in addition to "optional". It seems to me that a macro is to be inserted and not words (see below).
3. Prune, parse, extract links. No linkages are created yet.
4. If parsed with nulls, remove the phantom words and continue to step 6.
5. We now have a full linkage. Inspect the parse data structure (create by extract_links() ) to see if there is still a complete parse w/o each phantom word. If there is, remove the phantom word.
6. Create linkages and continue as usual.

As an example, I tried to inspect this algo for the sentence I will, if he goes to the store from PR #1234.
This is said to be converted to: I will.v [context-verb] , if she goes first.
The question is what to insert as this [context-verb]. The list from en/words/words.v.1.1 is too big, and also contains relevant disjuncts that may cause unintended parses. So I thought to create a special macro for it, say <verb-infinitive>.
There is also the question of how we can know which phantom word to insert. I cannot see how a special z connector may help here.

[linas]

There seem to be several plausible solutions (including inserting words during tokenization). I don't know which is best or easiest. Let's explore each in greater detail. So, this #224 (comment) is rather cryptic. Let me try to reconstruct what I was thinking.

Given the sentence I will, if she goes first, the desired result is

                                        +------------Xc------------+
    +------->WV------------>+----MVs----+----CV-->+                |
    +->Wd--+-Sp*i+-----I----+       +-Xd+-Cs+--Ss-+--MVa-+         |
    |      |     |          |       |   |   |     |      |         |
LEFT-WALL I.p will.v [context-verb] , if.r she goes.v first.a RIGHT-WALL

For now, I don't care about the specific format: besides [context-verb], perhaps <do-something> or {zero-verb} ... some way of indicating an unresolved reference to an action (cf. other discussions about reference resolution).

The existing 4.0.dict is modified to contain the following:

LEFT-WALL: (zWV+ & Wd+) or ...;
will.v: (Sp- & zI+) or ...;
[context-verb]: zI- & zWV- & zMVs+;
if.r: (Xd- & zMVs- & Cs+ & CV+ & Xc+) or ...;

During dictionary read, a special class of zero-words is created: these can be spotted in one of two ways: (1) they are surrounded by square brackets e.g. [context-verb] or better yet, (2) all connectors in the disjunct start with a z. This class of zero-words is kept in a special list.

Counting is done as usual, with a slight modification: if a disjunct has a connector starting with z, that connector is ignored during pruning -- its "invisible". As if it was never there. Linkage generation is done as usual, except that islands are allowed. For the above example, one of the linkages will be

                                        +------------Xc------------+
                                        +----CV-->+                |
    +->Wd--+-Sp*i+                  +-Xd+-Cs+--Ss-+--MVa-+         |
    |      |     |                  |   |   |     |      |         |
LEFT-WALL I.p will.v                , if.r she goes.v first.a RIGHT-WALL

During linkage generation, look at the chosen disjuncts, and see if any of them contain z connectors. If so, then scan the list of zero words, to see if any of them can satisfy the "invisible" unconnected connectors.

That is, while generating the above linkage, it will be discovered that LEFT-WALL has an invisible zWV+ on it, and will.v has an invisible zI+ on it, and that if.r has an invisible zMVs-. These are conjoined: this linkage has a zero-disjunct of zWV+ & zI+ & zMVs-. Flip all the direction indicators, and search the list of zero-words for some word that has that disjunct. If so, then we are done.

Well, almost done. Some open questions:

• Do we want to enforce the no-link crossing constraint on these invisible links? I don't know...
• How should it be drawn, in ascii-art? Conceptually, the zero-verb is kind of floating in outer space, not having any linear position in the sentence. It only has links tying it back down to earth. Thus, for example:

                      [context-verb]
                            ^
    +------->WV-------------+
    ^            +----I-----+           +------------Xc------------+
    ^            ^          +----MVs----+----CV-->+                |
    +->Wd--+-Sp*i+                  +-Xd+-Cs+--Ss-+--MVa-+         |
    |      |     |                  |   |   |     |      |         |
LEFT-WALL I.p will.v                , if.r she goes.v first.a RIGHT-WALL

In the above, [context-verb] is not the third or the fourth word in the sentence, its just in orbit around the sentence. Of course, this screws up the API, which wants to index the location of every word. More on this in the next comment.

[linas]

The above comment points out that we have an API issue. The current API is driven by word indexes: all words have a fixed location in a sentence, which means that location-independent linkages are not currently possible. Here is a futuristic example of a desirable linkage diagram:

                   +==============VR================>[context-verb]
                   $                                       ^
                   $                      +------->WV------+
                   $                      ^            +-I-+
                   $    +---Js---+        ^            ^   +-MVs----+----CV-->+
    +-->Wi--+---I--+-MVp+  +Ds**c+        +->Wd--+-Sp*i+        +-Xd+-Cs+--Ss-+--MVa-+
    |       |      |    |  |     |        |      |     |        |   |   |     |      |
LEFT-WALL let's jump.v off a  cliff.n LEFT-WALL I.p will.v      , if.r she goes.v first.a

Here, VR is a new link type. It stands for "verb reference". we are employing it for reference resolution.

I would like to do things like the above, but the current infrastructure isn't suitable for that. We can kind-of do multiple sentences, today, but its ad hoc:

    +-------Xp-------+------Xp-----+
    +---->WV--->+    +-->WV-->+    |
    +->Wd--+-Ss-+    +>Wd+-Ss-+    |
    |      |    |    |   |    |    |
LEFT-WALL it walks.v !  it talks.v !

but there is no way to draw links to references:

           +>========================NR==========>+
           $             +>=====NR===============>+
           $             $             +>===NR===>+
           $             $             $          $
           $         +----------------Xp----------$----+
           $         +--------->WV-------->+      $    |
    +-------Xp-------+------Xp-----+       |      $    |
    +---->WV--->+    +-->WV-->+    +-->WV->+      $    |
    +->Wd--+-Ss-+    +>Wd+-Ss-+    +>Wd+-Ss+-Oste-+    |
    |      |    |    |   |    |    |   |   |      |    |
LEFT-WALL it walks.v !  it talks.v !  it 's.v Wilber.m !

where the NR links to Wilber are "Named Reference" or "Named Entity" links. Note that these are crossing links. I'm wandering off-topic here; what I'm trying to say is that we need some kind of infrastructure for more general graphs. There are four possible directions to go in, for more general infrastructure:

1. Extend the LG API's in a more general direction.
2. Switch to some complete different API -- this is what the AtomSpace does.
3. Pick some existing NLP framework, e.g. NLTK and work really hard to make sure LG inter-operates cleanly and elegantly with that framework.
4. Like 3. but with some top-o-the-pops Java library.

I have deep misgivings about 3 and 4 since it seems one must sacrifice huge amounts of representational power, in exchange for a grab-bag of mostly silly tools. However, there are vast numbers of programmers (and corporate executives) who love these frameworks, and view them as the answer to all problems.

The problem with 2. is that it is too abstract: it's not programmer-friendly, its missing graph visualization tools (despite over a decade of failed efforts), there aren't any API utilities, e.g. no easy way to jam it into a computer-game non-player character chatbot. There are companies that make good money creating fancy chatbots for computer games... its frustrating that we can't demo that.

The problem with 1. is that... is there a problem? Well, it's not 2,3 or 4. I guess that's a problem. But I know that you personally would have a lot more fun working on 1. than on 2,3,4 and so .. that's a good thing. Lets push the boundary. See how far we can go.