Still haven’t seen my original message relayed by the list, here is yet another attempt…

I thought this group might find the following interesting.

For the last several years I have been leading an R&D group in AI (essentially applied NLP) at a major Wall Street Bank. Since 2020, I have been struck by the stunning progress made by Large Language Models (LLMs; GPT-3, ChatGPT, GPT-4, …). Essentially it seems that a lot of the work we were doing in computational linguistics (at least some of us), particularly in terms of applications to real-world question answering problems, has now been swamped by LLM advances.

I have a separate talk on the kind of highly-specific text analyses GPT-4 can do directly for (financial) professionals with domain-specific text, without having to go through parsing / semantic representation steps. Here I thought it might be illuminating to provide a log of a conversation with GPT-4 about representing meanings of simple sentences in IL.

As you can see it doesn’t always get things right. But some discussion helps it set things right (in itself, this is another startling capability of GPT-4). Such conversations can lead to the development of an accurate labeled data set, and thence to a fine-tuned end-point which should end up being highly accurate. (GPT-4 is not yet available to be fine-tuned, but GPT-3 is, and I would hope -4 will be, soon…)

There are tons of things to discuss here, of course, if there is interest. Just putting this out as a conversation starter…

Best, Vijay

The examples are taken from https://aclanthology.org/E93-1013.pdf and https://aclanthology.org/E95-1005.pdf

VS: Parse the following sentence and generate its m meaning in Montague's Intensional Logic. -- Sentence Bill kissed Hillary. -- Meaning formula

GPT4: To represent the given sentence "Bill kissed Hillary" in Montague's Intensional Logic, we will first parse the sentence into its components: Subject: Bill Verb: kissed Object: Hillary Montague's Intensional Logic uses lambda calculus to represent meanings. We will use the following variables:

b for Bill k for the verb 'kiss' h for Hillary Now, we can represent the meaning formula as follows:

k(b, h)

This formula states that the kissing action (k) occurs with Bill (b) as the subject and Hillary (h) as the object. The given sentence can be translated into Montague's Intensional Logic as the application of the binary relation 'k' to the individuals 'b' and 'h’. VS: Good. Here is another one. -- Sentence Bill obviously kissed Hillary.

GPT4: To represent the given sentence "Bill obviously kissed Hillary" in Montague's Intensional Logic, we need to consider the adverb "obviously" as a modifier of the verb "kissed". Let's denote the adverb with the letter 'o' and use the previously defined variables: b for Bill k for the verb 'kiss' h for Hillary o for the adverb 'obviously' Now, we can represent the meaning formula as follows:

o(k)(b, h)

This formula states that the kissing action (k) occurs with Bill (b) as the subject and Hillary (h) as the object, and that this action is obviously true (o). The given sentence can be translated into Montague's Intensional Logic as the application of the binary relation 'k' to the individuals 'b' and 'h', modified by the adverb 'o’. VS: Hmm.. shouldn't this be o(k(b,h))? GPT4: Apologies for the confusion. You are correct. The correct representation of the sentence "Bill obviously kissed Hillary" in Montague's Intensional Logic should be: o(k(b, h))

This formula states that the kissing action (k) occurs with Bill (b) as the subject and Hillary (h) as the object, and that this action is obviously true (o). The given sentence can be translated into Montague's Intensional Logic as the adverb 'o' modifying the result of applying the binary relation 'k' to the individuals 'b' and 'h’. VS: Good. Now let's do: -- Sentence Hillary let Bill write a letter -- Meaning formula

GPT4: To represent the given sentence "Hillary let Bill write a letter" in Montague's Intensional Logic, we will first parse the sentence into its components: Subject: Hillary Verb: let Object: Bill Infinitive verb: write Object of the infinitive verb: a letter We will use the following variables:

h for Hillary l for the verb 'let' b for Bill w for the verb 'write' a for a letter Now, we can represent the meaning formula as follows:

l(h, λx.w(x, a))(b)

This formula states that the letting action (l) occurs with Hillary (h) as the subject and a function representing Bill writing a letter (λx.w(x, a)) as the object. Then, the function is applied to the individual Bill (b).

The given sentence can be translated into Montague's Intensional Logic as the application of the ternary relation 'l' to the individuals 'h' and 'b', and the action of writing a letter (w) with 'a' as the object. VS: Good. Next: -- Sentence Bill supported, and Hillary opposed, NAFTA.

GPT4: To represent the given sentence "Bill supported, and Hillary opposed, NAFTA" in Montague's Intensional Logic, we need to parse the sentence into its components and consider the conjunction "and" that connects two separate actions: Subject 1: Bill Verb 1: supported Subject 2: Hillary Verb 2: opposed Object: NAFTA We will use the following variables:

b for Bill s for the verb 'support' h for Hillary o for the verb 'oppose' n for NAFTA ∧ for the conjunction 'and' Now, we can represent the meaning formula as follows:

(s(b, n) ∧ o(h, n))

This formula states that Bill (b) supports NAFTA (n), and Hillary (h) opposes NAFTA (n). The given sentence can be translated into Montague's Intensional Logic as two separate actions, the support action (s) with Bill (b) as the subject and NAFTA (n) as the object, and the oppose action (o) with Hillary (h) as the subject and NAFTA (n) as the object, connected by the conjunction 'and' (∧). VS: OK, next: -- Sentence Bill supported, and Hillary opposed, two trade bills.

GPT4: To represent the given sentence "Bill supported, and Hillary opposed, two trade bills" in Montague's Intensional Logic, we need to parse the sentence into its components and consider the conjunction "and" that connects two separate actions: Subject 1: Bill Verb 1: supported Subject 2: Hillary Verb 2: opposed Quantifier: two Object: trade bills We will use the following variables:

b for Bill s for the verb 'support' h for Hillary o for the verb 'oppose' tb for trade bills ∧ for the conjunction 'and' t for two Since we are dealing with a quantifier, we will use lambda notation to represent the two trade bills. Let x represent trade bills and y for the second trade bill:

Now, we can represent the meaning formula as follows:

∃x∃y(tb(x) ∧ tb(y) ∧ t(x, y) ∧ s(b, x) ∧ s(b, y) ∧ o(h, x) ∧ o(h, y))

This formula states that there exist two trade bills (x and y) such that Bill (b) supports both (s(b, x) and s(b, y)), and Hillary (h) opposes both (o(h, x) and o(h, y)). The given sentence can be translated into Montague's Intensional Logic as two separate actions, the support action (s) with Bill (b) as the subject and the two trade bills (x and y) as the objects, and the oppose action (o) with Hillary (h) as the subject and the two trade bills (x and y) as the objects, connected by the conjunction 'and' (∧). VS: Next: -- Sentence Hillary wanted, found, and supported two candidates. GPT4: To represent the given sentence "Hillary wanted, found, and supported two candidates" in Montague's Intensional Logic, we need to parse the sentence into its components and consider the conjunctions "and" that connect three separate actions: Subject: Hillary Verb 1: wanted Verb 2: found Verb 3: supported Quantifier: two Object: candidates We will use the following variables:

h for Hillary w for the verb 'want' f for the verb 'find' s for the verb 'support' c for candidates ∧ for the conjunction 'and' t for two Since we are dealing with a quantifier, we will use lambda notation to represent the two candidates. Let x represent the first candidate and y for the second candidate:

Now, we can represent the meaning formula as follows:

∃x∃y(c(x) ∧ c(y) ∧ t(x, y) ∧ w(h, x) ∧ w(h, y) ∧ f(h, x) ∧ f(h, y) ∧ s(h, x) ∧ s(h, y))

This formula states that there exist two candidates (x and y) such that Hillary (h) wanted both (w(h, x) and w(h, y)), found both (f(h, x) and f(h, y)), and supported both (s(h, x) and s(h, y)). The given sentence can be translated into Montague's Intensional Logic as three separate actions, the want action (w), find action (f), and support action (s) with Hillary (h) as the subject and the two candidates (x and y) as the objects, connected by the conjunctions 'and' (∧). VS: It would seem to me that the meaning should be the conjunction of two propositions, one expressing the wanting, and the second that the finding-and-supporting was successfully accomplished… GPT4: I apologize for my previous responses, and I understand now what you are looking for. To represent the given sentence "Hillary wanted, found, and supported two candidates" in Montague's Intensional Logic as the conjunction of two propositions, we can do the following: Subject: Hillary Intensional verb: wanted Extensional verb 1: found Extensional verb 2: supported Quantifier: two Object: candidates We will use the following variables:

h for Hillary w for the verb 'want' f for the verb 'find' s for the verb 'support' c for candidates t for two ∧ for the conjunction 'and' Now, we can represent the meaning formula as follows:

w(h, λx.t(c, λy.(f(h, y) ∧ s(h, y)), x)) ∧ ∃x∃y(c(x) ∧ c(y) ∧ t(x, y) ∧ f(h, x) ∧ f(h, y) ∧ s(h, x) ∧ s(h, y))

In this formula, we have two separate propositions joined by a conjunction. The first proposition represents the wanting, with the intensional verb 'want' as the function w(h, λx...), taking Hillary (h) as the subject and a property λx representing the existence of two candidates that Hillary found and supported as its argument. The second proposition represents the successful finding and supporting of two candidates, with the quantifier 'two' applied to the candidates, and for each candidate x and y, the extensional verbs 'found' and 'supported' are applied.

This representation treats the intensional verb 'wanted' separately from the extensional verbs 'found' and 'supported', expressing the conjunction of two propositions: one for the wanting, and the second for the successful finding and supporting of two candidates.

Sent from my iPhone

On May 1, 2023, at 3:59 AM, Miriam Butt <miriam.butt@uni-konstanz.de> wrote:

Hi,

really interesting post Ken! It seems it’s like with the text generation — very fluent posting and explanation of f-structures but with features we wouldn’t use in LFG (REL-CLAUSE, SUBORD-CLAUSE). I’d really like to know what the underlying technology for this is (beyond the basics).

I’ve started looking into prompt engineering — any pointers on this you might know?

Cheers,

Miriam

...

On 28. Apr 2023, at 10:53, Ken Kahn <toontalk@gmail.com> wrote:

Hi

I did get this notification of moderation 4 days ago.

Here's my message again:

Vijay's message got me wondering what else GPT-4 can do.

https://shareg.pt/VcJBEYK

Mary thought the result was at an undergraduate level.

Best,

-ken

---------- Forwarded message --------- From: <lfg-owner@listserv.linguistlist.org> Date: Mon, 24 Apr 2023 at 16:23 Subject: Your message to LFG awaits moderator approval To: <toontalk@gmail.com>

Your mail to 'LFG' with the subject

GPT-4 producing f-structures

Is being held until the list moderator can review it for approval.

The reason it is being held:

Post by non-member to a members-only list

Either the message will get posted to the list, or you will receive notification of the moderator's decision. If you would like to cancel this posting, please visit the following URL:

https://listserv.linguistlist.org/cgi-bin/mailman/confirm/lfg/7ecd6420d43b3b...

**************************************************************** Miriam Butt Department of Linguistics University of Konstanz Fach 184 Tel: +49 7531 88 5109 78457 Konstanz Fax: +49 7531 88 4865 Germany +49 7531 88 5115

miriam.butt@uni-konstanz.de https://www.ling.uni-konstanz.de/butt/

'I would suppose, sir,' he said cautiously, 'that Modesty Blaise might be a person awfully difficult for us---er---actually to get.' He blinked towards the big, grey-haired main who stood by the window, looking down at the night traffic, hurrying along Whitehall.

'For a moment,' Tarrant said, turning from the window, 'I hoped you might split that infinitive, Fraser.’

Peter O'Donnell, Modesty Blaise (p. 1) ****************************************************************

Hi Damir — Everything depends on how the problem is formulated. And which model you are using — the right prompts for GPT-4 are not necessarily right for GPT 3.5 or GPT 3. Anyway will give your examples a shot later today / tomorrow. An important point to note is (and I said this in my message): One has the possibility of further training these models on labeled data, e.g. human corrected LFG parses etc. I believe one of the LFG teams trained a statistical parser in this way 10-15 years ago. But with Language Model (LMs) technology now having progressed substantially we should be able to use LMs to develop really powerful tools for computational linguists.

On May 1, 2023, at 10:25 AM, Cavar, Damir <dcavar@indiana.edu> wrote:

Hi there,

Thanks for these notes! Very interesting!

In many ways GPT4 is useful, also for data formatting and annotation, or sample generation. You can give it some data format, then raw data, and ask it to reformat that accordingly. It also can give you more samples for some constructions if you need more material to train and test models.

However, just came back from Berlin, from the GGS, where I briefly discussed some of these points related to ChatGPT; not just are all the major parsers that I tested (constituent and dependency) problematic, but they are basically useless for complex constructions, only XLE for English being impressive! 😊 , ChatGPT is also not capable of dealing with specific constructions and ambiguities:

I am working on ellipsis, all kinds of gapping and sluicing constructions, and temporal scope in complex predicates, labeling for factivity and truth value, as well as generating presuppositions and implicatures for arguments (claims and evidence). It systematically fails to deal with ellipsis in coordination, deeply embedded pronoun bindings, and temporal or even reasoning. But, this is fine, since I would only use it as a front-end to users interfaces. What you can do is make it query a linguistic and knowledge backend (say SPARQL or knowledge graph language like Cypher), generate some formal results and make it articulate a response. I also play with the free and open versions, e.g. GPT4All or HuggingGPT, which in the case of GPT4All I can directly program with, and even integrate in a parser.

But, to give you some impression, when I said:

I stole a book and Peter kisses from Mary.

It gave me a moral lecture about why stealing is really bad and instructed me to return everything, in particular that I should return:

“Peter’s kisses from Mary”

This shows you that it took the coordination not to be of predicates, but of NPs [NP a book ] & [ Peter kisses from Mary ] and it did not consider that the PP modifies both predicates, that is, it does not get scope. And so on…

It is a good generator, though. If you provide some sample data format and tell it how to spell it out, it could be your eloquent text generator.

It does a surprisingly good job in generating presuppositions and implicatures. Just tell it something like this:

Miriam just benched 200 pounds in the gym. What are the implicatures of that?

Miriam just benched 200 pounds in the gym. What are the presuppositions of that?

But, since it just gives you word-sequence probability-based output, do not expect any real understanding. It is a dumb statistical model, not a smart parser and analyzer, although, we do not know all the details behind the front end. We know that they use two models in combination.

I would still bet on a good LFG parser, add to it probabilities, link it to professional knowledge systems, and let it interact with an LLM. Even though some people like Piantadosi like to tell us that linguistics and knowledge-based engineering approaches have failed, that is simply not true. I will jiust continue beating the drum… 😊

But, LLMs are important and very useful components. Instead of wasting my lifetime studying “query engineering” I would still say, real engineering and knowledge driven approaches are winning. LLMs will need them in the backend. It is like with the Kahneman model of Thinking Fast and Slow, is see it as there is one fast and probabilistic model, and there is a slow and knowledge-based model that interact with each other. ChatGPT 4 is the first attempt to get to a Fast Thinking module.

All the best

DC

From: LFG-list <lfg-list-bounces@mailman.uni-konstanz.de <mailto:lfg-list-bounces@mailman.uni-konstanz.de>> On Behalf Of Vijay Saraswat Sent: Monday, May 1, 2023 4:08 AM To: Miriam Butt <miriam.butt@uni-konstanz.de <mailto:miriam.butt@uni-konstanz.de>> Cc: lfg-list@mailman.uni-konstanz.de <mailto:lfg-list@mailman.uni-konstanz.de>; Mary Dalrymple <mary.dalrymple@ling-phil.ox.ac.uk <mailto:mary.dalrymple@ling-phil.ox.ac.uk>>; Ken Kahn <toontalk@gmail.com <mailto:toontalk@gmail.com>> Subject: Re: [LFG-list] Your message to LFG awaits moderator approval

Still haven’t seen my original message relayed by the list, here is yet another attempt…

I thought this group might find the following interesting.

For the last several years I have been leading an R&D group in AI (essentially applied NLP) at a major Wall Street Bank. Since 2020, I have been struck by the stunning progress made by Large Language Models (LLMs; GPT-3, ChatGPT, GPT-4, …). Essentially it seems that a lot of the work we were doing in computational linguistics (at least some of us), particularly in terms of applications to real-world question answering problems, has now been swamped by LLM advances.

I have a separate talk on the kind of highly-specific text analyses GPT-4 can do directly for (financial) professionals with domain-specific text, without having to go through parsing / semantic representation steps. Here I thought it might be illuminating to provide a log of a conversation with GPT-4 about representing meanings of simple sentences in IL.

As you can see it doesn’t always get things right. But some discussion helps it set things right (in itself, this is another startling capability of GPT-4). Such conversations can lead to the development of an accurate labeled data set, and thence to a fine-tuned end-point which should end up being highly accurate. (GPT-4 is not yet available to be fine-tuned, but GPT-3 is, and I would hope -4 will be, soon…)

There are tons of things to discuss here, of course, if there is interest. Just putting this out as a conversation starter…

Best, Vijay

The examples are taken from https://aclanthology.org/E93-1013.pdf and https://aclanthology.org/E95-1005.pdf

VS: Parse the following sentence and generate its m meaning in Montague's Intensional Logic. -- Sentence Bill kissed Hillary. -- Meaning formula

GPT4: To represent the given sentence "Bill kissed Hillary" in Montague's Intensional Logic, we will first parse the sentence into its components:

Subject: Bill

Verb: kissed

Object: Hillary

Montague's Intensional Logic uses lambda calculus to represent meanings. We will use the following variables:

b for Bill

k for the verb 'kiss'

h for Hillary

Now, we can represent the meaning formula as follows:

k(b, h)

This formula states that the kissing action (k) occurs with Bill (b) as the subject and Hillary (h) as the object. The given sentence can be translated into Montague's Intensional Logic as the application of the binary relation 'k' to the individuals 'b' and 'h’.

VS: Good. Here is another one. -- Sentence Bill obviously kissed Hillary.

GPT4: To represent the given sentence "Bill obviously kissed Hillary" in Montague's Intensional Logic, we need to consider the adverb "obviously" as a modifier of the verb "kissed". Let's denote the adverb with the letter 'o' and use the previously defined variables:

b for Bill

k for the verb 'kiss'

h for Hillary

o for the adverb 'obviously'

Now, we can represent the meaning formula as follows:

o(k)(b, h)

This formula states that the kissing action (k) occurs with Bill (b) as the subject and Hillary (h) as the object, and that this action is obviously true (o). The given sentence can be translated into Montague's Intensional Logic as the application of the binary relation 'k' to the individuals 'b' and 'h', modified by the adverb 'o’.

VS: Hmm.. shouldn't this be o(k(b,h))?

GPT4: Apologies for the confusion. You are correct. The correct representation of the sentence "Bill obviously kissed Hillary" in Montague's Intensional Logic should be:

o(k(b, h))

This formula states that the kissing action (k) occurs with Bill (b) as the subject and Hillary (h) as the object, and that this action is obviously true (o). The given sentence can be translated into Montague's Intensional Logic as the adverb 'o' modifying the result of applying the binary relation 'k' to the individuals 'b' and 'h’.

VS: Good. Now let's do: -- Sentence Hillary let Bill write a letter -- Meaning formula

GPT4: To represent the given sentence "Hillary let Bill write a letter" in Montague's Intensional Logic, we will first parse the sentence into its components: Subject: Hillary Verb: let Object: Bill Infinitive verb: write Object of the infinitive verb: a letter

We will use the following variables:

h for Hillary

l for the verb 'let'

b for Bill

w for the verb 'write'

a for a letter

Now, we can represent the meaning formula as follows:

l(h, λx.w(x, a))(b)

This formula states that the letting action (l) occurs with Hillary (h) as the subject and a function representing Bill writing a letter (λx.w(x, a)) as the object. Then, the function is applied to the individual Bill (b).

The given sentence can be translated into Montague's Intensional Logic as the application of the ternary relation 'l' to the individuals 'h' and 'b', and the action of writing a letter (w) with 'a' as the object.

VS: Good. Next: -- Sentence Bill supported, and Hillary opposed, NAFTA.

GPT4: To represent the given sentence "Bill supported, and Hillary opposed, NAFTA" in Montague's Intensional Logic, we need to parse the sentence into its components and consider the conjunction "and" that connects two separate actions: Subject 1: Bill Verb 1: supported Subject 2: Hillary Verb 2: opposed Object: NAFTA

We will use the following variables:

b for Bill

s for the verb 'support'

h for Hillary

o for the verb 'oppose'

n for NAFTA

∧ for the conjunction 'and'

Now, we can represent the meaning formula as follows:

(s(b, n) ∧ o(h, n))

This formula states that Bill (b) supports NAFTA (n), and Hillary (h) opposes NAFTA (n). The given sentence can be translated into Montague's Intensional Logic as two separate actions, the support action (s) with Bill (b) as the subject and NAFTA (n) as the object, and the oppose action (o) with Hillary (h) as the subject and NAFTA (n) as the object, connected by the conjunction 'and' (∧).

VS: OK, next: -- Sentence Bill supported, and Hillary opposed, two trade bills.

GPT4: To represent the given sentence "Bill supported, and Hillary opposed, two trade bills" in Montague's Intensional Logic, we need to parse the sentence into its components and consider the conjunction "and" that connects two separate actions: Subject 1: Bill Verb 1: supported Subject 2: Hillary Verb 2: opposed Quantifier: two Object: trade bills

We will use the following variables:

b for Bill

s for the verb 'support'

h for Hillary

o for the verb 'oppose'

tb for trade bills

∧ for the conjunction 'and'

t for two

Since we are dealing with a quantifier, we will use lambda notation to represent the two trade bills. Let x represent trade bills and y for the second trade bill:

Now, we can represent the meaning formula as follows:

∃x∃y(tb(x) ∧ tb(y) ∧ t(x, y) ∧ s(b, x) ∧ s(b, y) ∧ o(h, x) ∧ o(h, y))

This formula states that there exist two trade bills (x and y) such that Bill (b) supports both (s(b, x) and s(b, y)), and Hillary (h) opposes both (o(h, x) and o(h, y)). The given sentence can be translated into Montague's Intensional Logic as two separate actions, the support action (s) with Bill (b) as the subject and the two trade bills (x and y) as the objects, and the oppose action (o) with Hillary (h) as the subject and the two trade bills (x and y) as the objects, connected by the conjunction 'and' (∧).

VS: Next: -- Sentence Hillary wanted, found, and supported two candidates.

GPT4: To represent the given sentence "Hillary wanted, found, and supported two candidates" in Montague's Intensional Logic, we need to parse the sentence into its components and consider the conjunctions "and" that connect three separate actions:

Subject: Hillary

Verb 1: wanted

Verb 2: found

Verb 3: supported

Quantifier: two

Object: candidates

We will use the following variables:

h for Hillary

w for the verb 'want'

f for the verb 'find'

s for the verb 'support'

c for candidates

∧ for the conjunction 'and'

t for two

Since we are dealing with a quantifier, we will use lambda notation to represent the two candidates. Let x represent the first candidate and y for the second candidate:

Now, we can represent the meaning formula as follows:

∃x∃y(c(x) ∧ c(y) ∧ t(x, y) ∧ w(h, x) ∧ w(h, y) ∧ f(h, x) ∧ f(h, y) ∧ s(h, x) ∧ s(h, y))

This formula states that there exist two candidates (x and y) such that Hillary (h) wanted both (w(h, x) and w(h, y)), found both (f(h, x) and f(h, y)), and supported both (s(h, x) and s(h, y)). The given sentence can be translated into Montague's Intensional Logic as three separate actions, the want action (w), find action (f), and support action (s) with Hillary (h) as the subject and the two candidates (x and y) as the objects, connected by the conjunctions 'and' (∧). VS: It would seem to me that the meaning should be the conjunction of two propositions, one expressing the wanting, and the second that the finding-and-supporting was successfully accomplished…

GPT4: I apologize for my previous responses, and I understand now what you are looking for. To represent the given sentence "Hillary wanted, found, and supported two candidates" in Montague's Intensional Logic as the conjunction of two propositions, we can do the following:

Subject: Hillary

Intensional verb: wanted

Extensional verb 1: found

Extensional verb 2: supported

Quantifier: two

Object: candidates

We will use the following variables:

h for Hillary

w for the verb 'want'

f for the verb 'find'

s for the verb 'support'

c for candidates

t for two

∧ for the conjunction 'and'

Now, we can represent the meaning formula as follows:

w(h, λx.t(c, λy.(f(h, y) ∧ s(h, y)), x)) ∧ ∃x∃y(c(x) ∧ c(y) ∧ t(x, y) ∧ f(h, x) ∧ f(h, y) ∧ s(h, x) ∧ s(h, y))

In this formula, we have two separate propositions joined by a conjunction. The first proposition represents the wanting, with the intensional verb 'want' as the function w(h, λx...), taking Hillary (h) as the subject and a property λx representing the existence of two candidates that Hillary found and supported as its argument. The second proposition represents the successful finding and supporting of two candidates, with the quantifier 'two' applied to the candidates, and for each candidate x and y, the extensional verbs 'found' and 'supported' are applied.

This representation treats the intensional verb 'wanted' separately from the extensional verbs 'found' and 'supported', expressing the conjunction of two propositions: one for the wanting, and the second for the successful finding and supporting of two candidates.

Sent from my iPhone

On May 1, 2023, at 3:59 AM, Miriam Butt <miriam.butt@uni-konstanz.de <mailto:miriam.butt@uni-konstanz.de>> wrote:

Hi,

really interesting post Ken! It seems it’s like with the text generation — very fluent posting and explanation of f-structures but with features we wouldn’t use in LFG (REL-CLAUSE, SUBORD-CLAUSE). I’d really like to know what the underlying technology for this is (beyond the basics).

I’ve started looking into prompt engineering — any pointers on this you might know?

Cheers,

Miriam

On 28. Apr 2023, at 10:53, Ken Kahn <toontalk@gmail.com <mailto:toontalk@gmail.com>> wrote:

Hi

I did get this notification of moderation 4 days ago.

Here's my message again:

Vijay's message got me wondering what else GPT-4 can do.

https://shareg.pt/VcJBEYK

Mary thought the result was at an undergraduate level.

Best,

-ken

---------- Forwarded message --------- From: <lfg-owner@listserv.linguistlist.org <mailto:lfg-owner@listserv.linguistlist.org>> Date: Mon, 24 Apr 2023 at 16:23 Subject: Your message to LFG awaits moderator approval To: <toontalk@gmail.com <mailto:toontalk@gmail.com>>

Your mail to 'LFG' with the subject

GPT-4 producing f-structures

Is being held until the list moderator can review it for approval.

The reason it is being held:

Post by non-member to a members-only list

Either the message will get posted to the list, or you will receive notification of the moderator's decision. If you would like to cancel this posting, please visit the following URL:

https://listserv.linguistlist.org/cgi-bin/mailman/confirm/lfg/7ecd6420d43b3b...

**************************************************************** Miriam Butt Department of Linguistics University of Konstanz Fach 184 Tel: +49 7531 88 5109 78457 Konstanz Fax: +49 7531 88 4865 Germany +49 7531 88 5115

miriam.butt@uni-konstanz.de <mailto:miriam.butt@uni-konstanz.de> https://www.ling.uni-konstanz.de/butt/

'I would suppose, sir,' he said cautiously, 'that Modesty Blaise might be a person awfully difficult for us---er---actually to get.' He blinked towards the big, grey-haired main who stood by the window, looking down at the night traffic, hurrying along Whitehall.

'For a moment,' Tarrant said, turning from the window, 'I hoped you might split that infinitive, Fraser.’

Peter O'Donnell, Modesty Blaise (p. 1) ****************************************************************

I’m reminded of a comment that John Maxwell once made, about some piece of software: It’s like a talking dog. The fact that it talks at all is incredibly impressive—what it says, not so much.,

On May 1, 2023, at 9:47 AM, Cavar, Damir <dcavar@indiana.edu> wrote:

Hi there,

yes, powerful tools or toys, certainly, but not The Solution that replaces powerful engineers and renders smart linguists useless (or forces them into the hamburger wrapping or pizza delivery business).

My personal slogans are now more and more: Every time I fire a LLM expert, my systems run better. And, all statistical and neural parsers leak.

This after engineering them and running detailed evaluations and working with them for decades. XLE and the English grammar are still winning after 20 years of dependency corpora and training the smartest probabilistic/neural algorithms, or after piping almost the entire internet through an LLM like GPT 4.

(You all know that I love hyperbole, right?)

Now, it is funny:

I just tried again, saying "Give me the phrase structure tree for 'I stole a book and Peter kisses from Mary'" and it returned me:

ChatGPT I'm sorry, but the sentence "I stole a book and Peter kisses from Mary" is not a grammatically correct sentence. It appears to have some syntax errors, which make it difficult to construct a meaningful phrase structure tree. Could you please provide a revised sentence or clarify the intended meaning?

Now it also has introspection and can provide grammaticality judgements, it seems. Very interesting. So, I ask:

Me again: On a scale of 1 to 10, how ungrammatical is the sentence?

ChatGPT: The sentence "I stole a book and Peter kisses from Mary" is highly ungrammatical, as it contains multiple errors in syntax and structure. It seems to mix up different verb tenses and noun phrases, making it difficult to determine the intended meaning. I would rate its ungrammaticality as a 9 or 10 on a scale of 1 to 10, with 10 being the most ungrammatical.

Then I asked again, adding a comma: Draw me a phrase structure tree for "I stole a book, and Peter kisses from Mary"

ChatGPT: S _____|_____ | | NP VP | ____|___ | | | Pronoun V|P ConjP | | _____|_____ I stole | | | | | NP VP NP | | _|_ ___|___ | | | | | | | VP Det Noun | Verb Pronoun | | | | | a book and Peter kisses

In this tree, S represents the sentence as a whole, which is divided into two main branches: NP (noun phrase) and VP (verb phrase). The NP consists of a pronoun "I" and an NP headed by a determiner "a" and a noun "book". The VP is divided into two conjuncts by the coordinating conjunction "and". The first conjunct consists of a verb "stole" and a preposition "from" followed by an NP headed by the proper noun "Mary". The second conjunct consists of a proper noun "Peter" and a verb "kisses", forming a separate VP.

OK, so just one comma made all the difference in rating, now the sentence is good and the resulting analysis is quite interesting. But, let us try some simpler construction:

Me: Draw me a phrase structure tree for "Peter likes to think about algorithms and Mary about cats"

ChatGPT: S ____________|____________ | | NP(Peter) S(Mary) | __________|___________ VP(likes) | | | VP(to think) PP(about) S(to) and | ____________|________ NP(algorithms) NP(Mary) NP(cats) (about)

In this tree, S represents a sentence, NP represents a noun phrase, VP represents a verb phrase, and PP represents a prepositional phrase. The conjunction "and" is represented by S(to) and, where to is the conjunction, and S(Mary) represents the clause for Mary's action.

-------------------

Yes, yes, very good... (for me) :-) I won't even go into more complex constructions... And, well, XLE-Web with English does not get it, either, but it gets much more much better, with or without comma.

Some people argue that we should adjust our language so that AIs, LLMs, and some tools can process it. Others claim that ellipsis or gapping are weird constructions that only some crazy linguists can come up with. A colleague from a big US bank asked me why we put in these weird sentences into ChatGPT, why don't we just speak in plain English to it... :-)

Do we really want to study "prompts for different versions of GPT" and engineer around them, or do we want to engineer professional systems around our insights that we can falsify and test? I'll continue working on my LFG parser, quantum NLP, CGs, and real working tech, until ChatGPT figures that out. :-) I'd use LLMs for innocent problems where it cannot harm or create issues with generating problematic nonsense. It is a useful toy, not more than that (except the carbon emission...).

CU all!

DC

PS: I don't know how about you, but we got students submitting essays on different topics that start with:

"As an artificial intelligence language model, I do not have opinions on (...)"

Some folks just copy and paste, they do not even read what ChatGPT gives them. At least that is easy to grade...

On Mon, 2023-05-01 at 11:53 -0400, Vijay Saraswat wrote:

...

Hi Damir — Everything depends on how the problem is formulated. And which model you are using — the right prompts for GPT-4 are not necessarily right for GPT 3.5 or GPT 3. Anyway will give your examples a shot later today / tomorrow.

An important point to note is (and I said this in my message): One has the possibility of further training these models on labeled data, e.g. human corrected LFG parses etc. I believe one of the LFG teams trained a statistical parser in this way 10-15 years ago. But with Language Model (LMs) technology now having progressed substantially we should be able to use LMs to develop really powerful tools for computational linguists.

-- Damir Cavar <http://damir.cavar.me/> Associate Professor Indiana University at Bloomington NLP-Lab <https://nlp-lab.org/>

John Maxwell's remark cited by Ron is a variant of a well-known quotation illustrating historical prejudice: As recorded by Boswell on July 31, 1763, Johnson said, ''Sir, *a woman's preaching is like a dog's walking on his hinder legs*. It's not done well; but you are surprised to find it done at all. '' As a woman, I would like to see a more specific analysis of the problems of the software. I believe most people would struggle to parse some of the sentences offered without the appropriate intonation. signed, Mary about cats On Mon, May 1, 2023 at 12:42 PM Ron Kaplan <ron.kaplan@post.harvard.edu> wrote:

I’m reminded of a comment that John Maxwell once made, about some piece of software:

It’s like a talking dog. The fact that it talks at all is incredibly impressive—what it says, not so much.,

On May 1, 2023, at 9:47 AM, Cavar, Damir <dcavar@indiana.edu> wrote:

Hi there,

yes, powerful tools or toys, certainly, but not The Solution that replaces powerful engineers and renders smart linguists useless (or forces them into the hamburger wrapping or pizza delivery business).

My personal slogans are now more and more: Every time I fire a LLM expert, my systems run better. And, all statistical and neural parsers leak.

This after engineering them and running detailed evaluations and working with them for decades. XLE and the English grammar are still winning after 20 years of dependency corpora and training the smartest probabilistic/neural algorithms, or after piping almost the entire internet through an LLM like GPT 4.

(You all know that I love hyperbole, right?)

Now, it is funny:

I just tried again, saying "Give me the phrase structure tree for 'I stole a book and Peter kisses from Mary'" and it returned me:

ChatGPT I'm sorry, but the sentence "I stole a book and Peter kisses from Mary" is not a grammatically correct sentence. It appears to have some syntax errors, which make it difficult to construct a meaningful phrase structure tree. Could you please provide a revised sentence or clarify the intended meaning?

Now it also has introspection and can provide grammaticality judgements, it seems. Very interesting. So, I ask:

Me again: On a scale of 1 to 10, how ungrammatical is the sentence?

ChatGPT: The sentence "I stole a book and Peter kisses from Mary" is highly ungrammatical, as it contains multiple errors in syntax and structure. It seems to mix up different verb tenses and noun phrases, making it difficult to determine the intended meaning. I would rate its ungrammaticality as a 9 or 10 on a scale of 1 to 10, with 10 being the most ungrammatical.

Then I asked again, adding a comma: Draw me a phrase structure tree for "I stole a book, and Peter kisses from Mary"

ChatGPT:

S _____|_____ | | NP VP | ____|___ | | | Pronoun V|P ConjP | | _____|_____ I stole | | | | | NP VP NP | | _|_ ___|___ | | | | | | | VP Det Noun | Verb Pronoun | | | | | a book and Peter kisses

In this tree, S represents the sentence as a whole, which is divided into two main branches: NP (noun phrase) and VP (verb phrase). The NP consists of a pronoun "I" and an NP headed by a determiner "a" and a noun "book". The VP is divided into two conjuncts by the coordinating conjunction "and". The first conjunct consists of a verb "stole" and a preposition "from" followed by an NP headed by the proper noun "Mary". The second conjunct consists of a proper noun "Peter" and a verb "kisses", forming a separate VP.

OK, so just one comma made all the difference in rating, now the sentence is good and the resulting analysis is quite interesting. But, let us try some simpler construction:

Me: Draw me a phrase structure tree for "Peter likes to think about algorithms and Mary about cats"

ChatGPT:

S ____________|____________ | | NP(Peter) S(Mary) | __________|___________ VP(likes) | | | VP(to think) PP(about) S(to) and | ____________|________ NP(algorithms) NP(Mary) NP(cats) (about)

In this tree, S represents a sentence, NP represents a noun phrase, VP represents a verb phrase, and PP represents a prepositional phrase. The conjunction "and" is represented by S(to) and, where to is the conjunction, and S(Mary) represents the clause for Mary's action.

-------------------

Yes, yes, very good... (for me) :-) I won't even go into more complex constructions... And, well, XLE-Web with English does not get it, either, but it gets much more much better, with or without comma.

Some people argue that we should adjust our language so that AIs, LLMs, and some tools can process it. Others claim that ellipsis or gapping are weird constructions that only some crazy linguists can come up with. A colleague from a big US bank asked me why we put in these weird sentences into ChatGPT, why don't we just speak in plain English to it... :-)

Do we really want to study "prompts for different versions of GPT" and engineer around them, or do we want to engineer professional systems around our insights that we can falsify and test? I'll continue working on my LFG parser, quantum NLP, CGs, and real working tech, until ChatGPT figures that out. :-) I'd use LLMs for innocent problems where it cannot harm or create issues with generating problematic nonsense. It is a useful toy, not more than that (except the carbon emission...).

CU all!

DC

PS: I don't know how about you, but we got students submitting essays on different topics that start with:

"As an artificial intelligence language model, I do not have opinions on (...)"

Some folks just copy and paste, they do not even read what ChatGPT gives them. At least that is easy to grade...

On Mon, 2023-05-01 at 11:53 -0400, Vijay Saraswat wrote:

Hi Damir — Everything depends on how the problem is formulated. And which model you are using — the right prompts for GPT-4 are not necessarily right for GPT 3.5 or GPT 3. Anyway will give your examples a shot later today / tomorrow.

An important point to note is (and I said this in my message): One has the possibility of further training these models on labeled data, e.g. human corrected LFG parses etc. I believe one of the LFG teams trained a statistical parser in this way 10-15 years ago. But with Language Model (LMs) technology now having progressed substantially we should be able to use LMs to develop really powerful tools for computational linguists.

--

Damir Cavar <http://damir.cavar.me/> Associate Professor Indiana University at Bloomington NLP-Lab <https://nlp-lab.org/>

-- Joan Bresnan Stanford University http://web.stanford.edu/~bresnan/

Dear Damir and Joan (by the way how are you? - when are you going to come back to Venice again?), I tried with two a little bit more complex examples and the results are certainly interesting, but as someone else on the list remarked they need manual checking: ************************* Show me the semantic representation for the following short text: John and Mary went to see a movie. It was very bad, so she fell asleep. ∃x(person(x) ∧ name(x, "John")) ∧ ∃y(person(y) ∧ name(y, "Mary")) ∧ ∃z(movie(z) ∧ watch(x, z) ∧ watch(y, z)) ∧ quality(z, "bad") ∧ fall_asleep(y) Let's try another one: Frank hates Robert because he ruined his special bicycle by falling by accident into a river. ∃x(person(x) ∧ name(x, "Frank")) ∧ ∃y(person(y) ∧ name(y, "Robert")) ∧ hate(x, y) ∧ ∃z(bicycle(z) ∧ own(x, z) ∧ ruined_by(z, y, "falling into a river")) ********************* Apart from the mistaken subject assigned to the gerundive the remaining analyses are ok. But of course if I try to ask phoneme to grapheme translation of some poems then the answer is that it's too complex and requires a lot of manual checking in particular because poetry is difficult to translate. So eventually, I think that this stochastic parrot is good for other purposes but bad or insufficient for complex tasks as the ones required by linguists. Rodolfo Il giorno mar 2 mag 2023 alle ore 07:00 Cavar, Damir <dcavar@indiana.edu> ha scritto:

Joan,

You make an important point!

GPT3 was trained on text only. The new models include now image information, as well. Multi-modal data input is essential to figure that carrots are typically orange, for example. The issue is, if we talk online about things, we emphasize new and exciting facts. You might find "purple carrot" more frequently mentioned than "orange carrot" in text-based communication online, just because of pragmatic or conversational principles. Known and presupposed facts we do not talk about. Images of carrots would show orange as the default or dominant color. Bringing text and image together is essential

Also, without the image, if I would talk about "my new car" you might conclude that it could be a 2023 Porsche or such. Only when I show you the picture, would it be clear that it is not the car that is new, since it is a 1965 Ford Mustang.

So, good they are now training the models on text and image.

But, nobody - as far as I can tell - is training right now on phonetic, prosodic, and textual language data to potentially be able to catch intonational clues. Most likely because that kind of data does not exist on the necessary scale needed for such models.

In the previous post I was testing gapping with semantic mismatches of the coordinated predicates, and mismatches of phi-features. Most informants get it. Without such mismatches (semantic or agreement) things should be pretty easy to process for native speakers, they are in fact common in conversation, though not easy to parse for GPT, and, neither for Stanford CoreNLP, Stanza, spaCy, Benepar, etc. Our State of the Art NLP parsers fail on anything that involves silence elements that are common in real conversational behavior or in social media posts. Parsers are rarely used in real world applications, it turns out, and there are good reasons for that.

I am not opposing LLMs, and I see good application domains, just exploring the limits. The models are certainly useful, but also over-hyped.

All the best

DC

-- Damir Cavar <http://damir.cavar.me/> Associate Professor Indiana University <https://www.indiana.edu/> at Bloomington NLP-Lab <https://nlp-lab.com/>

------------------------------ *From:* LFG-list <lfg-list-bounces@mailman.uni-konstanz.de> on behalf of Joan Bresnan <bresnan@stanford.edu> *Sent:* Monday, May 1, 2023, 3:58 PM *To:* Ron Kaplan <ron.kaplan@post.harvard.edu> *Cc:* lfg-list@mailman.uni-konstanz.de <lfg-list@mailman.uni-konstanz.de> *Subject:* Re: [LFG-list] ChatGPT discussion

John Maxwell's remark cited by Ron is a variant of a well-known quotation illustrating historical prejudice:

As recorded by Boswell on July 31, 1763, Johnson said, ''Sir, *a woman's preaching is like a dog's walking on his hinder legs*. It's not done well; but you are surprised to find it done at all. ''

As a woman, I would like to see a more specific analysis of the problems of the software. I believe most people would struggle to parse some of the sentences offered without the appropriate intonation.

signed,

Mary about cats

On Mon, May 1, 2023 at 12:42 PM Ron Kaplan <ron.kaplan@post.harvard.edu> wrote:

...

I’m reminded of a comment that John Maxwell once made, about some piece of software:

It’s like a talking dog. The fact that it talks at all is incredibly impressive—what it says, not so much.,

On May 1, 2023, at 9:47 AM, Cavar, Damir <dcavar@indiana.edu> wrote:

Hi there,

yes, powerful tools or toys, certainly, but not The Solution that replaces powerful engineers and renders smart linguists useless (or forces them into the hamburger wrapping or pizza delivery business).

My personal slogans are now more and more: Every time I fire a LLM expert, my systems run better. And, all statistical and neural parsers leak.

This after engineering them and running detailed evaluations and working with them for decades. XLE and the English grammar are still winning after 20 years of dependency corpora and training the smartest probabilistic/neural algorithms, or after piping almost the entire internet through an LLM like GPT 4.

(You all know that I love hyperbole, right?)

Now, it is funny:

I just tried again, saying "Give me the phrase structure tree for 'I stole a book and Peter kisses from Mary'" and it returned me:

ChatGPT I'm sorry, but the sentence "I stole a book and Peter kisses from Mary" is not a grammatically correct sentence. It appears to have some syntax errors, which make it difficult to construct a meaningful phrase structure tree. Could you please provide a revised sentence or clarify the intended meaning?

Now it also has introspection and can provide grammaticality judgements, it seems. Very interesting. So, I ask:

Me again: On a scale of 1 to 10, how ungrammatical is the sentence?

ChatGPT: The sentence "I stole a book and Peter kisses from Mary" is highly ungrammatical, as it contains multiple errors in syntax and structure. It seems to mix up different verb tenses and noun phrases, making it difficult to determine the intended meaning. I would rate its ungrammaticality as a 9 or 10 on a scale of 1 to 10, with 10 being the most ungrammatical.

Then I asked again, adding a comma: Draw me a phrase structure tree for "I stole a book, and Peter kisses from Mary"

ChatGPT:

S _____|_____ | | NP VP | ____|___ | | | Pronoun V|P ConjP | | _____|_____ I stole | | | | | NP VP NP | | _|_ ___|___ | | | | | | | VP Det Noun | Verb Pronoun | | | | | a book and Peter kisses

In this tree, S represents the sentence as a whole, which is divided into two main branches: NP (noun phrase) and VP (verb phrase). The NP consists of a pronoun "I" and an NP headed by a determiner "a" and a noun "book". The VP is divided into two conjuncts by the coordinating conjunction "and". The first conjunct consists of a verb "stole" and a preposition "from" followed by an NP headed by the proper noun "Mary". The second conjunct consists of a proper noun "Peter" and a verb "kisses", forming a separate VP.

OK, so just one comma made all the difference in rating, now the sentence is good and the resulting analysis is quite interesting. But, let us try some simpler construction:

Me: Draw me a phrase structure tree for "Peter likes to think about algorithms and Mary about cats"

ChatGPT:

S ____________|____________ | | NP(Peter) S(Mary) | __________|___________ VP(likes) | | | VP(to think) PP(about) S(to) and | ____________|________ NP(algorithms) NP(Mary) NP(cats) (about)

In this tree, S represents a sentence, NP represents a noun phrase, VP represents a verb phrase, and PP represents a prepositional phrase. The conjunction "and" is represented by S(to) and, where to is the conjunction, and S(Mary) represents the clause for Mary's action.

-------------------

Yes, yes, very good... (for me) :-) I won't even go into more complex constructions... And, well, XLE-Web with English does not get it, either, but it gets much more much better, with or without comma.

Some people argue that we should adjust our language so that AIs, LLMs, and some tools can process it. Others claim that ellipsis or gapping are weird constructions that only some crazy linguists can come up with. A colleague from a big US bank asked me why we put in these weird sentences into ChatGPT, why don't we just speak in plain English to it... :-)

Do we really want to study "prompts for different versions of GPT" and engineer around them, or do we want to engineer professional systems around our insights that we can falsify and test? I'll continue working on my LFG parser, quantum NLP, CGs, and real working tech, until ChatGPT figures that out. :-) I'd use LLMs for innocent problems where it cannot harm or create issues with generating problematic nonsense. It is a useful toy, not more than that (except the carbon emission...).

CU all!

DC

PS: I don't know how about you, but we got students submitting essays on different topics that start with:

"As an artificial intelligence language model, I do not have opinions on (...)"

Some folks just copy and paste, they do not even read what ChatGPT gives them. At least that is easy to grade...

On Mon, 2023-05-01 at 11:53 -0400, Vijay Saraswat wrote:

Hi Damir — Everything depends on how the problem is formulated. And which model you are using — the right prompts for GPT-4 are not necessarily right for GPT 3.5 or GPT 3. Anyway will give your examples a shot later today / tomorrow.

An important point to note is (and I said this in my message): One has the possibility of further training these models on labeled data, e.g. human corrected LFG parses etc. I believe one of the LFG teams trained a statistical parser in this way 10-15 years ago. But with Language Model (LMs) technology now having progressed substantially we should be able to use LMs to develop really powerful tools for computational linguists.

--

Damir Cavar <http://damir.cavar.me/> Associate Professor Indiana University at Bloomington NLP-Lab <https://nlp-lab.org/>

-- Joan Bresnan Stanford University http://web.stanford.edu/~bresnan/

-- Rodolfo Delmonte Ph.D. Associate Professor of Computational Linguistics Director of Laboratory of Computational Linguistics Department of Linguistic Studies Ca' Bembo - Dorsoduro 1075 30123 VENEZIA (Italy) website: http//rondelmo.it

Thanks, Ken. Agree with your comments. I have a talk on the things that GPT4 can do, and those it cant. To your point, it can reason startlingly well about the real world, e.g. on a large variety of naive-physics examples, including several (should be) new, contrived ones. Interestingly, you can ask it for problems that it thinks GPT4 would be hard at, and indeed it turns out that it is not able to get those problems. Ilya Sutskever (OpenAI Chief Scientist) made the same point in a recent talk (as you probably know), their main focus in GPT4 was to get it to be much better at representing the world, and they feel they succeeded, compared to -3, -3.5.

On May 2, 2023, at 5:55 AM, Ken Kahn <toontalk@gmail.com> wrote:

Regarding "Multi-modal data input is essential to figure that carrots are typically orange, for example." I just asked ChatGPT 3.5 "Rank the most common colors for carrots".

Here's the reply:

The most common colors for carrots are: Orange Purple White Yellow Red Orange is by far the most common color for carrots. However, purple, white, yellow, and red carrots are also grown and eaten, although they are less common than orange carrots. Purple carrots, for example, are a good source of anthocyanins, which are antioxidants that may help protect against certain diseases. White carrots, on the other hand, are lower in beta-carotene than orange carrots, but they are still a good source of dietary fiber and other nutrients. Yellow and red carrots also contain various beneficial nutrients and are often used in culinary applications for their unique flavors and colors.

And text-davinci-003 doesn't seem to even consider other colors: <image.png>

I used these models because GPT-4 is multi-modal and is able to discuss images.

While multi-modal input is helpful, many studies are showing it isn't necessary. It can draw a unicorn in TikZ Latex. It can answer questions about what colors are close to other colors. It can reason about directions.

It seems pretty grounded despite being trained on text only. Predicting the next word well involves having good models of the world and people.

Best,

-ken

On Tue, 2 May 2023 at 06:00, Cavar, Damir <dcavar@indiana.edu <mailto:dcavar@indiana.edu>> wrote:

...

Joan,

You make an important point!

GPT3 was trained on text only. The new models include now image information, as well. Multi-modal data input is essential to figure that carrots are typically orange, for example. The issue is, if we talk online about things, we emphasize new and exciting facts. You might find "purple carrot" more frequently mentioned than "orange carrot" in text-based communication online, just because of pragmatic or conversational principles. Known and presupposed facts we do not talk about. Images of carrots would show orange as the default or dominant color. Bringing text and image together is essential

Also, without the image, if I would talk about "my new car" you might conclude that it could be a 2023 Porsche or such. Only when I show you the picture, would it be clear that it is not the car that is new, since it is a 1965 Ford Mustang.

So, good they are now training the models on text and image.

But, nobody - as far as I can tell - is training right now on phonetic, prosodic, and textual language data to potentially be able to catch intonational clues. Most likely because that kind of data does not exist on the necessary scale needed for such models.

In the previous post I was testing gapping with semantic mismatches of the coordinated predicates, and mismatches of phi-features. Most informants get it. Without such mismatches (semantic or agreement) things should be pretty easy to process for native speakers, they are in fact common in conversation, though not easy to parse for GPT, and, neither for Stanford CoreNLP, Stanza, spaCy, Benepar, etc. Our State of the Art NLP parsers fail on anything that involves silence elements that are common in real conversational behavior or in social media posts. Parsers are rarely used in real world applications, it turns out, and there are good reasons for that.

I am not opposing LLMs, and I see good application domains, just exploring the limits. The models are certainly useful, but also over-hyped.

All the best

DC

-- Damir Cavar <http://damir.cavar.me/> Associate Professor Indiana University <https://www.indiana.edu/> at Bloomington NLP-Lab <https://nlp-lab.com/>

From: LFG-list <lfg-list-bounces@mailman.uni-konstanz.de <mailto:lfg-list-bounces@mailman.uni-konstanz.de>> on behalf of Joan Bresnan <bresnan@stanford.edu <mailto:bresnan@stanford.edu>> Sent: Monday, May 1, 2023, 3:58 PM To: Ron Kaplan <ron.kaplan@post.harvard.edu <mailto:ron.kaplan@post.harvard.edu>> Cc: lfg-list@mailman.uni-konstanz.de <mailto:lfg-list@mailman.uni-konstanz.de> <lfg-list@mailman.uni-konstanz.de <mailto:lfg-list@mailman.uni-konstanz.de>> Subject: Re: [LFG-list] ChatGPT discussion

John Maxwell's remark cited by Ron is a variant of a well-known quotation illustrating historical prejudice:

As recorded by Boswell on July 31, 1763, Johnson said, ''Sir, a woman's preaching is like a dog's walking on his hinder legs. It's not done well; but you are surprised to find it done at all. ''

As a woman, I would like to see a more specific analysis of the problems of the software. I believe most people would struggle to parse some of the sentences offered without the appropriate intonation.

signed,

Mary about cats

On Mon, May 1, 2023 at 12:42 PM Ron Kaplan <ron.kaplan@post.harvard.edu <mailto:ron.kaplan@post.harvard.edu>> wrote:

...

I’m reminded of a comment that John Maxwell once made, about some piece of software:

It’s like a talking dog. The fact that it talks at all is incredibly impressive—what it says, not so much.,

...

On May 1, 2023, at 9:47 AM, Cavar, Damir <dcavar@indiana.edu <mailto:dcavar@indiana.edu>> wrote:

Hi there,

yes, powerful tools or toys, certainly, but not The Solution that replaces powerful engineers and renders smart linguists useless (or forces them into the hamburger wrapping or pizza delivery business).

My personal slogans are now more and more: Every time I fire a LLM expert, my systems run better. And, all statistical and neural parsers leak.

This after engineering them and running detailed evaluations and working with them for decades. XLE and the English grammar are still winning after 20 years of dependency corpora and training the smartest probabilistic/neural algorithms, or after piping almost the entire internet through an LLM like GPT 4.

(You all know that I love hyperbole, right?)

Now, it is funny:

I just tried again, saying "Give me the phrase structure tree for 'I stole a book and Peter kisses from Mary'" and it returned me:

ChatGPT I'm sorry, but the sentence "I stole a book and Peter kisses from Mary" is not a grammatically correct sentence. It appears to have some syntax errors, which make it difficult to construct a meaningful phrase structure tree. Could you please provide a revised sentence or clarify the intended meaning?

Now it also has introspection and can provide grammaticality judgements, it seems. Very interesting. So, I ask:

Me again: On a scale of 1 to 10, how ungrammatical is the sentence?

ChatGPT: The sentence "I stole a book and Peter kisses from Mary" is highly ungrammatical, as it contains multiple errors in syntax and structure. It seems to mix up different verb tenses and noun phrases, making it difficult to determine the intended meaning. I would rate its ungrammaticality as a 9 or 10 on a scale of 1 to 10, with 10 being the most ungrammatical.

Then I asked again, adding a comma: Draw me a phrase structure tree for "I stole a book, and Peter kisses from Mary"

ChatGPT: S _____|_____ | | NP VP | ____|___ | | | Pronoun V|P ConjP | | _____|_____ I stole | | | | | NP VP NP | | _|_ ___|___ | | | | | | | VP Det Noun | Verb Pronoun | | | | | a book and Peter kisses

In this tree, S represents the sentence as a whole, which is divided into two main branches: NP (noun phrase) and VP (verb phrase). The NP consists of a pronoun "I" and an NP headed by a determiner "a" and a noun "book". The VP is divided into two conjuncts by the coordinating conjunction "and". The first conjunct consists of a verb "stole" and a preposition "from" followed by an NP headed by the proper noun "Mary". The second conjunct consists of a proper noun "Peter" and a verb "kisses", forming a separate VP.

OK, so just one comma made all the difference in rating, now the sentence is good and the resulting analysis is quite interesting. But, let us try some simpler construction:

Me: Draw me a phrase structure tree for "Peter likes to think about algorithms and Mary about cats"

ChatGPT: S ____________|____________ | | NP(Peter) S(Mary) | __________|___________ VP(likes) | | | VP(to think) PP(about) S(to) and | ____________|________ NP(algorithms) NP(Mary) NP(cats) (about)

In this tree, S represents a sentence, NP represents a noun phrase, VP represents a verb phrase, and PP represents a prepositional phrase. The conjunction "and" is represented by S(to) and, where to is the conjunction, and S(Mary) represents the clause for Mary's action.

-------------------

Yes, yes, very good... (for me) :-) I won't even go into more complex constructions... And, well, XLE-Web with English does not get it, either, but it gets much more much better, with or without comma.

Some people argue that we should adjust our language so that AIs, LLMs, and some tools can process it. Others claim that ellipsis or gapping are weird constructions that only some crazy linguists can come up with. A colleague from a big US bank asked me why we put in these weird sentences into ChatGPT, why don't we just speak in plain English to it... :-)

Do we really want to study "prompts for different versions of GPT" and engineer around them, or do we want to engineer professional systems around our insights that we can falsify and test? I'll continue working on my LFG parser, quantum NLP, CGs, and real working tech, until ChatGPT figures that out. :-) I'd use LLMs for innocent problems where it cannot harm or create issues with generating problematic nonsense. It is a useful toy, not more than that (except the carbon emission...).

CU all!

DC

PS: I don't know how about you, but we got students submitting essays on different topics that start with:

"As an artificial intelligence language model, I do not have opinions on (...)"

Some folks just copy and paste, they do not even read what ChatGPT gives them. At least that is easy to grade...

On Mon, 2023-05-01 at 11:53 -0400, Vijay Saraswat wrote:

...

Hi Damir — Everything depends on how the problem is formulated. And which model you are using — the right prompts for GPT-4 are not necessarily right for GPT 3.5 or GPT 3. Anyway will give your examples a shot later today / tomorrow.

An important point to note is (and I said this in my message): One has the possibility of further training these models on labeled data, e.g. human corrected LFG parses etc. I believe one of the LFG teams trained a statistical parser in this way 10-15 years ago. But with Language Model (LMs) technology now having progressed substantially we should be able to use LMs to develop really powerful tools for computational linguists.

-- Damir Cavar <http://damir.cavar.me/> Associate Professor Indiana University at Bloomington NLP-Lab <https://nlp-lab.org/>

-- Joan Bresnan Stanford University http://web.stanford.edu/~bresnan/