Fwd: Using GPT4 as an assistant for computational linguists
Try 2...
Begin forwarded message:
From: Vijay Saraswat <vijay@saraswat.org> Subject: Using GPT4 as an assistant for computational linguists Date: April 24, 2023 at 6:17:08 AM EDT To: LFG@listserv.linguistlist.org, hpsg-l@listserv.linguistlist.org, Mary Dalrymple <mary.dalrymple@ling-phil.ox.ac.uk>, Ken Kahn <toontalk@googlemail.com>, fernando.pereira@gmail.com, lamping@gmail.com
I thought this group might find the following interesting.
For the last several years I have been leading an R&D group in AI (essentially applied NLP) at a major Wall Street Bank. Since 2020, I have been struck by the stunning progress made by Large Language Models (LLMs; GPT-3, ChatGPT, GPT-4, …). Essentially it seems that a lot of the work we were doing in computational linguistics (at least some of us), particularly in terms of applications to real-world question answering problems, has now been swamped by LLM advances.
I have a separate talk on the kind of highly-specific text analyses GPT-4 can do directly for (financial) professionals with domain-specific text, without having to go through parsing / semantic representation steps. Here I thought it might be illuminating to provide a log of a conversation with GPT-4 about representing meanings of simple sentences in IL.
As you can see it doesn’t always get things right. But some discussion helps it set things right (in itself, this is another startling capability of GPT-4). Such conversations can lead to the development of an accurate labeled data set, and thence to a fine-tuned end-point which should end up being highly accurate. (GPT-4 is not yet available to be fine-tuned, but GPT-3 is, and I would hope -4 will be, soon…)
There are tons of things to discuss here, of course, if there is interest. Just putting this out as a conversation starter…
Best, Vijay
The examples are taken from https://aclanthology.org/E93-1013.pdf and https://aclanthology.org/E95-1005.pdf
VS: Parse the following sentence and generate its m meaning in Montague's Intensional Logic. -- Sentence Bill kissed Hillary. -- Meaning formula
GPT4: To represent the given sentence "Bill kissed Hillary" in Montague's Intensional Logic, we will first parse the sentence into its components: Subject: Bill Verb: kissed Object: Hillary Montague's Intensional Logic uses lambda calculus to represent meanings. We will use the following variables:
b for Bill k for the verb 'kiss' h for Hillary Now, we can represent the meaning formula as follows:
k(b, h)
This formula states that the kissing action (k) occurs with Bill (b) as the subject and Hillary (h) as the object. The given sentence can be translated into Montague's Intensional Logic as the application of the binary relation 'k' to the individuals 'b' and 'h’. VS: Good. Here is another one. -- Sentence Bill obviously kissed Hillary.
GPT4: To represent the given sentence "Bill obviously kissed Hillary" in Montague's Intensional Logic, we need to consider the adverb "obviously" as a modifier of the verb "kissed". Let's denote the adverb with the letter 'o' and use the previously defined variables: b for Bill k for the verb 'kiss' h for Hillary o for the adverb 'obviously' Now, we can represent the meaning formula as follows:
o(k)(b, h)
This formula states that the kissing action (k) occurs with Bill (b) as the subject and Hillary (h) as the object, and that this action is obviously true (o). The given sentence can be translated into Montague's Intensional Logic as the application of the binary relation 'k' to the individuals 'b' and 'h', modified by the adverb 'o’. VS: Hmm.. shouldn't this be o(k(b,h))? GPT4: Apologies for the confusion. You are correct. The correct representation of the sentence "Bill obviously kissed Hillary" in Montague's Intensional Logic should be: o(k(b, h))
This formula states that the kissing action (k) occurs with Bill (b) as the subject and Hillary (h) as the object, and that this action is obviously true (o). The given sentence can be translated into Montague's Intensional Logic as the adverb 'o' modifying the result of applying the binary relation 'k' to the individuals 'b' and 'h’. VS: Good. Now let's do: -- Sentence Hillary let Bill write a letter -- Meaning formula
GPT4: To represent the given sentence "Hillary let Bill write a letter" in Montague's Intensional Logic, we will first parse the sentence into its components: Subject: Hillary Verb: let Object: Bill Infinitive verb: write Object of the infinitive verb: a letter We will use the following variables:
h for Hillary l for the verb 'let' b for Bill w for the verb 'write' a for a letter Now, we can represent the meaning formula as follows:
l(h, λx.w(x, a))(b)
This formula states that the letting action (l) occurs with Hillary (h) as the subject and a function representing Bill writing a letter (λx.w(x, a)) as the object. Then, the function is applied to the individual Bill (b).
The given sentence can be translated into Montague's Intensional Logic as the application of the ternary relation 'l' to the individuals 'h' and 'b', and the action of writing a letter (w) with 'a' as the object. VS: Good. Next: -- Sentence Bill supported, and Hillary opposed, NAFTA.
GPT4: To represent the given sentence "Bill supported, and Hillary opposed, NAFTA" in Montague's Intensional Logic, we need to parse the sentence into its components and consider the conjunction "and" that connects two separate actions: Subject 1: Bill Verb 1: supported Subject 2: Hillary Verb 2: opposed Object: NAFTA We will use the following variables:
b for Bill s for the verb 'support' h for Hillary o for the verb 'oppose' n for NAFTA ∧ for the conjunction 'and' Now, we can represent the meaning formula as follows:
(s(b, n) ∧ o(h, n))
This formula states that Bill (b) supports NAFTA (n), and Hillary (h) opposes NAFTA (n). The given sentence can be translated into Montague's Intensional Logic as two separate actions, the support action (s) with Bill (b) as the subject and NAFTA (n) as the object, and the oppose action (o) with Hillary (h) as the subject and NAFTA (n) as the object, connected by the conjunction 'and' (∧). VS: OK, next: -- Sentence Bill supported, and Hillary opposed, two trade bills.
GPT4: To represent the given sentence "Bill supported, and Hillary opposed, two trade bills" in Montague's Intensional Logic, we need to parse the sentence into its components and consider the conjunction "and" that connects two separate actions: Subject 1: Bill Verb 1: supported Subject 2: Hillary Verb 2: opposed Quantifier: two Object: trade bills We will use the following variables:
b for Bill s for the verb 'support' h for Hillary o for the verb 'oppose' tb for trade bills ∧ for the conjunction 'and' t for two Since we are dealing with a quantifier, we will use lambda notation to represent the two trade bills. Let x represent trade bills and y for the second trade bill:
Now, we can represent the meaning formula as follows:
∃x∃y(tb(x) ∧ tb(y) ∧ t(x, y) ∧ s(b, x) ∧ s(b, y) ∧ o(h, x) ∧ o(h, y))
This formula states that there exist two trade bills (x and y) such that Bill (b) supports both (s(b, x) and s(b, y)), and Hillary (h) opposes both (o(h, x) and o(h, y)). The given sentence can be translated into Montague's Intensional Logic as two separate actions, the support action (s) with Bill (b) as the subject and the two trade bills (x and y) as the objects, and the oppose action (o) with Hillary (h) as the subject and the two trade bills (x and y) as the objects, connected by the conjunction 'and' (∧). VS: Next: -- Sentence Hillary wanted, found, and supported two candidates. GPT4: To represent the given sentence "Hillary wanted, found, and supported two candidates" in Montague's Intensional Logic, we need to parse the sentence into its components and consider the conjunctions "and" that connect three separate actions: Subject: Hillary Verb 1: wanted Verb 2: found Verb 3: supported Quantifier: two Object: candidates We will use the following variables:
h for Hillary w for the verb 'want' f for the verb 'find' s for the verb 'support' c for candidates ∧ for the conjunction 'and' t for two Since we are dealing with a quantifier, we will use lambda notation to represent the two candidates. Let x represent the first candidate and y for the second candidate:
Now, we can represent the meaning formula as follows:
∃x∃y(c(x) ∧ c(y) ∧ t(x, y) ∧ w(h, x) ∧ w(h, y) ∧ f(h, x) ∧ f(h, y) ∧ s(h, x) ∧ s(h, y))
This formula states that there exist two candidates (x and y) such that Hillary (h) wanted both (w(h, x) and w(h, y)), found both (f(h, x) and f(h, y)), and supported both (s(h, x) and s(h, y)). The given sentence can be translated into Montague's Intensional Logic as three separate actions, the want action (w), find action (f), and support action (s) with Hillary (h) as the subject and the two candidates (x and y) as the objects, connected by the conjunctions 'and' (∧). VS: It would seem to me that the meaning should be the conjunction of two propositions, one expressing the wanting, and the second that the finding-and-supporting was successfully accomplished… GPT4: I apologize for my previous responses, and I understand now what you are looking for. To represent the given sentence "Hillary wanted, found, and supported two candidates" in Montague's Intensional Logic as the conjunction of two propositions, we can do the following: Subject: Hillary Intensional verb: wanted Extensional verb 1: found Extensional verb 2: supported Quantifier: two Object: candidates We will use the following variables:
h for Hillary w for the verb 'want' f for the verb 'find' s for the verb 'support' c for candidates t for two ∧ for the conjunction 'and' Now, we can represent the meaning formula as follows:
w(h, λx.t(c, λy.(f(h, y) ∧ s(h, y)), x)) ∧ ∃x∃y(c(x) ∧ c(y) ∧ t(x, y) ∧ f(h, x) ∧ f(h, y) ∧ s(h, x) ∧ s(h, y))
In this formula, we have two separate propositions joined by a conjunction. The first proposition represents the wanting, with the intensional verb 'want' as the function w(h, λx...), taking Hillary (h) as the subject and a property λx representing the existence of two candidates that Hillary found and supported as its argument. The second proposition represents the successful finding and supporting of two candidates, with the quantifier 'two' applied to the candidates, and for each candidate x and y, the extensional verbs 'found' and 'supported' are applied.
This representation treats the intensional verb 'wanted' separately from the extensional verbs 'found' and 'supported', expressing the conjunction of two propositions: one for the wanting, and the second for the successful finding and supporting of two candidates.
Hi again, and this, what got Ken started, is of course also extremely interesting. Though I am not really sure what to say about it. In one way, it’s impressive. In another, it’s just a deterministic translation of one thing into another. As it is, you go through each example and have to check it and correct it if necessary. Might be saving work, but might it not also be just as simple (for these simple examples) to write a few deterministic rules for which you then don’t have to check every single example again? Did you try more complex things? But the self-correction is impressive — I’ve been thinking of playing to see how well these language models deal with ambiguity (they try to avoid it) and whether one could force them to reach back into themselves to self-correct a wrongly resolved ambiguity. This is what might be going on in the example that Vijay posted as well? Cheers, Miriam
On 28. Apr 2023, at 10:58, Vijay Saraswat <vijay@saraswat.org> wrote:
Try 2...
Begin forwarded message:
From: Vijay Saraswat <vijay@saraswat.org> Subject: Using GPT4 as an assistant for computational linguists Date: April 24, 2023 at 6:17:08 AM EDT To: LFG@listserv.linguistlist.org, hpsg-l@listserv.linguistlist.org, Mary Dalrymple <mary.dalrymple@ling-phil.ox.ac.uk>, Ken Kahn <toontalk@googlemail.com>, fernando.pereira@gmail.com, lamping@gmail.com
I thought this group might find the following interesting.
For the last several years I have been leading an R&D group in AI (essentially applied NLP) at a major Wall Street Bank. Since 2020, I have been struck by the stunning progress made by Large Language Models (LLMs; GPT-3, ChatGPT, GPT-4, …). Essentially it seems that a lot of the work we were doing in computational linguistics (at least some of us), particularly in terms of applications to real-world question answering problems, has now been swamped by LLM advances.
I have a separate talk on the kind of highly-specific text analyses GPT-4 can do directly for (financial) professionals with domain-specific text, without having to go through parsing / semantic representation steps. Here I thought it might be illuminating to provide a log of a conversation with GPT-4 about representing meanings of simple sentences in IL.
As you can see it doesn’t always get things right. But some discussion helps it set things right (in itself, this is another startling capability of GPT-4). Such conversations can lead to the development of an accurate labeled data set, and thence to a fine-tuned end-point which should end up being highly accurate. (GPT-4 is not yet available to be fine-tuned, but GPT-3 is, and I would hope -4 will be, soon…)
There are tons of things to discuss here, of course, if there is interest. Just putting this out as a conversation starter…
Best, Vijay
The examples are taken from https://aclanthology.org/E93-1013.pdf and https://aclanthology.org/E95-1005.pdf
VS: Parse the following sentence and generate its m meaning in Montague's Intensional Logic. -- Sentence Bill kissed Hillary. -- Meaning formula
GPT4: To represent the given sentence "Bill kissed Hillary" in Montague's Intensional Logic, we will first parse the sentence into its components: • Subject: Bill • Verb: kissed • Object: Hillary Montague's Intensional Logic uses lambda calculus to represent meanings. We will use the following variables:
• b for Bill • k for the verb 'kiss' • h for Hillary Now, we can represent the meaning formula as follows:
k(b, h)
This formula states that the kissing action (k) occurs with Bill (b) as the subject and Hillary (h) as the object. The given sentence can be translated into Montague's Intensional Logic as the application of the binary relation 'k' to the individuals 'b' and 'h’. VS: Good. Here is another one. -- Sentence Bill obviously kissed Hillary.
GPT4: To represent the given sentence "Bill obviously kissed Hillary" in Montague's Intensional Logic, we need to consider the adverb "obviously" as a modifier of the verb "kissed". Let's denote the adverb with the letter 'o' and use the previously defined variables: • b for Bill • k for the verb 'kiss' • h for Hillary • o for the adverb 'obviously' Now, we can represent the meaning formula as follows:
o(k)(b, h)
This formula states that the kissing action (k) occurs with Bill (b) as the subject and Hillary (h) as the object, and that this action is obviously true (o). The given sentence can be translated into Montague's Intensional Logic as the application of the binary relation 'k' to the individuals 'b' and 'h', modified by the adverb 'o’. VS: Hmm.. shouldn't this be o(k(b,h))? GPT4: Apologies for the confusion. You are correct. The correct representation of the sentence "Bill obviously kissed Hillary" in Montague's Intensional Logic should be: o(k(b, h))
This formula states that the kissing action (k) occurs with Bill (b) as the subject and Hillary (h) as the object, and that this action is obviously true (o). The given sentence can be translated into Montague's Intensional Logic as the adverb 'o' modifying the result of applying the binary relation 'k' to the individuals 'b' and 'h’. VS: Good. Now let's do: -- Sentence Hillary let Bill write a letter -- Meaning formula
GPT4: To represent the given sentence "Hillary let Bill write a letter" in Montague's Intensional Logic, we will first parse the sentence into its components: • Subject: Hillary • Verb: let • Object: Bill • Infinitive verb: write • Object of the infinitive verb: a letter We will use the following variables:
• h for Hillary • l for the verb 'let' • b for Bill • w for the verb 'write' • a for a letter Now, we can represent the meaning formula as follows:
l(h, λx.w(x, a))(b)
This formula states that the letting action (l) occurs with Hillary (h) as the subject and a function representing Bill writing a letter (λx.w(x, a)) as the object. Then, the function is applied to the individual Bill (b).
The given sentence can be translated into Montague's Intensional Logic as the application of the ternary relation 'l' to the individuals 'h' and 'b', and the action of writing a letter (w) with 'a' as the object. VS: Good. Next: -- Sentence Bill supported, and Hillary opposed, NAFTA.
GPT4: To represent the given sentence "Bill supported, and Hillary opposed, NAFTA" in Montague's Intensional Logic, we need to parse the sentence into its components and consider the conjunction "and" that connects two separate actions: • Subject 1: Bill • Verb 1: supported • Subject 2: Hillary • Verb 2: opposed • Object: NAFTA We will use the following variables:
• b for Bill • s for the verb 'support' • h for Hillary • o for the verb 'oppose' • n for NAFTA • ∧ for the conjunction 'and' Now, we can represent the meaning formula as follows:
(s(b, n) ∧ o(h, n))
This formula states that Bill (b) supports NAFTA (n), and Hillary (h) opposes NAFTA (n). The given sentence can be translated into Montague's Intensional Logic as two separate actions, the support action (s) with Bill (b) as the subject and NAFTA (n) as the object, and the oppose action (o) with Hillary (h) as the subject and NAFTA (n) as the object, connected by the conjunction 'and' (∧). VS: OK, next: -- Sentence Bill supported, and Hillary opposed, two trade bills.
GPT4: To represent the given sentence "Bill supported, and Hillary opposed, two trade bills" in Montague's Intensional Logic, we need to parse the sentence into its components and consider the conjunction "and" that connects two separate actions: • Subject 1: Bill • Verb 1: supported • Subject 2: Hillary • Verb 2: opposed • Quantifier: two • Object: trade bills We will use the following variables:
• b for Bill • s for the verb 'support' • h for Hillary • o for the verb 'oppose' • tb for trade bills • ∧ for the conjunction 'and' • t for two Since we are dealing with a quantifier, we will use lambda notation to represent the two trade bills. Let x represent trade bills and y for the second trade bill:
Now, we can represent the meaning formula as follows:
∃x∃y(tb(x) ∧ tb(y) ∧ t(x, y) ∧ s(b, x) ∧ s(b, y) ∧ o(h, x) ∧ o(h, y))
This formula states that there exist two trade bills (x and y) such that Bill (b) supports both (s(b, x) and s(b, y)), and Hillary (h) opposes both (o(h, x) and o(h, y)). The given sentence can be translated into Montague's Intensional Logic as two separate actions, the support action (s) with Bill (b) as the subject and the two trade bills (x and y) as the objects, and the oppose action (o) with Hillary (h) as the subject and the two trade bills (x and y) as the objects, connected by the conjunction 'and' (∧). VS: Next: -- Sentence Hillary wanted, found, and supported two candidates.
GPT4: To represent the given sentence "Hillary wanted, found, and supported two candidates" in Montague's Intensional Logic, we need to parse the sentence into its components and consider the conjunctions "and" that connect three separate actions: • Subject: Hillary • Verb 1: wanted • Verb 2: found • Verb 3: supported • Quantifier: two • Object: candidates We will use the following variables:
• h for Hillary • w for the verb 'want' • f for the verb 'find' • s for the verb 'support' • c for candidates • ∧ for the conjunction 'and' • t for two Since we are dealing with a quantifier, we will use lambda notation to represent the two candidates. Let x represent the first candidate and y for the second candidate:
Now, we can represent the meaning formula as follows:
∃x∃y(c(x) ∧ c(y) ∧ t(x, y) ∧ w(h, x) ∧ w(h, y) ∧ f(h, x) ∧ f(h, y) ∧ s(h, x) ∧ s(h, y))
This formula states that there exist two candidates (x and y) such that Hillary (h) wanted both (w(h, x) and w(h, y)), found both (f(h, x) and f(h, y)), and supported both (s(h, x) and s(h, y)). The given sentence can be translated into Montague's Intensional Logic as three separate actions, the want action (w), find action (f), and support action (s) with Hillary (h) as the subject and the two candidates (x and y) as the objects, connected by the conjunctions 'and' (∧). VS: It would seem to me that the meaning should be the conjunction of two propositions, one expressing the wanting, and the second that the finding-and-supporting was successfully accomplished… GPT4: I apologize for my previous responses, and I understand now what you are looking for. To represent the given sentence "Hillary wanted, found, and supported two candidates" in Montague's Intensional Logic as the conjunction of two propositions, we can do the following: • Subject: Hillary • Intensional verb: wanted • Extensional verb 1: found • Extensional verb 2: supported • Quantifier: two • Object: candidates We will use the following variables:
• h for Hillary • w for the verb 'want' • f for the verb 'find' • s for the verb 'support' • c for candidates • t for two • ∧ for the conjunction 'and' Now, we can represent the meaning formula as follows:
w(h, λx.t(c, λy.(f(h, y) ∧ s(h, y)), x)) ∧ ∃x∃y(c(x) ∧ c(y) ∧ t(x, y) ∧ f(h, x) ∧ f(h, y) ∧ s(h, x) ∧ s(h, y))
In this formula, we have two separate propositions joined by a conjunction. The first proposition represents the wanting, with the intensional verb 'want' as the function w(h, λx...), taking Hillary (h) as the subject and a property λx representing the existence of two candidates that Hillary found and supported as its argument. The second proposition represents the successful finding and supporting of two candidates, with the quantifier 'two' applied to the candidates, and for each candidate x and y, the extensional verbs 'found' and 'supported' are applied.
This representation treats the intensional verb 'wanted' separately from the extensional verbs 'found' and 'supported', expressing the conjunction of two propositions: one for the wanting, and the second for the successful finding and supporting of two candidates.
**************************************************************** Miriam Butt Department of Linguistics University of Konstanz Fach 184 Tel: +49 7531 88 5109 78457 Konstanz Fax: +49 7531 88 4865 Germany +49 7531 88 5115 miriam.butt@uni-konstanz.de https://www.ling.uni-konstanz.de/butt/ 'I would suppose, sir,' he said cautiously, 'that Modesty Blaise might be a person awfully difficult for us---er---actually to get.' He blinked towards the big, grey-haired main who stood by the window, looking down at the night traffic, hurrying along Whitehall. 'For a moment,' Tarrant said, turning from the window, 'I hoped you might split that infinitive, Fraser.’ Peter O'Donnell, Modesty Blaise (p. 1) ****************************************************************
participants (2)
-
Miriam Butt -
Vijay Saraswat