glad so awkwardly worded tweet was redeemed by a good discussion still i shouldn't post half-asleep in the middle of the night while trying to calm down a shaking dog scared of severe thunderstorms
Can GPT-3 invent new words that make sense? If not I’m not interested
2
0
0
7
in reality what i was trying to ask if GPT-3 does a word playing, punning etc
2
0
0
2
Tl;dr Word tokenisation is the main obstacle, as GPT-3 is aware of puns, but can't necessarily tell which words are actually similar due to not seeing characters.
1
0
0
0
Replying to @NineOfNein @gwern
One thing I've noticed is that when generating other alphabets such as Russian, the tokens are generally one character in size I think. Haven't tried it yet, but I wonder if puns and rhyme would actually work better in Russian and other languages.

7:02 PM · Jul 24, 2020

2
0
0
2
Replying to @NineOfNein @gwern
On the other hand strength of the model definitely not as good outside of English.
0
0
0
0
You’re unable to view this Tweet because this account owner limits who can view their Tweets.
Yes I was a bit overoptimistic. Tokens are definitely shorter than English, but the performance even worse. Getting it to explain its thinking, it clearly can't tell at all which sentences/words sound the same, which is odd, since homonyms tend to have the same letters in Russian
1
0
0
2