Generating text
Recently, text generation models made a major impact when they came into the public consciousness with OpenAI’s success with ChatGPT in 2022. However, text generation was among the first uses of AI. Eliza was the first chatbot ever developed, back in 1966, before all but the most technically inclined people had even seen a computer themselves. The personal computer wouldn’t even be invented for another 5 years, in 1971. However, it’s only recently that truly impressive chatbots have been developed.
Recent developments
A type of model called transformers is responsible for the recent burst in language models. Transformers are neural networks that are comprised entirely of a layer called an attention layer. Attention layers work sort of like a spotlight, focusing on the part of the data that is most likely to be important. This lets transformers (and other models that use attention layers) be a lot deeper without losing “focus”...
