ChatGPT is an artificial intelligence large language model developed by OpenAI that can understand and generate human-like text across a wide range of topics. GPT is an acronym for Generative, Pre-Trained Transformer. The word, Generative, indicates that the model generates text. Pre-trained refers to the model being trained with data before being released. Transformer points to its function of simplifying input, transforming it into numbers and, after passing it through a neural network, creating a response.
If it seems like ChatGPT popped up overnight, it's because it has. Five years after its introduction, it has 100 million active monthly users.
Let's say we ask ChatGPT a question about our favorite sports team, "Tell me a fun fact about the Cleveland Browns in two sentences." It replies within seconds, "The Cleveland Browns' spirited fan tradition, the "Dawg Pound," originated in the 1980s when the team's defense was dubbed the "Dawgs." This lively section in the stadium, known for fans wearing dog masks, remains a symbol of energetic support for the Browns."
How did the model come up with that reply? It assessed the question and predicted what words, phrases and sentences are best associated with it. Then, it output words and sentences that should best answer the question.
If we ask the same question again, it responds, "The Cleveland Browns have a passionate fan section called the "Dawg Pound," known for its rowdy atmosphere and fans wearing dog masks. This tradition started in the 1980s when the team's defense earned the nickname "Dawgs." This response is slightly different but offers the same information.
Let's look at what's happening behind the curtain. ChatGPT first tries to determine what words would most likely be expected after comparing your input to the data set used to train it: about 10 billion web pages and volumes of books and other data.
ChatGPT needs all the data it can get. As of this writing, all it "knows" is limited by that huge data set, which goes up to January 2022. I wrote, "knows," because ChatGPT doesn't know anything. It can't think or reason. It's just really good at predicting the next word in a sentence.
If you ask it to complete, "The study of physics is…" It assesses the text and calculates what word might come next. It looks for matches in context and meaning. The result is that it looks at each next word's "probability" and produces a ranked list of words that could follow. When prompted, "The study of physics is…," it might choose the next word to be: "described," "a," "based," etc.
ChatGPT turns to an impressive word model to determine how to select an appropriate word. It's not a 3-D or a 4-D model but a 12,288-dimensional model that stores all the words it knows in a giant matrix. It contains words grouped according to how frequently they appeared together in the training data. For example, "computer" and "screen" might appear close together, like "car" and "driver." Those word groups, however, might be stored further away from each other in the matrix. So, each time ChatGPT adds a word to the sentence, it returns to its word model to find the next word until it's finished. Interestingly, it doesn't always choose a word with the highest probability. So, while "the" might be the most probable word to follow, "the study of physics is…" it might select "a."
This randomness in word selection rarely affects the accuracy of the response. It does, however, make ChatGPT appear to be more human. Because it has word options, the model's answers will be different almost every time the same question is asked.
As you may have guessed, the sentence completion model isn't enough. If we ask it something like, "Explain how physics works," it must use a different strategy, as there is no "next word" prediction scenario. To accomplish this requires training beyond knowing all the words and their relationships in context.
To answer questions like this, humans are hired to train the model and play the roles of both a user and a chatbot. These user-chatbot conversations are entered into the model, which helps it learn to maximize the probability of picking the correct sequence of words and sentences. Through this process, the model learns patterns about the context and the meaning of various inputs so that it can respond appropriately. This is another reason why an exchange with ChatGPT can feel like an exchange with a human.
The output from this stage is then finetuned in a second stage, where developers teach ChatGPT to assign a ranking, or weight, to each output. If they ask the model to "describe a belt," the potential answers might be: A) It's a continuous loop of a sturdy material used to transfer energy to a device from an engine or a motor. B) It's a strap wrapped around a person's waist to hold up pants. C) It's a vague measure of liquor. The trainer could rank this output from best to worst by telling the model that A is greater than B, which is greater than C.
This helps ChatGPT to evaluate what the best output might be. However, the problem with using human trainers for supervised learning is scale. Humans would have to anticipate all the inputs and outputs of any potential question for hundreds of years.
With ChatGPT, you can ask it to write a short story about almost anything, and it will usually offer up something that makes sense. So, how does it do that? This employs a type of unsupervised learning called "reinforcement learning." This trains it so no specific output is associated with any given input. Instead, the model is trained to learn the underlying context and patterns in the input data based on its earlier, human-taught pre-training.
In other words, the model uses pre-training to determine what it will output for the unsupervised training stage. Here, a model can process a large volume of data and learn the patterns from texts and sentences of an unlimited number of subjects. The best part is that it does it on its own.
ChatGPT-3.5 used a dataset of about 45 terabytes. This doesn't seem to be much by today's standards, where you can buy a terabyte flash drive for $20, but this is a massive amount of text. To put it into context, one terabyte is equal to 83 million pages of information.
This training helped ChatGPT learn patterns and relationships between words and phrases on a massive scale, such that it can produce relatively meaningful outputs for almost any question. As good as ChatGPT is now, the following versions will be even better.
One thing many companies don't consider when using ChatGPT is something called "Shadow AI." If your staff uses ChatGPT to generate letters or documents using any of your sensitive information, they are sending that information outside your organization, outside of your firewall.
EOX Vantage understands artificial intelligence and how to use it securely. If you are concerned about the security of your company or want to introduce AI innovation to your organization, contact us at info@eoxvantage.com. We're happy to discuss how AI can be used safely to increase your bottom line.
