History of GenAI

With the buzz around all things GenAI, it seems to have sprung into existence overnight. But in reality, it’s been around for a good amount of time. Neural networks date back to the seminal work "Parallel DIstributed Processing" by Rumelhart and McCLelland in 1987.

One of the earliest pioneers in AI is Geoffrey Hinton, known to many as the Godfather of AI. He spoke about deep learning models and back propagation as well as the work that was done in the original residual network. Back then people were thinking about machine learning as a much more imperative approach and less of a data driven approach. Hinton and his grad students created ResNet, a machine learning network for identifying images.

From there, we move from that deep learning into the space of recursive neural networks (RNNs). They have the problem of maintaining context and training as they get deeper, as well as the breadth of knowledge. As words got further away from each other the system lost the ability to associate them together. With this, you could take traditional convolutional neural networks and combine them into (RNNs) to get some semblance of understanding of language.

There then came the creation of Long Short-Term Memory which had little sliding windows over the set of words to try to maintain the association between words that were not right next to each other. But again, there was a problem with determining if you were getting the full context of the question that's being asked. With this, models looked at one token or one word (or very few words) at a time, not necessarily understanding the semantic meaning that a native speaker might infer on the sentence as a whole. All the stemming and the vector of those tokens was still happening in an imperative fashion.

This all starts to change as we get to models like Word2vec. This model would start to relate words like ‘King’ which is closely related to ‘Man’ and ‘Queen’ which is closely related to ‘Woman’, and would plot all of the semantic concepts that exist in all of these languages in a vector space and would create this gigantic vector space. 

Due to this, you have a means to tokenize your information across this multi dimensional space and it becomes addressable. Once it's addressable, you can do some training and back testing, but you will still have the problem of being able to create meaning for more than just one word. 

At the same time in a different field for sound, video and images there was this concept of generative adversarial networks (GANs). GANs were generators whose job was to fool the adversary, so it trained both of them to go against each other. In that fashion of training the generator, it trained the adversary to detect fakes, ultimately making a much better kind of network that was able to create this. This type of training echoes a technique long used in the animal world. One bird, for example, who has learned a task that results in obtaining a treat is paired with a novice bird. The new bird is competing for the treat and learns the task faster than it would without a competitor for the treat.

In 2017, the concept of Transformer architecture was introduced, which is a form of weight expansion and weight compression. This asks ‘who’ ‘what’ ‘when’ and ‘where’ questions of the input tokens. For example, if you put in the sentence ‘Mary was 15 years old when she started 10th grade’, all of those tokens get put together and the transformer, in order to expand its knowledge in real time, is taking each of those tokens ‘Mary’ ‘10’ ‘years.’ It is then asking ‘who’ ‘what’ ‘when’ and ‘where’ relationship questions and is expanding this topology. 

This type of model is not an imperative model, but emerges as you continue to run this learning. In terms of large language models, the transformer architecture really advanced the state of the art.

The transformer architecture is the basis for a lot of things. But many failed to realize early on that these generative pretrained transformers like feeding everything forward and then back propagating. This was not thought to be a viable approach because these models actually converge the learning rate, so it is lower at first. If you look at the learning rate of a traditional convolutional neural network or RNN, the curve continues increasing. Whereas with the transformer models, you must train the model for a longer amount of time before you will see any movement.

What brought about this Generative AI revolution that we are seeing now is the interface. The transformer architecture eventually became the generative pretrained architecture until we came to Chat GPT when the interface became available to the public. With this availability, there's almost an exponential curve in terms of adoption, and in terms of the excitement about the abilities of these models.

How Caylent Can Help

Are you exploring ways to take advantage of Analytical or Generative AI in your organization? Partnered with AWS, Caylent's data engineers have been implementing AI solutions extensively and are also helping businesses develop AI strategies that will generate real ROI. For some examples, take a look at our Generative AI offerings.