With all the recent revolutions of technology, Artificial intelligence has become the most important topic around the world. Artificial intelligence is the technology where enabling the ability for machines to mimic human natural intelligence.
There are several subfields of Artificial intelligence that are being developed in real time. one of these major developments is Large Language Models.
what are Large Language models:
LLMs are artificially designed systems that are built to train on large amounts of data to understand and generate natural human language or other types of content to automate various tasks. The most popular use case of LLMs is Open AI ‘s Chat-GPT.
LLMs are a class of foundation models, which are trained on enormous amounts of data to provide the foundational capabilities needed to drive multiple use cases and applications, as well as resolve a multitude of tasks. This is in stark contrast to the idea of building and training domain specific models for each of these use cases individually, which is prohibitive under many criteria (most importantly cost and infrastructure), stifles synergies and can even lead to inferior performance.
In simple terms, A LLM is a language model that is known for its substantial scale, enabling the integration of billions of parameters to build intricate artificial neural networks. These networks harness the potential of advanced AI algorithms, employing deep learning methodologies and drawing insights from extensive datasets for the tasks of assessment, normalization, content generation, and precise prediction.
History of modern LLMs dates back to cold war era. In 1966, the MIT introduced a language model called ELIZA. The developments of modern LLMs started with the introduction of deep learning and neural networks.
Modern LLMs are working as back bones of Natural Language Processing (NLP). They empower users to input queries in natural language, prompting the generation of coherent and relevant responses.
There are three different LLMs in the world.
Autoregressive models:
predict the next word in a sentence based on previous words, making them ideal for tasks like text generation.
Autoencoding models:
focus on encoding and reconstructing text, excelling in tasks like sentiment analysis and information retrieval.
Hybrid LLMs:
combine the strengths of both approaches, offering versatile solutions for complex applications.
imagine you are studying for the a exam on topic and have to read many books to understand and gather information that are on text format on that topic into your brain. The LLMs work in similar way. So, they can be versatile for many modern applications.
Building your own Large Language Model:
before starting to build a LLM we need to understand that there is a Hugh need for vast amount of data to consume for generating content. For example Open AI’s Chat-GPT model was trained by more than 6 million tokens.
Tokens:
Tokens are like cells in our body. Tokens are words, character sets, or combinations of words and punctuation that are generated by large language models (LLMs) when they decompose text. Tokenization is the first step in training.
Tokenization:
Tokenization is the process of breaking text into smaller pieces, like words or characters, allowing the model to effectively process and understand each part.
Where we can find input data for training:
- internet/web-based data
- books/e-books
- codebases
- academic texts
- archive documents
- released PDFs
- Public APIs
Data pre-processing:
data pre-processing the process cleaning and re-organizing data structures before they are being used for training purposes. For this we use previously mentioned tokenization.
tokenization can be achieved in many stages. this is the first step of building a modern LLM.
we can tokenize letters/characters and call each character as a token. this is called Character Tokenization.
We can divide words into sub words and make them tokens. This called sub-word tokenization.
We can also tokenize words and make them tokens. This is called word-level tokenization.
Embedding:
Embedding is a process of converting customer reviews into numerical vectors that capture sentiment and meaning, helping the model analyze feedback and improve recommendations.
Attention:
Attention is the stage which focusing on the most important parts of a sentence, ensuring the model accurately grasps key sentiments, such as distinguishing between product quality and service issues.
The training process is a loop. This training loop is contained with following stages. the first stage is Data pre-processing.
Data ingestion: this is the process of collecting, gathering and loading data from various sources.
Data cleaning: This is the major part of data pre-processing. This is the the stage where Removing noise, handle missing data, and redact sensitive information.
Normalization: After cleaning data, then we should standardize text, handle categorical data, and ensure data consistency.
Chunking: Then the data should be Split large texts into manageable chunks while preserving context.
Tokenization: Then the the tokenization process Converts text chunks into tokens for model processing.
Load data: Finally after pre-processing, Efficiently load and shuffle data for optimized training, using parallel loading when necessary.
Then the second stage of training loop starts. This is loss calculation.
Calculate Loss: this is the process of Comparing predictions to true labels using a loss function, converting the difference into a “loss” or “error” value.
Performance Indicator: this is where Higher loss indicates poor accuracy; lower loss suggests better alignment with actual targets.
The third stage of training loop is the hyperparameter tuning.
Learning Rate: Controls weight update size during training — too high may cause instability; too low slows down training.
Batch Size: Number of samples per iteration — larger batches stabilize training but require more memory; smaller batches introduce variability but are less resource-intensive.
The next stage is parallelization and resource management.
Data Parallelization: Split datasets across multiple GPUs for faster processing.
Model Parallelization: Divide the model across GPUs to handle large models.
Gradient Checkpointing: Reduce memory usage during training by selectively storing intermediate results.
The final stage of training loop is Iterations and Epochs.
Iterations: Process batches of data, updating weights each time.
Epochs: Complete passes through the dataset, refining the model’s parameters with each pass.
Monitoring: Track metrics like loss and accuracy after each epoch to guide adjustments and prevent overfitting.
How to evaluate the LLM:
The evaluation of LLM performance after training them is one of the essential tasks that is required for building accurate LLM models with better performances.
There are several standards for evaluating including:
- MMLU (Massive Multitask Language Understanding): Assesses natural language understanding and reasoning across a wide range of subjects.
- GPQA (General Purpose Question Answering): Tests the model’s ability to handle diverse, complex questions across domains.
- MATH: Measures the model’s mathematical reasoning by solving multi-step problems.
- HumanEval: Evaluates coding proficiency by assessing the model’s ability to generate accurate, functional code.
When fine-tuning LLMs for specific tasks, metrics should align with the application’s objectives. For example, in a medical setting, accuracy in matching disease descriptions with codes could be prioritized.
Final Notes:
- LLMs are one of the most demanded fields in modern AI revolution.
- Building a LLM is a very complex task with several tasks and procedures.
- For training a LLM, there is a Hugh need of vast amount of input data.
In this article, we have discussed about the basic procedures of building a LLM. During the next article, we are going to talk more about building a LLM with examples.