Syllabus

Module 1 - Basic RAG with Langchain and LlamaIndex

This module covers the basic concepts of Langchain and Llamaindex and prepares you to build a basic RAG application with both frameworks. This is recap on Langchain concepts covered in the earlier LangChain & Vector Databases in Production course, together with a brief introduction to the Llamaindex framework. It also contains a summary of the different strengths and focus of each framework. This course will be focussed on advanced RAG topics so we recommend taking our earlier course and reading examples from LangChain and Llamaindex documentation to complement this module.

Lesson 1 - LangChain: Basic Concepts Recap

In this lesson, students will recap on the functionalities and components of LangChain, a framework designed to work with generative AI and language model-based (LLM) applications. This material was covered in depth with code and project examples in our earlier course LangChain & Vector Databases in Production. Students will be introduced to preprocessing techniques like document loading and chunking, understanding the indexing of document segments and embedding models, as well as the structure and functionality of chains, memory modules, and vector stores. Additionally, students will gain insight into working with chat models, LLMs, embedding models, and constructing sequential chains to automate complex interactions.

Lesson 2 - LlamaIndex Introduction: Precision and Simplicity in Information Retrieval

In this lesson, students will learn about the LlamaIndex framework, designed to enhance the capabilities of Large Language Models (LLMs) by integrating them with Retrieval-Augmented Generation (RAG) systems. The framework allows LLM-based applications to fetch accurate and relevant information using vector stores, connectors, nodes, and index types for better-informed responses. The lesson covers vector stores and their importance in semantic search, the role of data connectors and LlamaHub in data ingestion, the creation of Node objects from documents, and the indexing of data for quick retrieval. Students will also learn about the practical construction and usage of query engines, routers, and the distinction between saving indexes locally and on the cloud. Finally, it compares LlamaIndex with the LangChain framework, and concludes by discussing the practical application and effectiveness of LlamaIndex in LLM applications.

Module 2 - Advanced RAG

The "Advanced Retrieval Augmented Generation" module offers an in-depth exploration into optimizing large language models (LLMs) with advanced Retrieval-Augmented Generation (RAG) techniques. Across four lessons, it encompasses a range of topics including query transformation, re-ranking, optimization techniques like fine-tuning, the implementation of Activeloop's Deep Memory, and other advanced strategies using LlamaIndex. Students will gain practical experience in enhancing RAG system performance, from query refinement to production deployment and iterative optimization. The module is designed to provide a comprehensive understanding of building, refining, and deploying efficient RAG systems, integrating hands-on examples with theoretical knowledge to prepare students for real-world applications.

Fine-tuning vs RAG; Introduction to Activeloop’s Deep Memory;

In this lesson, students will explore various optimization techniques to maximize the performance of large language models (LLMs), such as prompt engineering, fine-tuning, and retrieval-augmented generation (RAG). The lesson begins with identifying the benefits and challenges of each method. It further examines the limitations of RAG systems and introduces Activeloop's Deep Memory as a solution to these challenges, particularly in improving retrieval precision for user queries. Students will see a step-by-step guide on how to implement Deep Memory in experimental workflows, including creating a synthetic training dataset and running inference with the trained Deep Memory model. A significant portion of the lesson is dedicated to hands-on examples using code to demonstrate the increased recall rates when Deep Memory is applied in a RAG system. The lesson concludes with a comparison of empirical data, highlighting the advantages of Deep Memory over traditional methods and emphasizing its role in advancing the capabilities of LLMs.

Mastering Advanced RAG Techniques with LlamaIndex

In this lesson, students will learn about the advanced techniques and strategies that enhance the performance of Retrieval-Augmented Generation (RAG) systems, using LlamaIndex as a framework. They will explore the concepts of query construction, query expansion, and query transformation to refine the information retrieval process. Students will also be introduced to advanced strategies like reranking with Cohere Reranker, recursive retrieval, and small-to-big retrieval to further improve the quality and relevance of search results. The lesson includes hands-on examples of setting up a query engine from indexing to querying, as well as creating custom retrievers and utilizing reranking. The conclusion underlines the importance of these techniques and strategies in developing more efficient RAG-based applications.

Production-Ready RAG Solutions with LlamaIndex

In this lesson, students will learn about the challenges, optimization strategies, and best practices for Retrieval-Augmented Generation (RAG) systems in production. The discussion includes dealing with dynamic data management, diverse representation in latent space, regulatory compliance, and model selections for system efficiency. The lesson emphasizes the importance of fine-tuning both the embedding models and the Language Large Models (LLMs) to improve retrieval metrics and generate more accurate responses. Additionally, students will explore the role of Intel® technologies in optimizing neural network models on CPUs, and they will acquire knowledge on utilizing generative feedback loops, hybrid searches, and the continuous evaluation of RAG system performance. Practical use cases, data management tools, and integration of metadata in retrieval steps are also highlighted, with LlamaIndex being presented as a comprehensive framework for building data-driven LLM applications.

Iterative Optimization of LlamaIndex RAG Pipeline: A Step-by-Step Approach

In this lesson, you will learn the process of iteratively optimizing a LlamaIndex Retrieval-Augmented Generation (RAG) pipeline to enhance its performance in information retrieval and generating relevant answers. The lesson guides you through establishing a baseline pipeline, experimenting with retrieval values and embedding models like "text-embedding-ada-002" and "cohere/embed-english-v3.0," and incorporating techniques like reranking and deep memory to refine document selection. Additionally, you will learn about performance metrics, such as Hit Rate and Mean Reciprocal Rank (MRR), and evaluate faithfulness and relevancy of answers using GPT-4 as a judge. The lesson provides hands-on code examples for each optimization step and concludes with the overall enhancement observed in the RAG pipeline's accuracy.

Module 3 - RAG Agents

The "Retrieval Augmented Generation Agents" module offers a comprehensive exploration into the creation and enhancement of AI agents, with a focus on integration and application in various fields. It begins by introducing LangChain, where students learn about agents, tools, and the initiation of OpenGPTs, gaining practical insights into setting up and customizing AI assistants. Then, it delves into the LlamaIndex framework, teaching students how to build efficient RAG systems by integrating OpenAI agents with various data sources and creating custom functions for enhanced decision-making. The module also covers the use of the OpenAI Assistants API and Hugging Face Inference API.

LangChain Overview: Agents, Tools, and OpenGPT Introduction 

In this lesson, students will learn about the fundamental concepts of LangChain, focusing on agents, tools, and the initiation of OpenGPTs. They will examine how agents integrate chains, prompts, memory, and tools to execute tasks and understand the different types of agents, such as Zero-shot ReAct and Conversational Agent, designed for various scenarios. The lesson also covers the available tools and how to customize them for specific needs, benefiting from functionalities like Python tool, JSON tool, and CSV tool. Additionally, students will get practical insights into setting up and creating LangChain OpenGPTs through cloning the repository and customizing prompts, providing a comprehensive understanding of how to configure and use AI assistants similarly to OpenAI GPTs for tailored interactions.

Utilizing AI Agents with the LlamaIndex Framework for Enhanced Decision-Making

In this lesson, students will learn how to leverage agents within the LlamaIndex framework to build a more efficient and insightful RAG (Retrieval-Augmented Generation) system. They will gain insights into integrating OpenAI agents with various data sources and create custom functions to enhance the agent's capabilities in areas such as mathematical operations. The lesson provides guidance on installing necessary packages, configuring API keys, defining data sources, employing query engines, and setting up agents. Students will also explore an interactive chat interface with the agent and the creation of a dataset, using custom functions as tools that the agent can invoke as required. Finally, students will gain exposure to LlamaHub for further expanding the functionalities of their agents.

Crafting AI Assistants via OpenAI and Hugging Face API

In this lesson, students will explore the capabilities of the OpenAI Assistants API, including the Code Interpreter, Knowledge Retrieval, and Function Calling features. The lesson offers a step-by-step guide for creating and configuring AI assistants integrating OpenAI's tools, revisiting fundamental concepts such as Threads, Messages, and Tools for individual interactions. Additionally, the lesson introduces other advanced models by OpenAI like Whisper, Dall-E 3, and GPT-4 Vision that can be valuable integrations for comprehensive AI product development. We also cover how to use the Hugging Face Inference API to leverage a broad spectrum of machine learning models for tasks like text summarization, sentiment analysis, and text-to-image generation. By the conclusion of the lesson, students will possess the necessary understanding to harness these tools for their own sophisticated AI projects.

Project; Multimodal Financial Document Analysis and Recall

In this lesson, students will learn how to use tools such as GPT-4 vision to enhance Retrieval-augmented Generation (RAG) for processing financial documents like Tesla's Q3 financial report PDF, involving the extraction of text, tables, and graphs, and transforming them into a query-able format using a vector database for efficient information retrieval by an AI chatbot. The lesson covers using tools such as Unstructured.io for text and table extraction, GPT-4V for graph information extraction, and the use of Deep Lake and LlamaIndex for storing and recalling the processed data to address user queries effectively. We also show how to use Deep Memory to enhance retrieval accuracy. The techniques detailed equip students to develop AI applications capable of analyzing and recalling complex multimodal data from financial documents.

Module 4 - Rag Evaluation and Observability

The module "Retrieval Augmented Generation Evaluation and Observability" provides a comprehensive exploration of advanced techniques and tools in the field of AI, focusing on enhancing chatbots and question-answering systems through Retrieval-Augmented Generation (RAG) systems. It explores the critical aspects of evaluating these systems, emphasizing the importance of faithfulness, relevance, and the prevention of hallucinations in AI responses. The module introduces tools like the FaithfulnessEvaluator and RAGAS, along with the Golden Context Dataset, offering insights into effective evaluation methodologies, including indexing, embedding, and generation metrics. Additionally, the module covers the LangChain framework and the LangSmith platform, providing practical knowledge on building and testing LLM-powered applications. Students will learn about the components of LangChain, such as Models, Vector Stores, and Chains, and the functionalities of the LangChain Hub.

RAG - Metrics & Evaluation

In this lesson, you will learn about Retrieval-Augmented Generation (RAG) systems and their evaluation metrics, with a focus on improving chatbots and question-answering systems. The lesson introduces you to different approaches to analysis, the importance of faithfulness and answer relevancy, nuances of indexing and embedding metrics, and generation metrics aimed at preventing hallucinations in AI responses. It discusses the significance of the FaithfulnessEvaluator tool for checking the alignment of AI responses with retrieved context and introduces RAGAS and the Golden Context Dataset for system evaluation. Additionally, real-world setups for assessing and improving RAG systems are explored through examples of community-based tools, including comprehensive evaluation of retrieval metrics, holistic approach evaluations, and the custom RAG pipeline evaluation.

LangSmith and LangChain Fundamentals for LLM Applications

In this lesson, you will learn about the fundamentals of the LangChain framework and the newly introduced LangSmith platform to build and test LLM-powered applications. We will review LangChain components, such as Models, Vector Stores, and Chains, as well as principles of the LangChain Hub, including prompt exploration and versioning for collaborative prompt development. The lesson will guide you through setting up the LangSmith environment, creating an API key, and basics of prompt versioning and tracing. You will also learn how to use LangServe to deploy applications as a REST API and go through the process of reading and processing data from a webpage, storing it into Deep Lake vector store, and using prompts from the LangChain Hub to build a QuestionAnswering Chain application.