Building LLM Apps with Open AI Alternatives using LiteLLM

During the course of this bootcamp, many learners highlighted that OpenAI has now stopped giving free credits. As an easy option you can experiement by purchasing $5 credits. If that doesn't work, you don't need to worry as this sub-module helps you with options.

A great option to explore is the use of open source models or go for APIs offered by providers like Cohere, Replicate, Google, etc. This guide will show you how to integrate the LiteLLM connector with the Replicate API, providing a smooth transition.

Step 1: Update and Install LiteLLM

First, let's ensure you have the latest version of LiteLLM, which will help you access the most up-to-date features and improve compatibility:

# This command updates LiteLLM to the latest version.
!pip install -U litellm

# Alternatively, you can specify a minimum version like this: 
!pip install litellm>=1.37.20

Step 2: Obtain and Set your API Key

• Obtaining your API key: Visit the Replicate website to find your API key. It's typically located in the right sidebar on the documentation pages.

• Setting your API key in Python:

import os 
os.environ["REPLICATE_API_KEY"] = "<YOUR_API_KEY>" 
# As a best practice, consider adding your API key to a `.env` file to keep it secure.
# Don't forget to load the env file with the `load_dotenv`!

Step 3: Implementing LiteLLM in Python

Here's how you can set up LiteLLM for a simple chat application using Pathway. This example uses the Llama 3 model provided by Replicate:

import pathway as pw 
from pathway.xpacks.llm.llms import LiteLLMChat, prompt_chat_single_qa 

# Initialize the LLM chat with the chosen model 
chat = LiteLLMChat(model="replicate/meta/meta-llama-3-8b")  # Example model 

# Prepare a query
query_table = pw.debug.table_from_markdown(''' 
txt 
Wazzup? 
''') 

# Generate a response from the LLM 
response = query_table.select(response=chat(prompt_chat_single_qa(query_table.query))) 

# Output the results to see the conversation in action 
print(response)

Integrating with Existing Applications

If you're already using OpenAI LLMs in your apps, switching to LiteLLM is straightforward. Simply replace the model initialization line in your existing code:

# Replace this for OpenAI: 
chat = OpenAIChat(model="gpt-4") 

# With this for Replicate:
chat = LiteLLMChat(model="replicate/meta/meta-llama-3-8b")

This change involves only a few lines, making it an efficient way to adapt to new LLM providers without overhauling your existing applications. If you have any questions or need further assistance, feel free to reach out via #iit-delhi-bootcamp channel on Pathway Discord server.