Project Milestone 1

Goal

Develop an initial prototype of an LLM inference API + chat interface.

Deliverables

1. A word, google doc, or pdf file write up (likely 1-2 pages) summarizing progress and questions
2. A GitHub repo with your project code

Only 1 member of the team needs to submit for the group.

Requirements

1. Repository Setup

• Create a new repo in the course's GitHub organization
• One repo per group
• Use the course's modal Llama 3 8B Instruct API inference repo as a starting point

2. For LLMOps/AI App Developers

• Explore the api.py file
• Review Modal's documentation
• Understand basic Modal conventions and Python patterns

3. For Product Manager & Data/Prompt Engineer

• Examine the client.py file
• Study the provided notebook for API usage
• Test different prompts and API interactions

4. Write-up Requirements (1-2 pages)

1. Document successful replication of Llama3 inference repo and/or Streamlit app
2. Provide examples of tested prompts:
   • At least 1 successful output
   • At least 1 case where the model underperformed
3. List open questions and challenges with the code/Modal
4. Outline 2-3 project ideas for next milestone

Extra Credit

Integrate LangFuse into your endpoint for logging traces of all interactions.

Submission

• Due: Sept 12
• Points: 2
• Submission Types: Website URL or file upload (docx/pdf)