LLM Application Frameworks
The developer toolkits for building production LLM applications. From orchestration chains to observability platforms, these frameworks are the backbone of every AI product shipping today.
Core Orchestration Frameworks
| Name |
Description |
Links |
| LangChain |
The agent engineering platform. Framework for developing applications powered by LLMs with chaining, tools, memory, and retrieval. 133k+ stars. |
Docs |
| LangGraph |
Graph-based framework for building stateful, multi-actor LLM agents as directed graphs. Supports cycles, branching, persistent state, and human-in-the-loop. |
Docs |
| LlamaIndex |
Leading document agent and RAG platform. Connect custom data sources to LLMs with indexing, retrieval, and query interfaces. 48k+ stars. |
Docs |
| DSPy |
Stanford's paradigm-shifting framework for programming (not prompting) LLMs. Declarative modules with automatic prompt optimization. 20k+ stars. |
Paper, dspy.ai |
| Haystack |
Production-grade NLP/LLM framework for RAG pipelines, QA, and conversational AI. Composable pipeline architecture with 60+ integrations. |
Docs |
| Semantic Kernel |
Microsoft's enterprise SDK for LLM integration. Plugin/skill architecture powering Microsoft Copilot. Python, C#, Java. 23k+ stars. |
Docs |
| smolagents |
HuggingFace's minimalist agent library. Code-first agents that write and execute Python. ~1000 lines core. 14k+ stars. |
Docs |
| PydanticAI |
Agent framework with type safety, structured outputs via Pydantic, and dependency injection for testing. Async-first. |
Docs |
| Agno |
Full-stack agent framework (formerly Phidata). Lightning-fast runtime, multimodal agents, native teams, 100+ tools. 20k+ stars. |
Docs |
| Name |
Description |
Links |
| Dify |
Open-source LLM app development platform with visual workflow orchestration, RAG, and agent framework. 137k+ stars. |
dify.ai |
| Flowise |
Drag-and-drop no-code builder for LLM apps and chatbots using visual node graphs. 52k+ stars. |
flowiseai.com |
| Ollama |
The de-facto standard for running open-weight LLMs locally. Dead-simple CLI. OpenAI-compatible API. 110k+ stars. |
ollama.com |
| Open WebUI |
User-friendly AI interface supporting Ollama, OpenAI API, and more. Self-hosted ChatGPT alternative. 131k+ stars. |
openwebui.com |
| Jan.ai |
Open-source, privacy-first desktop AI assistant. Runs 100% offline on-device. OpenAI-compatible local API. 25k+ stars. |
jan.ai |
| LiteLLM |
Unified API for 100+ LLMs using OpenAI SDK format. Proxy server with load balancing, cost tracking, and key management. |
Docs |
| Name |
Description |
Links |
| Instructor |
Get structured, typed outputs from LLMs using Pydantic models. Handles retries, validation, and streaming. 10k+ stars. |
Docs |
| Outlines |
Structured text generation at the token level using FSM-guided decoding. Guarantees valid JSON/regex output. 11k+ stars. |
Docs |
| guidance |
Microsoft's language for constrained and structured LLM generation with token-level control. 20k+ stars. |
GitHub |
LLM Observability & Evaluation
| Name |
Description |
Links |
| LangFuse |
Open-source LLM engineering platform. Tracing, prompt management, evals, metrics. Self-hostable. 8k+ stars. |
langfuse.com |
| Phoenix |
Open-source LLM observability with OpenTelemetry-native tracing, retrieval evaluation, and experiment comparison. By Arize. |
Docs |
| Ragas |
Leading RAG evaluation framework. Metrics: faithfulness, relevancy, precision, recall. Synthetic test set generation. 8k+ stars. |
Docs |
| LangSmith |
LangChain's platform for debugging, testing, evaluating, and monitoring LLM chains and agents. |
Docs |
| Opik |
Open-source LLM observability and evaluation. Tracing, evaluation, and prompt optimization. |
Docs |
| Helicone |
LLM observability via one-line proxy. Cost tracking, caching, rate limiting. |
helicone.ai |