Show HN: We made glhf.chat – run almost any open-source LLM, including 405B

Try it out! https://glhf.chat/ Hey HN! We’ve been working for the past few months on a website to let you easily run (almost) any open-source LLM on autoscaling GPU clusters. It’s free for now while we figure out how to price it, but we expect to be cheaper than mo

reissbaker · 161

Show HN: Mutable.ai Codebase chat that uses a Wiki for RAG

Hi HN! We wanted to introduce our codebase chat that uses a wiki of your repo to do retrieval instead of traditional vector or keyword RAG. We have been pleasantly surprised by the quality of responses. We think creating a language knowledge base for LLMs to do RAG is the future.

oshams · 48

Ask HN: Are you training and running custom LLMs and how are you doing it?

I have been researching methods and projects built around training and running LLMs locally. I'm interested in what others have been using on this front (including straight up Pytorch/Transformers). Here's what I've gathered so far: Engines/APIs: - vllm:

kordlessagain · 15

vLLM v0.28.0

mrrrcs · 108

vLLM introduces memory optimizations for long-context inference

addisud · 5

vLLM IR: A Functional Intermediate Representation for vLLM

matt_d · 4

vLLM 0.29.0: Model Runner V2 is now the default

joshcsimmons · 3

Vllm: High-throughput and memory-efficient inference and serving engine for LLMs

tosh · 3