Most inferencing packages have their own REST API, but having an OpenAI compatible API is useful for using a variety of clients, or to be able to easily switch between providers.

Are there any with full support: assistants, function calling, chat completions?



  • General llama.cpp Python wrapper
  • Has a good enough OpenAI API server as well

  • Python
  • llama.cpp and exllama compatibility
  • on-demand model loading

  • Docker-focused
  • For HF models

  • Python
  • Uses various backends (CTransformers, llama-cpp-python, not well documented)

Some clients provide an OpenAI API compatibility layer:

  • Uses conda

  • Mac or Windows GUI App
  • But w/ an OpenAI API layer
  • Not open source but free to use