Running models locally with ollama

Augmented Software Engineering Meetup

20-Mar-2025

Stage-Setting

  • Who knows Ollama? Is using it?
  • (One of these) Run/Manage models locally solutions
    • LM-Studio, vllm, …
  • I am using it, because there are (good enough) IDE integrations

Demo

  • Install, …, push/pull, list, run, …
  • Base, instruct, code, …, 3b, 7b, 32b, …, Q4-K-M, …
  • Configuration, …,
  • Test(s), …
  • VsCode, …

Take-a-ways

  • ollama is (just) one option
  • Use code(r)/instruct models
  • Use (up to) 7b (with Q4-K-M)
  • Configure it right!!! (FLASH-ATTENTION, KV-CACHE-TYPE)
  • (On my laptop) (Very) Limited stop-gap measure

Questions?