Day 1 Running Ollama Locally

This is the first post in my series “Building My Personal AI Assistant - Dave”, where I document how I’m putting together a local AI interface that integrates into my desktop and hopefully mobile daily life.

The idea is simple: I want my own personal AI assistant that can help with search, automation, and daily tasks. Not a cloud service or relying on an API, because I don’t particularly trust these companies but instead something that runs entirely on my hardware, with full control over the models and data.

It’ll have to start of small with my limited hardware but if it works I can’t see why I can’t expand this with various hardware upgrades or even my own encrypted cloud server.

Why I’m Doing This

My initial trello note for this project was basically:

“I want a personal desktop AI interface. Ollama should be the backend server. LM Studio should be my general chatbot UI. A Flutter app will manage settings that integrate with Ubuntu. And all my AI models should live in one shared drive so I can manage them together.”

That last point is important. I also use ComfyUI for image workflows, and I don’t want models scattered across different hidden folders. Instead, I have an existing shared folder (Windows & Linux access) that I’ve added all my AI models (text + image) to, so I can easily back them up, swap machines, or share them.

For me, it’s as follows:

/media/shared/ai/models/

  • llms/ → large language models (Ollama, LM Studio, etc.)
  • checkpoints/
  • diffusion_models/
  • loras
  • (and so on…)

This way, everything is in one predictable place.

Installing Ollama

Ollama is a lightweight server for running large language models locally. It exposes an OpenAI-compatible API on port 11434, which makes it easy to integrate with tools like LM Studio, custom apps, or scripts.

Install:

# install ollama (Linux)
curl -fsSL https://ollama.com/install.sh | sh

And then obvioulsy we need to get it running.


# enable the server on startup
sudo systemctl enable --now ollama

Redirecting Models to a Shared Drive

By default, Ollama stores models under /var/lib/ollama. I redirected mine to my shared drive:

sudo systemctl edit ollama

and I added the following

[Serivce]
Environment=OLLAMA_MODELS=/media/shared/ai/models/llms

Then reload:

sudo systemctl daemon-reload
sudo systemctl restart ollama

This ensures all LLMs sit alongside my ComfyUI models, making the setup portable and easier to manage.

Pulling a model

For the first test, I grabbed Llama 3.1 (8B) — roughly in the 7B class, and it runs well on my RTX 3060 12 GB:

ollama pull llama3.1:8b

Check available models:

curl http://127.0.0.1:11434/v1/models

and I get the response:

{"object":"list","data":[{"id":"llama3.1:8b","object":"model","created":1756635630,"owned_by":"library"}]}

Which is just proving my model exists and is available for use from the intended location

Talking to the Model

Ollama supports the OpenAI /v1/chat/completions endpoint and it’s super easy to test, as follows:

curl -s http://127.0.0.1:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model":"llama3.1:8b",
    "messages":[{"role":"user","content":"Say hi in 5 words."}]
  }'

Response:

{"id":"chatcmpl-38","object":"chat.completion","created":1756635818,"model":"llama3.1:8b","system_fingerprint":"fp_ollama","choices":[{"index":0,"message":{"role":"assistant","content":"Hello, I am here to help!"},"finish_reason":"stop"}],"usage":{"prompt_tokens":17,"completion_tokens":9,"total_tokens":26}}

That’s it - a working local AI server. Now we can begin the fun stuff

What’s Next?

In the next post, I’ll connect a frontend of some kind on to my Ollama server so I can move away from day to day use of ChatGPT. I was thinking of LM Studio but I think that wont provide what I’m looking for as a “chat interface” without some kind of work so I’ll explore my options using Ollama as my brain for this project.

There is a possibility that I see if I can work on my own interface but I want to keep this as light as possible until I get an MVP working

Later in the series, I’ll expand into:

  • A search/indexing service that Ollama can query for my documents.
  • A Flutter desktop app that integrates with Ubuntu settings + AI toggles.
  • Shared workflows with ComfyUI using the same model directory
  • And whatever else comes to mind to make this awesome.