Page 3 of 5

Building a Private AI Knowledge Platform with RAGFlow and Ollama

Part 2 – Ubuntu Installation, Docker Deployment, Ollama Configuration and RAGFlow Setup

A Practical Enterprise Tutorial for SMEs, Engineers, Researchers and IT Professionals

13. Introduction

In Part 1, we introduced the concepts of Retrieval-Augmented Generation (RAG), Large Language Models (LLMs), Ollama and RAGFlow, and discussed why local AI deployments are becoming increasingly important for organisations that need secure access to proprietary information.

This chapter moves from theory to implementation by guiding readers through the installation and configuration of a production-ready AI platform based on Ubuntu Linux.

By the end of this chapter, readers will have:

  • Prepared an Ubuntu server for AI workloads.
  • Installed Docker and Docker Compose.
  • Installed and configured Ollama.
  • Downloaded and managed local AI models.
  • Installed RAGFlow using Docker.
  • Connected RAGFlow to Ollama.
  • Verified the deployment.
  • Prepared the platform for document ingestion.

14. System Requirements

The hardware requirements depend on the size of the language models and the anticipated workload.

Minimum Development System

  • Ubuntu 24.04 LTS or Ubuntu 26.04 LTS
  • Quad-core CPU
  • 16 GB RAM
  • 250 GB SSD
  • Gigabit Ethernet

Recommended for testing and learning.

Recommended SME Production Server

  • Intel Xeon or AMD EPYC
  • 32–64 GB RAM
  • NVMe SSD (1 TB or larger)
  • NVIDIA RTX GPU (optional but recommended)
  • Ubuntu Server LTS

This configuration comfortably supports multiple users and medium-sized document repositories.

Enterprise Deployment

For larger organisations:

  • Multiple application servers
  • Dedicated vector database storage
  • GPU inference server
  • Reverse proxy
  • Backup server
  • Monitoring server

This architecture supports high availability and future growth.

15. Ubuntu Preparation

Update the operating system before installing additional software.

sudo apt update sudo apt upgrade -y

Install common administration tools.

sudo apt install \ git \ curl \ wget \ unzip \ vim \ htop \ net-tools \ build-essential \ ca-certificates \ software-properties-common -y

Reboot if the kernel has been updated.

sudo reboot

16. Installing Docker

Docker provides application isolation and simplifies deployment.

Install Docker:

sudo apt install docker.io docker-compose-v2 -y

Enable Docker:

sudo systemctl enable docker sudo systemctl start docker

Add the current user to the Docker group:

sudo usermod -aG docker $USER

Apply the group membership:

newgrp docker

Verify the installation:

docker version docker compose version

17. Installing Ollama

Ollama manages local language models and provides a REST API used by RAGFlow.

Install:

curl -fsSL https://ollama.com/install.sh | sh

Enable the service:

sudo systemctl enable ollama sudo systemctl start ollama

Check service status:

sudo systemctl status ollama

Verify:

ollama list

Initially, no models will be installed.

18. Downloading AI Models

Chat Model

ollama pull llama3.2

Embedding Model

ollama pull bge-m3

Larger Reasoning Model

ollama pull qwen2.5:14b

Coding Assistant

ollama pull qwen2.5-coder

Viewing Installed Models

ollama list

Example:

NAME SIZE llama3.2 4.7 GB bge-m3 1.2 GB qwen2.5:14b 9.5 GB

19. Configuring Ollama

By default, Ollama listens on the local interface.

For Docker-based deployments, edit the service configuration.

sudo systemctl edit ollama

Add:

[Service] Environment="OLLAMA_HOST=0.0.0.0:11434"

Reload:

sudo systemctl daemon-reload sudo systemctl restart ollama

Verify:

curl http://localhost:11434/api/tags

Expected output:

{ "models":[ { "name":"llama3.2" } ] }

20. Installing NVIDIA GPU Support (Optional)

GPU acceleration significantly improves inference speed.

Install the NVIDIA driver:

ubuntu-drivers autoinstall

Reboot:

sudo reboot

Install the NVIDIA Container Toolkit if Docker containers will use the GPU.

Verify:

nvidia-smi

Typical output includes:

  • Driver version
  • CUDA version
  • GPU temperature
  • Memory usage

If no GPU is available, Ollama automatically uses CPU inference.

21. Installing RAGFlow

Download the project:

git clone https://github.com/infiniflow/ragflow.git

Move into the project directory:

cd ragflow

Start the containers:

docker compose up -d

Docker downloads the required images automatically.

Check running containers:

docker ps

22. Understanding the RAGFlow Components

A standard deployment includes services such as:

  • RAGFlow Web Application
  • PostgreSQL
  • Redis
  • MinIO Object Storage
  • Elasticsearch
  • Background Worker

Each service performs a dedicated function, making the platform easier to maintain and scale.

23. Accessing the Web Interface

Open your browser:

http://localhost>If the server is remote:http://SERVER-IP>Log in and create the administrator account.

24. Connecting RAGFlow to Ollama

Open:

Settings → Model Providers → Ollama

Configure:

Parameter

Value

Provider

Ollama

Base URL

http://host.docker.internal:11434

Chat Model

llama3.2

Embedding Model

bge-m3

If both applications run directly on the host:

http://localhost:11434

If Ollama runs on another server:

http://192.168.x.x:11434

Replace the IP address with the actual address of the Ollama server.

25. Testing the Connection

Use the Test Connection option within RAGFlow.

If successful, the configured models will appear in the available model list.

If the test fails:

  • Verify the Ollama service.
  • Confirm firewall rules.
  • Check Docker networking.
  • Ensure the base URL is correct.
  • Confirm that the models have been downloaded.

26. Creating the First Knowledge Base

Select:

Knowledge Bases → Create

Enter:

  • Name
  • Description
  • Language
  • Embedding Model

Save the configuration.

The knowledge base is now ready to receive documents.

27. Uploading Documents

Supported formats include:

  • PDF
  • DOCX
  • TXT
  • Markdown
  • HTML
  • CSV
  • PowerPoint
  • Excel

RAGFlow automatically:

  1. Parses the document.
  2. Extracts text.
  3. Divides the content into chunks.
  4. Generates embeddings.
  5. Stores vectors for semantic search.

28. Verifying Document Processing

Upload a sample PDF.

Observe the processing pipeline:

Upload ↓ Parsing ↓ Chunking ↓ Embedding ↓ Vector Storage ↓ Ready for Search

Once complete, the document is available for conversational retrieval.

29. Troubleshooting

Ollama Cannot Be Reached

Check:

sudo systemctl status ollama

Verify:

curl http://localhost:11434/api/tags

Docker Containers Not Running

Check:

docker ps -a

View logs:

docker compose logs

Models Not Found

List models:

ollama list

Download missing models:

ollama pull llama3.2

Slow Responses

Possible causes include:

  • Insufficient RAM
  • CPU-only inference
  • Large language models
  • Slow storage

Consider using an NVIDIA GPU or selecting a smaller model for interactive workloads.

30. Practical Deployment Example

A 50-person engineering consultancy deploys RAGFlow and Ollama on an Ubuntu server with 64 GB RAM and an NVIDIA RTX GPU. They upload:

  • ISO standards
  • Engineering drawings
  • Maintenance manuals
  • Project reports
  • Internal procedures
  • Proposal templates

Engineers can then ask questions such as:

  • "Which documents describe the commissioning process?"
  • "Summarise the electrical safety requirements."
  • "Compare Revision 3 and Revision 5 of the installation manual."

The platform retrieves relevant passages from internal documents and generates responses grounded in the organisation's own knowledge base, reducing search time and improving consistency.

31. How KeenComputer.com Can Help

KeenComputer.com can assist organisations with deploying and supporting a production-ready RAG platform by providing:

  • Ubuntu server installation and hardening
  • Docker and container orchestration
  • Private cloud infrastructure design
  • GPU server configuration
  • AI platform deployment
  • Network security and firewall configuration
  • SSL/TLS implementation
  • Backup and disaster recovery planning
  • Performance optimisation
  • Staff training and managed support

These services help SMEs adopt AI with a secure, scalable infrastructure aligned to business needs.

32. How IAS-Research.com Can Help

IAS-Research.com complements implementation with advanced engineering and applied AI expertise, including:

  • AI architecture design
  • Retrieval-Augmented Generation (RAG) strategy
  • Model evaluation and benchmarking
  • Embedded AI and Industrial IoT integration
  • Digital twin development
  • Technical research and innovation
  • Engineering knowledge-base design
  • Technical documentation and white-paper development
  • Proof-of-concept projects for research and industry

Together, KeenComputer.com and IAS-Research.com provide organisations with both practical implementation capabilities and research-driven engineering support.

33. Summary

This chapter established a complete Ubuntu-based deployment of RAGFlow and Ollama, covering system preparation, Docker installation, model management, platform configuration and verification. The environment is now ready for indexing organisational documents and supporting secure, AI-assisted knowledge retrieval.

In Part 3, we will explore document ingestion, chunking strategies, embedding models, vector search, prompt engineering, reranking, and techniques for improving answer quality using engineering manuals, research papers and business documentation.

References

  1. RAGFlow Documentation – Deployment and Configuration.
  2. Ollama Documentation – Installation and Model Management.
  3. Docker Documentation – Containers and Compose.
  4. Ubuntu Server Documentation.
  5. NVIDIA CUDA Documentation.
  6. BAAI. BGE-M3 Technical Documentation.
  7. Meta AI. Llama 3 Documentation.
  8. Alibaba Cloud. Qwen2.5 Technical Report.