Generative AI for the Uninformed
(by the Uninformed)

Topics

Requirements by Use Case

OS Matters!

• Using a headless Linux install automatically grants you up to an additional 1 GB of VRAM when compared to Windows.

Introductory Concepts

Generative AI is Big Statistics

It's statistics all the way down.

Determinism

Generative AI models are deterministic.

Equal inputs = Equal outputs.

What is a Vector?

A vector is an ordered list of numbers.

[10, 96, 72, 31, 98]

Or...

[489.1512, 82.582, 17.124, 857.1723, 15.234]

Vectors have dimension. These examples have 5 dimensions each.

More on ONNX

From ONNX.ai:

ONNX is an open format built to represent machine learning models.

• Strong backing from many technical industry leaders

Large Language Models

Prompt

• User-provided text that the LLM responds to.

							$ Tell me about the Roman Empire.

							Absolutely! The Roman Empire was...
						

System Prompt

• System-configured prompt to provide instructions or context to subsequent prompts.
• Prefixes or is combined with every user prompt
• Generally not visible or adjustable by the end user.

							You are an assistant designed to help the user with
							any request or question. Responses should be
							accurate and concise. Refuse to answer any request
							or question that might be used to harm the user or
							another person.
						

Tokens

• A word, part of a word, or even punctuation.
• Determined by the LLM's tokenizer.
• Each token is assigned and referenced by an integral ID.

Image borrowed from TeeTracker@Medium

Context

• A summary of the history of the conversation with the LLM or additional tokens provided by other sources.
• Different LLMs can have varying maximum context windows.

							[113, 101, 102, 103, 104, 105, 106, 107, 108, 109, 104, 110, 111, 112, 114]
						

Temperature

How likely an LLM will select the "best" series of tokens in its response.

Retrieval Augmented Generation (RAG)

1. Convert external data to embeddings.
2. Retrieve the embeddings.
3. The embeddings are used to augment the user's prompt.
4. The LLM generates a response to the prompt.

Embeddings

Embeddings are similar to tokenizing the data, but embeddings also preserve the relationship between tokens.

Image borrowed from Tokenization vs. Embedding: Understanding the Differences and Their Importance in NLP.

Model Context Protocol (MCP)

• Standardized protocol to contextualize and provide external callable functionality to LLMs.
• Based on JSON-RPC 2.0.

							User: Turn on the living room lights.

							Assistant: Ok, I turned on the living room lights.
						

1. MCP Server Lists Functions

							{
							  "jsonrpc": "2.0",
							  "id": 1,
							  "result": {
							    "tools": [
							    {
							      "name": "control_lighting",
							      "title": "Smart Home Control: Lighting",
							      "description": "Turn lights on or off in a room.",
							      "inputSchema": {
							        "type": "object",
							        "properties": {
							         "room": {
							           "type": "string",
							           "description": "The room to turn the lights on or off in.",
							          },
									  "enabled": {
										"type": "bool",
										"description": "Whether to turn the lights on or off."
									  }
							        },
							        "required": ["room"]
							      }
							    }
							  ]
							}						  }
						

2. MCP Host Calls a Function

							{
							  "jsonrpc": "2.0",
							  "id": 1,
							  "method": "tools/call",
							  "params": {
							    "name": "control_lighting",
							    "arguments": {
							      "room": "Living Room",
							      "enabled": true
							    }
							  }
							}
						

3. MCP Server Executes Function and Returns Result

							{
							  "jsonrpc": "2.0",
							  "id": 1,
							  "result": {
							    "room": "Living Room",
							    "enabled": true
							  }
							}
						

Hugging Face

Online repository for various types of machine learning models.

Llama.cpp

• Inference engine for the Meta-developed LLaMA foundational LLM models
• Open-source
• CLI application, server, or standalone library, and more
• Used in nearly all open-source/self-hosted LLM solutions

Ollama

• "Just works" LLM runtime
• Server manager for queueing requests across multiple models
• OpenAI compatible API

ollama pull

• Download a large language model from Ollama's repository.
• Model names based on Docker naming convention with tags - :latest is the default tag.

							$ ollama pull llama3:8b
						

ollama run

• Runs an interactive shell to prompt the LLM.
• Pulls the model if missing.

							$ ollama run llama3:8b
						

ollama serve

Runs an OpenAI compatible API to facilitate LLM interactions

							$ ollama serve
						

Open Web UI

Web frontend for Ollama and other LLM backends

Llamafile

Single-file multi-platform self-contained executable LLM with a CLI or web based frontend

LM Studio

• Desktop application
• Probably the easiest way to get started with local LLMs
• Supports starting an OpenAI compatible API
• Closed source

Jan

• Desktop application similar to LM Studio
• Another approachable way to get started with local LLMs
• Supports starting an OpenAI compatible API
• Open source

Image Generation

Latent Space

A low-dimensional or set of low-dimensional vectors representing an image or image canvas

							[
							  0, 0, 0,
							  0, 0, 0,
							  0, 0, 0
							]
						

UNet Model

• The main generation model, typically a CNN (convolutional neural network).
• Predicts an image by iteratively removing noise in the latent image.
• U-like shape of the image encoding and decoding process.

The UNet architecture. Don't ask me how it works yet. Borrowed from Wikipedia's U-Net article.

Text Encoder

Tokenizes the prompt into a vector.

Image borrowed from TeeTracker@Medium

KSampler

1. Add visual noise based on seed and noise strength.
2. Load UNet model.
3. Iterate in steps based on positive and negative guidance (prompts).

Seed

An integer used to generate the starting noise pattern.

657835674309455

Scheduler

Schedulers control the level of noise at each step.

simple

karras

normal

sgm_uniform

Guidance Scale

The priority level of the prompt.

3.5

7.0

14.0

28.0

Variational Auto Encoder (VAE)

Converts the latent image to a pixel bitmap.

									[
									  -3.609193, -6.2027211, -3.2266970,
									  9.5763211, 1.2311611, -2.2856392,
									  -5.266179, 2.9949563, -5.5448007,
									  etc...
									]
								

→

Stable Diffusion

Long-time open-source/open-weight image generation model from Stability AI.

Flux

Open and closed source/weight models from Black Forest Labs that challenged Stability AI's Stable Diffusion 3.0 release.

SD.Next

A great beginner-friendly image generation web application with support for the latest models.

ComfyUI

Advanced industry-standard node-based generative AI toolkit.

SwarmUI

Assistive wrapper around ComfyUI.

Invoke

• Canvas
• Textbox
• Selection Tool

Paint with AI-driven inpainting.

"Spacetime Latent Patches"

Adding dimensionality around space and time to latent images helps produce video.

Borrowed from OpenAI's Video Generation Models as World Simulators.

Mochi

Early open source video model from Genmo.

Wan

More recent and more capable open source video model from Alibaba.

Audio Generation

Meta Audiocraft

									Whistling with wind blowing
								

									Pop dance track with catchy melodies,
									tropical percussion,
									and upbeat rhythms,
									perfect for the beach
								

ACE-Step

Open source/weight foundational music model with official support for ComfyUI.

Official example output from ACE-Step.

							(Verse 1)
							🎵🎵🎵
							It's not just a dream, it's the start of a way,
							Building the future where music will play.
							A model that listens, a model that grows,
							ACE-Step is the rhythm the new world knows.

							(Pre-Chorus)
							No more limits, no more lines,
							A thousand songs in a million minds.
							We light the spark, we raise the sound,
							A new foundation breaking ground.

							(Chorus)
							ACE-Step, we take the leap,
							Into a world where the music speaks.
							Fast and free, we shape the skies,
							Creating songs that never die.
							ACE-Step — the beat goes on,
							Foundation strong, a brand-new dawn.
							🎵🎵🎵


							(Verse 2)
							🎵
							Not just end-to-end, but a canvas to paint,
							A million colors, no need to wait.
							For every artist, for every sound,
							ACE-Step lays the sacred ground.

							(Pre-Chorus)
							🎵
							No more limits, no more lines,
							A thousand songs in a million minds.
							We light the spark, we raise the sound,
							A new foundation breaking ground.

							(Chorus)
							🎵
							ACE-Step, we take the leap,
							Into a world where the music speaks.
							Fast and free, we shape the skies,
							Creating songs that never die.
							ACE-Step — the beat goes on,
							Foundation strong, a brand-new dawn.

							(Bridge)
							🎵
							From every beat to every rhyme,
							We build the tools for endless time.
							A step, a song, a dream to keep,
							ACE is the promise we will leap.

							(Final Chorus)
							🎵
							ACE-Step, we take the leap,
							Into a world where the music speaks.
							Fast and free, we shape the skies,
							Creating songs that never die.
							ACE-Step — the beat goes on,
							Foundation strong, a brand-new dawn.
							ACE-Step — the future's song.
						

3D Model Generation

Hunyuan3D 2.0

An open source/weight 3D model generation model by Tencent.