A lot comes when you are training a model to generate 4k images: Translation from the positional embedding Tiling is the only way here : Cause sd3 training was whole on images that are under 2MP So tiling requires an conditional full size image as well to see the surrounding context to make correct edits there Position embedding limitation : SD3 : fixed sincos SD3.5 : Uses RoPE SD3.5 vs SD3 ...
GPU resources https://github.com/karpathy/llm.c https://siboehm.com/articles/22/CUDA-MMM Tiling1 : https://docs.nvidia.com/cuda/cuda-programming-guide/02-basics/writing-tile-kernels.html Tiling2 : https://cvw.cac.cornell.edu/cuda-intro/gpu-performance-topics/tiling Tiling video : http://youtube.com/watch?v=ccHyFnEZt7M Cornell cuda intro : https://cvw.cac.cornell.edu/cuda-intro/ Tensor cores : https://youtu.be/Yt1A-vaWTck Tensor cores2 : https://tgautam03.github.io/2024/10/30/TensorCores/ tensor cores3 : https://www.glennklockwood.com/garden/tensor-cores Theory of GPU Basic Terms : Host : CPU device Device : GPU device CUDA : compute unified device architecture, these are used when we need to do write code for nvidia gpus ROCm : Radeon open compute platform, used when we need to write for amd gpus MSL : Metal shading language to run directly on apple graphics, for macos Logical : a software construct, something that is written in software to make it simpler / easier for people to understand Physical : its a physical / hardware level definition that we make here means this is present in hardware Terms: Threads : So smallest execution unit on a GPU is a thread, its role is to complete an atomic instruction on SM Warp : Group of 32 threads that are sequential and in same thread block is called a warp ( they are faster cause they use Shared memory) .. this is useful if the data needs to be shared among all like in reduce operations. Active mask : So this is used to define which all threads to use in a warp so a mask looks like this 0x00000001 or this 0xFFFFFFFF blocks : Group / Collection of threads (cuda limits this no. of threads to 1024) make up a thread block grid : group of blocks in a gpu makes up a grid ( where blocks are arranged ) Streaming Multiprocessors : the fundamental building block of an NVIDIA GPU, this is where the operations are executed on cuda cores Tiling : A software memory strategy. It is a technique where you break down large datasets into small chunks (“tiles”) that fit inside fast, local SRAM (Shared Memory or Registers) to avoid pulling repeatedly from slow VRAM. (memory bound ops with high data reuse) SIMT (Single Instruction, Multiple Threads) means that multiple threads execute the same instruction at the same time, but each thread operates on its own data. Here we need a global thread index , loads SIMD (Single instruction multiple data) : TODO Tile kernel : level of entire tile block, load a whole tile, perform ops on that tile and store back the tile Single thread : There is nothing as single thread in a GPU execution. even if you define <<<1,1>>> its basically initiating a full 32 thread hardware warp to execute a single thread. Just that active mask changes in this case Cuda stream : single stream vs multiple streams TODO sync vs async GPU operations Memory layout in GPU memory dimension in GPU ...
Following this : https://jax-ml.github.io/scaling-book/inference/#the-basics-of-transformer-inference Beam search Speculative decoding used at inference time by expanding the most probable nodes for 3 to 5 iterations and chooses the best iteration out so here we calculate the sum of log probabilities rather than taking just the normal probabilities
World models Its a framework ( emphasised by yann le cun ) that works autoregressively (that is we pass in last x frame and model predicts x_t+1 frame) and World model usually cares about dynamics, causality, and actions so a world models learns next state must we consistent with the world rules ( a ball in air must come down also ) Action conditioning in the predictor model
So ffmpeg is one of the best open source tool that is used to create video / change codec , change pixel-format , change crf factor etc and this is written in cpp so its already one of the most fastest software that we have here FFMPEG commands
Logging / profiling and tracing Nightly build : every night at a fixed time a cron runs that takes all the commits of the prev day and make a new compiled binary ( this is usually done for companies where the work is GPU heavy and they cant bog down there whole system for this). This is done in addition to github actions that run on each commit and it has some precommit hooks there so that makes helps to identify most of the basic issues and this nightly build is used to check for hardware issue , security issues that are not possible to run on each commit. Torch tracing to get better efficiency so we can log / trace the model / network to see where the torch graph is breaking and where we can make it fast ...
Audio Modality Codec : piece of hardware/software that compresses / decompresses digital data to reduce file size FFT : fast fourier transform converts from time-amplitude domain to frequency-amplitude domain Resampling : an audio was recorded at 44100 Hz, but we want to resample it to 16000 Hz, so that is called resampling Spectrogram : converting from time-amplitude domain to frequency-amplitude domain Mels : mel scale, approximates how humans percieve pitch and in this freq axis is converted to mel scale Channel : no. of seperate audio how many microphones were used to record the audio Mono channel : single sound , more like one headphone sound Stereo channel : surround sounds , more like two headphones sound (tv , songs , youtube vids ) Sampling Rate : no. of sound point extracted from 1 sec of audio Waveform : so this is audio plot , on x-axis we have time , on y-axis we have decibels, pitch . This is what we hear and what music players shows Spectogram : so we convert from time domain to freq domain using FFT , Mel-spectogram : Inspired from how humans listen to sound, and we listen on a logscale so therefore mel-spectogram is made for humans to listen Example : SAMPLE_RATE = 16000 HOP_LENGTH = 256 # number of audio samples between spectrogram frames between 2 short time frame windows N_FFT = 1024 MAX_MEL_FRAMES = 512 # no. of timestamps in a mel spectrogram So in this audio is sampled at 16Khz N_FFT : tells in a 1 fft how many samples to analyse, ...
Resolution means a lot for image domains an upscale from 512 to 1204 can make or break your AI model U2Net still works better for segmentation than most of the SAM3 models when it comes to high defined smooth edges. SAM models are data hungry, need a lot data to do something nice from it Convolution operation and kernels are underrated, a lot can be done if the filter / kernel values are set correctly ...
Terms Ephemeral (efimeral) : short lived functions like google-cloud functions , that are running for per request and has a fixed timeout and at that timeout it sends a SIGTERM to close it. Presigned / Signed URL : Is used for upload / download directly to storage (w/o involving backend in this all), its a time based URL that expires with time, the generator of the link should have access and uploader can just upload … so this is used in Customer support as well , when we upload images of a damaged item, this is used to reduce load to backend and solve this ...
Alignment / RLHF part of the models , and how to do those ! Pretrained model to getting to Instruct GPT model the whole flow of going from here to there Enabling better, tighter controls over LM output Post-training : SFT : if you want to imitate expert demostration you better have expert demonstration of what that looks like , once we have the data how to adapt to it ? https://youtu.be/Dfu7vC9jo4w?t=337 All policies tries to find / estimate the Advantage value .. PPO does that by telling how better this action is compared to the average .. Advantage = ( R - Baseline ) ...
Encoding Its a way to send / transmit information The transfering of data across internet is done in byte stream and we have a defined format to encode those in a byte stream Like if its a text data then we have utf-8 encoding , if its a audio file its mp3 encoding, image file its png encoded like this we define encoding for all dataformats and on client side decoding happens to output it in a compatible way to the end user ...
Embeddings Vector Embeddings https://www.pinecone.io/learn/what-is-similarity-search/ Searching over structured data that is easy we can use Data structures for it like Binary tress / arrays (sorted order). This was done in internet 2.0 , sql , mysql , mongodb these leveraged it so well Now for unstructured data we need something that represents more deeper concept / representation of the data Using sentence-transformers (and models like Word2Vec , BERT model) So in the bert model we train it using the [CLS] token / prefix, we take the trained model and then extract this token embedding. Encoder only architecture : This is used in models like BERT ( that is bidirectional ) and its useful for NLU tasks ( that is natural language understanding ) and to generate more tokens out from this we use [MASK] as a token header. And in word2vec model, we use cbow and skip-gram that depend on the proximity of similar words Vector Search Terminologies : IndexPQ ( product quantizers ) , IndexIVFPQ ( Inverted File with Product Quantization ) ...