vimix/docs/RENDERING_ARCHITECTURE.md

Vimix Rendering Architecture Overview

Executive Summary

Vimix is a real-time video mixing application with a sophisticated three-tier rendering pipeline:

1. Source Rendering Layer - Individual sources render to frame buffers via GStreamer
2. Composition Layer - Session compositing sources together into a single output frame
3. UI/Display Layer - OpenGL window rendering with ImGui UI overlay

The main rendering loop operates at a fixed ~62 FPS framerate with callbacks for each stage.

---

1. Main Application Loop

Loop Structure

// Main rendering loop (lines 261-262)
while ( Rendering::manager().isActive() )
    Rendering::manager().draw();

Initialization Flow (lines 209-247)

1. Rendering::manager().init()          // OpenGL/GLFW initialization
2. UserInterface::manager().Init()       // ImGui setup
3. Audio::manager().initialize()         // Audio system
4. pushBackDrawCallback(prepare)         // Register update callback
5. pushBackDrawCallback(drawScene)       // Register scene rendering callback
6. pushBackDrawCallback(renderGUI)       // Register UI rendering callback
7. Rendering::manager().show()           // Display windows
8. Mixer::manager().load(_openfile)      // Load session
9. Main loop starts

---

2. Main Rendering Loop Implementation

Location

• Method: Rendering::draw() (lines 608-673)

Rendering Pipeline Flow

Rendering::draw()
    ├─ glfwPollEvents()                    // Poll input/window events
    │
    ├─ TabletInput::instance().pollEvents() // Tablet pressure support
    │
    ├─ main_.changeFullscreen_()           // Handle fullscreen requests
    │
    ├─ OUTPUT WINDOWS RENDERING
    │  └─ for (output windows):
    │      └─ output.draw(Mixer::session()->frame())  // Render to output windows
    │
    ├─ MAIN WINDOW CONTEXT
    │  └─ main_.makeCurrent()              // Switch to main window GL context
    │
    ├─ EXECUTE DRAW CALLBACKS (key stage)
    │  └─ for (draw_callbacks_):
    │      1. prepare()
    │         └─ Control::manager().update()
    │         └─ Mixer::manager().update()
    │         └─ UserInterface::manager().NewFrame()
    │
    │      2. drawScene()
    │         └─ Mixer::manager().draw()
    │
    │      3. renderGUI()
    │         └─ UserInterface::manager().Render()
    │
    ├─ SCREENSHOT CAPTURE
    │  └─ if (request_screenshot_):
    │      └─ screenshot_.captureGL()
    │
    └─ glfwSwapBuffers()                   // Display frame
    
    └─ FPS LIMITER (15600 microseconds)   // Soft-cap at ~62 FPS
       static GTimer for frame timing

Framerate Control

• Location: Lines 666-672 in RenderingManager.cpp
• Method: GLib GTimer (g_timer_*)
• Target: ~62 FPS (15600 microseconds between frames)
• Implementation: Busy-sleep if frame completes too quickly

---

3. Scene Update and Rendering (Mixer)

Location

• Methods: Mixer::update() (lines 86-276), Mixer::draw() (lines 278-283)

Update Phase (prepare() callback)

Executed in Mixer::update():

Mixer::update(dt_)
    ├─ Session::update(dt_)          // Update all sources
    │  └─ for (each source):
    │      └─ Source::update(dt)     // Update individual source
    │
    ├─ FrameGrabbing::grabFrame()    // Grab frames for recording
    │
    ├─ Process failed sources
    │  ├─ FAIL_RETRY: Attempt to recreate
    │  ├─ FAIL_CRITICAL: Detach from mixer
    │  └─ FAIL_FATAL: Delete source
    │
    └─ Update views for deep refresh
       └─ View::update(dt) for dirty views

Framerate tracking:
    dt_ = elapsed time since last frame (ms)
    dt__ = stabilized dt (EMA: 5% new + 95% old)
    fps = 1000 / dt__

Draw Phase (drawScene() callback)

Executed in Mixer::draw():

Mixer::draw()
    └─ current_view_->draw()         // Render current view

The actual rendering is delegated to the View's draw method.

---

4. Source Rendering Pipeline

Source Architecture

Base Class: vimix/src/Source/Source.h

Source hierarchy:

Source (base class)
├─ MediaSource              // Video/image files via GStreamer
├─ DeviceSource             // Camera/webcam input
├─ StreamSource             // Generic GStreamer pipeline
├─ PatternSource            // Generated patterns
├─ TextSource               // Rendered text
├─ ShaderSource             // GLSL shader output
├─ RenderSource             // Renders nested session
├─ CloneSource              // References another source
├─ SessionSource            // Embedded session file
├─ ScreenCaptureSource      // Screen capture
├─ MultiFileSource          // Image sequence
├─ NetworkSource            // Network stream
└─ SrtReceiverSource        // SRT protocol input

Source Update/Render Cycle

Location: Source.h (lines 162-169)

// Each source has:
virtual void update(float dt);      // Called once per frame
virtual void render();              // Render to frame buffer
virtual FrameBuffer *frame() const; // Get output texture

MediaSource (Primary Video Source) Flow

File: vimix/src/Source/MediaSource.cpp

MediaSource::update(dt)
    └─ mediaplayer_->update()       // Update GStreamer pipeline

MediaSource::init()
    └─ Create FrameBuffer matching media resolution
    └─ Link texture from MediaPlayer to source framebuffer

MediaSource::render()
    └─ Render media texture to source's frame buffer

---

5. GStreamer Integration

MediaPlayer Class

Location: /vimix/src/MediaPlayer.h (Lines 59-434)

Key components:

struct MediaInfo {
    guint width, height, bitrate;
    guint framerate_n, framerate_d;
    std::string codec_name;
    bool isimage, interlaced, seekable;
    GstClockTime dt, end;    // Frame duration and end time
};

class MediaPlayer {
    GstElement *pipeline_;           // GStreamer pipeline
    GstBus *bus_;                    // Pipeline message bus
    GstAppSink *appsink_;            // Video output sink
    
    Frame frame_[N_VFRAME];          // Frame buffer stack
    uint textureid_;                 // OpenGL texture
    
    struct TimeCounter {
        GTimer *timer;               // FPS measurement
        gdouble fps;
    } timecount_;
};

Pipeline Structure

File: /vimix/src/MediaPlayer.cpp

GStreamer Pipeline:
    [uridecodebin]
        ├─ Video path:
        │   ├─ [videoscale]
        │   ├─ [videoconvert]
        │   ├─ [videofilter (optional)]
        │   └─ [glsinkbin] (GL rendering) or [appsink] (CPU)
        │
        └─ Audio path:
            └─ [audioconvert]
            └─ [appsink]

GStreamer OpenGL Integration

Location: RenderingManager.cpp (lines 42-52, 172-221)

// GL context sharing with GStreamer
#ifdef USE_GST_OPENGL_SYNC_HANDLER
static GstGLContext *global_gl_context = NULL;
static GstGLDisplay *global_display = NULL;

// Linking pipeline to rendering instance
void Rendering::LinkPipeline(GstPipeline *pipeline)
{
    GstBus* m_bus = gst_pipeline_get_bus(pipeline);
    gst_bus_set_sync_handler(m_bus, bus_sync_handler, pipeline, NULL);
}

GPU Hardware Decoding

Location: RenderingManager.cpp (lines 500-515)

// Enable GPU decoding if available
std::list<std::string> gpuplugins = 
    GstToolkit::enable_gpu_decoding_plugins(
        Settings::application.render.gpu_decoding);

// Supported plugins logged
if (Settings::application.render.gpu_decoding)
    Log::Info("Hardware decoding enabled.");

Frame Handling

File: MediaPlayer.cpp

// Frame stack with mutex protection
struct Frame {
    GstBuffer *buffer;
    FrameStatus status;    // SAMPLE, PREROLL, EOS, INVALID
    bool is_new;
    GstClockTime position;
    std::mutex access;
};
Frame frame_[N_VFRAME];    // N_VFRAME = 5 frames

// Callback when new sample arrives
static GstFlowReturn callback_new_sample(GstAppSink *sink, gpointer data)
{
    // Get sample, map to GPU texture
    // Fill texture via PBO (Pixel Buffer Object) upload
}

---

6. Session Composition and Output

Session Class

Location: /vimix/src/Session.h

class Session {
    SourceList sources_;              // All sources in session
    RenderView render_;               // Composition renderer
    
    // Main output framebuffer
    FrameBuffer *frame() const { 
        return render_.frame(); 
    }
    
    void update(float dt);            // Update all sources
};

RenderView (Composition)

Location: /vimix/src/View/RenderView.h

class RenderView : public View {
    FrameBuffer *frame_buffer_;       // Output framebuffer
    Surface *fading_overlay_;         // Fading effect overlay
    
    void draw() override;             // Composite all sources
};

Composition Flow

RenderView::draw() (called from current View)

RenderView::draw()
    ├─ frame_buffer_->begin()         // Bind FBO
    │
    ├─ DrawVisitor traversal
    │  └─ Visit and draw all source nodes
    │     └─ Each source renders its texture to composition
    │
    ├─ Apply blending/effects
    │  └─ ImageProcessingShader
    │  └─ BlendingShader
    │
    ├─ Apply fading overlay
    │  └─ fading_overlay_->draw()
    │
    └─ frame_buffer_->end()           // Unbind FBO
    
Result: frame() returns the composed image

---

7. GUI Rendering (ImGui)

Location

• Init: UserInterface::Init() (lines 117-150)
• Frame Start: UserInterface::NewFrame() (lines 866-871)
• Render: UserInterface::Render() (lines 1046-1047)

ImGui Integration

// Initialization
ImGui::CreateContext();
ImGui_ImplGlfw_InitForOpenGL(window, true);
ImGui_ImplOpenGL3_Init(VIMIX_GLSL_VERSION);

// Per-frame sequence (in renderGUI callback)
void UserInterface::NewFrame()
{
    ImGui_ImplOpenGL3_NewFrame();
    ImGui_ImplGlfw_NewFrame();
    ImGui::NewFrame();
}

void UserInterface::Render()
{
    ImGui::Render();
    ImGui_ImplOpenGL3_RenderDrawData(ImGui::GetDrawData());
}

UI Rendering Stages

renderGUI() callback
    ├─ UserInterface::NewFrame()
    │  └─ Poll input, start ImGui frame
    │
    ├─ Build UI (in UserInterface::Render())
    │  ├─ Main menu
    │  ├─ View navigator
    │  ├─ Source panels
    │  ├─ Output windows
    │  └─ Various controls
    │
    └─ ImGui::Render() + RenderDrawData()
       └─ Draw ImGui command buffer to OpenGL

---

8. Existing Timing and Performance Measurement

GLib GTimer Usage

Location: Multiple files

// RenderingManager.cpp (lines 667-671)
static GTimer *timer = g_timer_new();
double elapsed = g_timer_elapsed(timer, NULL) * 1000000.0;
if (elapsed < 15600.0 && elapsed > 0.0)
    g_usleep(15600 - (gulong)elapsed);
g_timer_start(timer);

// Mixer.cpp (lines 201-206)
static GTimer *timer = g_timer_new();
dt_ = g_timer_elapsed(timer, NULL) * 1000.0;  // Frame delta time in ms
dt__ = 0.05f * dt_ + 0.95f * dt__;           // Smoothed delta time (EMA)
g_timer_start(timer);

FPS Calculation

Location: Mixer.h (line 87)

inline int fps() const { return int(roundf(1000.f/dt__)); }

Also in MediaPlayer for individual sources:

// MediaPlayer.h (lines 369-378)
struct TimeCounter {
    GTimer *timer;
    gdouble fps;
public:
    TimeCounter();
    ~TimeCounter();
    void tic();
    inline gdouble frameRate() const { return fps; }
};

UI Metrics Display

Location: UserInterfaceManager.cpp

#define PLOT_ARRAY_SIZE 180  // Rolling window of 180 frames

// RenderMetrics() method displays:
- Current FPS
- Frame time plot
- GPU memory usage
- Render statistics

Memory Monitoring

Location: RenderingManager.cpp (lines 786-850)

static glm::ivec2 getGPUMemoryInformation()
{
    // NVIDIA: GL_NVX_gpu_memory_info extension
    // ATI: GL_ATI_meminfo extension
    return { current_available_MB, total_available_MB }
}

static bool shouldHaveEnoughMemory(glm::vec3 resolution, int flags)
{
    // Check if enough VRAM before creating framebuffer
}

Source Update Timing

Location: Mixer.cpp (lines 200-206)

// Detailed frametime tracking
dt_ = g_timer_elapsed(timer, NULL) * 1000.0;   // Instantaneous
dt__ = 0.05f * dt_ + 0.95f * dt__;             // Exponential Moving Average

// Accessible via
Mixer::manager().dt()   // Raw delta
Mixer::manager().dt__() // Smoothed delta
Mixer::manager().fps()  // Calculated FPS

---

9. Callback Architecture

Rendering Callbacks

Location: RenderingManager.h (lines 129-131)

typedef void (* RenderingCallback)(void);
void pushBackDrawCallback(RenderingCallback function);

// Callbacks are executed in sequence each frame
std::list<RenderingCallback> draw_callbacks_;

Current Registered Callbacks

Location: main.cpp (lines 241-243)

1. prepare()           // Update phase
2. drawScene()         // Render scene phase
3. renderGUI()         // Render UI phase

Each callback is invoked in order during Rendering::draw() (lines 651-655).

---

10. Rendering Call Flow Diagram

┌─ Main Loop (main.cpp:261)
│
└─ Rendering::draw() [RenderingManager.cpp:608]
   │
   ├─ Event handling
   │  ├─ glfwPollEvents()
   │  ├─ TabletInput::pollEvents()
   │  └─ Handle fullscreen/window state changes
   │
   ├─ Output Windows Rendering
   │  └─ For each output window:
   │      └─ RenderingWindow::draw(frame)
   │          └─ Render session()->frame() to monitor
   │
   ├─ Main Window Context Activation
   │  └─ main_.makeCurrent()
   │
   ├─ Execute Draw Callbacks (in sequence)
   │  │
   │  ├─ [1] prepare() [main.cpp:46]
   │  │   └─ Mixer::update(dt) [Mixer.cpp:86]
   │  │      └─ Session::update(dt) [Session.h:96]
   │  │         ├─ For each source:
   │  │         │  ├─ Source::update(dt) [Source.h:162]
   │  │         │  │  ├─ MediaSource::update(dt) [MediaSource.cpp:181]
   │  │         │  │  │  └─ MediaPlayer::update() [MediaPlayer.cpp]
   │  │         │  │  │     └─ Poll GStreamer pipeline
   │  │         │  │  │     └─ Update texture from latest frame
   │  │         │  │  │     └─ Measure FPS via TimeCounter
   │  │         │  │  │
   │  │         │  │  └─ SourceCallback execution
   │  │         │  │
   │  │         │  ├─ RenderView internal update
   │  │         │  │  └─ Update mixing nodes positions/transforms
   │  │         │  │
   │  │         │  └─ View::update(dt) [View.h:38]
   │  │         │
   │  │         ├─ Frame delta time calculation
   │  │         │  └─ GTimer: dt_, dt__ (smoothed), fps
   │  │         │
   │  │         └─ Handle failed sources
   │  │
   │  │   └─ UserInterface::NewFrame() [UserInterfaceManager.cpp:866]
   │  │      ├─ ImGui_ImplOpenGL3_NewFrame()
   │  │      ├─ ImGui_ImplGlfw_NewFrame()
   │  │      └─ ImGui::NewFrame()
   │  │
   │  ├─ [2] drawScene() [main.cpp:53]
   │  │   └─ Mixer::draw() [Mixer.cpp:278]
   │  │      └─ current_view_->draw() [View.h:39]
   │  │         │
   │  │         └─ RenderView::draw() [RenderView.cpp]
   │  │            ├─ frame_buffer_->begin() [FrameBuffer.cpp:58]
   │  │            │  └─ Bind FBO, set viewport, clear
   │  │            │
   │  │            ├─ DrawVisitor traversal
   │  │            │  ├─ Visit scene root
   │  │            │  └─ Draw all source nodes
   │  │            │     └─ For each source:
   │  │            │         ├─ Source::render() [Source.h:169]
   │  │            │         │  └─ Render texture to composition FB
   │  │            │         │
   │  │            │         └─ Apply blending shader
   │  │            │            ├─ ImageProcessingShader
   │  │            │            └─ BlendingShader effects
   │  │            │
   │  │            ├─ Apply fading overlay
   │  │            │  └─ fading_overlay_->draw()
   │  │            │
   │  │            └─ frame_buffer_->end()
   │  │               └─ Unbind FBO
   │  │
   │  │            Result: Session frame buffer updated
   │  │
   │  └─ [3] renderGUI() [main.cpp:58]
   │      └─ UserInterface::Render() [UserInterfaceManager.cpp:1046]
   │         ├─ Build ImGui window hierarchy
   │         │  ├─ Menu bar
   │         │  ├─ View navigator
   │         │  ├─ Source control panels
   │         │  ├─ Inspector windows
   │         │  └─ Other UI elements
   │         │
   │         └─ ImGui::Render() + RenderDrawData()
   │            └─ Execute OpenGL draw calls for UI
   │
   ├─ Screenshot Capture (if requested)
   │  └─ screenshot_.captureGL()
   │
   ├─ Buffer Swap
   │  └─ glfwSwapBuffers()
   │
   └─ Framerate Limiting
      └─ GTimer: sleep until 15600 microseconds elapsed (~62 FPS)

---

11. Key Files and Their Roles

Component	File	Key Responsibility
Main Loop	main.cpp	Application entry, callback registration
Rendering Manager	RenderingManager.cpp/.h	Window management, GL context, callback execution
Mixer	Mixer.cpp/.h	Scene management, source updating, FPS calculation
Session	Session.cpp/.h	Source composition, session state
RenderView	View/RenderView.cpp/.h	Final composition/rendering to FBO
Source Base	Source/Source.cpp/.h	Source interface, framebuffer attachment
MediaSource	Source/MediaSource.cpp	Video/image file handling
MediaPlayer	MediaPlayer.cpp/.h	GStreamer pipeline control, texture management
Stream	Stream.cpp/.h	Generic GStreamer stream handling
FrameBuffer	FrameBuffer.cpp/.h	OpenGL FBO creation and management
UI Manager	UserInterfaceManager.cpp/.h	ImGui initialization and rendering
DrawVisitor	Visitor/DrawVisitor.cpp/.h	Scene graph traversal for rendering

---

12. Profiling Opportunities

Existing Instrumentation Points

1. GLib GTimer - Used for frame timing

   • Location: RenderingManager.cpp, Mixer.cpp
   • Already measures: Frame delta time, FPS, framerate limiting

2. MediaPlayer TimeCounter - FPS per source

   • Location: MediaPlayer.h (lines 370-378)
   • Already measures: Update framerate per media player

3. GPU Memory - VRAM usage queries

   • Location: RenderingManager.cpp (getGPUMemoryInformation)
   • Uses: NVIDIA/ATI GPU memory extensions

Recommended Profiling Points

1. Per-stage timing (prepare, draw, GUI)

   • Insert GTimer at start/end of each callback
   • Track: Update time, render time, UI time

2. Source-level profiling

   • Time each Source::update() and Source::render()
   • Track: GStreamer decoding, texture upload, GPU rendering

3. View composition timing

   • Time RenderView::draw() phases
   • Track: DrawVisitor traversal, shader execution

4. GStreamer pipeline monitoring

   • Query pipeline state and buffer queue depth
   • Track: Frame drops, decoding lag

5. ImGui rendering stats

   • Use ImGui metrics window
   • Track: Draw call count, vertex count, UI layout time

---

13. Architecture Summary

Vimix's rendering is organized as:

1. Event-Driven Main Loop

   • Framerate-limited (~62 FPS)
   • Callback-based architecture

2. Three-Stage Pipeline Per Frame

   • Update phase: Process all sources and state
   • Render phase: Compose sources to session framebuffer
   • UI phase: Render ImGui overlay

3. GStreamer-Based Source Handling

   • Asynchronous decoding via GStreamer pipelines
   • GPU texture management (GL interop)
   • Hardware decoding support

4. Layered Rendering Model

   • Individual source → source framebuffer
   • All sources → session framebuffer (composition)
   • Session → output windows (display)
   • UI overlay on top (ImGui)

5. Visitor Pattern for Scene Traversal

   • DrawVisitor renders scene graph
   • Depth-sorted source ordering
   • Flexible for different rendering modes

This architecture enables efficient real-time video mixing with flexible source types and advanced effects processing.