Dynamic Patchification in VLMs
Early vision-language architectures process visual inputs by forcing incoming images into fixed square grids. Standard Vision Transformers scale, crop, or stretch images to fit pre-defined canvas sizes. Forcing images into fixed dimensions harms model performance. Distorting aspect ratios warps spatial geometry, while downsampling destroys the fine text, small symbols, and table layouts critical for document understanding.
Instead of resizing the image, dynamic patchification splits it into a flexible grid of uniform tiles that preserve the original aspect ratio. This lets vision encoders process images at native resolutions without warping shapes or losing fine details.
The Failure of Fixed-Grid Image Resizing
Standard Vision Transformers divide input images into a fixed array of non-overlapping patch blocks. When processing an image with a rigid square constraint, an incoming visual input must fit a rigid token sequence count.
Processing a non-square document (such as an architectural blueprint, wide chart, or multi-column document scan) under a fixed square constraint forces the system into two inefficient choices:
Aspect-Ratio Skewing: Squashing a wide or tall image into a square canvas changes the geometry of spatial objects, warping text characters and blurring fine lines beyond recognition.
Padding and Subsampling: Padding the short axis with empty white space wastes compute cycles on uninformative background tokens while downscaling the long axis discards high-frequency visual details.
In document parsing, chart analysis, and fine-grained visual reasoning, downsampling leads directly to optical character recognition failure.
The Mechanics of Dynamic Tile Partitioning
Dynamic patchification frees vision models from fixed canvas sizes. Instead of stretching or shrinking an entire image to fit one rigid shape, the system looks at the original aspect ratio and breaks the image into a flexible grid of tiles.
Working within a set token budget, the pipeline picks a grid layout that preserves the image’s original shape:
Layout Selection: Chooses a grid layout (like 1x2 or 2x2) that closely matches the image's native aspect ratio.
Tile Fitting: Slightly adjusts the image boundaries so each section fits standard tile dimensions.
Patch Conversion: Splits each tile into small pixel patches and converts them into token embeddings.
Because token count scales with image size, a small icon uses only a few tokens, while a dense, high-resolution document gets the detailed sequence it needs.
Spatial Indexing via 2D Rotary Position Embeddings (2D-RoPE)
Standard 1D Rotary Position Embeddings (RoPE) work well for text but start to introduce ambiguity when used with 2D images. Flattening a grid of tiles into a linear token sequence strips away vertical alignment, treating neighboring vertical patches as if they were separated by long token distances.
2D Rotary Position Embeddings solve this spatial tracking problem by splitting positional frequency channels into decoupled vertical and horizontal coordinate axes.
For an embedding vector representing a visual patch located at specific grid coordinates, 2D-RoPE splits the channel dimensions into two equal halves:
Vertical Channels: One half encodes the vertical row coordinate.
Horizontal Channels: The second half encodes the horizontal column coordinate.
Standard rotary transformations run independently on each half. When self-attention calculates dot products between tokens, it measures exact relative distance across both axes at once. This lets the transformer track 2D spatial geometry accurately, regardless of the grid shape or aspect ratio.
Sequence Packing and Cross-Modal Attention
Once dynamic image tiles turn into visual tokens indexed by 2D-RoPE coordinates, the system packs these tokens into the context window alongside standard text tokens. Specialized boundary markers or 2D spatial attention masks determine tile transitions. The language model trunk processes the packed sequence, applying standard causal attention over text tokens while using 2D spatial attention over visual patch tokens.
Combining dynamic tile partitioning with 2D-RoPE gives vision-language models 4 key advantages:
Zero Aspect-Ratio Warping: Eliminates distortion, preserving geometrical structure in technical diagrams and natural scenes.
High-Density Text Extraction: Retains fine spatial resolution necessary for reading small document fonts, table headers, and dense code snippets.
Computational Efficiency: Allocates token budgets proportionally to visual complexity, spending minimal tokens on low-resolution images.
Flexible Resolution Scaling: Allows vision-language models to scale context windows naturally across multi-image inputs and high-resolution video frames.
Moving from rigid resizing to dynamic tiling and 2D-RoPE gives vision-language architectures a cleaner, far more efficient way to process complex visual data.