Spatial splat [WIP]

Goal: generate splats which perfectly preserve relative position/scale of objects.

Idea

As cool as it is to generate a gaussian splat from an image, you sacrifice the reliable nature of regular gaussian splatting. 3/10 times my generation scenes would get rotated wrong, get the floor height ever so slightly off, or add curvature to my straight edges. I tried 4 different ways to try and get the model to generalize the corrections. Some worked better then others.

The 4 ideas

LoRA (heavy) (using geometric primitives + gpt image)
LoRA (light) (using geometric primitives)
Controlnet (using geometric primitives)
Super low rank adaptation (attempting to isolate rotation/scale "parameters" in latent vector)

Links:

Prelim

Triposplat is weird to train on top of, as there were 2 phases in it's training.

3d model -> latent -> gaussians, which turned the original dataset of 3d models into a fixed sized latent, and learns to decode it imto gaussians. This is the internal language, where it learned to represent 3d models in a fixed size vector.
image -> latent. Predicting these latent tokens from images to give the internal representation of the 3d model.

All I care about is phase 2. Phase 1 is developing the internal representation of 3d models in latent form. Phase 2 however is more of a "world model", hallucinating up with this 3d representation from a single image.
I don't have the resources to retrain phase 1 and teach a new language, but IF the latent contains the nessecary parameters to rotate/scale/position the world, using a LoRA should be enough.

LoRA (heavy)

Curating training data for this LoRA was a 4 step process,

Build a scene of geometric primitives
Then get a real-life scene interpretation of our primitives with an image model
Now by simply feeding this interpretation into triposplat, we can check the baseline accuracy compared to the original primitives (red lines are primitives):
We optimize for correct depth from 6 key angles, comparing the splat with the original primitives. The result is a spatially-accurate splat (80% accuracy -> 96% accuracy):

Even after only ~50 examples the LoRA quickly discovered a way to correct bad rotations, check out this tree scene. Left is original, right is my LoRA. This splat was not part of training, it has never seen it before.

Triposplat	Spatialsplat LoRA

Though this is evidence that my training method does generalize to any splat, the issue is that splat quality goes down the more corrections are done. Ideally, we could reinforce consistent visuals while still correcting the geometry.