Video generation as a system

Our research behind Cue begins with a choice of representation: a model writes a program that renders a film. For a product demonstration, the program reproduces an interface from real references and changes it over time. This puts the model’s ability to write code to work on video generation, while making the construction available for inspection and revision.

We took that approach into real footage, recorded speech and music. The model now had to select recorded material and combine it with coded graphics and motion. The question for X-Arc was what the applied system needed around the model to make those layers work together, and what the experiment could teach us about Cue’s next research steps.

Between 9 and 13 October, we developed this workflow with Louise, our AI agent, through delegated analyses, builds and human reviews. This is an account of that system taking shape. We can trace particular failures and corrections; we did not run a controlled comparison of models or isolate every component’s contribution.

A new domain tests the surrounding system

Cue shipped as a way to make precise product demonstrations. The research behind it then examined direction, a shared clock for picture and score, and inspection of the rendered film. Our later experiment asked which context and controls remained useful as a model improved. This work continues those questions in a medium where the system authors some material and edits other material that already exists.

The new inputs were 99 clips, spoken passages, music and visual references. They brought problems that a screenshot did not: useful action inside a long take, words crossing an edit boundary, camera movement, private screen content, and sound whose effect depended on its placement. Writing the composition was only one part of the job.

Here, the model’s generated output was an executable composition. Recorded images and sound remained source material. Code specified which intervals to use, their transformations, the geometry to add and the timing of the whole. We use harness for the arrangement that lets the model do that work and receive evidence: source access, execution tools, intermediate artifacts, checks and review records. A failure could then be investigated as a choice, missing context, an execution defect or an inadequate check.

Give the model evidence it can inspect

A folder of videos was not yet a useful working context. We made lightweight copies, contact sheets, transcripts and a catalogue of usable intervals. Records described visible action, camera movement, image quality and privacy concerns, while retaining the location of the original material. The model could select a moment and return to its source instead of relying on a filename or a summary alone.

Cue’s interface elements trace to screenshots. Here, provenance applied to footage and recorded speech, while geometric metaphors were deliberately invented in code. The general principle was to distinguish facts to preserve from forms the model was free to create. Both could enter the same composition without becoming the same kind of evidence.

The reference film was also an object to investigate. We separated voice from music, examined frames around cuts and compared its pictures with the footage pool. Feature matching showed that its live-action shots came from the same collection. That finding changed the brief: those moments had already been used. A source-interval ledger made that constraint available to later builds.

These representations served different questions. A contact sheet exposed composition; a transcript exposed wording; motion measurements helped locate an action. None contained the whole experience. The useful system retained the path back to the source and the limits of each representation. Otherwise, an interpretation could quietly become the only evidence the next agent received.

Make the plan a reviewable artifact

Once the material was inspectable, the system needed a way to inspect its proposed use. Louise’s workflow developed four directions, pairing spoken passages with actual footage, motion sketches and reasons for the choices. The board connected an abstract intention to a concrete sequence before a full render.

The person

The distance

The material

The assembly

The three pillars in this brief illustrate how directing principles can guide the combination of code and real footage. They gave the model reasons for choices, in the role that narrative context plays in Cue:

• Attention: Establish a subject worth following. The opening placed a real person before the abstract account of an idea becoming a product.
• Retention: Develop one argument across successive beats. A recurring coded object let the viewer compare its changing states without learning a new visual subject each time.
• Emotion: Connect the abstraction to human experience. Hands, a keyboard and the creator’s profile gave the argument a relationship to real work.

The principles connected intent to material selection, treatment and later review. Their expression was inspectable in the film: where the person appeared, whether the object remained recognisable, and how real action related to the spoken idea. They were directing hypotheses for this example, not measured audience effects or gates that every future Cue film should satisfy.

Craftsmanship was a direction we returned to after other builds disappointed us. A sphere could represent an idea, a road the work between idea and product, and a changing shape its evolution. Real screen and keyboard footage grounded the materials. The creator’s profile from the opening became the outline containing thousands of points.

By this stage, the 14-frame board came from the actual coded timeline. We were inspecting the composition that would be rendered, with wording, source selections and geometry already present. That reduced the distance between a proposed treatment and its implementation. The board could reveal a weak visual relationship; the moving film and sound were still needed to judge timing and flow.

Give the layers a shared execution contract

Cue’s earlier research put picture and score on a shared clock. Here, that clock also joined recorded and code-generated material. Footage and speech were addressed by source interval; geometry and motion were calculated for a requested time. Crops, playback speeds, captions and music met in the same timeline.

The central operation was frame = render(time, edit, assets). Browser code assembled the picture, a renderer stepped through frames, and an audio process constructed the mix. This made a reported defect addressable: a late caption had a timestamp, a crop had coordinates, and a speech cut had source boundaries. A revision could change the decision responsible for the failure.

We checked this contract at three sampled times, requesting each frame twice in the same environment. Each pair was byte-identical. That supports a narrow claim about repeatability at those points. It does not establish identical rendering across machines, browsers or every frame.

Execution also exposed a system gap. Rendering in chunks made interrupted work resumable, but the cache checked whether a chunk file existed, not whether its inputs had changed. Reusing a directory after an edit could therefore preserve stale pictures. Separate version directories helped avoid that; safe automatic reuse would require tracking dependencies. The fact that the system produced a working renderer did not establish that every part of its execution machinery was reliable.

Verify in the medium where failure occurs

The clearest failure was in sound. We removed an example from the craftsmanship voiceover and checked the transcription. The words looked right. Listening to the export revealed roughly 0.17 seconds of the deleted phrase still present: speech recognition had omitted the fragment.

The sound beyond the intended cut

The check had answered whether the intended words were recognised. It had not answered whether unwanted sound remained. We inspected the waveform, repaired the boundary and reviewed the other joins, finding a clipped consonant and another residual ending. For Cue’s inspection research, this adds a specific requirement: recorded speech edits need signal-level evidence alongside the text that describes them.

A later rerun of the waveform checker flagged the earlier cut and passed all seven segments in the corrected specification. This is evidence that a targeted check detected these known faults. It is not a general accuracy estimate, and the checker cannot establish that every join sounds natural. Listening remains part of the review.

Another revision showed why agreement is not enough. Captions and motion shared word timings and agreed with each other, yet both lagged behind the speech because their common timing estimate was late. Checking selected onsets against the waveform gave us evidence from outside that shared error. More checking agents using the same mistaken representation would not, by itself, make the result independently verified.

For the applied system, the lesson was precise: choose the observation that can reveal the failure being tested. Read text for wording, inspect frames for layout, examine sound for speech boundaries, and review the assembled sequence for relationships that only exist over time.

Separate implementation from judgment

The system also needed a place for questions that mechanical checks could not settle. A film could render exactly as specified while its chosen representation weakened the explanation.

04 sThe idea

18 sThe idea changes

37 sThe finished form

In the craftsmanship film, one object changed state while a faint original made the change comparable. We later replaced the finished square with a diagram of an AI agent and its tools, making the image more specific to our work. After seeing it rendered, we returned to the square. The tool sequence introduced a second subject to decode; the simpler form preserved the breadth of the original point.

That correction did not call for a rule to always use abstract shapes. It called for context explaining why this representation had worked in this argument. Likewise, a request for more of the creator initially produced extra footage by quantity. Later review clarified the need: place real action where it gives the idea human meaning.

These cases locate a different kind of failure. Source access and execution were sufficient to produce the requested image. The choice still needed evaluation against the purpose of the work. The three pillars made that purpose discussable; human review supplied a judgment and its reason. A technical check could verify placement or continuity without establishing that the film was engaging or moving.

Preserve evidence across agents and sessions

The work crossed delegated analyses and resumed sessions. Its durable context lived in the catalogue, edit data, exports and decision records. These artifacts let later work recover which source interval was meant, what had actually been built, and why a previous attempt had been rejected.

Keeping the reason mattered as much as keeping the action. “Restore the square” preserves a change; the explanation that a literal tool diagram introduced a second subject preserves something a future model can reason with. Compaction should not turn a local correction into a permanent law of style.

State also needed to remain current. A music-only reel entered the confirmed backlog on 12 October and was rejected on 13 October. A later session inheriting only the approval would begin from the wrong premise. Rendered, reviewed, approved, published and measured are different facts, and a revised export should not silently inherit the judgment attached to an earlier one.

The system we would reuse therefore includes more than a prompt and a renderer. It includes source-linked observations, inspectable intermediate artifacts and decisions attached to the work they concern. That continuity supports reasoning across sessions without requiring each session to repeat the whole investigation.

Test what each layer contributes

“Good enough” needs a task and an observable criterion. For source access, can a selection be traced to the intended interval? For a treatment, do its choices trace to the evidence and form a coherent visual argument? For execution, does the requested state match the rendered state? For verification, does the check catch relevant defects without rejecting valid work? For judgment, does review explain how the artifact serves its purpose? For continuity, can the next session recover the current artifact and the reason for the last decision?

The next experiment should test one of these layers directly. For example, compare a transcript-based speech-edit check with one that also receives waveform evidence and boundary tools. Keep the model, source clips and edit requests fixed. Use a held-out set of joins containing both defects and valid cuts, with reference judgments established through listening. Measure missed defects, false alarms, successful repairs, review effort and execution cost.

This follows the audit logic of our preceding Cue study: keep task facts fixed and vary one part of the surrounding system. Repeating the comparison across models would extend that audit, testing whether the added inspection helps beyond known cases and whether its contribution changes with the model. Neither comparison was completed here. Our evidence consists of particular corrections, sampled execution checks and human judgments from iterative builds.

Several things changed together during those builds. One film shortened from 50 to 42 seconds while its words, alignment, picture structure and music also changed. The completed artifact demonstrates that the assembled system could produce it. It does not tell us how much each intervention helped, or establish a reliable rate of success on future films.

What this contributes to Cue’s research

The result for video generation is a working example of an executable composition with two kinds of material. Real footage and sound stay source-linked; coded form stays revisable; both meet on a timeline that can be inspected at the frame and signal level. The board exposes the proposed argument, while review records preserve why a representation worked or failed. This extends the research question behind Cue without claiming a new shipped capability.

It gives us concrete additions to investigate in Cue: source-interval context for recorded material, boards extracted from the actual composition, speech-boundary checks, and decisions attached to the versions they concern. The three directing pillars illustrate how a brief guides that system. Their particular visual expression belongs to this film; the reusable method is to preserve the intent, the choice and the evidence for revising it.

Some support expresses task facts that a stronger model still needs. Other support compensates for a current limitation and should be retested as models change. The research programme is to find which representation, tool or check changes the delivered result, at what cost, and under which conditions. This experiment supplies concrete cases for that work.

The result

The reviewed craftsmanship film brings the layers together: selected footage, recorded voice, coded geometry, captions and sound on an executable timeline. It is the concrete outcome of this extension of Cue’s research. The failures and corrections above explain what the surrounding system needed to produce it.

Contact

If these questions touch your work, we would love to hear from you.