Post by Master.dev (Formerly Frontend Masters)

Giving your AI agent "eyes" on a live canvas Here's a neat pattern for building AI features that need to understand visual, stateful UI, like a canvas or whiteboard. The problem: your AI agent needs to know what's currently on the canvas (shapes, positions, etc.) before it can answer questions about it or act on it. But that state lives on the frontend, not in your normal message history. The fix is simpler than it sounds. Every time the UI sends a message, it also attaches a data part containing the current canvas state. Then on the backend, you write one small utility function, extractCanvasState, that grabs the last message in the array (which is always the user's message), finds the specific data part with type "data-canvas-state," and pulls out those elements. That extracted state then gets passed straight into your streaming agent alongside the regular messages. The result: you draw a square, an arrow, and a circle on the canvas, ask the agent "tell me what you see," and it correctly responds with exactly those three elements. It's a good example of how a lot of "AI understands my app" moments come down to a small, well-placed data pipe, not some deep model magic.

Video Content