Litter to Life.

Connecting a sketch to an interactive world

Litter to Life is a collaborative Unity VR experience that connects environmental restoration with drawing-driven object creation. The interesting integration problem sits between the drawing canvas and the world: a player's sketch must become a model input, the prediction must be interpreted, and the application needs a useful response when that prediction cannot be acted on.

My contribution includes the classifier integration, prediction receiver, drawing workflow and brush-related work. The original changes are preserved within the integration commit, which provides a more useful account of my work than the repository's aggregate commit count. The broader environment and gameplay are team contributions. Contribution history.

Preprocessing is part of the interface

A drawing texture cannot simply be passed to a model without considering its representation. The player's mark can occupy a small part of the canvas, have a different aspect ratio, or use a coordinate orientation that does not match the classifier's expected input. The preprocessing path is therefore part of the application's behavior, not just a data conversion at its edge.

My classifier code finds the ink region, crops it, scales it while preserving aspect ratio, and places the result into a 28 × 28 input. It also inverts grayscale and flips vertical orientation before inference. These operations make the transformation from the canvas to the model explicit and inspectable. Preprocessing implementation.

Preserving aspect ratio avoids stretching every drawing to fill the square. Cropping reduces the influence of where the sketch happened to be placed on the canvas. Neither operation guarantees recognition: drawing style, incomplete marks and inputs outside the supported categories still create uncertainty. A production evaluation would need to measure the full canvas-to-prediction path using representative player drawings.

Local inference with a clear boundary

The Unity integration loads an ONNX model into a CPU inference worker. After preparing the input, it executes the model, converts output scores into probabilities and emits a prediction event for the rest of the application. The classifier also handles disposal of its inference resources. Classifier and resource lifecycle.

That event boundary separates numerical inference from the game-specific response. The classifier can report a label and score without deciding everything the scene should do with them. The prediction receiver and drawing UI can then apply the rules relevant to the interaction. Prediction receiver.

Local inference keeps this path inside the Unity application rather than introducing a prediction API call. It also makes device-side resource use part of the design. The source establishes a CPU execution path; it does not establish a frame-time budget or measured latency across headsets. Those would need device-specific measurements.

A prediction is not automatically an action

The receiver and UI include a configurable confidence threshold, with 0.55 as the source default. The drawing UI also distinguishes unsupported predictions and objects that have already been created. These conditions give the player feedback when a label cannot produce the intended world response. Receiver threshold, drawing feedback.

The threshold is an application rule, not proof that the model's probabilities are calibrated. A confident prediction can still be wrong, and confidence alone cannot establish that an unfamiliar drawing belongs to a supported category. Likewise, a UI message about an existing object is evidence of that feedback path, not a guarantee about every scene's object-spawning behavior.

These boundaries are where the ML feature becomes ordinary software engineering: maintain a clear input contract, manage resources, pass results through a defined interface, and give the user a coherent outcome for each supported state.

What the demonstration establishes

The recorded drawing demonstration shows the interaction and its feedback within the project. The code links above document the transformations and decision points behind it. Together, they support an integration story without turning a recording into a claim about recognition accuracy or broad hardware coverage.

My focus here is the inference and drawing integration. A model-quality claim would require an identified checkpoint, matching exported artifact, clear data split and evaluation of the actual deployment input path. Next engineering steps would include fixture-based checks for empty and off-center drawings, preprocessing parity, low-confidence handling and resource cleanup, followed by measurements on the intended hardware. The existing work makes those boundaries visible and gives each one a concrete place to test.