BrickQuest - Hackathon Submission, Somehow Made It to the Finals

A new experiment has joined the Quest series. This time, LEGO. I entered BrickQuest in the 4th Agentic AI Hackathon with Google Cloud on Zenn, and I got lucky: it made the final Top 10, so I'm going to the finals on March 19.

Table 1

01Let's enter a hackathon

There's a platform called Zenn that a lot of developers in Japan use. It's a tech blog plus developer community, and it runs hackathons regularly. This time, Zenn hosted the 4th Agentic AI Hackathon with Google Cloud. The theme was Agentic AI: building AI agents that figure things out and act on their own.

While I was wondering what to enter, something that had been bugging me came to mind. I have two kids, 6 and 8. When we buy them a LEGO set, they build it once and the instructions vanish somewhere. Then everything starts getting mixed into one bin, and in the end it's a pile of bricks where you can't tell which set is which. Every time I looked at that bin I thought, "I wish we could build something new with this." Why not try making that? That's how it started.

02What BrickQuest does

I built two modes.

Design mode turns a photo into a LEGO design. It draws front, side and rear views, then produces a 3D assembly guide. It also generates a BrickLink parts list. Complex models still come out a bit disappointing, though.

MyBrick mode recognizes the bricks lying around your house, one by one, from photos. "2x4 red brick," "1x2 blue plate," that kind of thing. As they pile up, it suggests "what you can build with the parts you have right now," but recognition accuracy still has a long way to go.

03Trying to patch AI with a physics engine

Ask Gemini to design LEGO and the designs themselves come out pretty good. But it has no sense of space. Bricks overlap, float in midair or sit off the LEGO grid quite often.

So I put together a feedback loop of my own.

%% The Gemini and physics engine feedback loop · BrickQuest as entered
flowchart TD
  A["Gemini · coordinates and sizes via JSON Schema"] --> B["Grid snap"]
  B --> C["Sort by height"]
  C --> D["Gravity check"]
  D --> E["Collision resolve"]
  E --> F{"Too many blocks deleted?"}
  F -->|"Yes"| G["Send dropped bricks and reasons to Gemini"]
  G -->|"Regenerate · up to 3 times"| A
  F -->|"No"| H["Use the best result"]

The physics engine checks in 4 stages, from grid snap to collision resolve. If too many blocks get deleted, it sends "this brick was dropped here for this reason" back to Gemini and has it regenerate. It repeats up to 3 times and uses the best result.

The idea seems decent, but when I actually run it, it's still shaky. What I pictured in my head and what comes out are a bit different.

04Tech stack

Table 2

I used Firestore's onSnapshot so each block shows up in the 3D viewer the moment the AI makes it, and that part seems to work fairly well.

05Got lucky and made the finals

When I looked at the project list, it showed 254 projects. I made the Top 30 in the first round, passed the second round and landed in the final Top 10.

Honestly, I was really happy when the result email came. I started building the day before the deadline, so I wasn't expecting anything. I think I just got lucky. Looking at other people's projects, a lot of them are far better made, and I still don't know why mine was picked.

The judges' comments from the second round came along with it.

手持ちのレゴブロックを画像認識で分析し、そこから作れるものを提案するというアプローチが、既存のブロック組み立て体験にはない独自の価値として評価されました。リアル世界との接合によって生まれるユーザー体験の魅力を最終ピッチでお聞きしたいと思いました。

"The approach of analyzing the LEGO bricks you own with image recognition and suggesting what can be built from them was valued as something unique that existing block-building experiences don't offer. We'd like to hear in the final pitch about the appeal of the user experience that comes from connecting with the real world." That's roughly what it says. I'm grateful.

The finals are on March 19 at Google Cloud Agentic AI Summit '26 Spring. I'm not just attending. I have to get on stage and pitch. The prizes are a Grand Prize of ¥500,000 (1 team), Excellence Awards of ¥250,000 each (3 teams) and Encouragement Awards of ¥100,000 each (5 teams). It looks like 9 teams get prize money and only last place doesn't. 9 out of 10 isn't bad odds... but finishing last would sting a bit. There's also an exhibition booth for each project at the venue, so I need to make an A1 panel to put up.

The day-of schedule came in too.

Table 3

The prep deadlines are tight too. The exhibition panel is due March 9 and the pitch deck March 13. Not much time.

06What I learned

Using a system to catch what AI can't do is a pretty fun pattern. I can imagine using it for things like architectural modeling or furniture layout beyond LEGO, but that's just a thought for now, and actually doing it is a long way off.

Also: prepare for hackathons early instead of racing the deadline. I felt that one keenly this time.

07Until the finals

While I make the panel and the pitch deck (Google Slides), I also need to stabilize the demo. If it crashes live, it's over. I want to push brick recognition accuracy further too. And I need to practice the presentation. I have to pitch in Japanese, so I'm nervous.

I got lucky to make it this far. I'll polish the rough parts and give it my best. A prize would be great, and even without one it's a good experience.