Setup

• What I want: I wanted two-tone candles with the tones separated horizontally rather than vertically. This required a two-part overmolding process: mold and make one candle, then insert it into a second mold and pour the second candle around it.

• Mold Design: I first designed the candle shape in 3D, then designed the mold using that shape. The mold consists of a cavity and a sealing pin. Once the wax sets, the sealing pin is removed and an ejector pin pushes the candle out.

• Sealing Pin: In practice, I left a bit of space and used Teflon tape around the sealing pin because a plastic-to-plastic hard seal would be less reliable and harder to remove.

• Sealing Pin Shape: Use a circular-ish sealing pin shape - complex shapes are much harder to seal with Teflon tape. It doesn’t have to be a perfect circle; the shape below worked fine.

• Ejector Pin Surface Area: The ejector pin needs as large a surface area as possible; otherwise, it just indents the wax.

• Mold Material: PLA and PETG were easier to work with. I tried TPU, but it was prone to air bubbles from moisture in the filament, which caused leaks. I could have dried the filament properly, but that seemed like more effort than it was worth.

• Print Settings: Standard print settings will leak. To get a watertight mold, I increased the wall loops to 5 and used slight over-extrusion (a flow factor of 1.05 for PLA and 1.02 for PETG).

Couple images from the process:

Other Learnings

• Wick Size: I was surprised by how much this matters - there’s more to candle making than I thought. There’s a subtle balance of wicking speed, material constituents, etc. I don’t fully understand it yet, but it was fun to get a glimpse into what’s going on behind the scenes in candle design.

• Wax Additives: A hardening agent helps with mold release, improves shrinking, and makes the candle harder and easier to work with. I added 2% stearic acid by weight to my paraffin wax. Apparently this also improves opacity, fragrance retention, etc. Not really sure about the details on this. As I said, there’s more to candles than I thought!

• Draft Angle: An 8-degree draft angle worked well for ejecting the candle. 2 degrees were too hard to eject.

• Cooling Time is Critical: Wait for the wax to cool and shrink. Ejecting after three hours resulted in high ejection force even with 8 degree draft, launching the candle halfway across the room. After waiting overnight, it just needed a gentle tap.

• Mold Release: I used a paintbrush to apply oil onto the 3D printed mold as a release agent.

• Excess Wax: Don’t throw away wax drippings - they’re useful for repairing cosmetic defects like sink holes. I used a hot air gun to drip wax onto the problem area and fix it.

• I give the tool a product idea

  • It performs market research and competitor research

  • It provides initial research into operational requirements (licensing requirements, any other operational concerns etc)

  • It generates branding (brand name, colour combination, logo, taglines)

  • It generates product concepts and images

  • It generates a landing page

  • It generates ad copy and visuals

While normally using the tool, after each stage it stops and asks for input on what to change before moving on to the next step. For now to demo the tool I’ve let it run on an example fully autonomously without my input (except 2 things - I’ll talk about those later).

I’ve added the output of the workflow below. I think the quality of the initial output is a 5/10 but I’m quite happy with this workflow as a first iteration and I think there’s easy ways to improve the quality massively:

1. Giving it feedback between each stage already improves the output quality significantly - it was an interesting experiment to run it on full autonomous mode to see the output + it made writing this blog post easier

2. I can likely also improve the prompts to give it more context and “training” (eg. feeding it a couple of articles on copywriting techniques before asking it to write copy)

3. Using specialized tools like Leonardo.ai API for specialized portions of the work like image generation (Leonardo Blueprints are amazing)

4. I also need to ask it to include more detailed citations for the the research bits (eg. competitor research) so that I can cross reference important figures later

I’m quite excited about the potential this type of workflow has on product development - being able to validate a product before building anything is going to get much much easier.

The Output

First, here is the full output: https://github.com/IcedRooibos/Mushroom-Chips-Validation-Workflow-Output-Example.

I will highlight some interesting bits below.

If you want to see the chat that led to the output, I’ve also attached a copy of the conversation at the end of the article - it’s quite cool to see it do its thing with minimal input.

Market Research and Competitor Research

• It’s tried to estimate market size (but only found global market size info which isn’t very useful in this context).

• It also found an interesting list of competitors to use as a starting point to further understand the market

Operational Considerations

It talks about import regulations and packaging requirements as well as estimates costs across different manufacturing models. I think the estimates are off but the line items being included are a good starting point to fill in better numbers.

Branding

It came up with a brand name and some logo options - I think both the brand name and logo are mediocre and this is likely where human input would have been very helpful.

Product Concepts and Images

As well as some product renders - but the branding isn’t consistent across those and the images aren’t great. This bit needs work and integration with Leonardo.ai will likely go a long way here.

Landing Page

It came up with a landing page. Link to the site: https://umamichips-bfppijdq.manus.space/

Images below in case I ever take the site down later.

Ad Copy

It came up with ad copy and images to go with those ads.

One variant here, but there’s a whole library of them - including platform specific ads eg. Facebook, Instagram, HKTVMall (local Hong Kong online shopping platform), Google, etc.

The Chat

I think it’s quite cool to see the whole chat play out so I’ve attached an image gallery here with the chat.

Human Input

After sending it the initial prompt I saw it was looking into manufacturing in Hong Kong which I knew is not a good path to go down so this stopped it and told it not to consider that as an option.

In a very human-like moment of forgetfulness it forgot to make the landing page so I reminded it to do that as well. It’s OK, AI, it’s only human to forget.

For 20 minutes and ~3.7USD worth of work this is a really interesting starting point. I am sure this will only get better.

A product validation approach I’ve been thinking about that uses AI to speed up the boring parts. The idea isn’t to remove the human from the loop—it’s to make the loop so much faster that it feels like a completely different process.

I think 10x faster/cheaper tends to have more of an impact on how you work than just “a bit faster.” When we moved from film photography to digital the workflow changed. With film you were more careful about what you shot, you had 36 shots in a roll. With digital you click as many pictures as you want and delete the ones you don’t need later. When something takes a weekend instead of a month, you’re more willing to try it. You’re more willing to throw it away if it doesn’t work. That’s the goal here. (Philosophically, I think it’s an open debate as to which approach is better but certainly the second approach wasn’t possible before so I am interested in exploring it).

I haven’t battle-tested this yet so take it with a grain of salt—but I think the approach is interesting enough to write about now and then follow up with results later after I have run through this a couple times.

The Three Phases

I’ve broken this down into three chunks: Initial Research, Validation Marketing, Iteration.

Here are the goals:

• Initial Research: This is for you to look into the idea and learn about the lay of the land and decide if you want to take this to the next step

• Validation Marketing: I call this validation marketing because the goal isn’t to sell but to learn what works and collect an audience along the way (e-mail lists, followers, etc.)

• Iteration: This is the human in the loop - take the learnings from the previous two stages and decide what to do next. Tweak the product concept? Change the ads? Start over with another product concept entirely? If done right, you now have an audience from step two to talk to about your tweaks before you start the loop again.

A Quick Note About Tools

Here are the tools I find useful and have used for items in the rest of the post below.

(Note: I gain nothing from you signing up to them, this is just to share my current workflow).

• Kagi - privacy friendly search engine

  • Their search-assisted AI is the best I have found so far. I use the Ultimate Plan as it has a special research mode.

• OpenWebUI - self hosted chatGPT UI

  • This is the “heart” of the system and how the output from a lot of these sections talk to each other. I add output from all AI models to an openWebUI folder (called a “knowledge base”) that the model can reference later

• Canva

  • Logo generation

• Leonardo.ai

  • Image and video AI models - I really like the outputs from this tool

• Huemint - colour palette generator

  • This comes up with colours palettes very quickly and I can cycle through a few before I hit something I think is a good fit for a specific idea

• Carrd / GetResponse:

  • For landing pages.

  • I have used Carrd for a long time and it’s a good tool

  • GetResponse claims to be more integrated from an analytics and email marketing perspective. It even offers basic ad management. Seems interesting enough that I am taking it for a spin.

Initial Research

The goal here is to quickly figure out:

1. Are people talking about this problem? — Find mentions of the problem or similar problems online. Forums, Reddit, Twitter, product reviews complaining about something. You want to know if real people actually care.

2. What competitor products exist? — Who’s already selling something in this space? What do their products look like? What do customers say about them?

3. What competing solutions exist for the same problem? — Sometimes the competition isn’t a direct competitor—it’s a workaround, a DIY solution, or just people tolerating the problem.

This is also a good place to add constraints that are important to you (for example “Could this go from idea to revenue in 90 days?”).

How?

• Kagi’s research assistant is great for the research phase to try and find information and put together a summary to the questions you feel are important to have answered

• Once I have done AI-assisted research I will move the collected information to OpenWebUI so that future conversations can reference this research

The output of this phase: enough context to decide whether to keep going, and enough material to inform your product positioning.

Marketing

Once you’ve decided the idea is worth testing, you need to get something in front of people. The goal is minimum viable marketing—just enough to run a credible test.

Minimum Viable Branding

You don’t need a brand agency. You need:

• A brand name (openWebUI brainstorm with AI with previous conversation context, pick one)

• A tagline (same)

• A color palette (I use Huemint)

• A logo (Canva’s AI logo generator works fine)

This takes maybe 15 minutes. It won’t win design awards but it’s good enough to pass the sniff test and get the audience to look at the product and decide if they want to demonstrate interest (eg. by signing up to your e-mail list).

Audience Definition

Who exactly are you selling to? I work through this with AI as well:

• What’s the problem? What are current alternatives? Why are they inadequate?

• Come up with 3-5 candidate customer profiles

• Pick one primary target (you should do this one, not AI)

• Write a one-sentence customer definition

• Sanity check: are they reachable? Is the market large enough? Use your advertising platform of choice’s targeting tools to check.

In reality I do this before branding because this should feed into the branding but to read a blog post it flows better this way so I have left branding first.

Asset Generation

This is where AI image/video generation is genuinely useful. You can create product visuals without having a product.

• Pull reference images from competitors or suppliers

• Use AI tools to generate product shots, carousel images, lifestyle images, with the branding you want

• Generate short videos from images if needed

I’ve been using Leonardo.ai and the quality is good enough for ads and landing pages.

It’s quite believable that AI could do marketing research and come up with some audience definition information or landing page copy. It’s possibly a bit harder to believe that it could do it well.

But asset generation seems like the kind of thing AI can’t get right. It was the missing piece for a while, in my opinion. But the models are good enough now to get what we need out of them.

As an example, if this is the input to the system:

Here’s what we can get out of it.

Video for advertisement:

Image carousel for landing page:

Copy Generation

• Landing page copy: I feed the knowledge base context to openWebUI brainstorm with Opus to write landing page copy

  • I have a skeleton structure for the landing page that I feed to the AI as well so it knows what kind of things to generate. In theory we are leaving some creativity on the table in terms of structure but a well formatted landing page is a good starting point and my hope is the speed we get out of this will make it worth it.

  • Here’s an image of the prompt I use (+ it has all the knowledge of conversations prior to this):

• Ad copy: Same process, but formatted for ads

• Ad scripts: If you’re doing video ads, get AI to draft scripts based on your audience and positioning then generate then using your tool of choice

Landing Page

I have made a template in two website builders based on the structure I shared above and can plug in the copy and assets. Nothing fancy. The point is to have somewhere to send traffic to.

In terms of tools I am currently testing both Carrd and GetResponse. I have used Carrd for a long time and it’s solid, but GetResponse claims better integration with analytics and email so I am taking it for a spin. I’m quite happy to fall back to Carrd if GetResponse causes issues. This is an implementation detail anyway and not really core to the experiment.

Iteration: Human in the Loop

At this point you’re ready to run ads and start learning. The human is still very much in the loop. You’re making decisions about what to test, how to interpret results, whether to iterate or kill the idea.

The difference is the loop is now a lot faster and cheaper to execute. You can test an idea in a day instead of a couple weeks. You can test multiple ideas in parallel. And when something doesn’t work, you haven’t sunk much into it. This also sounds like the type of process that is easy to operationalize and scale; run by a small team testing multiple ideas in parallel. Something to explore later.

What’s Next?

I’m going to run a few ideas through this process and see what happens. Will report back on what worked and what didn’t. I think there are some parts of this process that raise a few questions (how do I know if it’s a product issue or a ad targetting issue, for example?) but I am hoping as I run this playbook a bit more I can learn about what happens in the real world.

If you have thoughts or have tried something similar, I’d be curious to hear about it.

Battery powered portable lights that can sync with one another and sync to the music. I think it's cool that portable speakers have become so common that you can organize an event in fairly remote locations and have decent music. I think it would be cool if the same were true of lights.

This video below shows what the setup looks like for now - the hardware works fine but now I need to spend some time experimenting with different effects to see which ones look good together. A couple interesting effect combinations from my testing are shown below.

I also found this setup looks best in the dark but I haven't been able to figure out how to use my phone to record it without overexposing parts of the video - apologies!

Why did I make it?

The main aim was to have something simple and easy to carry to an NYE event. Initial plan was to just make one but after I made one I realized it would be way more interesting if there were a bunch of them synced up (4? 10? 1 million? Not sure).

I am quite into music and thought this would also be interesting to set up during performances just to add a bit of visual interest.

Process

For this project I wanted results quickly so I made sure to view it as an integration project rather than a development project. This made it feel like I sidestepped everything technical and so I almost don't want to write about it. Except I think making things work while avoiding the difficult bits is also an important engineering skill - so here we are.

There were still some hiccups along the way. One fun issue I had was a shipment got lost on the way so to get one unit ready in time for an event I had to use a Pi Pico solely as a voltage regulator to power an ESP32 because the Pico dev board had a wider input voltage range (unfortunately doing that type of thing is my idea of fun). In any case - the hiccups are irrelevant to the outcome so let’s focus on the final design.

Here are the different components that were used to make each light:

The Xiao ESP32 S3 (https://wiki.seeedstudio.com/xiao_esp32s3_getting_started/) is the selected dev board. The Xiao is a tiny little development board that is designed to be able to run off battery voltage levels which is very convenient.

I used an 18650 cell because they're easy to source and can be easily charged in an external charger so I don't have to bother adding fast charge into the device (though there is a backup slow charger that comes "free" with the Xiao - good to have for after the event where you can just plug in a USB C cable and let it charge overnight)

The 18650 holder has a protection circuit built in.

There's a switch that connects and disconnects battery power from the circuit for to quickly toggle power to the board.

I found WS2812 LED boards that had LEDs arranged into concentric circles and came with wires to connect them up together. Very convenient but the wires are longer than I'd like. They are rated for 5V but work just fine off battery power (saves the cost / space / current draw of having a boost converter on board)

And there's an INMP441 mic that operates off the 3V3 of the Xiao board and communicates with the ESP32 over I2S.

WLED - Open source software which has been amazing to get this up and running quickly. It allowed me to quickly experiment with different light effects and start working out what looks good together. Huge thanks to the WLED contributors. Link if you'd like to learn more: https://kno.wled.ge/

Chassis / diffuser - I created some rough designs for the housing and 3D printed them. There's a hook to be able to hang the lights as well as prevent it from rolling off if it's placed somewhere.

There's also a detachable light diffuser.

No diffusion looks cool if the brightness is turned down / if it's still daytime / or it's facing away from the viewers since the environment then does a decent job of diffusing the light. Here's the same effect with and without a diffuser.

The slicer settings used to 3D print the diffuser were quite important - different infill settings make the lights feel slightly different and for now I have settled on the infill method shown on the right in the video below. I think there's still room for improvement here.

Here's a rough connection diagram:

Another Variant

My mom liked these and wanted to take a couple to a monastery she volunteers at. I made a non-battery powered version to give to them since I was a bit concerned about these being used in the rain or thrown around and didn't want anyone to take on the associated safety risks. These are powered by USB-C and if they want it to be portable they can of course just plug it into a power bank so it's still good enough for their use case. Since I didn't need the battery capabilities of the Xiao here I went with the Waveshare ESP32 S3 which worked fine as well.

What's next?

• Custom PCB?

  • I am wondering whether I want to spend the time putting this down onto a PCB so I can make way more of these. Tempting. Right now the assembly is manual and being able to put everything onto a PCB will make things a lot simpler to assemble. I should probably do the lights and the WLED control board as two separate boards that way I can have different light attachments that can be swapped out for different events.

  • Not sure it's worth the time as I don't know if I want more of these so I will first start using the few that I have built so far and then decide whether I want more.

• Human Input

  • I think it would be very cool for me to be able to “play” the lights as an instrument by playing a 8x8 MIDI controller like the one below (for anyone unfamiliar with what’s shown below - it’s traditionally used to play music and is similar to an electric piano but in a different form factor)

I didn't really care what the model did so I chose handwriting detection as an arbitrary test case. There is a PCB with a bunch of pads that I use as touch sensors and then use my finger to draw alphabets on them and try and get the model to guess which letter I drew.

Couple things I find cool about this:

• Model was trained on PyTorch on my laptop and inference performed on an ESP32

• The way this is implemented it would be quite easy to completely change the model architecture and reconfigure it to do other things like audio recognition or object detection

• The ML related custom code-base is extremely small (~120 lines of MicroPython code and ~235 lines of C optimizations)

• Good enough for real-time inference (needs 31ms for inference)

• Has 95% validation set accuracy (as a reference point - a simpler technique like measuring distance from the mean of each category got only 31% accuracy)

Hardware

• ESP32 connected to a display

• Pads on a PCB that act as touch sensors

  • Only 13 of the 16 pads are connected due to MCU peripheral limitations - the MCU only has 13 capacitive touch input input pins

• USB for power

Data

• To collect training data I used my finger to draw out specific shapes for each letter and stored that data onto the ESP

• Each "gesture" has touch sensing data from each of the pads collected at 50ms intervals for 1.5s

• The data array is 13 sensors x 30 datapoints, each datapoint being 50ms apart.

• Since this is a 2D array of floats, the techniques that apply to image classification models will also apply here.

• To see why, let's visualize the data. Below is what one sample of each 1.5s "recording" of the input data looks like when visualized for each letter. The more green something is, the higher the capacitance i.e. the stronger the touch detection signal (larger contact area will lead to larger signal amplitude)

• Data is stored onto the ESP32 as a CSV file then transferred onto desktop for training

• I collected 120 samples of each letter which takes about 3 minutes of writing the same letter onto the pad each time (Note: Since I had to manually record the data I have only implemented detection for the alphabets A, B, C, X, Y, and Z. Doing all 26 alphabets would be around 1.5 hours of just writing alphabets onto the touchpad - not necessary for a proof of concept on a for-fun blog post)

Training - PyTorch / Laptop

• Here is the model architecture I landed on after trying a couple.

• This model has 1585 parameters in total and requires 11,287 floating point operations.

• I wouldn't be surprised if this architecture can be optimized further to have fewer parameters (reduce storage requirements) and reduce the number of floating point operations (run faster by reducing the amount of computation required)

• It got 95% validation set accuracy but in real life testing there's still plenty of room for improvement because it occasionally confuses one letter for another or is not confident enough about the letter it predicts

• I should collect more training data with more varied ways of writing the letters and that will probably improve the model. eg. B needs to be written a specific way because that's how I ended up writing it while I was recording training data.

• This model took 13 minutes to train on my laptop with an RTX4060 to get to 95% accuracy (I could get to around 90% accuracy by training for 2.5 minutes)

I won't get too into the details of how the model was trained - that's not really the focus of this article. The key thing to note is there is a model that takes 13 x 30 matrix of touch sensor data as input and can predict which of the 6 alphabets it is with 95% accuracy (in theory - in practice it is worse).

The default way to store the weights for a PyTorch models is a .pt file but since we want to write our own weights loader in micropython it will be a lot easier to save the weights as a JSON file. Luckily this is pretty easy and can be done with:

with open(f'{fname}.json', 'w') as json_file:
   json.dump(model.state_dict(), json_file,cls=EncodeTensor)

We will look at how to load these into our MicroPython model later but first - we need to actually make the MicroPython model.

Inference - MicroPython / ESP32

• To recreate the model for inference on the ESP32 here's what we need to do:

  1. Recreate the different model layers (nn.Conv2d, nn.Linear, nn.ReLU, nn.Flatten)

  2. Write a weights loader that can read the JSON weights file and load the weights into the model layers from step 1

  3. Performance optimizations to speedup inference (optional, of course)

  4. Validate results against running the model with same data on desktop to ensure same output

Creating different blocks

• I really just needed 5 types of layers - nn.Conv2d, nn.Linear, nn.ReLU, nn.Flatten, and a dummy layer that does nothing to simulate the nn.Dropout layers that don't do anything during inference and only make a difference during training.

I first wrote all the code in pure MicroPython so that I could validate my logic and find out which bits of code were slow and needed to be sped up using C.

I'm going to show how the most complicated layer (the convolution layer) works below since I want to show that even that isn't really a lot of code and is actually quite easy to implement. At the end of the article I'll include all the layers as an appendix so anyone that wants to can have a look.

For those who don't care to read the code (I get it!) here's a quick walkthrough showing you what's being done. Note that I'm going to assume you already know what a convolution is - if not, check this out (link). Alternatively for the purposes of this article you can think of it like this: a convolution takes in a 2D array as an input and applies a function to it and gives back another 2D array as an output. The function can be something like edge detection and the output can be a 2D array where all edges are white and any pixel that isn't an edge is black.

Back to the layer implementation - an actual convolution layer can have multiple input channels but I've shown a simplified case with one input channel and multiple output channels below to keep things easy to understand. With one input channel what a convolution layer does is apply n_out_channel number of kernels (functions) to the input "image". As an example - if I had a convolution layer that had 3 output channels and the channels detected left edges, top edges, and outlines then this is what that would look like.

The first layer of our model also has one input and 3 outputs but the kernels seem more complicated than just edge detection. Here is what the first convolution layer of the model I trained actually does. I could take a guess as to what it might be doing but honestly I'm not confident so let's leave that out for now. There are techniques to figure what the different layers are doing but that's probably a whole blog post in and of itself.

In code, the nn.Conv2d equivalent is:

Similarly I've implemented the other layers from my PyTorch model and you can have a look at the code in the appendix. To put all of them together I have a Model class. Let's look at how it works first then we can look at how it's implemented.

This is how you'd use the Model class.

And this is how it's implemented - pretty simple.

And that's it - we have a working model. We'll need to figure out how to load weights into this next.

Load weights

PyTorch stores the model weights as a dictionary. Here's an example:

model_filename = "model_conv_medium_2_300_epochs_bs_5.pt"
state_dict = torch.load(model_filename)
state_dict.keys()
---
output:
---
odict_keys(['0.weight', '0.bias', '2.weight', '2.bias', '4.weight', '4.bias', '7.weight', '7.bias', '9.weight', '9.bias', '11.weight', '11.bias', '13.weight', '13.bias'])
---

The weights for the model are stored in a dictionary where the keys are {layer_number}.weight or {layer_number}.bias. We save the weights as a JSON file and then write a simple weights loader.

Here's how we can extend the Model class to load the weights from a JSON file:

Load the JSON file into a dictionary and go through the layers one at a time. If weights for that layer exist, pass the weights to that layer's load_state_dict function.

Then we need to implement load_state_dict for each layer where we take in the weights and biases and store them for use during inference.

This is how it works for a linear layer:

And this is how it works for a convolution layer:

Validate ESP32 output against output from PyTorch model

It's pretty important to make sure that the model has the same outputs on an ESP32 and on PyTorch.

I wrote test scripts with dummy data for each type of layer. Example test for linear layer:

The really nice thing about using MicroPython is that I can import code that is supposed to run on my ESP32 into regular old desktop Python and execute it to validate it. That's what the from model import linearLayer line is doing - importing MicroPython code into desktop Python since it's also perfectly valid desktop Python code. The ability to develop like this makes MicroPython extremely powerful.

Similarly, I picked one datapoint from my dataset and ran it through each model I was testing to make sure the output of the model on MicroPython matches the output of the original PyTorch model for that datapoint.

These tests were especially helpful later when I was writing C optimisations - I just had to rerun the tests to make sure I didn't break anything.

Output from benchmark_model (not the same as our final inference model) used during development that was run on PyTorch:

tensor([-309.3656,   66.2828,  -38.7419,  -72.0498], grad_fn=<AddBackward0>)

Output from benchmark_model used during development that was run on ESP32:

[-309.3655, 66.28277, -38.74186, -72.04981]

For anyone wondering why the outputs don't match exactly eg. -309.3656 vs -309.3655 - this is because of small errors that are inherent to floating point calculations on GPUs partially due to limitations of floating point numbers and partially due GPUs preferring speed over absolute accuracy so they make optimisations that get "close enough" return values but get them faster.

Here's an example where desktop Python gets a floating point calculation slightly wrong:

>>> 0.7+0.6
1.2999999999999998

Correct answer is:

1.3

So the answer is off by 0.0000000000000002.

But interestingly enough if you do the same calculation but break it up, you get the "correct" answer:

>>> 0.2+0.2+0.2+0.7
1.3
# Even this isn't entirely correct - the actual value in floating point notation is 5854679515581645 / 4503599627370496 which translates to around 1.300000000000000044408920985 - but Python truncates it to 1.3 for display purposes

This is because in floating point operations, the order of operations matters because each number is just an approximate representation of the true value. As you stack more calculations up the errors get larger.

Does it matter? These types of errors can cause issues during training but not so much during inference so for now we can safely ignore it.

Performance optimization

Inference on the benchmark model I was initially using for testing took 194ms. Is that good enough? Maybe. Depends on the use case. But being faster would make this a lot more useful across other embedded use cases and honestly optimization is just fun. Perks of this being on my blog rather than in a product is I can do it just because I want to.

For performance optimization you definitely want to start with a target. Mine was to have inference complete in less than 50ms because that means that you could run inference realtime if you wanted since you collect data every 50ms.

There are three approaches to speeding up inference:

• Rewrite specific function calls in C.

  • eg. the Convolution2D function that convolves a single 2D matrix is a great candidate to be rewritten in C because that has a bunch of loops and rewriting it in C still allows us to retain all of the flexibility of python since we can continue to restructure our model while just calling the function as if it were any other Python function.

  • There are a few other functions like this that I'll talk about later in the article as well as the exact impact they had on execution time.

  • The rewritten C functions were compiled as a native module called "mlops" which is a collection of operations useful for machine learning applications. A native module is essentially just a library but compiled into a binary rather than Python code. I upload the "mlops.mpy" library onto the ESP32 like any other MicroPython file and can use it as a module by importing it like any other module with import mlops

• The second approach is to do inference entirely in C and just call the model from MicroPython eg. c_model.predict(data). This would definitely be the fastest but least flexible - I'd have to rewrite and recompile the C module each time I change model architecture.

• The third approach is to apply application specific optimizations'. eg. if I was to predict gestures each time a datapoint is collected (every 50ms) then only the last entry of the 1.5s long data recording would change so I just need to update calculations that depend on the last row.

For my use case (a proof of concept on a for-fun blog) I felt the first type of optimization was good enough. I can just predict once every 1.5s and it still works fine.

So now we need to rewrite specific Python functions in C. Let's look at rewriting the the ReLU function in C. As a quick refresher ReLU is a function that replaces all negative elements of a matrix with 0 and leaves the positive elements untouched.

The original Python version is:

The C optimized forward function is below.

And after we compile that and load the new mlops.mpy library, we rewrite the python class as shown below. This is why I love this approach - it's a really happy medium between having the development speed of Python and the runtime execution speed of C.

Here is a list of performance optimizations. All measurements below were taken at 160MHz clock speed because that's what I started off with and later figured I'll consider the speed increase from the switch to 240MHz during inference as "margin of error" in case things go wrong. I'm glad I did that since I later had to increase model size and therefore it took longer to compute but I was still under my 50ms criteria.

Here are the optimizations and then a chart showing how much time each optimization saved.

• 194ms was the original inference time

• 138ms after using the native code emitter in micropython (out of which 56ms/40% of the time is spent in the Convolution2D function)

• 129ms after specific small performance optimizations (eg. cache values in loops, flatten in C)

• 74ms after Convolution2D implemented in C

• 66ms after ReLU forward call implemented in C

• 49ms after linearLayer matrix multiplication implemented in C

• 32ms after implementing a matrix summation function in C

• 21ms after changing the input normalization function to C

These optimizations brought inference time down from 194ms to 23ms @ 160MHz. Inference time was 15ms @ 240MHz.

After I switched to a bigger model inference took 31ms @ 240MHz. If you call this once every 1.5s that's just using 2% of one of the two cores on the ESP32. Plenty of cycles left to do lots of other stuff.

Side Notes

Here's a few small asides.

How many samples are needed?

I looked into whether I really needed 120 samples for each gesture and it does seem like it's roughly correct. Here is model accuracy vs number of samples. These were trained for 2 minutes (50 epochs with one cycle LR scheduler) rather than 13 minutes.

It looks like 75 samples may be enough too but went with higher just to be safe - 96 training samples and 24 validation samples (120 total).

Note: I don't think the slight drop from 75 to 100 is significant, with more runs to sample I feel like it would likely cancel out. From a decision making perspective - 75 or 100 both are probably fine and 100 is safer.

Using coins while I was waiting for the PCB to arrive

After I had the idea to try this and before I received the touch pad PCBs I sent to get made I wanted to carry on with the firmware and software development so I soldered a bunch of coins and put them in a 3D printed casing to start working on the software while I wait for the hardware. It was too big and didn't feel good to "write" on but definitely fun to use coins as touch sensors!

Confidence Threshold

• On the ESP32 after the model makes a prediction I only allow it to update the display if it is at least 75% confident that the answer is correct. This does two things:

  • Prevents it from interpreting ambiguous signals like my palm resting on the touchpad as an alphabet

  • If it sees an alphabet drawn in a way that confuses the model, it prevents the model from forcing a guess - I found it was often wrong with these types of guesses. It must be quite confident about the prediction to display it on the screen.

Libraries

I tried to look at other libraries like ndarray or upytorch but none of them could do what I needed yet:

• uPyTorch doesn't have inference yet

• ndarray actually made things slower - code that was pure MicroPython with native emitter that took 138ms to run without ndarray took 179ms after using ndarray.

Appendix - All the other layers

Earlier in the article I mentioned I would show the code for all the other layers - so I've put the code down here along with their C equivalents.

Linear Layer

Linear Layer with C Optimization

ReLU Layer

ReLU Layer with C Optimization

Dummy Layer

Flatten Layer

Flatten Layer with C Optimization

Input Normalization

Input Normalization with C Optimization

Conv2D Layer

Conv2D Layer with C optimization

There are two things that could be making code run slow:

1. Waiting for external input like an HTTP request or GPIO button press

2. Lots of computations

To speed up code in scenario #1, you can use asyncio to run code concurrently. There's lots of information about this online so there's no need for me to write about it.

Here I'm looking at scenario #2 where there's lots of computation going on. Just in case anyone wants to follow along I am using an ESP-WROOM-32 development board to run this code.

As a spoiler, I tested 3 different methods to improve performance and here are the results I got:

I also provide the complete source towards the end in case you'd like to check the source for more information.

Baseline Function

Let's set up a baseline first.

Here's a function that tests whether a number is prime or not (source):

def testPrime(num): 
    if num == 1: 
        # 1 is defined as not prime 
        return False 
         
    elif num > 1: 
        # check for factors 
        for i in range(2, num): 
            if (num % i) == 0: 
                # if factor is found, this is not a prime number 
                return False 
        # No factors found - number is prime 
        return True

Let's benchmark how fast it runs on MicroPython.

Here's the test code - we measure how many CPU clock cycles it takes for the function to run and convert it to a milliseconds estimate based on clock frequency.

def compareRunTime(num): 
    print("------------") 
    print(f"TESTING whether {num} is prime") 
    print("------------") 
    start_time = time.ticks_cpu() 
    output1 = testPrime(num) 
    end_time = time.ticks_cpu() 
    regular = time.ticks_diff(end_time, start_time) # How many clock cycles did it take to run the function 
    time_ms = regular/cpu_frequency*1000 
    print(f"Regular script took {time_ms} ms / {regular} clock cycles")

compareRunTime(71569) # This is a prime number

Here's the output:

------------ 
TESTING whether 71569 is prime 
------------ 
Regular script took 487.2375 ms / 77958003 clock cycles

So our baseline time to check a 5 digit number is 487ms.

Native Code Emitter

Next we are going to use Python's native code emitter - documentation can be found here. All we need to do is add a decorator. What this does is converts the code to machine op-codes rather than using Python bytecode as an intermediary which then needs to be converted to machine op-codes (which is what happened when we ran our baseline example). The trade-off here is larger compiled code size which shouldn't really be an issue if you only use this in the performance-critical sections of code.

@micropython.native 
def testPrimeNative(num):
# No changes to anything below this
    if num == 1: 
        # 1 is defined as not prime 
        return False 
         
    elif num > 1: 
        # check for factors 
        for i in range(2, num): 
            if (num % i) == 0: 
                # if factor is found, this is not a prime number 
                return False 
        # No factors found - number is prime 
        return True

Making code run with the native emitter is extremely easy - lots of code should run fine without any changes and the only features that can't be used are context managers and generators (so no async code either since that runs as a generator under the hood).

Here's the performance comparison code in case you want to see it.

def compareRunTime(num): 
    print("------------") 
    print(f"TESTING whether {num} is prime") 
    print("------------") 
    start_time = time.ticks_cpu() 
    output1 = testPrime(num) 
    end_time = time.ticks_cpu() 
    regular = time.ticks_diff(end_time, start_time) 
    time_ms = regular/cpu_frequency*1000 
    print(f"Regular script took {time_ms} ms / {regular} clock cycles") 
    start_time = time.ticks_cpu() 
    output2 = testPrimeNative(num) 
    end_time = time.ticks_cpu() 
    native = time.ticks_diff(end_time, start_time) 
    time_ms = native/cpu_frequency*1000 
    print(f"Native script took {time_ms} ms / {native} clock cycles")

And here are the results:

------------ 
TESTING whether 71569 is prime 
------------ 
Regular script took 487.2414 ms / 77958627 clock cycles 
Native script took 203.7882 ms / 32606107 clock cycles

So that's over twice as fast and with essentially no changes to the code.

Viper Code Emitter

Next we can use the viper code emitter. This produces machine op-codes similar to the native emitter but it optimises things further. To do that it needs to know the variable types and you will need to follow a few other restrictions (link). You will have to write your code to work specifically with the viper emitter and it may not run on the regular desktop version of CPython.

Here is the code:

@micropython.viper 
def testPrimeViper(num:int) -> bool: 
    if num == 1: 
        # 1 is defined as not prime 
        return False 
         
    elif num > 1: 
        # check for factors 
        for i in range(2, num): 
            if (num % i) == 0: 
                # if factor is found, this is not a prime number 
                return False

Here are the results:

------------ 
TESTING whether 71569 is prime 
------------ 
Regular script took 487.2413 ms / 77958605 clock cycles 
Native script took 203.7885 ms / 32606159 clock cycles 
Viper script took 30.02811 ms / 4804497 clock cycles

That's a lot faster - we are already running 16x faster than the initial MicroPython code and we didn't really have to do too much work.

Micropython C module

There's one last thing we can do - we can write our function in C and import it into MicroPython and use that to run the calculation. The nice thing about this method is most of the firmware can still be written in MicroPython and you can use all the nice high-level Python features for most of the firmware. You only write the performance-critical sections of code in C.

Once you have a compiled C module it's pretty simple to use in MicroPython:

import test_prime_c 
is_prime = test_prime_c.test_prime(71569)

Compiling it takes a bit more effort, however. You need to set up the toolchain and then write your C module. Setting up the tool-chain shouldn't take more than 2 hours. As for writing the C module - the simple one here took about 5 minutes but something more complex could take considerably longer. I'd recommend using the MicroPython examples (link) and modifying them to suit your needs. Make sure you can compile and import their example module before you try and make your own.

Here is what the C code looks like (and here is the documentation to start building your own C modules - link).

test_prime_c.c shown below.

// Include the header file to get access to the MicroPython API
#include "py/dynruntime.h"

// Helper function to check whether a number is prime
STATIC mp_int_t prime_helper(mp_int_t x) {
    if (x == 1){
	// 1 is not prime
        return 0;
    }
	for (int i = 2; i<x; i++){
		if(x%i == 0){
			// factor found - not a prime number
			return 0;
		}
	}
	return 1; // if none of the above conditions determined a factor, then this is a prime
}

// This is the function which will be called from Python, as test_prime(x). We pass it a micropython object (everything is an object in python - int is an object of type int)
STATIC mp_obj_t test_prime(mp_obj_t x_obj) {
    // Get the C compatible integer from the MicroPython integer object
    mp_int_t x = mp_obj_get_int(x_obj);
    // Check whether it is prime
    mp_int_t result = prime_helper(x);
    // Convert the result to a MicroPython boolean object and return it
    return mp_obj_new_bool(result);
}

// Define a Python reference to the function above
// so that you can call it through test_prime_c.test_prime
STATIC MP_DEFINE_CONST_FUN_OBJ_1(test_prime_obj, test_prime);

// This is the entry point and is called when the module is imported
mp_obj_t mpy_init(mp_obj_fun_bc_t *self, size_t n_args, size_t n_kw, mp_obj_t *args) {
    // This must be first, it sets up the globals dict and other things
    MP_DYNRUNTIME_INIT_ENTRY

    // Make the function available in the module's namespace
    mp_store_global(MP_QSTR_test_prime, MP_OBJ_FROM_PTR(&test_prime_obj));

    // This must be last, it restores the globals dict
    MP_DYNRUNTIME_INIT_EXIT
}

We compile that for our target MCU and it will generate a test_prime_c.mpy  file that we upload onto our MCU. We can then import the module into MicroPython and start using it like any other module. Here's a copy of the module I compiled for the ESP32: link.

Here is the performance benchmark. It’s also printing the results to make sure that all of them return the same result.

------------ 
TESTING whether 71569 is prime 
------------ 
Regular script took 487.2414 ms / 77958627 clock cycles 
Native script took 203.7882 ms / 32606107 clock cycles 
Viper script took 30.02839 ms / 4804543 clock cycles 
Compiled C module took 4.476794 ms / 716287 clock cycles 
------------ 
RESULTS 
------------ 
True True True True 
------------

108.8x faster than the baseline - not bad!

Let's check with a larger number and a smaller number to see how the performance compares for larger and smaller workloads.

compareRunTime(313) # This is a prime number - should be quick to calculate

------------ 
TESTING whether 313 is prime 
------------ 
Regular script took 2.238606 ms / 358177 clock cycles 
Native script took 0.9453438 ms / 151255 clock cycles 
Viper script took 0.1852 ms / 29632 clock cycles 
Compiled C module took 0.03605625 ms / 5769 clock cycles 
------------ 
RESULTS 
------------ 
True True True True 
------------

compareLongRunTime(4951277) # This is a prime number - should take much longer to calculate

------------ 
TESTING whether 4951277 is prime 
------------ 
Regular script took 34 s 
Native script took 14 s 
Viper script took 2 s 
Compiled C module took 0s / 305.7551 ms  
------------ 
RESULTS 
------------ 
True True True True 
------------

With a small workload we see a 62x improvement and with a large workload we see a 112x improvement.

Conclusion

In general I would try and stick to the native emitter or viper emitter because they are decent enough improvements but still quite quick and easy to implement.

There are also some other things you can do such as caching references to global objects that would make things faster without having to move to C (link).

However if there are some specific performance critical sections that are too slow then C is worth a shot. I haven't tried accessing hardware through C just yet so that should make an interesting experiment in the future.

Here are the results of the above tests in graph form.

In case anyone needs these for reference, here are source files of the micropython test script, the compiled C module, and C module source file:

test_prime_script.py

test_prime_c.c (uncompiled C code that you could compile)

test_prime_c.mpy (compiled C module that you can import onto an ESP-WROOM-32)

I wanted to make a robot that can play music.  This is what the robot can do so far - there's plenty of issues that need to be ironed out.

Because this is not a trivial project, it's got a long way to go before it's complete.  However, given how long this post has gotten already it made sense to document the progress so far.

The goal

• My goal with this project was to make a cheap robot that was capable of playing the piano.

• I wanted to be able to place it anywhere and have it walk to to the piano and then play it

• I definitely did not want to have a robot with 50 fingers where I just install the robot onto the piano and each finger plays one key - instead I wanted the robot to move it's arm around to play music

• I wanted to the option to operate these in a swarm later (i.e. many robots coordinate to play music).

The early choices

• I wanted this to be cheap enough to build a swarm later so I knew I had to minimize the number of servos and use generic/cheap drivers

• I also decided early on that there would be a "big brain + little brain" control system.

  • The little brain would take instructions from the big brain about when and where to move it's arms

  • The big brain would detect the location of the piano and the robot through an overhead camera and command to robot to move to it

  • The big brain would then decide what music to play (notes + rhythm) and then tell the little brain how to move it's arms to play that music

  • The big brain can later be extended to control multiple robots at the same time

  • The big brain and little brain would communicate over WiFi

This makes the two systems somewhat independent - I just need to write motion control firmware for the robot to allow it to take commands from the big brain and move it's arms around. All the knowledge about what a piano is, what it looks like, how to use it to play music, etc would reside on the big brain.

However for this to work the the big brain needs to know the position of the robot and the piano somewhat precisely. For this I used Aruco markers - I placed these QR code type labels on the robot and the piano which then allowed me to track their position.

To minimize the error in piano location, I added two codes onto the piano far apart from each other. I've only added one Aruco marker onto the robot and it seems to work fine.

The reason two QR codes is better than one is because we have a much longer distance to calibrate the camera. If we use just one 12mm large code our calibration distance is ~25 pixels but if we use two codes far apart then we get a 180pixel calibration distance and so the impact of having a one or two pixel error is much smaller.

Separately, I also wanted to have a digital simulation of the robot. I mostly wanted this for fun but it did end up being useful when I was trying to debug the motion control firmware - more on that later.

The setup

It turns out I didn't need to design a robot from scratch. There was a robot available for purchase that matched my requirements quite well - cheap, battery powered, and I could upload custom firmware for it on it’s ESP8266 controller. This means that I didn't really have too much to do mechanically apart from the fact that this robot came with incorrect and outdated documentation so I had to figure out how a bag of parts goes together to make a robot.

Camera - at the beginning I used a Logitech C270 webcam since I had one lying around. I later switched to a CMOS camera with a lens to make the position tracking more precise. The CMOS camera was pretty affordable too - about ~40USD including the lens. Neither of them are particularly high end but you can definitely see the difference in detail between the two cameras. The CMOS camera had significant lens distortion which I accounted for through camera calibration.

Piano - an Akai MPK Mini 2 that I had lying around.

Big brain - Python running on my computer

Connectivity - The robot and the big brain communicate over my home WiFi network.

Here’s a diagram showing how the setup works:

Robot Simulator

• There are a couple options for robot simulators but I picked Webots

• I set things up so that a lot of the firmware I write for the real robot can be uploaded onto the digitally simulated robot with just a few modifications, this made it pretty useful for debugging firmware issues.

One nice benefit of having a simulation was: if I ran into an issue and I wasn't sure if it was a software or hardware issue, I could just run the same code on the simulated robot. If the issue exists on the real robot and not on the simulation then the issue is probably mechanical or electrical. If the issue exists on both then it's more likely to be software related.

For example I had an issue where the path the robot followed while walking was curved rather than straight. Since I didn't see the issue on the digital simulation it seemed likely it was a physical issue. And it turned out it was - one of the legs was mis-aligned by around 10 degrees and that caused the robot to turn a little bit after each step leading to the curved path.

Robot Firmware

I ended up writing custom firmware for the robot.

• It was programmed as a state machine with 7 states

  • Standby - do nothing

  • Walk forward - continuously walk forward

  • Turn left - continuously rotate left

  • Turn right - continuously rotate right

  • Close stance walk - this is a type of walk where it's front legs are closer to the robot chassis. This is useful because it allows the robot to get closer to the piano.

  • Close stance standby - same as above, front legs are closer to the robot chassis but it just stands there waiting

  • Music mode - I'll explain this in a minute

Close Stance vs Regular Stance

Here's an image that shows the difference between the close stance and regular stance states. The close stance allows the robot to get much closer to the piano and allows it to reach a lot more keys compared to the regular stance where it would just reach 1 to 2 keys.

Music Mode

For the robot to play music, it had to have somewhat accurate timing for motion control. If there is supposed to be a 0.25 second gap between two notes that are played then the second note does actually need to be played somewhat close to 0.25 seconds after the first. But the start time of the song can be offset by as much as 5 seconds and it wouldn't be a problem.

Music mode is a way of loading up commands into the robot's queue so that it can focus on playing music "live".

The big brain will translate high level commands like "play the note C4" into about 5 to 10 more specific movements such as "lift your left hand by 30° then turn your left arm by 38°" and so on. If we try to issue these commands to the robot exactly at the time the command needs to be executed, WiFi latency + program execution latency become an issue. Both of them are variable enough that it would lead to timing issues where we can't predict exactly when the note will play even if we try and account for the latency. This means we need to queue up these commands for the robot to execute later, when the exact time comes. This means the commands the big brain actually issues to the little brain is closer to "lift your left hand by 30° in 300 milliseconds then turn your left arm by 38° in 650ms".

And that's what music mode does - commands can be queued up at any time and once the robot switches to music mode it will start executing those commands sequentially.

One thing to note here is both the big brain and small brain are keeping track of time. They do this by counting the number of milliseconds that have passed from the moment they get switched on. However because they won't be switched on at the exact same time their internal clocks will not be the same. This means the big brain and little brain need to sync-up about what time it is. When the big brain first connects to the little brain it asks the little brain for the time and keeps track of it. Instead of saying "lift your arm in 300ms", the big brain knows the current time according to the little brain's internal clock is, for example, 11066 milliseconds. So it sends the command "lift your arm at 11366ms". It's done this way to minimize timing errors due to latency variation.

Big Brain

The big brain has three main purposes:

• It is a GUI for me to interact with the robot

• It commands the robot to walk it to the piano

• Once at the piano it generates some music and commands the robot to play it

Let's break the three of those down:

GUI

It is a GUI for me to interact with the robot

This one is pretty simple - I built a GUI to make it easier for me to interact with the robot. It allows me to set the robot to a specific state using buttons or allows me to issue high level commands like "GoToKeyboard" after which the big brain will start directing the robot to the keyboard.

Walk to piano

It commands the robot to walk it to the piano

• The big brain knows where the piano is because of the overhead camera.

  • It uses the camera to locate the two aruco codes on the piano

  • It also knows the dimensions of the piano and exact location of the codes stuck on to the piano.

• It also knows where the robot is because of the aruco code on the robot.

• It keeps tracking both the robot and piano in real time. It issues commands to the robot such as "turn left, walk straight", etc to direct the robot to the piano

• There was also some optimization to increase the accuracy of keyboard and spider position tracking by filtering the noisy data

Music and Commands

Once at the piano it generates some music and instructs the robot on how to play it.

• It checks which keys are within reach for the robot through inverse kinematics i.e. it tries to calculate how to hit the each key and if no solution exists - it is not within reach.

• It generates music to be played based on the keys that are within reach i.e. what note should be played at what time (the music it generates is extremely basic at the moment)

• It converts that music into motion through inverse kinematics. For example - to play note C4 the left arm will have to be move to 30 degrees from resting position and the left finger will need to move down by 15 degrees

• It does extremely basic collision avoidance eg. it knows to lift up the outer arms sufficiently before moving over the keyboard so as to not crash into it

• It then streams those commands to the robot to queue them up

Current state + Issues

This is where things are at now:

Plenty of issues that need to be worked on:

• It sometimes misses the keys and ends up pressing in between two keys

  • Potentially need to improve the inverse kinematics algorithm or improve the computer vision computations in the big-brain

• It often aims for the correct key but doesn't actually trigger the key press because it either doesn't press down enough or aims too high up on a key where it needs too much force to trigger the key.

The reason for this is there is just a 1mm margin where they key press is triggered. If the arm stops above that then the note won't be triggered. However if it pushes the key below more than 1mm below the trigger point, the key will bottom out and the robot will be pushed away. This means all future key presses will be based on incorrect robot position calculations. Right now we see both these issues, sometimes it presses the key too lightly and sometimes it presses it too hard and moves the spider away a little bit. Those small position changes add up over time.

One way to resolve this might be to use a somewhat flexible end effector that can move down and bend away once it hits the bottom of the keystroke.

This would also allow me to remove the weights that are currently on the robot since they only exist to reduce how much the robot moves away after it hits the bottom of the stroke.

• Another issue is the motion doesn't look natural - this doesn't really affect the functionality but small changes to the software might make it seem more natural and fluid. This could be things like:

  • allowing the arms to rest down when it is not going to play a note for a couple of seconds

  • adding slight movement of the arm between notes

All of this is going to be a lot of work to fine tune and it's probably a bit more than I should do for a blog but if I can find the time I will keep making small changes. Let's see how it plays out!

How it works

• An ESP32 microcontroller is used to control the LEDs.

• The microcontroller is connected to the internet through my phone hotspot.

• The microcontroller will constantly check the communication server for messages. Once it gets a message it interprets the command to figure out what, if anything, it should do.

• On the webapp, I have buttons set up for a couple different effects and pressing any of these will send a message to the MQTT server that tells the microcontroller what to do.

• Anyone can go to the webapp (which could be hosted on any domain eg. spookyhalloween.com) and control the lights
  
  Because I am a lot more familiar with Python than Javascript, I felt it was easiest to do the heavy lifting through Python so what the webapp really does is calls a postman script that is written in Python and tells it what message to send to the communication server. I happened to use Google Cloud Functions as my "server" for the postman script.

I could definitely have done this without the postman script but it was much quicker for me to do it this way and because this was just a Halloween costume - quick and easy was the way to go.

Demo

Here's a longer demo of the costume:

Next I just add on the light diffusers and we are good to go.

What didn't work

1. The way you would do this if you could just connect to your home WiFi would be much easier. You wouldn't need an MQTT server - you can just enable port-forwarding on your router and send the HTTP requests directly from the website to your microcontroller. Unfortunately because I had to connect to a phone hotspot this isn't possible. Network providers block incoming requests so any requests sent from the website to the microcontroller over cellular would just by dropped by the network provider. You can buy Machine-To-Machine (M2M) sim cards but that is both overkill and unnecessary for a Halloween costume so mobile hotspot + MQTT it is!

2. I also tried to use aREST which is a service that would have made this pretty turnkey but their documentation is sparse and their server seems down.

3. As a side note, one nice thing about this project is I didn't really buy (and waste) any parts, just used parts parts that I had lying around. After Halloween, those parts went back into my bin to be used for other projects. This is why I'm using plain old LEDs rather than using WS2812 I2C controlled LEDs where I would have much fewer wires to connect.

In this post we will:

1. Fix the leakage issue

2. Design a basic microfluidic droplet generator (video shown below)

3. Benchmark the performance of the droplet generator to have something to compare future improvements to

Solving the leakage issue

First let's talk about leakage.

Previously, liquid would sometimes leak out from between the two layers of the circuit like shown here.

Adding a screw near each corner and using a nut to clamp down the circuit improved the sealing. I haven't seen much leakage since.

Now let's use this to make a droplet generator.

Droplet Generator Circuit Overview

If you haven't read the previous post - a droplet generator will take two fluids, eg. oil and water, and create small droplets of water inside a stream of oil. Example video shown here (credit: Churchman, Adam H. (2018) doi:10.5518/153).

This circuit has three input ports - two are used for oil and one for colored water. We use a syringe to provide suction and pull the liquids through to the output port.

Next let's create a an internal fluid channel design that actually works. Because prototyping these is really fast, we can just come up with a basic design and iterate over it.

Channel Designs

Channel Design #1

Design and Expected Behavior

The first design + how I expected it to behave is illustrated below. I expected the two streams of oil would break the free-flowing stream of red water into droplets around the constriction.

Test Outcome

In reality - the three streams just flow side-by-side.

Adding red dye to the water makes it thicker (aka more viscous). My theory is this increased viscosity makes it harder to break down into droplets.

Channel Design #2

Design and Expected Behavior

I thought increasing the flow rate of oil would force the water to break down into droplets. So I made the oil channels wider.

Test Outcome

Close - but it turns out the droplets formed before the constriction. The constriction then forced them to combine back into a larger stream of liquid.

Channel Design #3

Design and Expected Behavior

So I just removed the constriction.

Test Outcome

This made the water and 2 oil streams flow side by side.

Note: You’ll see little drops of red water coming in from the oil input ports – this is just because I was also testing whether this could generate oil droplets in a water stream and there was residual water in the oil input ports.

However it turns out if we replace water with air, this circuit can form droplets.

My theory is the oil stream still wasn't able to break the water into droplets because of the increased viscosity of water after being mixed with dye.

Channel Design #4

Design and Expected Behavior

I added a constriction before the intersection to force the water stream to break up into droplets.

Test Outcome

Which ended up working.

Performance Benchmarking

This isn't a great droplet generator because:

1. Droplet sizes are inconsistent

2. The droplets are generated quite slowly

This may be because I pull on the syringe manually. This causes the suction pressure to fluctuate - which could be responsible for the inconsistent droplet size.

But let's add some performance benchmarks anyway so that we have something to compare to in the future.

1. Droplet generation rate (Hz): How many droplets are formed every second?

2. Droplet volume (µL): How many microliters is each droplet?

Let's start with the results and then I will discuss the measurement method. After that I also talk about about three weird little artifacts you might have noticed while looking at the videos so far.

Droplet Volume
Roughly 0.05µL
1 tablespoon of water would create ~300,000 droplets

Droplet Generation Rate
Roughly 0.8Hz
It would take ~4.3 days to convert 1 tablespoon of water into 300,000 droplets

Let's talk about measurement methods.

Droplet Generation Rate

To measure droplet generation rate, we can add a red line and count the time it takes for a droplet to cross the line.

The droplet generation frequency is roughly 0.8Hz (i.e. 1 droplet formed every ~1.25 seconds)

Droplet Volume

First we need to know the size of one pixel in millimeters. We use a calibration ruler like shown below - 10mm is about 852.53 pixels.

Then we calculate the area (in mm2) of the entire region the camera sees.
(let's ignore perspective distortion to keep things simple)

Then we need to count how many pixels make up one droplet. This particular droplet occupies 1586 pixels

After measuring the depth of the channel (0.26mm), combine all the data to calculate the final volume of the droplet

Unfortunately the droplet volume isn't very consistent - many are around the ~0.06µL mark with the occasional ~0.02µL droplet.

Replacing the syringe with a more constant suction source might help improve the droplet volume consistency. Or maybe increasing the suction pressure would prevent the formation of the occasional smaller droplet.

Minor Artifacts

Three interesting artifacts were seen during the experiment.

Droplet Splitting

The smaller ~0.02µL droplet is usually formed because an additional small droplet breaks off from the main fluid stream while forming the "normal" droplet. A potential fix to this might be just increasing the suction pressure.

Air Bubble

On the final droplet generator you can see an air bubble enter the flow stream.

With the slowed down video shown below you can see the air bubble enter from between the two layers of the circuit.

Some theories:

1. Leakage: Maybe a nut wasn't clamped tight enough or the thermal processing had an issue which caused bonding issues between the two layers

2. Trapped Air: Maybe there was a residual packet of air trapped between the layers during fabrication and it was dragged into the flow stream

Flow Splitting

For channel design #2, the droplet's were merging at the constriction. However after the constriction they split into separate streams around the edges of the channel. This might be because the channel is deeper around the edges - but that needs to be confirmed through testing.

Someone showed me a science experiment I had never seen before. The setup is: you have a bowl with water in it and you grind some pepper onto the surface. Watch what happens when you add a little bit of soap onto the surface of the water:

What causes this to happen?

When soap is introduced onto the surface of water, it will quickly start spreading over the surface. The macro view of why this happens is that the soap film reduces the surface tension of the water and puts it into a lower energy state. Things prefer to be at a lower energy state the same way a ball at the top of a hill will roll downhill.

The micro view is that soap molecules have both a hydrophilic end and a hydrophobic end. The way this interacts with the water is what causes the soap film to spread on the water surface.

Here’s a more detailed explanation for anyone interested:

Can this be used to move objects?

After looking at the initial experiment, I wondered if this could be used to propel an object forward. As it turns out, it can:

But this still feels like cheating because the soap propels the object forward and I put it there using my hand. The easy solution is to add soap onto the "boat" and use a wick to release the soap into the water slowly.

Controlling the motion

This is getting better, but there's no way to control when the boat moves forward.

It would be really cool if we had electronics that dipped soap onto the water only when we wanted it to move. First though, would this even move if it had the weight of electronics on it?

The test above shows that 15gms is a reasonable weight limit for a 8cm x 8cm square of foam.

Next we need to whip up some electronics.

To communicate with the micro-controller I would normally pick Bluetooth, Wifi, or USB. But because of the weight limit a WiFi module would be too heavy. Instead I went with a light sensor. I shine a laser onto the light sensor when I want the boat to move. I've shown an example video of an older prototype below. This one doesn't have soap and so doesn't move.

For the actuator I used a piezo-vibrator because it was the lightest moving thing I could find. Because vibrators aren't really designed for this use case, I had to try a bunch of different actuator combinations for this to work. The final set-up looked like this (the boat doesn't move because no soap is used in this demo):

It's a vibrator on a piece of paper that acts like a lever and also has a thin metal rod attached to it. The vibrator moving up and down moves the paper, which amplifies the motion and also moves the metal rod. I dab a bit of soap on to the metal road so that the soap only touches the water when the vibrator is on.

Final Result

Here's the final result in action!

It would be really cool if we could also stop the boat; but once the soap touches the water it continues getting drawn in to the water. We could likely solve this issue with a vibration setup that has a larger range of motion. Another issue is that the soap runs out in the demo above. I would probably connect a larger soap source to the actuator through some tubes or a wick.

For the moment, I'm quite happy with where this got. In the future if I have the time to find or make a better actuator I might test it out in a similar system to try and improve it.

Happy accident

For now, here's another prototype that had a really interesting outcome. The actuator had a flap that acted like oars on a boat and this let the boat move forward without any soap involved.