Start by adding matcha powder to a mixing bowl. Then add hot water and use a whisk to get rid of any lumps. Add coconut milk, oat milk, and agave and stir. Taste and add more agave if desired.
Pour the mixture to fill each popsicle mold. Place lid over molds, then insert popsicle sticks. Place in the freezer for at least 6 hours before serving.
Remove lid and release one popsicle mold, run under warm water (not hot water) and gently tug on the popsicle stick until it easily slides out of the mold.