Preheat the oven to 350°F (177℃). Grease a 9-inch round cake pan or spring form pan and line the bottom with parchment paper, then grease the paper. Set aside.
In a mixing bowl, toss the sliced figs with melted butter, brown sugar, ¼ cup almonds, and balsamic vinegar until evenly coated. Then arrange the figs and almonds, cut side down, in the bottom of the prepared cake pan.
In a separate small bowl, mix the almond flour, all-purpose flour, baking powder, and baking soda. Set aside.
Using a stand mixer or hand mixer, whip eggs and sugar until pale yellow and fluffy. Then mix in lemon zest, lemon juice, olive oil, and vanilla until incorporated. With a spatula or wooden spoon, gently fold in the dry ingredients. Pour the batter over the figs and bake for 45 minutes.
Allow the cake to cool and invert it onto a plate. You may need to cut some of the top of the cake off before flipping it so that it sits level on the serving plate.
Evenly sprinkle remaining almonds over the top of the cake.