Preheat oven to 350°F (175°C). Line a 9” round pan with parchment paper and grease the sides.
In a bowl, rub lemon zest into granulated sugar with your fingertips until fragrant and slightly damp.
In a separate medium bowl, whisk together gluten-free flour, baking powder, baking soda, and salt.
In a large bowl, whisk eggs until lightly beaten. Slowly stream in olive oil while whisking continuously until combined.
Whisk the lemon sugar mixture into the eggs and oil until smooth.
Whisk in the fresh lemon juice and milk until just combined.
Gradually add the dry ingredients to the wet ingredients, whisking until just blended. Do not overmix.
Pour batter into the prepared pan and bake for 30-40 minutes, or until a toothpick inserted into the center comes out clean.