The valve nut on a presta valve is used to secure the presta valve to the rim. While not absolutely necessary, it is helpful to secure it as it will prevent the valve from shifting from its perpendicular position against the rim which can ultimately result in possible tearing of the valve from the tube. The nut will also help stabilize the valve when attaching a pump head to the valve during inflation.
When installing the valve nut it is important to remember not to tighten it too much which can result in the valve separating from the tube. It is best to only tighten the valve nut after inflating the tube and only tight enough to snug the nut down against the rim.
Tubeless valves typically will be in the presta valve form and the valve nut should be tightened enough keep the rubber seal air tight against the inside of the rim.