The rotoscoping filter lets you create masks using cubic Bézier curves and keyframes.
The initial mask can be finished (after creating some points by clicking on the (edit) monitor) with a right mouse button click, or by clicking on the point that was created first.
To fine tune you can move the individual points and their handles. Multiple points can be selected by holding down shift while dragging the mouse to create a rectangular shape. You can remove points from that selection by clicking on them with ctrl pressed. The opposite (adding points to the selection with ctrl + mouse click) is not yet possible.
Points can be deleted with a right mouse button click. New points can be added by clicking on the spline.
The following shortcuts might become handy for fine tuning (also available in "Pan & Zoom" and "Corners"):
All keyframes should have the same number of points (unless you really know what you are doing)! In the future (post 0.8) some controls might be added to make life with this limitation a bit easier.
A smooth transition at the edge of the mask can be created by setting the feather width (no keyframes). Feathering is just a simple average/box blur applied to the mask, so it might not look terribly good. Therefore you have the possibility to do multiple blur/feathering passes (3 passes will give a good approximation of a Gaussian blur).