There are no options to limit memory, except on a global level with global_allocation_limit. HANA manages its own memory pool and assigns to the processes as it sees fit.
See page 7: http://www.saphana.com/docs/DOC-2299
As the others have suggested, I would focus on optimizing the query, rather than trying to manipulate how HANA is handling the resources.