Query Performance Issue
I would eliminate the splitting operation from your enquiries by saving the result of dbo.split to a temporary table with the matching data type (int) and suitable indexing.
Run the query with OPTION (RECOMPILE) to enable the parameter-embedding optimization. This may well help by compiling a plan for the specific runtime values of @GroupID and @CourseIDs. The existing general plan shape for all possible variable values does result in a moderately high number of executions for some plan operators.
Check the sys.dm_os_waiting_tasks DMV during execution, paying particular attention to the node ID in the resource_description for CXPACKET waits. These waits seem on the high side, and it may be that the DMV can help identify where this is occurring. Further analysis would be required to determine any deeper cause here.
The difference is estimate vs actual rows for the scans is probably not cause for concern. Most of the scans have pushed-down static bitmap filters, which are added after query optimization (so the estimates do not account for their effects). On the face of it, the bitmap + scan strategy seems a reasonable one.
Aside from the above, I would also break this query up a bit, along the natural transitions in the query plan. This may help improve performance, but it is more aimed at understanding where the performance problem is occurring. Depending on what you find by doing that, you may choose to leave it that way, or rebuild the monolithic query incorporating any corrective measures.
The spilling hash joins and sort does not appear to be causing a performance problem, but eliminating these may be a handy side-effect of the measures outlined above.
There are also a number of small implicit conversions due to typing mismatches. I doubt these are having a major performance effect, but again, it is worth correcting these just in case.
Sorry there isn't a single obvious cause (to me, anyway).