In computing, a system call is how a program requests a service from an operating system’s kernel. This may include hardware related services (e.g. accessing the hard disk), creating and executing new processes, and communicating with integral kernel services (like scheduling). System calls provide an essential interface between a process and the operating system.
Whenever system call is executed from user level a trap is generated and it gets handled at file Exception.S (i386\i386)
syscall() is a function in Trap.c (i386\i386) is a machine dependent code area.
Function syscall() is called with trap frame which has input values to system call.
Function syscall() now gets all the values of system call input and store in kernel so that kernel code can access the values while processing system call.
Then check we have direct or indirect system call. Indirect system call is one which is like loadable kernel module which loads system call. Once loaded then it can pass the system call number as argument which can be injected in here.
Then given the system call code we check if we have the code as valid number. If system call code is valid then lookup in system call table which system call is referred else simply store the pointer to indirect system call which is system call 0.
where sv_table entry will have following values
Once we know which system it is then we determine how many arguments it have
narg = callp->sy_narg;
Then we copy all the user space arguments into kernel space using function copying which validates the addresses of source and destination etc.
Now, check if we saw any error so far. If we did find the error then we need to return the error but, at this time in kernel it does not know how to return in user space. So, we set the register 0 to be non zero which is the error number and we set another register which is carry bit(eflag).
So, when system call is returned, C lib which issued the system call will check if carry bit(eflag) is set or not. If it is set then it will get the error number from register 0 and map it to human readable error errorno and then override the register 0 with -1.
So, say if open system call encounters error then its going to get the error value from register 0; translate it to meaningful error and then override the register 0 with -1. So open returned the value -1 which is failure.
Now we call the actual system call > error = (*callp->sy_call)(td, args);
Here, td is the thread pointer args are the arguments we have copied above from user space to kernel space
We check error and if all fine then we return the values back to user space
Now we fall back to assembly program from where this function was called.