trap invalid opcode rip rsp

Prakash picture Prakash · Sep 21, 2010 · Viewed 11.7k times · Source

We see a couple of below mentioned messages in /var/log/messages for one of our application:

Sep 18 03:24:23 <machine_name> kernel: application_name[14682] trap invalid opcode rip:f6c6e3ce rsp:ffc366bc error:0
...
Sep 18 03:19:35 <machine_name> kernel: application_name[4434] general protection rip:f6cd43a2 rsp:ffdfab0c error:7b2

I am not able to make what’s these output means and how we can track the function / code that is causing the issue. Further what is 'trap invalid opcode' and 'general protection' means?

Answer

sbi picture sbi · Sep 21, 2010

Usually that means that your program's instruction pointer points to data or garbage. That's commonly caused by writing to stray pointers and such.

One scenario would be that your code writes (through a stray pointer) over some class' virtual table, replacing the member function addresses with nonsense. The next time you call one of the class' virtual functions, your program will interpret the garbage as an address and jump to that address. If whatever data lies at this address happens to not to be a valid machine code instruction for your processor, you would see this error.