FLIRT progress

Last semester, my goal was to get a working FLIRT implemetnation up and running. However, with it being my most intense semester to date, I didn't get as far as I wanted Most of the work involved code fixes and some crude code fixes to the Red Plait (rpat) pattern generator.
Really though, rpat needed a full rewrite to be truly useful in my project. While I could have maintained in its original C form, I really would rather have had a highly functionalized version using STL data structures. As of commit e719d6ddba27bf1fa4d1ace18d3426a356d19fa5, this is done. Example output:

[mcmaster@gespenst bin]$ ./uvobj2pat.dynamic --input=../obj2pat/main_d.o
5589E583EC18E8........C1E81F84C07426C744240C........C74424085800 2B E01B 004B :011F _Z14initProgConfigv ^0007 _Z21initFLIRTSharedConfigv ^0016 ^0026 ^0032 uv_err_ret_handler ^0039 g_config ^0040 5589E583EC18........FFC1E81F84C07426........5E0100........085800........240406000000C7
5589E55383EC24C745E8FFFFFFFFC745EC00000000C745F000000000C745F4FF FF F3C5 0349 :01A4 _Z6uvmainiPPc ^0024 _Z13UVDGetVersionv ^0037 strcmp ^0045 _Z13UVDGetVersionv ^005B ^0060 printf ^0065 stdout ^006D fflush ^0072 _Z7UVDInitv ^0081 ^0091 ^009D uv_err_ret_handler ^00A7 g_config ^00B8 ^00C8 ^00D4 uv_err_ret_handler ^00DE _Z14initProgConfigv ^00ED ^00FD ^0109 uv_err_ret_handler ^0127 _ZN9UVDConfig9parseMainEiPKPc ^0138 ^0148 ^0154 uv_err_ret_handler ^0173 ^017F printf_debug_level ^0186 g_flirt ^018B _ZN8UVDFLIRT8getFLIRTEPPS_ ^0199 ^019E puts ^01AF g_flirt ^01BB ^01CB ^01D7 uv_err_ret_handler ^01E1 g_uvd ^01ED ^01FD ^0209 uv_err_ret_handler ^0213 g_uvd ^0222 ^0232 ^023E uv_err_ret_handler ^0248 g_uvd ^0261 ^0271 ^027D uv_err_ret_handler ^0290 _ZNKSt6vectorISsSaISsEE5emptyEv ^029B ^02A0 puts ^02A5 _Z7UVDHelpv ^02AD ^02BD ^02C8 uv_err_ret_handler ^02DD ^02E2 puts ^02EF _Z9UVDDeinitv ^02FE ^030E ^031A uv_err_ret_handler ^0324 ^0334 ^033F uv_err_ret_handler 5589E553........45E8FFFFFFFFC745EC00000000C745........00C745F4FFFFFFFFE8FC........C3E82DFEFFFF895C2404890424E8FCFFFFFF........C0........E8........89C3E80C........5C........2404C7042460000000E8FC........00000000890424E8FCFFFFFF........FFC1E81F84C07429........57010000C744........0000C744240406000000C70424........E8FCFFFFFFE99D020000A100........45EC837DEC007529........57010000C744........0000C744240406000000C7........FFFFE8FCFFFFFFE966020000........FFC1E81F84C07429........57010000C744240881000000C744240406000
5589E583E4F083EC20C7442404........C7042405000000E8........8B450C 3C F345 005C :04ED main ^000D ^0019 printf_debug_level ^002B _Z6uvmainiPPc ^0037 ^0043 printf_debug_level 5589E583E4F083EC20C744........0000C70424050000........FFFF8B450C894424........890424E8FCFFFFFF8944241CC74424044B010000C7
---

Which seems consistent with the assembly:
000004ed

:
4ed: 55 push %ebp
4ee: 89 e5 mov %esp,%ebp
4f0: 83 e4 f0 and $0xfffffff0,%esp
4f3: 83 ec 20 sub $0x20,%esp
4f6: c7 44 24 04 3e 01 00 movl $0x13e,0x4(%esp)
4fd: 00
4fe: c7 04 24 05 00 00 00 movl $0x5,(%esp)
505: e8 fc ff ff ff call 506
50a: 8b 45 0c mov 0xc(%ebp),%eax
50d: 89 44 24 04 mov %eax,0x4(%esp)
511: 8b 45 08 mov 0x8(%ebp),%eax
514: 89 04 24 mov %eax,(%esp)
517: e8 fc ff ff ff call 518
51c: 89 44 24 1c mov %eax,0x1c(%esp)
520: c7 44 24 04 4b 01 00 movl $0x14b,0x4(%esp)
527: 00
528: c7 04 24 05 00 00 00 movl $0x5,(%esp)
52f: e8 fc ff ff ff call 530
534: 83 7c 24 1c 00 cmpl $0x0,0x1c(%esp)
539: 79 07 jns 542
53b: b8 01 00 00 00 mov $0x1,%eax
540: eb 05 jmp 547
542: b8 00 00 00 00 mov $0x0,%eax
547: c9 leave
548: c3 ret

However, the correctness of this won't be known until I actually try processing it. This version should be much easier to maintain and be more stable. There are some assorted points to fix, but it was definitely a step in the right direction. Next steps will be to implement pattern generation for libuvudec based disassembling and verify FLAIR's makesig produces the same output and/or is valid for input to pat2sig. One of the challenges I will of course face is how to support all of the architectures. I'm thinking the best way will be for any arch IDA does not support, I'll put a different magic number at the start of the file and add a new architecture field
Part of the reason I got working on this again is someone pointed out to me someone posted the .sig binary format here. I was hoping to do this last semester, but mostly due to time constraints, didn't get to it. With the format out there now, it will be much easier to implement a pat2sig equivalent and the matching loader.
Finally, I made a quick stab at a Qt GUI:

All it does is display the function list. I was trying to figure out how to get the window to resize to fit the whole screen but needs more work. I don't have a lot of GUI experience, so there will be a learning curve to get things decent. Hopefully though, if this project becomes popular, it shouldn't be too hard for someone to design something better and swap it out. The major issue that became readily apparent is the engine is currently designed for one shot analysis. It cannot respond to annotations. Thus, some sort of analysis event system will have to be implemented. This will be important for plugins as well as they may need to respond to various engine events if they want to do more than just respond to keystrokes. In the near future for the GUI, I'd like to:
-Get the window to resize correctly
-Display a dead listing
-Ability to double click on functions and have that location appear in the dead listing
Next:
-Write comments
-Load and save a workspace (I briefly toyed with this)
-Load FLIRT signatures. This may be sooner than later as I'm currently writing uvpat2sig which will mean I'll have at least most of first revision signature engine up.

New utility program: uvstructoffset

Under util/uvstructoffset/uvstructoffset.py
Here and there I find the need to verify the APIs I create have the correct structure alignment. I would do this manually with something like:
printf("some_member: 0x%.4X\n", offsetof(struct my_struct_t), some_member);
But this can be automated if there was a program to parse the C code. pycparser seemed like the most convenient C parser to use. After some coding, I could transform structures:

#include

struct sig_header_t
{
uint8_t dell1800FP;
short atomicForceMicroscope;
uint16_t fearAndLoathingInLasVegas;
uint8_t oatmeal;
char paperTowels;
uint16_t jointedGlasswares;
uint8_t micrometer;
int orangina;
char accidentWaitingToHappen[16];
uint8_t SATADrive;
uint16_t fastSteeringMirror;
uint32_t hamburgers;
} __attribute__((__packed__));

Into this:

struct sig_header_t
dell1800FP @ 0x0000
atomicForceMicroscope @ 0x0001
fearAndLoathingInLasVegas @ 0x0003
oatmeal @ 0x0005
paperTowels @ 0x0006
jointedGlasswares @ 0x0007
micrometer @ 0x0009
orangina @ 0x000A
accidentWaitingToHappen @ 0x000E
SATADrive @ 0x001E
fastSteeringMirror @ 0x001F
hamburgers @ 0x0021

Formatting doesn't line up on the web, but you get the idea. The C parser only parses the structure and doesn't try to guess the actual offsets. This is done with gcc since this is really the only reliable way to do it based on all of the different data types and such are considered. For decompiler use, config files will have to be referenced for data sizes and alignment.
After working with JSON, I've decided it is the sort of data exchange format I've been looking for for a while. I've never been a fan of XML because I find it overcompicated to parse and work with for general use. If I need any sort of convient data exchange between programs where performance in't an issue but convenience is, I'll probably use it as the format. As required, these can be migrated to higher performance formats. This will include configuration files and structures definitions.
Eventually I will be needing to parse out structures in the decompiled/disassembled files. Since these files don't need to be parsed often and for other data exchange reasons, I will be using the aforementioned Python parser to output a JSON structure definition.

ELF object dump repaired

As commented in the previous post, the old ELF code was very ugly and a thorn in my side. It is now fixed and running better than other and much easier to maintain. Example object file:

[mcmaster@gespenst bin]$ objdump --syms --reloc analysis/sub_0EC3.elf

analysis/sub_0EC3.elf: file format elf32-i386

SYMBOL TABLE:
00000000 l df *ABS* 00000000 candela_pltl1_rev_3.bin
00000000 l d .text 00000000 .text
00000000 *UND* 00000000 sub_0FCB
00000000 *UND* 00000000 sub_0FE5
00000000 *UND* 00000000 sub_101C
00000000 *UND* 00000000 sub_75E3
00000000 g F .text 00000107 sub_0EC3


RELOCATION RECORDS FOR [.text]:
OFFSET TYPE VALUE
0000009e R_386_16 sub_0FCB
000000c0 R_386_16 sub_0FCB
00000103 R_386_16 sub_0FE5
000000a1 R_386_16 sub_101C
000000c3 R_386_16 sub_75E3

I'm still using 386 object file format and still unsure how I want to deal with that in the future. For now it seems like the logical thing so that tools like objdump will play with it. I could in theory generate a patch for binutils, but that would be annoying for decompiler users and a pain to maintain. Maybe I can make a plugin patch for binutils and try to get it into the mainline? It ha some very limited (read: not useful enough) plugin capability. I didn't look into it too much, but from a quick grep only nm and ar support binutils plugins. At first it didn't look like there is any notion of full architecture plugins, but now I'm thinking its just that only that for some odd reason only certain tools allow use of uninstalled plugins. Maybe I'll send an e-mail to their mailing list for advice. clang/llvm is a more modern project and might work better with this stuff. Unfortunately, I haven't yet spent any time to learn it. From a quick look at it, they expose a lot of API stuff that might be good if I wanted to allow my arch files compile executables, but they seem to still depend on binutils for day to day object inspection. clang can do some linking, but I don't know where this functionality comes from. For the short term, I might consider making a very basic binutils skeleton to work with these files, probably in Python. I'm afraid this will feature creep and never get replaced though and is ultimately not the right way to do things.

Edit:
After some more thinking and research, I've decided by and by the far easiest and cleanest way to do this is to simply provide wrapper programs over binutils. The basic idea will be to parse out binutils files and redirect them to files temporarily fixed up to be compatible with binutils. Basically, this will involve temporarily changing the object type to EM_386 since that's what all constants will be based off of. While this will obviously run into a number of issues, it unfortunate seems by far the cleanest solution.

Summer progress: installer, licensing, fixing bugs

This summer I'm doing malware analysis. The first time I've held a full time reverse engineering job, I'm getting time to try out a lot more tools and develop skills. These will undoubtably be valuable to this project.
The following is written with a huge patch queue sitting on my laptop (probably over 4k lines of git diff, maybe more once I'm done). I was hoping to stabilize the ELF code before commiting, hopefully my laptop won't blow up in between.
I played around briefly with Installjammer (website, github) and at first glance it seems pretty nice. It does seem more targetted towards Windows, but I was able to make an InstallShield type installer within minutes for Linux and was impresed. The biggest thing lacking I saw at first glance was no support for .rpm or .deb files, so I might consider asking what it would take to get those supported if I am still interested. Screenshot of the quick test installer running on Fedora 13:
I will probably be dropping support for SpiderApe in the near future. My previous system had unstable Python code which towards the end of its life I was working on fixing, but this was unbeknonst to me when I first tried using the Python APIs and was quite dissapointed with them. I'm still dissapointed in their lack of good error handling (Python_Init() or w/e its called returns void), but when it works it seems to work decent. Fedora 13 doesn't seem to ship a static lib, so I might try to do a prefixed installation to grab that.
On that note, I fixed the build process to be a lot cleaner with regards to using PREFIX variables on dependencies. The original reason why I started playing with using package uninstalled was mostly related to binutils not exporting everything I'd like it to that was needed for the binutils .sig generation when using rpat as a reference.
For a number of legal reasons, I'm going to dual license the project under BSD/GPL. I still like the freedom that BSD provides, but for a couple of reasons, including the otherwise inability to distribute binaries, I'm going to tack on a GPL licensing option.
The ELF/object generation system is being rewritten. It used a fixup based linking approach that just didn't work well. The code is much larger than it could have been using a sevearl pass approach. This is likely responsible for the errors that have been occuring and the rewrite seems nearly done and is resulting in much cleaner code.
Tonight I will hopefully finish the ELF rewrite or be very close. But, tonight is the night of fire, and I like fire. So I might be busy. Happy 4th!

Bug Trac-ing

I've been meaning to try out Trac for some time and decided this project would be a good excuse. After looking around for a bit, I found free Trac hosting at Assembla which seems to provide pretty decent free hosting. While I would have preferred it through a non-profit organization, I decided its not a big deal since I can just export the Trac database and move hosting if I like Trac and want to move hosting later. The uvudec bug tracking system can be found here. The system just has a few issues in it now and I'll be building up the database over time each time I see one of my XXX or FIXMEs in code.

uvromstat: statistical ROM image integrity testing

I've become aware of several different issues that can arise during ROM imaging. There are three main classes of errors I've come across:
-Blank or nearly blank ROM
-Missing address pin. Results in repeat blocks
-Bad selection switches, improper placement, or other read routing issues. With my Willem programmer anyway, this results in bad rips with several interesting, human recognizable properties. More on this later.
Blank ROMs are easy to detect. I do a simple frequency analysis. If only one byte showed up, its definitely blank. If below a certain threshold, defaulted to 16, I consider this a mostly blank ROM that is either severely damaged or otherwise poorly used. Frequency distribution from a nearly blank ROM:
Raw data looks something like this:

00000000 04 7f 25 7f 45 7f 7f 7f 04 7f 7f 7f 7f 7f 7f ff |..%.E...........|
00000010 04 7f 7f 7f 67 7f 7f ff 65 7f 7f ff 7f 7f 7f ff |....g...e.......|
00000020 04 7f 7f 7f 7f 7f 7f 7f 65 7f 7f 7f 7f 7f 7f ff |........e.......|
00000030 45 7f 7f 7f 7f 7f 7f ff 7f 7f 7f ff 7f ff 7f ff |E...............|
00000040 04 7f 7f 7f 7f 7f 7f 7f 45 7f 7f 7f 7f 7f 7f ff |........E.......|
00000050 45 7f 7f 7f 7f 7f 7f ff 7f 7f 7f ff 7f ff 7f ff |E...............|

Missing address pin is also fairly easy to detect. I'll assume that pins have been disconnected and will hold a value, say 0, when disconnected. I do not consider the case where it floats and randomly selects data. For the assumed behavior, just look for repeated blocks of data of size 2**n. For example, say the original data was 0123 4567 89AB CDEF. If the highest address pin was disconnected, we will get 0123 4567 0123 4567. If the second highest address pin was disconnected, you'd get 0123 0123 89AB 89AB. Both broken would result in 0123 0123 0123 0123.
For the last case , this can occur several ways. The first observed was when a 27C eprom was put into the flash socket by mistake. A separate adapter board is actually required to rip 27C PLCC32 chips. Additionally, similar output appears for some reason I have yet to look into when no chip is inserted into the Willem and told to rip anyway. An excert from such a rip of a 27C put into the flash socket on a Willem:

000003d0 d5 d5 00 00 d5 d5 00 00 d6 d6 00 00 d7 d7 00 00 |................|
000003e0 e0 e0 00 00 e1 e1 00 00 e2 e2 00 00 e2 e2 00 00 |................|
000003f0 e3 e3 00 00 e4 e4 00 00 e5 e5 00 00 e6 e6 00 00 |................|
00000400 04 06 04 06 04 06 04 06 04 06 04 06 04 06 04 06 |................|
*
00000500 14 16 14 16 14 16 14 16 14 16 14 16 14 16 14 16 |................|
*
00000600 04 06 04 06 04 06 04 06 04 06 04 06 04 06 04 06 |................|
*
00000700 14 16 14 16 14 16 14 16 14 16 14 16 14 16 14 16 |................|
*

Graphical frequency analysis (optional command line option that uses gnuplot-python) showed some obvious trends on such chips:



These chips skip frequencies in blocks. In the first one, the first 8 bytes are represented, the second 8 aren't. This continues for the rest of the set. The same trend is seen in the second one, but with a block size of 128 instead of 8.
These algorithms are being developed in util/uvromstat/uvromstat.py (UVet ROM STATistics). Here are some of the ones it detected:

[mcmaster@gespenst ROM]$ uvromstat $(find . -name '*.bin' |fgrep -v .svn) |fgrep ERROR
./buffer/quantum_first_prob_bad.bin: ERROR: expect did not position for rip correctly! (27C check)
./buffer/quantum_first_prob_bad.bin: ERROR: block size: 16
./buffer/prompro7_junk/27C512_IBM_sticker.bin: ERROR: expect did not position for rip correctly! (27C check)
./buffer/prompro7_junk/27C512_IBM_sticker.bin: ERROR: block size: 256
./buffer/prompro7_junk/27C512_IBM_sticker.bin: ERROR: an address pin is probably missing
./buffer/prompro7_junk/27C512_IBM_sticker.bin: ERROR: repeated block size: 0x2000 (8192), address bit: 0x000D (13)
./buffer/prompro7_junk/2732_1.bin: ERROR: few distinct numbers, expect nearly blank
./buffer/prompro7_junk/2732_1.bin: ERROR: expect did not position for rip correctly! (27C check)
./buffer/prompro7_junk/2732_1.bin: ERROR: block size: 8
./buffer/prompro7_junk/27C256_0.bin: ERROR: an address pin is probably missing
./buffer/prompro7_junk/27C256_0.bin: ERROR: repeated block size: 0x2000 (8192), address bit: 0x000D (13)
./buffer/prompro7_junk/27C256_0.bin: ERROR: an address pin is probably missing
./buffer/prompro7_junk/27C256_0.bin: ERROR: repeated block size: 0x4000 (16384), address bit: 0x000E (14)
./arch/8051/80C32/cisco_catalyst_5000__supervisor_I/80C32.bin: ERROR: an address pin is probably missing
./arch/8051/80C32/cisco_catalyst_5000__supervisor_I/80C32.bin: ERROR: repeated block size: 0x20000 (131072), address bit: 0x0011 (17)

Another thing to note is that if one check fails, other might as well. This did in fact discover several images that I didn't realize had issues. The Catalyst 5000 image I had recognized during my initial rip was having issues. I was using a breadboarded ripper at the time and could not figure out why the image appeared to repeat twice. It is possible that I messed something up and would double check with the WIllem now, but it appears that is the only chip out of the set from that board I didn't keep. In any case, it does appear to be unused at the end, so the image is not likely truncated.
In conclusion, even a simple utility has proven useful for discovering bad rips. When I rip ROMs in the future, I'll try to run them through it to discover common issues before I throw the ROM away.

ROM ripping rampage

When I bought the PromPro-7, it came with a lot of EPROM and related MCU chips. Unfortunately, the picture of it didn't migrate over when I switched accounts with this blog and I can't find the original picture. Since the PromPro-7 didn't work very well, they never got ripped. Although I really wasn't looking to get a parallel port unit, I finally got the tried and true Willem programmer since other things weren't working out. My Linux file server hasn't been up for a while since the software only works in Windows and I didn't feel like reconfiguring all of my shares and drives.
In addition to the chips the Prompro 7 came with, I've had a bunch of other boards and chips laying about my room. The largest was probably a stack of boards from a Siemens PBX that was thrown out and I got to torture. One of the first stumbling blocks I encountered from this was that there was a 1Mb EPROM on it. This is an issue since the stock Willem programmer can only handle 512Kb EPROMs. So, I ordered an adapter board and ripped it. I had a few other large EPROMs, so this turned out to be a good idea. One notable was from a Rockwell Collins Pro 2000-SL transciever/GPS unit that I found cheap ($20 shipped) on eBay a bit back. This unit should prove interesting for a variety of RE purposes since it seems to be military grade. Here is a picture of it:
The until is composed of 4 main board assemblies: power supply (+module), CPU board, GPS translator, and GPS. The GPS translator board is the only board that contained an easily accessible (socketed) ROM:
ROM data like GPS_SOLUTION_LOCK gives it a military feel. A 28F flash chip is also visible. Especially since I broke some of the flimsy cables on the PSU module, I might just rip the unit to shreds and rip all of the chips since its probably a better use of time than getting the unit running anyway. With some luck, there will be assorted ASICs using rare standard cell libraries that I can submit to Silicon Zoo.
There are several types of chips that are still in my rip queues though. The first are 24C SOIC EPROMs. This is what a small group of us has been doing to experiment with some radio units we've been messing with. Hopefully, more on that later as it develops. Although I have some adapter boards, I don't feel like soldering to them each time I want a clean rip. I might be able to get away with just gently placing them on the pads, but really I need a ZIF socket. I'll hopefully be ordering 150 mil and 200 mil SOIC8 compatible (probably SOIC16 and I'll just slide it into position as needed) ZIF sockets.
The second type are 27C PLCC32 type EPROMs. I didn't realize originally that I needed a seperate adapter for those and that the flash sockets wouldn't work for those. Next, I have the stack of MCS-48 combo MCU/EPROM ICs that also require an adapter to work with the Willem. Finally, I have some Altera ROMs I'm trying to rip. The bus pirate might be a good choice, but I actually have some old Altera programmers (Altera Programming Units (APU)) that I was hoping would be a nice way to rip them. Unfortunatly, it seems I can't see to read the ROMs unless I setup a project or something. Lame. Maybe there is a better behaved command line utility in the suite or I can port the code to an open source Linux program.
Of all of the boards and ROMs I've ripped, my favorite so far has come from an IBM RS/6000 43P Model 260. In one of the ROMs, I found the string:

16-6, Go Horns !
Take that, Dave

Probably thought that nobody would ever read that. This ROM I think was labeled 0373.