Summer progress: installer, licensing, fixing bugs

This summer I'm doing malware analysis. The first time I've held a full time reverse engineering job, I'm getting time to try out a lot more tools and develop skills. These will undoubtably be valuable to this project.
The following is written with a huge patch queue sitting on my laptop (probably over 4k lines of git diff, maybe more once I'm done). I was hoping to stabilize the ELF code before commiting, hopefully my laptop won't blow up in between.
I played around briefly with Installjammer (website, github) and at first glance it seems pretty nice. It does seem more targetted towards Windows, but I was able to make an InstallShield type installer within minutes for Linux and was impresed. The biggest thing lacking I saw at first glance was no support for .rpm or .deb files, so I might consider asking what it would take to get those supported if I am still interested. Screenshot of the quick test installer running on Fedora 13:
I will probably be dropping support for SpiderApe in the near future. My previous system had unstable Python code which towards the end of its life I was working on fixing, but this was unbeknonst to me when I first tried using the Python APIs and was quite dissapointed with them. I'm still dissapointed in their lack of good error handling (Python_Init() or w/e its called returns void), but when it works it seems to work decent. Fedora 13 doesn't seem to ship a static lib, so I might try to do a prefixed installation to grab that.
On that note, I fixed the build process to be a lot cleaner with regards to using PREFIX variables on dependencies. The original reason why I started playing with using package uninstalled was mostly related to binutils not exporting everything I'd like it to that was needed for the binutils .sig generation when using rpat as a reference.
For a number of legal reasons, I'm going to dual license the project under BSD/GPL. I still like the freedom that BSD provides, but for a couple of reasons, including the otherwise inability to distribute binaries, I'm going to tack on a GPL licensing option.
The ELF/object generation system is being rewritten. It used a fixup based linking approach that just didn't work well. The code is much larger than it could have been using a sevearl pass approach. This is likely responsible for the errors that have been occuring and the rewrite seems nearly done and is resulting in much cleaner code.
Tonight I will hopefully finish the ELF rewrite or be very close. But, tonight is the night of fire, and I like fire. So I might be busy. Happy 4th!

Bug Trac-ing

I've been meaning to try out Trac for some time and decided this project would be a good excuse. After looking around for a bit, I found free Trac hosting at Assembla which seems to provide pretty decent free hosting. While I would have preferred it through a non-profit organization, I decided its not a big deal since I can just export the Trac database and move hosting if I like Trac and want to move hosting later. The uvudec bug tracking system can be found here. The system just has a few issues in it now and I'll be building up the database over time each time I see one of my XXX or FIXMEs in code.

uvromstat: statistical ROM image integrity testing

I've become aware of several different issues that can arise during ROM imaging. There are three main classes of errors I've come across:
-Blank or nearly blank ROM
-Missing address pin. Results in repeat blocks
-Bad selection switches, improper placement, or other read routing issues. With my Willem programmer anyway, this results in bad rips with several interesting, human recognizable properties. More on this later.
Blank ROMs are easy to detect. I do a simple frequency analysis. If only one byte showed up, its definitely blank. If below a certain threshold, defaulted to 16, I consider this a mostly blank ROM that is either severely damaged or otherwise poorly used. Frequency distribution from a nearly blank ROM:
Raw data looks something like this:

00000000 04 7f 25 7f 45 7f 7f 7f 04 7f 7f 7f 7f 7f 7f ff |..%.E...........|
00000010 04 7f 7f 7f 67 7f 7f ff 65 7f 7f ff 7f 7f 7f ff |....g...e.......|
00000020 04 7f 7f 7f 7f 7f 7f 7f 65 7f 7f 7f 7f 7f 7f ff |........e.......|
00000030 45 7f 7f 7f 7f 7f 7f ff 7f 7f 7f ff 7f ff 7f ff |E...............|
00000040 04 7f 7f 7f 7f 7f 7f 7f 45 7f 7f 7f 7f 7f 7f ff |........E.......|
00000050 45 7f 7f 7f 7f 7f 7f ff 7f 7f 7f ff 7f ff 7f ff |E...............|

Missing address pin is also fairly easy to detect. I'll assume that pins have been disconnected and will hold a value, say 0, when disconnected. I do not consider the case where it floats and randomly selects data. For the assumed behavior, just look for repeated blocks of data of size 2**n. For example, say the original data was 0123 4567 89AB CDEF. If the highest address pin was disconnected, we will get 0123 4567 0123 4567. If the second highest address pin was disconnected, you'd get 0123 0123 89AB 89AB. Both broken would result in 0123 0123 0123 0123.
For the last case , this can occur several ways. The first observed was when a 27C eprom was put into the flash socket by mistake. A separate adapter board is actually required to rip 27C PLCC32 chips. Additionally, similar output appears for some reason I have yet to look into when no chip is inserted into the Willem and told to rip anyway. An excert from such a rip of a 27C put into the flash socket on a Willem:

000003d0 d5 d5 00 00 d5 d5 00 00 d6 d6 00 00 d7 d7 00 00 |................|
000003e0 e0 e0 00 00 e1 e1 00 00 e2 e2 00 00 e2 e2 00 00 |................|
000003f0 e3 e3 00 00 e4 e4 00 00 e5 e5 00 00 e6 e6 00 00 |................|
00000400 04 06 04 06 04 06 04 06 04 06 04 06 04 06 04 06 |................|
*
00000500 14 16 14 16 14 16 14 16 14 16 14 16 14 16 14 16 |................|
*
00000600 04 06 04 06 04 06 04 06 04 06 04 06 04 06 04 06 |................|
*
00000700 14 16 14 16 14 16 14 16 14 16 14 16 14 16 14 16 |................|
*

Graphical frequency analysis (optional command line option that uses gnuplot-python) showed some obvious trends on such chips:



These chips skip frequencies in blocks. In the first one, the first 8 bytes are represented, the second 8 aren't. This continues for the rest of the set. The same trend is seen in the second one, but with a block size of 128 instead of 8.
These algorithms are being developed in util/uvromstat/uvromstat.py (UVet ROM STATistics). Here are some of the ones it detected:

[mcmaster@gespenst ROM]$ uvromstat $(find . -name '*.bin' |fgrep -v .svn) |fgrep ERROR
./buffer/quantum_first_prob_bad.bin: ERROR: expect did not position for rip correctly! (27C check)
./buffer/quantum_first_prob_bad.bin: ERROR: block size: 16
./buffer/prompro7_junk/27C512_IBM_sticker.bin: ERROR: expect did not position for rip correctly! (27C check)
./buffer/prompro7_junk/27C512_IBM_sticker.bin: ERROR: block size: 256
./buffer/prompro7_junk/27C512_IBM_sticker.bin: ERROR: an address pin is probably missing
./buffer/prompro7_junk/27C512_IBM_sticker.bin: ERROR: repeated block size: 0x2000 (8192), address bit: 0x000D (13)
./buffer/prompro7_junk/2732_1.bin: ERROR: few distinct numbers, expect nearly blank
./buffer/prompro7_junk/2732_1.bin: ERROR: expect did not position for rip correctly! (27C check)
./buffer/prompro7_junk/2732_1.bin: ERROR: block size: 8
./buffer/prompro7_junk/27C256_0.bin: ERROR: an address pin is probably missing
./buffer/prompro7_junk/27C256_0.bin: ERROR: repeated block size: 0x2000 (8192), address bit: 0x000D (13)
./buffer/prompro7_junk/27C256_0.bin: ERROR: an address pin is probably missing
./buffer/prompro7_junk/27C256_0.bin: ERROR: repeated block size: 0x4000 (16384), address bit: 0x000E (14)
./arch/8051/80C32/cisco_catalyst_5000__supervisor_I/80C32.bin: ERROR: an address pin is probably missing
./arch/8051/80C32/cisco_catalyst_5000__supervisor_I/80C32.bin: ERROR: repeated block size: 0x20000 (131072), address bit: 0x0011 (17)

Another thing to note is that if one check fails, other might as well. This did in fact discover several images that I didn't realize had issues. The Catalyst 5000 image I had recognized during my initial rip was having issues. I was using a breadboarded ripper at the time and could not figure out why the image appeared to repeat twice. It is possible that I messed something up and would double check with the WIllem now, but it appears that is the only chip out of the set from that board I didn't keep. In any case, it does appear to be unused at the end, so the image is not likely truncated.
In conclusion, even a simple utility has proven useful for discovering bad rips. When I rip ROMs in the future, I'll try to run them through it to discover common issues before I throw the ROM away.

ROM ripping rampage

When I bought the PromPro-7, it came with a lot of EPROM and related MCU chips. Unfortunately, the picture of it didn't migrate over when I switched accounts with this blog and I can't find the original picture. Since the PromPro-7 didn't work very well, they never got ripped. Although I really wasn't looking to get a parallel port unit, I finally got the tried and true Willem programmer since other things weren't working out. My Linux file server hasn't been up for a while since the software only works in Windows and I didn't feel like reconfiguring all of my shares and drives.
In addition to the chips the Prompro 7 came with, I've had a bunch of other boards and chips laying about my room. The largest was probably a stack of boards from a Siemens PBX that was thrown out and I got to torture. One of the first stumbling blocks I encountered from this was that there was a 1Mb EPROM on it. This is an issue since the stock Willem programmer can only handle 512Kb EPROMs. So, I ordered an adapter board and ripped it. I had a few other large EPROMs, so this turned out to be a good idea. One notable was from a Rockwell Collins Pro 2000-SL transciever/GPS unit that I found cheap ($20 shipped) on eBay a bit back. This unit should prove interesting for a variety of RE purposes since it seems to be military grade. Here is a picture of it:
The until is composed of 4 main board assemblies: power supply (+module), CPU board, GPS translator, and GPS. The GPS translator board is the only board that contained an easily accessible (socketed) ROM:
ROM data like GPS_SOLUTION_LOCK gives it a military feel. A 28F flash chip is also visible. Especially since I broke some of the flimsy cables on the PSU module, I might just rip the unit to shreds and rip all of the chips since its probably a better use of time than getting the unit running anyway. With some luck, there will be assorted ASICs using rare standard cell libraries that I can submit to Silicon Zoo.
There are several types of chips that are still in my rip queues though. The first are 24C SOIC EPROMs. This is what a small group of us has been doing to experiment with some radio units we've been messing with. Hopefully, more on that later as it develops. Although I have some adapter boards, I don't feel like soldering to them each time I want a clean rip. I might be able to get away with just gently placing them on the pads, but really I need a ZIF socket. I'll hopefully be ordering 150 mil and 200 mil SOIC8 compatible (probably SOIC16 and I'll just slide it into position as needed) ZIF sockets.
The second type are 27C PLCC32 type EPROMs. I didn't realize originally that I needed a seperate adapter for those and that the flash sockets wouldn't work for those. Next, I have the stack of MCS-48 combo MCU/EPROM ICs that also require an adapter to work with the Willem. Finally, I have some Altera ROMs I'm trying to rip. The bus pirate might be a good choice, but I actually have some old Altera programmers (Altera Programming Units (APU)) that I was hoping would be a nice way to rip them. Unfortunatly, it seems I can't see to read the ROMs unless I setup a project or something. Lame. Maybe there is a better behaved command line utility in the suite or I can port the code to an open source Linux program.
Of all of the boards and ROMs I've ripped, my favorite so far has come from an IBM RS/6000 43P Model 260. In one of the ROMs, I found the string:

16-6, Go Horns !
Take that, Dave

Probably thought that nobody would ever read that. This ROM I think was labeled 0373.

Install magic and RCOS goes 1337

I did a small amount of work to make a basic installable form. Although the project is prepped for automake, I haven't bothered with that yet and I'm not sure when I will. That aside, this is what I did.
Default install location is to /opt/uvudec/. A symlink is also created /opt/uvudec/current to the last installed version. This then contains a bin and an arch directory. The bin directory contains the executable binaries and library files necessary to run uvudec. The arch dir contains the architecture configuration files that are used to parse binaries into intermediate representation. I might like to later move the libraries into a lib dir instead, but this is a minor change. The biggest issue right now is that if a dynamic build is used, they need to know how to find them. I personally don't like using rpath because its hard to change without specialized tools. I'd consider LD_LIBRARY_PATH, but that has its own set of issues. The main one is that I'd have to have helper scripts or such to set it, which is also somewhat annoying. However, this is the approach many enterprise grade installable Linux software packages such as Firefox take and I will probably do that eventually.
Using makefile configuration variables and later through autoconf, you can do the standard PREFIX and such that allows you to install applications to the directory of choice. However, I personally like programs that I can shove in any folder on the hard drive and still work without hard coded paths. So, I added some detection of where the executable is and derrive the install directory from that.
On a random note, in honor of RCOS, I added a second random ASCII art choice to the --useless-ascii-art option (the first is a UVNet logo). Using an ASCII art generator at http://www.glassgiant.com/ascii/ , I made a 40 X 40 char ASCII art RCOS logo and then cleaned it up a bit. Here is the current art:
I might remove the dots and leave as white space instead.

New blog, undergraduate research symposium

After fitting an extensive battle, I finally got my Dia format poster to plot. Had to do a series of workarounds, but got something to work. The biggest issue was since they are large images, some operations took a while. Maybe I should have used some smaller test images? Dunno, never thought I'd have so many problems. Here's my rant about what it took for me to get my poster plotted. I just sent it off too, I don't even know if this went through, but, if it wasn't correct, should be minimal effort from here.

-After some experimenting, tiff seemed to be a good format both my computers and library Windows computers were happy with
-Dia export under a lot of formats (but not all including jpg) under Linux is buggy, but appears not to be under Windows. The bug is that it will misalign chars randomly. Created a .dia under Linux and moved it to Windows VM
-Gimp under Windows doesn't seem to export svg to tiff with correct resolution, but Linux gimp does. Move .svg to Windows
-Load .svg in Gimp. Don't change the resolution, it will just get angry and waste a lot of time. Even if you do wait long enough, which is several minutes (for just changing an option without hitting anything? what the heck?), it seems to asepct lock it to 1:1, screwing up the poster. Leave it at the default of 90 in/pix
-Once loaded, all operations are smooth. Export as a .tiff
-Copy tiff to USB stick and move to Windows computer
-Plot the stupid image with ARCH E type paper. Seems to only allow setting to 600 dpi..w/e hopefully that will still print and not run out of memory
Gimp can actually export .jpg's for some reason fine under my version of Linux. Here is a scaled down version of the final poster:

Oh...and I wish Dia had spellchecking support...but I'm probably too lazy to add it

Friendly competition

There is finally another open source decompiler on the market! Metasm (http://metasm.cr0.org/) has released a C decompiler module for its assembly manipulation suite. Although I had heard of it before, I was not terribly familiar with this toolsuite. So, I will be looking into this in the near future to learn what worked well in what they did and what I can improve upon.
IDA .sig format has some good progress. Need to get a block of time to finish it though.