There is now a minimal GUI up and running:
The demonstration here is not so much that I've yet made a usable program, but that I learned the basics of Qt and can start moving forward with what it will take to make a usable GUI.
As part of this process, I wanted to review some of the leading disassembler environments out there and what I like and don't like about their interfaces. I might interject a few things that are more feature related, but try to stick to usability.
The biggest open source player is probably Metasm. I had forgotten about their GUI component until someone reminded me of it recently. I had mostly looked at them before because, during the course of this project, they released a decompiler module. I only played with it briefly, but porting it to a plugin might server as a good starting point for me. In any case, this is what I came up with their GUI disassembling objdump:
Hmm it didn't disassemble it. Not a huge deal, it wasn't hard to figure out how to start it disassembling:
And the decompile feature was pretty easy to use as well:
To be fair, they do label it as as "sample application" of the metasm library rather than a full application bundled with metasm. That said, here are my comments:
-I would expect it to disassemble by default. If the user had a truly good reason why it shouldn't, maybe they should use a "open wizard" menu option that gives advanced open options.
-The graph view is decent. I'd like to see thicker lines
-It doesn't seem like the GUI is a main focus. If it wants to be a serious competitor to IDA, a full GUI should be made. But, RE's typically don't like GUI stuff, so I can see why it might not happen
-Needs more keyboard shotcuts
-Couldn't get the comment function to work. For some reason it seems to decompile
-Crashed on me after using it for only a few minutes
-I'd probably prefer tabs to floating UI elements across the desktop. At the very least, I'd prefer them bound to a main window. Some software. such as Xilinx ISE allows either if people really cared.
-The underlying engine seems pretty powerful and with work, a nicer GUI could be made for it. As such, I might consider writing a metasm plugin if it seems I can really leverage some of their features, such as the decompiling. I'd need to see if I can get uvsync and metasmsync plugins though so they can cleanly exchange analysis data and maybe avoid the issue of linking C++ against Ruby.
-I shouldn't need to know Ruby to run it. Since this is intended as a Ruby sample, seems acceptable given their target audience
Here's what Hex-Rays shows for IDA (http://www.hex-rays.com/idapro/pix/idalarge.gif):
Overall, the GUI is decent since it has so many features. I'm not a high IDA user as I don't have a license for it, but there is the older version available. Also, I've just used it on occasion as some of the companies I've done work for have had a license I can use. So, some of my complaints may be inaccurate and I might not just know how to custimize it / where the correct command is. Some of the areas I think I could improve on:
-Price tag: $$$
-Structure definitions. Supposedly you can import C header files instead of using their janky structure defining tool. I couldn't figure it out. My other complaint about structures is that it doesn't seem you can define a structure on the stack
-Debugging: Far behind OllyDbg. WinDbg is okay, but seriously it should support GDB server out of the box. And WTF is the Bochs debugger?
-The IDA Pro book is pretty clear the closest thing to undo in IDA is to close the workspace and open it again
-IDC script: fortunately, IDAPython exists. Admittedly though, IDC was made at a time when such tools at Python weren't (well?) developed. Plus, IDC is still a million times better than OllyScript
-Graph view is overall decent. I see it sometimes will go out of its way to interleave unrelated loop basic blocks together to try, I think, to make consistent loop entry point shapes.
I use OllyDbg quite a bit of Windows RE. Since I'm too lazy to fire it + VMWare up, grabbed a screenshot from http://www.joestewart.org/morphine-dll/:
For a free product, its nice, but I don't really understand why they don't release the source code. Speaking of that, don't they admit to using a GPL'd assembler?!?!? Seems like anyone could clearly request the source code from them. Anyway what I think:
-Overall, debugging interface is way better than IDA
-I like the CPU view especially
-Stack view is good. I find it has a tendency to force ESP view upon you and for some programs its nicer to use EBP view. Showing function args is very nice.
-User interface could be more flexible. I go long stretches without using the binary view below, making it wasted space
I'd use Immunity debugger more since I like Python, but the phone home bit creeps me out. Granted, I do typically operate on the assumption the machines I'm working on are owned, but its the principle I guess. Anyway, here's their screenshot of it (http://www.immunityinc.com/images/adwithtext5.jpg):
I don't know enough about this product to really write about it. I do know they tried to conform to OllyDbg stuff, so their GUI is similar. I think theres is more powerful though with such additions as the graph view shown above. The real kicker for me not using it is that it seems to claim some compatibility with OllyDbg plugins, but I found they mostly just crash.
The last big one I know of is PyDbg. If i recall though, it depends on PaiMei, which is a tutorial in itself to get running. Or maybe the dependency is the other way around. I think PaiMei in turn requires IDA, another discouragement to me. In any case, when someone showed it to me before, I wasn't very experienced in RE and it was over my head. So, I can't make any real observations about it other than I remember it being difficult to setup due to a large number of dependencies. Here's a token screenshot of PaiMei:
I must say, it has a somewhat appealing UI.
So, in summary, here are the features most lacking in other environments I want:
-Undo. Sure, it might be hard to undo a function call during debugging. But, is it really so hard to undo converting between code and data or undo a function rename?
-Good keyboard shortcuts. I'd like to think of what would be my "ideal" shortcut layout given no history, and then think about what I'd like to do to preserve compatibility. At the very least, I'd like a config menu where you can select IDA or such compatibility mode.
-Efficient use of screen area. While I tend to write source code generously spaced, I like data tightly packed. If someone wants to have a sparser UI, they should be able to drag widgets around to make it fit their taste. IDA and Metasm both have some elements of this, but there is still room for improvement.
-Easy to setup/run. I could be able to run some shell script or executable and it should just run. I shouldn't need to know anything about what language its implemented in.
-Multi platform. Many of these are Windows only
-Free base product. I don't mind paying for plugins if I want some extra kick, but the base product should be free.
Those are the main points for now. I'll probably update this more as I figure out what's important.