Visualizing Binary Files

Posted on 2026-08-05

I have developed a hex editor called bine. In its original state, it was quite austere—essentially just black and white.

The Quest for Color

Some time ago, the xxd tool in Vim introduced the ability to colorize output (though this isn't available in the Vim Classic version I prefer). I felt that bine would benefit from a similar feature.

My first attempt at a draft was a step in the right direction, as it made identifying blocks of ASCII text significantly easier. However, it wasn't perfect:

• I needed a broader color palette.
• The layout needed adjustment; removing the spaces between hex columns would make the data more compact.

Technical Hurdles

My TUI framework, movwin, is built on Python but utilizes ncurses (a C library) for its backend. I chose ncurses specifically for its long-term stability. However, I encountered a performance bottleneck:

Calls from Python to ncurses are surprisingly expensive.

Whether this latency is inherent to ncurses or a byproduct of the Python wrapper is irrelevant; the result was that bine struggled. While the editor wrote entire hex lines in a single call to maintain speed, the framework wasn't designed for rapid, multi-colored updates. Furthermore, I would have had to manage separate palettes for movwin's native light and dark themes.

The Alternative: Character-Based Visualization

Instead of fighting with colors, I pivoted to using specific Unicode characters in the ASCII column to represent non-printable data. Since these characters don't appear in standard binary files, there are no collisions.

Mapping Logic:

While this doesn't mirror every xxd case, it provides a sufficient overview. NUL bytes are immediately obvious, and the distinction between "plain text" and binary data is clear.

---

Macro-Scale Visualization

I wanted a way to visualize massive files—ranging from a few megabytes to several gigabytes—at a glance. My solution? Force the computer to treat the binary file as an image.

By mapping every byte in the range $[0x00, 0xFF]$ directly to a pixel, I could create a visual map of the data without any complex processing. I used the binary version of the PGM (Portable Gray Map) format.

Implementation

The "logic" was minimal: I simply calculated an appropriate width and height and prepended the PGM header to the raw file data.

printf 'P5\n%s %s 255\n' "$width" "$height" < cat "$infile"

Workflow Diagram

Case Studies

1. GORILLAS.BAS (EXE)

Note: This image represents the entire game and could theoretically be converted back into a working EXE.

Observations:

• The Top: A "noisy" area with high contrast (very dark and very bright pixels). I suspect it's a Relocation Table, though its position is odd. Since this is an EXEPACK file, that adds a layer of complexity.
• The Bottom: A consistent "dark gray" region. This is the ASCII string section; because the high bit is never set, the values are lower than the executable code.

2. Tarballs and Binaries

• Tarball: A visualization of a tarball containing a text file and an image reveals distinct patterns in the text area.
• Ruff Binary: This 25 MB file resulted in a PGM of $4096 \times 6225$ pixels.
  • A prominent black stripe near the top indicates a concentration of NUL bytes, likely representing aligned data tables.
  • I scaled this image down for the blog, though GIMP and nsxiv handle the full resolution easily.

3. Large Scale Files

I tested the approach on a 690 MB .iso file.

• PGM Creation: $0.2\text{s}$
• ImageMagick Resize: $6\text{s}$

The Limit: I attempted this with a 4 GB file, but ImageMagick crashed due to RAM exhaustion.

Final Thoughts

To handle files in the 4 GB+ range, one would need to iteratively resize the image (processing only a few rows at a time) before compressing. I haven't built this yet, and ImageMagick doesn't seem to support this specific workflow.

Regardless, for most of my needs, this is a viable tool. If a file is too large, I can simply split it.

Example: 8 GB Disk Image Mildly interesting, but potentially useful depending on the data pattern being sought.

Summary of Goals

• Implement ASCII-based visualization in bine.
• Create a raw binary-to-PGM converter.
• Analyze various file types (EXE, ISO, Tarball).
• Develop an iterative resizer for $\text{RAM} < \text{File Size}$ scenarios.