How do you edit a large file?

Editing large files is difficult because the people developing editors are building artificial limits into their editors. Here are the tricks of the trade for editing large files.

How large is large?

I regularly work with files greater than a GigaByte and find people calling 1 MB a large file. Where is the boundary between small and large?

Some old file systems have a file size limit of 2 GB. Everything larger than 2 GB is large. Modern file systems can handle files larger than the largest disk and the 2 GB limit is gone.

Working on Linux today, I cannot edit a file that is only 360 MB. On Linux, 360 MB is large. There are several choices for editing that size file in Windows and I have used some of them to edit files greater than 800 MB. 360 MB is not large on Windows.

Some editors edit only in memory and crash when memory runs out. If you have several applications open, you can run out of memory before you reach the end of the file you want to edit. If you have 1 GB of memory, 600~800 MB of memory can disappear just with the operating system, you email client, and the other applications you use every day. The memory remaining for the editor it tiny.

You can easily build a desktop computer with 64 GB of memory but notebooks do not give you the same range. 8 GB is the current practical limit for a notebook and constant use of common applications all day chews up 2 GB, leaving you with 6 GB for editing.

Editors often use double the size of the file because they open the original and create a copy. 6 GB of spare memory might let you edit only a 3 GB file with the typical editor.

Some editors crash when there is plenty of memory. My main Linux machine has 2.9 GB of memory unused after loading up my normal set of applications. All the standard Linux editors crash with files only one tenth of the size of the available memory.

Windows or Linux?

Windows gives you a better choice of editors for large files. You will find, on many projects, it is easier to copy the large files to a Windows machine and edit there. There are no problems created by the copy to a Windows machine when using modern software. If you use FTP, you will use Filezilla and you set the transfer to binary to preserve the same line endings in text files. Every other copy technique copies the correct way or offers the equivalent to the Filezilla binary option. Set your editor to keep the original format.

Using Windows and Linux along side each other is easy. I see lots of people work on Linux computers, rave about Linux, then open up their notebook computer and use the Windows supplied with the notebook.

Editing large files is one of the few things keeping Windows alive on one of my computers. Currently there is no choice of Windows or Linux. The answer is Windows and Linux.

I could buy a better editor for Linux but none of the commercial choices for Linux offer the type of Licence I like. I still need Windows for some other programs and will use Windows for large file editing until I find a good open source Linux alternative.

Editors

Bluefish slows down to the point of unusable with a 360 MB file. Bluefish also, eventually, crashes when working on a large number of medium size files or many small files. Bluefish is good for small and fast on any operating system then quickly slows down with minor increases in file size. I have not experimented with the various editing and display options to make editing of larger files faster.

EditPad Lite, www.editpadlite.com, is similar to Bluefish and is recommended by some users for large files. I have not yet tested EditPad Lite with my 360 GB file. EditPad Lite is free for non profit use and $25 for commercial use, is written for Windows and runs under Wine in Linux. If EditPad Lite does work with large files and Bluefish could be enhanced to work with the same size files, I do not think there is anything else in EditPad Lite that is better than Bluefish for my work.

Gedit immediately crashes with the 360 GB file.

Netbeans just sat there trying to read my 360 MB test text file. A mass of memory disappeared but there was no processor usage. After many minutes I gave up and closed Netbeans.

Ultraedit

Ultraedit handles large files and, with the right settings, can work at a reasonable speed. Unfortunately Ultraedit costs $99 for a version that works on every operating system and it is not open source. You can test it for 30 days free. Think about what you want to test before you start the 30 days.

When you are on the move from Windows to Linux, perhaps dual booting, you need the $99 version. If you are already committed to Linux or Mac Unix, you can buy a $60 version for a specific operating system.

The price becomes reasonable when buying many copies for a large team but you then have the problem of tracking licences. The price ends up so cheap that it is easier to buy more than you need and give away copies for use on spare computers, personal computers, and contractor's computers.

Windows freeware

There were a lot of free (freeware) and low cost (shareware) editors developed for Windows. PFE was the free project of a developer at a university and provided some superb features at that time. Crimson editor was another editor free I used for a specific purpose. Some of those editors handled the largest files you could create at that time, either the 700 MB limit of a file on a CD or the 2 GB limit of a file on the old FAT file system.

Back then Windows users were using NT and the NTFS file system without the 2 GB limit, making the 4.7 GB size of a DVD the largest practical limit. I did work in one file that compressed to 4.6 GB. The uncompressed original was 12 GB. I managed to edit it using one of the several editors I had on my NT computer.

None of the old editors exist because the developers moved on. Some produced commercial versions. Some moved to working on projects at commercial companies. The new projects all appear to focus on Web developers working on several small files. There are no current general purpose free safe always works editors for all files on the latest versions of Windows.