Herodotus DB

Unique Features

Crash proof

Herodotus DB uses an append-only file format. So when a transaction is committed, Herodotus DB appends a chunk of data onto the end of the database file. Before the transaction is committed, Herodotus DB prepares the data chunk in memory, so the disk-write operation can be done almost instantaneously. So the chance that the computer crashes during the write operation (e.g. due to a power failure) is extremely small.

Even so, we wanted to make Herodotus DB 100% crash proof. So before each append operation, Herodotus DB creates a small companion file next to the database file that contains the current valid length of the database file. Then Herodotus DB appends the data chunk to the database, and finally Herodotus DB deletes the companion file.

When Herodotus DB opens a database, it looks for the companion file. If the companion file exists, then it can check that the database file length matches the length written inside the companion file. If there is a mismatch, then Herodotus DB can automatically roll-back the interrupted transaction. This mechanism makes Herodotus DB completely crash proof, even if the computer crashes during the disk-write operation.

If your computer crashes, just start it up again and Herodotus DB will always be able to open up your databases.

Undo/Redo

Herodotus DB applications can provide Undo and Redo commands. For example, GuiDb is a program supplied as part of Herodotus DB. This application lets you create, view and modify Herodotus databases either locally or remotely. All commands in GuiDb can be undone and redone. So if you delete a Table, and then realize that you need that table, then you can Undo the delete operation.

Online Backup

With other database systems that do not use an append-only file format, the backup process often needs to lock the whole database for the duration of the backup.

But Herodotus DB uses an append-only file format, so it is easy to backup a database even while it is being used. The HdbBackup program provides functions for full backups and incremental backups.

Serializable Transactions

SQL/92 defines four isolation levels for transactions:

READ UNCOMMITTED
READ COMMITTED
REPEATABLE READ
SERIALIZABLE

Each level increases the isolation between concurrent transactions. With concurrent transactions, more isolation is better, since it means that transactions will not be polluted with data from other transactions that may or may not be committed. Of course, the best isolation level for reliable applications is SERIALIZABLE, so all transactions in Herodotus DB are SERIALIZABLE. Other database systems provide support for the unsafe lower levels to allow greater throughput.

Memory handling

Herodotus DB is programmed entirely in Java, which provides advanced memory handling. Herodotus DB does not "leak" memory, so in theory a Herodotus DB Server can run uninterrupted forever. With 64-bit hardware and a 64-bit JVM, Herodotus DB will run unchanged and will be able to use more than 4 GB of RAM.

Exception handling

The details of this feature are technical, but the end result is that the Herodotus DB software is extremely robust.

For the technical minded, Herodotus DB is programmed entirely in Java, which provides fully typed exceptions as a basic part of the language.

No database size limit

The Herodotus DB file format is designed to have no size limit. All of the pointers from one part of the file to another part of the file are stored as variable length integers. So near pointers are stored in 1 byte, and far pointers are stored in two, three or more bytes. So as well as having no size limit, Herodotus DB files are also compact.

Although the file format is designed to have no size limit, the current software implementation (i.e. the Java code that reads and writes the database file) uses 64-bit integers to navigate around the database file. This means that the current software is "limited" to about 18000 petabytes. In practice, this is more than the total capacity of all the hard disks ever manufactured, and will probably not be a limit for many years to come. However, if it ever does become a limit then the software can be upgraded without changing the database file format.

Distributed Transactions

Herodotus DB supports distributed transactions. This means that a single transaction can involve linked changes in more than one database. Herodotus DB uses a Two Phase Commit process to ensure that the transaction is committed in all databases, or rolled back in all databases.

Replicatable

Traditionally, one way to "scale up" a database server is to replicate the database onto more servers. In this way, many servers can serve up replica copies of the original database. This technique is often used for Web servers (sometimes called "Mirror Servers").

In theory, replication is a simple way to let more clients access a single logical database simultaneously. But what happens when the original database is modified? All of the replica databases must have the same modification applied. With other database systems, keeping all of the servers "in sync" becomes more and more complicated as the number of replicas grows.

With Herodotus DB, managing a replicated database is easy and scalable. This is because Herodotus DB uses an append-only file format. So when one of the server databases is modified, all that the system needs to do to keep the replica servers in sync is to copy the appended data chunk from the modified database to the replica databases.

In addition, the simplicity of replicating Herodotus databases means that replica servers can be arranged in a "tree". Each node in the tree is a replica server. Each replica server can have its own child replica servers further down the tree. In this way, a single logical database can be served by thousands of server machines, and can be accessed by millions of clients simultaneously.

Multi-User

Herodotus DB uses an append-only file format. This means that while one user is modifying the database, other users can be querying old versions of the database. None of the users needs to block or delay the database being accessed by the other users.

To be fair, this type of non-blocking multi-user functionality is provided by other client-server databases, as the server can cache the uncommitted modifications and/or the pre-modified data in the server's memory. However, Herodotus DB provides non-blocking multi-user access for local file database connections, as well as client-server database connections.

Multi-Threaded

Herodotus DB is programmed in Java, so it can easily use multi-threaded algorithms. The Java language provides support for fine grained multi-thread synchronization. This means that a Herodotus DB Server can efficiently support lots of concurrent users, especially when running on a multi-CPU machine.

Platform independent

Herodotus DB is programmed entirely in Java, so it can immediately be run on many different operating systems, and different hardware.

In addition, the database file format is platform independent. This means that a Win32 Herodotus DB Server can be replicated by a Linux Herodotus DB Server.

Language independent

By default, text data in Herodotus DB is stored using Unicode, so text data can be processed independently of language or locale. This means that it is much easier to write multi-national applications with Herodotus DB.

However, for applications that wish to use different character sets, Herodotus DB supports over 50 character set encodings and over 40 character set collations.

SQL/92

The SQL/92 standard defines three language levels: Full SQL, Intermediate SQL and Entry SQL. Herodotus DB supports the Full SQL level of the SQL/92 standard. This means that applications written for generic databases should be portable to Herodotus DB.

JDBC

Currently, Herodotus DB can be programmed by Java applications using SQL and JDBC. Herodotus DB supports JDBC 3, including scrollable and updatable cursors.

Programmers can easily develop, debug and test their applications using local-file Herodotus databases. Then, when the application is ready for field testing and deployment, they can switch to using a client-server Herodotus database by changing a single line of code.

Local and Remote access

Herodotus DB can work in two modes:

Local mode, using direct file access
Remote mode, using Herodotus Net

In Local mode, the application can open a database as a file anywhere in the Local Area Network. Several different users in the LAN can open and read the same database file simultaneously. In Local mode only one of the users can edit the file at a time. So Local mode supports "N-Readers And 1-Writer" concurrency, using file-locking.

In Remote mode the server process opens the database file, and the client processes access the database via the server. The clients connect to the server using "Herodotus Net" protocol. Herodotus Net is based on TCP/IP, so it can work over a Local Area Network, or over the Internet. The Internet Assigned Numbers Authority (IANA) has assigned port 3921 for Herodotus Net. Since the server process can use row-locking, Remote mode supports "N-Readers And N-Writers" concurrency.

Both Local mode and Remote mode use the same database format. In fact, a single database file could be used in Local mode and Remote mode at the same time (the server process would count as one of the local users).

Historical

Append-only files have a special property: all old versions of the file can be recovered by simply cutting off the tail of the file. The more tail you cut off, the further back into the history of the file you go. It's a bit like archeology, where the deeper you dig, the further back in time you go.

Herodotus DB uses an append-only format for the database file. In addition, Herodotus DB stores information about how long the database file was after each transaction. This means that Herodotus DB can instantly access all historical versions of the database, simply by ignoring tail-end portions of the file.

Tamper proof

One security danger faced by database users is "data tampering". This is where a miscreant modifies data in the database, without the proper authorities being aware of the change. An example would be a bank database, where a rogue employee moved money from other accounts into their own account. The insidious aspect of this sort of data tampering is that the miscreant could completely cover their tracks, so their actions cannot be detected after the event.

But with Herodotus DB, data tampering is much harder. Herodotus DB automatically logs information about every transaction, such as who did it, when it was done, and what machine it was done from. This makes it almost impossible for miscreants to hide their tracks after data tampering.

Top of Page