Thursday, March 19, 2009

Filesystem Specifications, or EXT4 "Losing Data"

This is a bit off-topic for this blog, but the particular issue at hand bugs the heck out of me.

EXT4 "meets" the POSIX specifications for filesystems. The specification does not make any requirements for data to be written out in any order - and for very good reason. If the application developer -requires- data to be written out in order, they should serialise their operations through use of fsync(). If they do -not- require it, then the operating system should be free to optimise away the physical IO operations.

As a clueful(!) application developer, -I- appreciate being given the opportunity to provide this kind of feedback to the operating system. I don't want one or the other. I'd like to be able to use both where and when I choose.

Application developers - stop being stupid. Fix your applications. Read and understand the specification and what it provides -everyone- rather than just you.


  1. What about shell scripts? No shell (I checked ksh93, bash and zsh) fsyncs before close, so `echo foo >> bar.tmp && mv bar.tmp bar` is vulnerable. Calling sync(1) is no good, since that just calls sync(2) which can return before it completes.

  2. Then the shells need to be fixed? :)

  3. Which leads to the "everyone calling fsync all the time" which leads to people suggesting "yes-I-really-mean-it" flags for fsync.