Begin pontification!
There are a few golden rules in high performance server development:
- use the right algorithm / data structure for the job
- precompute as much as you can
- avoid system calls
Usually this boils down to an offline (or high latency) system that sends data to a fast serving tier via message replication or file push.
And basically the only way to read data from a file without syscalls is with mmap.
Too bad Java’s mmap implementation was so half-assed. In Java 1.5 (and possibly 1.6, I’m not sure), you…
- can’t map files larger than 2 GB (yay signed integers!)
- can’t do a bulk copy from an absolute position
- can’t even unmap an mapped file!
The rationale for the non-resolution or won’t-fix status of these bugs is pretty sad/hilarious, or as I prefer, “sadlarious”.
It’s basically “we designed NIO in 1973″ and “obscure security issue means we’ll reclaim that memory when we feel like it”.
5 comments ↓
I don’t understand most of this post! BUT OH EM GEE YOU CHANGED YOUR LAYOUT D:
The various mmap issues are just the price of the cross platform nature of Java. Java: Jack of all OSes, Master of None.
The signed 32 bit integer size limit on Buffers is a real bitch… I wrote about it here:
http://www.jroller.com/murphee/date/20080211
The solution for the next Java version is NIO2, which essentially copy/pastes the _whole_ NIO Buffer tree and replaces ‘int’ with ‘long’ to fix the issue… the blog post explains all the fun effects that’ll bring.
(Also… Java wouldn’t be in this silly situation without its primitives and the fact that people use 32 bit integers when they actually want something like a BigInteger…).
Oh well…
AFAIK you can only map 2GB at a time. I think it is possible to select any 2GB slice of the file by using an offset. It involves more code, but is doable.
You could always write your own via JNI. Most general purpose programming languages break down, when they are not used for general programming. That’s where good systems programmers come in to enhance the language. I’d rather pay 50 developers average going rates for writing in a common standard language and a few premium system programmers to do a few tweaks; then pay 50 programmers above average rates for an obscure language that’s not maintainable by the average joe programmer.
JSR-203
http://www.youtube.com/watch?v=yNRS1ssLPdQ
http://jcp.org/en/jsr/detail?id=203
Leave a Comment