I am trying to enforce maximum memory a program can consume on a Unix system. I thought ulimit -v should do the trick. Here is a sample Java program I have written for testing :

import java.util.*;
import java.io.*;

public class EatMem {

  public static void main(String[] args) throws IOException, InterruptedException {
    System.out.println("Starting up...");
    System.out.println("Allocating 128 MB of Memory");
    List<byte[]> list = new LinkedList<byte[]>();
    list.add(new byte[134217728]); //128 MB
    System.out.println("Done....");
  }
}

By default, my ulimit settings are (output of ulimit -a) :

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31398
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31398
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

When I execute my java program (java EatMem), it executes without any problems. Now I try to limit max memory available to any program launched in the current shell to 512MB by launching the following command :

ulimit -v 524288

ulimit -a output shows the limit to be set correctly (I suppose):

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31398
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31398
virtual memory          (kbytes, -v) 524288
file locks                      (-x) unlimited

If I now try to execute my java program, it gives me the following error:

Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

Ideally it should not happen as my Java program is only taking around 128MB of memory which is well within my specified ulimit parameters. If I change the arguments to my Java program as below:

java -Xmx256m EatMem

The program again works fine. While trying to give more memory than limited by ulimit like :

java -Xmx800m EatMem

results in expected error. Why the program fails to execute in the first case after setting ulimit ?

I have tried the above test on Ubuntu 11.10 and 12.0.4 with Java 1.6 and Java 7