Google Answers Logo
View Question
 
Q: Understanding ulimit output ( Answered 5 out of 5 stars,   1 Comment )
Question  
Subject: Understanding ulimit output
Category: Computers > Operating Systems
Asked by: davious-ga
List Price: $20.00
Posted: 27 Feb 2004 10:38 PST
Expires: 28 Mar 2004 10:38 PST
Question ID: 311442
I'm writing a perl/cgi script that will allow me detect what resource
limits exist on various virtual web hosts.  We need to deal with many
many different types of hosting environments with varied and often
arbitrary resource limits and being able to tell in advance what
limits exist and how they might affect perl/cgi scripts would be very
helpful.

Along those lines, I'm writing a script to parse the output from
"ulimit -a" and I need a better understanding of all the values that
are returned.  I'd like a detailed explanation of all of the following
values and comment on how they relate to perl/cgi scripts.

cpu time
file size
data seg size
stack size
core file size
max memory size
locked memory
max user processes
open files
virtual mem size
sbsize
pipe size

Please feel free to add any I may have missed.  I noticed some values
such as sbsize and pipe size are reported by some systems but not
others.

It would seem that some values (such as CPU time) would have a big
impact on cgi scripts, where as other values (such as core file size)
wouldn't really matter much at all.

Explain the different values in layman?s terms and how it relates to
cgi scripts. Especially for all the different memory values. Here's an
example for "cpu time":

CPU TIME - Limits the number of CPU seconds a process can use (not
clock seconds).  CPU time is the amount of time the CPU actual spends
executing processor instructions and is often much less than the total
program "runs time".  A program that did math for 10 seconds would use
a lot of CPU time, where as a program that simply paused for 10
seconds before display a message would use next to none.

When a process exceeds the soft CPU time limit it will be sent a XCPU
signal (and terminated if the signal isn't caught).  When it exceeds
the hard limit it will be sent a kill signal.

Request for Question Clarification by majortom-ga on 27 Feb 2004 11:08 PST
I would like to help you with this question and I have the relevant
expertise. However, it would be difficult to answer this with
certainty for all Unix variants without spending an inordinate amount
of time on this one question. Would you accept a thorough answer
applying to Linux systems? The information would certainly be useful
on all systems, however units used and the specific options available
may vary.

Clarification of Question by davious-ga on 27 Feb 2004 11:24 PST
Majortom, 

Sure, a general or linux centric answer is fine, but also post some
brief comments on what you think those possible differences with other
UNIX variations may be (if you know of any).
Answer  
Subject: Re: Understanding ulimit output
Answered By: majortom-ga on 27 Feb 2004 13:33 PST
Rated:5 out of 5 stars
 
Hello, davious-ga,

Thanks for the opportunity to answer this interesting question. 

(I have reproduced your answer for "cpu time," just for the record,
and added a few additional comments to it. The remainder of the
material is new, of course.)

CPU TIME - Limits the number of CPU seconds a process can use (not
clock seconds).  CPU time is the amount of time the CPU actual spends
executing processor instructions and is often much less than the total
program "runs time".  A program that did math for 10 seconds would use
a lot of CPU time, where as a program that simply paused for 10
seconds before display a message would use next to none. 

When a process exceeds the soft CPU time limit it will be sent a XCPU
signal (and terminated if the signal isn't caught).  When it exceeds
the hard limit it will be sent a kill signal.

VIRTUAL MEMORY SIZE - the most useful of the memory-related
limitations, because it includes all types of memory, including the
stack, the heap, and memory-mapped files. Attempts to allocate memory
in excess of this limit will fail with an out-of-memory error.
Specified in kilobytes, in the bash shell, as found in Linux and many
other systems.

DATA SEGMENT SIZE - Limits the amount of memory that a process can
allocate on the heap, as with malloc, calloc, C++ "new," and most
object creation in higher-level languages. Specified in kilobytes, in
the bash shell, as found in Linux and many other systems.

STACK SIZE - Limits the amount of memory a process can allocate on the
stack, as in the case of local variables in C, C++ and many other
languages. Limiting the stack size stops runaway recursive function
calls; however, it is possible to legitimately allocate large arrays
of data on the stack without unreasonable recursion. Specified in
kilobytes, in the bash shell, as found in Linux and many other
systems.

FILE SIZE - Limits the maximum size of any one file a process can
create. Specified in 512-byte blocks, in the bash shell, as found in
Linux and many other systems.

MAX USER PROCESSES - limits the number of processes the current user
is permitted to have in the process table at one time. Attempts to
start additional processes will fail.

OPEN FILES - limits the number of file descriptors belonging to a
single process. "File descriptors" includes not only files but also
sockets for Internet communication. Attempts to open additional file
descriptors will fail.

CORE FILE SIZE - Limits the size of a "core" file left behind when a
process encounters a segmentation fault or other unexpected fatal
error. Core files are images of the entire memory space used by the
process at the time of the crash, and can be used to examine the state
of the process at that time with a debugger, such as gdb. In most
cases core files are of limited utility if the process was not
compiled with debugging flags enabled. Disabling core files with a
core file size limit of zero is reasonable when the processes expected
are not C/C++ applications compiled with -g or similar or no such
debugging capability is desired. Specified in 512-byte blocks.

LOCKED MEMORY - this parameter limits the maximum amount of memory
that can be "locked down" to a specific address in physical memory by
a given process. In practice, only the root user can lock memory in
this fashion. Specified in kilobytes, in the bash shell, as found in
Linux and many other systems.

MAX RESIDENT SET SIZE - this parameter limits the amount of memory
that can be "swapped in" to physical RAM on behalf of any one process.
While patches have been offered, this limit is not actually enforced
in mainstream Linux kernels; it briefly appeared in the 2.6 series but
has been removed for now due to problems. Specified in kilobytes, in
the bash shell, as found in Unix and many other systems.

MAX TOTAL SOCKET BUFFER SIZE - this parameter limits the total amount
of memory that may be taken up by buffers holding network data on
behalf of a given process. Originated as a response to
denial-of-service attacks which worked by initiating large numbers of
connections and flooding them with input; the operating system would
buffer this data patiently while waiting for the receiving process to
cope with it. Implemented in FreeBSD; not implemented in Linux, which
does not address this issue via a configurable limit. Specified in
bytes.

PIPE SIZE - when two Unix processes communicate via a pipe or FIFO
(first in first out) buffer, as in the simple case of paging through a
directory listing with the command "ls | more", the output of the
first command is buffered before transmission to the second. The size
of this buffer, in bytes, is the pipe size. This is typically not an
adjustable parameter, except at kernel compile time.

* * *

Thanks again, I enjoyed answering this question.

Sources of information:

(re: Linux and RLIMIT_RSIZE:) LWN: Patch: mm swapping improvements
http://lwn.net/Articles/68843/

(re: SBSIZE and FreeBSD:) 923Mbits/s across the ocean
http://www.cctec.com/maillists/nanog/current/msg00217.html

The GNU bash manpage

The FreeBSD sh manpage

(Painful) personal experience

Request for Answer Clarification by davious-ga on 27 Feb 2004 19:22 PST
DATA SEGMENT SIZE is per process.  What about VIRTUAL MEMORY SIZE?  Is
that per process or is it a max cap for all the processes a specific
user might run?

Clarification of Answer by majortom-ga on 28 Feb 2004 06:26 PST
My apologies for leaving out that information. The maximum virtual
memory size is for a single process. (Source: bash manpage.)

Request for Answer Clarification by davious-ga on 28 Feb 2004 09:30 PST
A couple clarifications: 

So there is no limit that would restrict the total memory pool
available to all processes from a certain user?  But rather this would
be done with the "process memory limits" and the "max process"
settings?

As I understand it, Apache's RLimitMEM directive sets the DATA SEGMENT
SIZE, any ideas why it set that over VIRTUAL MEMORY SIZE?

For the purposes of determining how much memory is available to a
perl/cgi process what values should I look at?  Would it be the lower
of DATA SEGMENT SIZE and VIRTUAL MEMORY SIZE?

Clarification of Answer by majortom-ga on 28 Feb 2004 11:09 PST
Hello, davious-ga,

You're correct, there is no one limit that controls total memory
available to all processes belonging to one user, at least not via the
ulimit command as found in bash.

Apache uses RLIMIT_AS (described as "address space limit", or virtual
memory size, in /usr/include/asm/resource.h) by preference, only using
RLIMIT_DATA or RLIMIT_VMEM if RLIMIT_AS is not defined by the host
operating system. This is according to my inspection of
main/util_script.c in Apache 1.3.29. Modern 2.x versions of bash's
"ulimit -v" command set RLIMIT_AS when built on linux, or RLIMIT_VMEM
on SysV derived systems.

The lower of (data segment size + stack size) and (virtual memory
size) is a good yardstick for the memory available to you.
davious-ga rated this answer:5 out of 5 stars
Thanks for all the info!

Comments  
Subject: Re: Understanding ulimit output
From: majortom-ga on 28 Feb 2004 11:09 PST
 
Hello, davious-ga,

You're correct, there is no one limit that controls total memory
available to all processes belonging to one user, at least not via the
ulimit command as found in bash.

Apache uses RLIMIT_AS (described as "address space limit", or virtual
memory size, in /usr/include/asm/resource.h) by preference, only using
RLIMIT_DATA or RLIMIT_VMEM if RLIMIT_AS is not defined by the host
operating system. This is according to my inspection of
main/util_script.c in Apache 1.3.29. Modern 2.x versions of bash's
"ulimit -v" command set RLIMIT_AS when built on linux, or RLIMIT_VMEM
on SysV derived systems.

The lower of (data segment size + stack size) and (virtual memory
size) is a good yardstick for the memory available to you.

Important Disclaimer: Answers and comments provided on Google Answers are general information, and are not intended to substitute for informed professional medical, psychiatric, psychological, tax, legal, investment, accounting, or other professional advice. Google does not endorse, and expressly disclaims liability for any product, manufacturer, distributor, service or service provider mentioned or any opinion expressed in answers or comments. Please read carefully the Google Answers Terms of Service.

If you feel that you have found inappropriate content, please let us know by emailing us at answers-support@google.com with the question ID listed above. Thank you.
Search Google Answers for
Google Answers  


Google Home - Answers FAQ - Terms of Service - Privacy Policy