Some days ago, our program crash. I found the crash in lua code. So I check lua code, found the stack overflow.

Please look this code In function luaD_precall:

1  if (!cl->isC) {  /* Lua function? prepare its call */
2       CallInfo *ci;
3       StkId st, base;
4       Proto *p = cl->p;
5       luaD_checkstack(L, p->maxstacksize);
6       func = restorestack(L, funcr);
7       if (!p->is_vararg) {  /* no varargs? */
8              base = func + 1;
9              if (L->top > base + p->numparams)
10             L->top = base + p->numparams;
11      }
12      else {  /* vararg function */
13             int nargs = cast_int(L->top - func) - 1;
14             base = adjust_varargs(L, p, nargs);
15             func = restorestack(L, funcr);  /* previous call may change the stack */
16      }
17      ci = inc_ci(L);  /* now `enter' new function */
18      ci->func = func;
19      L->base = ci->base = base;
20      ci->top = L->base + p->maxstacksize;
21      lua_assert(ci->top <= L->stack_last);
22      L->savedpc = p->code;  /* starting point */
23      ci->tailcalls = 0;
24      ci->nresults = nresults;
25      for (st = L->top; st < ci->top; st++)
26             setnilvalue(st);
27      L->top = ci->top;

In my program, the p->maxstacksize is 79 before line 5, the current stacksize is 51, after call luaD_checkstack, the stacksize grow to 130.

The lua function use vararg, so will run to line 14. Function adjust_varargs will be called.

static StkId adjust_varargs (lua_State *L, Proto *p, int actual) {
        int i;
        int nfixargs = p->numparams;
       Table *htab = NULL;
       StkId base, fixed;
       for (; actual < nfixargs; ++actual)
           setnilvalue(L->top++);
#if defined(LUA_COMPAT_VARARG)
       if (p->is_vararg & VARARG_NEEDSARG) { /* compat. with old-style vararg? */
           int nvar = actual - nfixargs;  /* number of extra arguments */
           lua_assert(p->is_vararg & VARARG_HASARG);
           luaC_checkGC(L);
       htab = luaH_new(L, nvar, 1);  /* create `arg' table */

In function adjust_varargs(), the lua function use “arg”, So luaC_checkGC will be called. In luaC_checkGC, the current lua stack size will be reduce to 65! The call stack like this:

luaC_step()
singlestep()
propagatemark()
traversestack()
checkstacksizes()
luaD_reallocstack()

But the p->maxstacksize is 79, the stacksize is not enough… When the program run to line 27,the L->top is bigger than L->stack_last, in the next operation, will cause crash!