I am struggling a little coming to grips with the OTP development model as I convert some code into an OTP app.

I am essentially making a web crawler and I just don't quite know where to put the code that does the actual work.

I have a supervisor which starts my worker:

-behaviour(supervisor).
-define(CHILD(I, Type), {I, {I, start_link, []}, permanent, 5000, Type, [I]}).

init(_Args) ->          
  Children = [
    ?CHILD(crawler, worker)
  ],  
  RestartStrategy = {one_for_one, 0, 1},
  {ok, {RestartStrategy, Children}}.

In this design, the Crawler Worker is then responsible for doing the actual work:

-behaviour(gen_server).

start_link() ->
  gen_server:start_link(?MODULE, [], []).

init([]) ->
  inets:start(),        
  httpc:set_options([{verbose_mode,true}]), 
  % gen_server:cast(?MODULE, crawl),
  % ok = do_crawl(),
  {ok, #state{}}.

do_crawl() ->
  % crawl!
  ok.

handle_cast(crawl}, State) -> 
  ok = do_crawl(),
  {noreply, State};

do_crawl spawns a fairly large number of processes and requests that handle the work of crawling via http.

Question, ultimately is: where should the actual crawl happen? As can be seen above I have been experimenting with different ways of triggering the actual work, but still missing some concept essential for grokering the way things fit together.

Note: some of the OTP plumbing is left out for brevity - the plumbing is all there and the system all hangs together