Search Results

Search found 434 results on 18 pages for 'jo erlang'.

Page 5/18 | < Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >

• HTTP crawler in Erlang

  - by ctp

  I'm coding on a simple HTTP crawler but I have an issue running the code at the bottom. I'm requesting 50 URLs and get the content of 20+ back. I've generated few files with 150kB size each to test the crawler. So I think the 20+ responses are limited by the bandwidth? BUT: how to tell the Erlang snippet not to quit until the last file is not fetched? The test data server is online, so plz try the code out and any hints are welcome :) -module(crawler). -define(BASE_URL, "http://46.4.117.69/"). -export([start/0, send_reqs/0, do_send_req/1]). start() -> ibrowse:start(), proc_lib:spawn(?MODULE, send_reqs, []). to_url(Id) -> ?BASE_URL ++ integer_to_list(Id). fetch_ids() -> lists:seq(1, 50). send_reqs() -> spawn_workers(fetch_ids()). spawn_workers(Ids) -> lists:foreach(fun do_spawn/1, Ids). do_spawn(Id) -> proc_lib:spawn_link(?MODULE, do_send_req, [Id]). do_send_req(Id) -> io:format("Requesting ID ~p ... ~n", [Id]), Result = (catch ibrowse:send_req(to_url(Id), [], get, [], [], 10000)), case Result of {ok, Status, _H, B} -> io:format("OK -- ID: ~2..0w -- Status: ~p -- Content length: ~p~n", [Id, Status, length(B)]); Err -> io:format("ERROR -- ID: ~p -- Error: ~p~n", [Id, Err]) end. That's the output: Requesting ID 1 ... Requesting ID 2 ... Requesting ID 3 ... Requesting ID 4 ... Requesting ID 5 ... Requesting ID 6 ... Requesting ID 7 ... Requesting ID 8 ... Requesting ID 9 ... Requesting ID 10 ... Requesting ID 11 ... Requesting ID 12 ... Requesting ID 13 ... Requesting ID 14 ... Requesting ID 15 ... Requesting ID 16 ... Requesting ID 17 ... Requesting ID 18 ... Requesting ID 19 ... Requesting ID 20 ... Requesting ID 21 ... Requesting ID 22 ... Requesting ID 23 ... Requesting ID 24 ... Requesting ID 25 ... Requesting ID 26 ... Requesting ID 27 ... Requesting ID 28 ... Requesting ID 29 ... Requesting ID 30 ... Requesting ID 31 ... Requesting ID 32 ... Requesting ID 33 ... Requesting ID 34 ... Requesting ID 35 ... Requesting ID 36 ... Requesting ID 37 ... Requesting ID 38 ... Requesting ID 39 ... Requesting ID 40 ... Requesting ID 41 ... Requesting ID 42 ... Requesting ID 43 ... Requesting ID 44 ... Requesting ID 45 ... Requesting ID 46 ... Requesting ID 47 ... Requesting ID 48 ... Requesting ID 49 ... Requesting ID 50 ... OK -- ID: 49 -- Status: "200" -- Content length: 150000 OK -- ID: 47 -- Status: "200" -- Content length: 150000 OK -- ID: 50 -- Status: "200" -- Content length: 150000 OK -- ID: 17 -- Status: "200" -- Content length: 150000 OK -- ID: 48 -- Status: "200" -- Content length: 150000 OK -- ID: 45 -- Status: "200" -- Content length: 150000 OK -- ID: 46 -- Status: "200" -- Content length: 150000 OK -- ID: 10 -- Status: "200" -- Content length: 150000 OK -- ID: 09 -- Status: "200" -- Content length: 150000 OK -- ID: 19 -- Status: "200" -- Content length: 150000 OK -- ID: 13 -- Status: "200" -- Content length: 150000 OK -- ID: 21 -- Status: "200" -- Content length: 150000 OK -- ID: 16 -- Status: "200" -- Content length: 150000 OK -- ID: 27 -- Status: "200" -- Content length: 150000 OK -- ID: 03 -- Status: "200" -- Content length: 150000 OK -- ID: 23 -- Status: "200" -- Content length: 150000 OK -- ID: 29 -- Status: "200" -- Content length: 150000 OK -- ID: 14 -- Status: "200" -- Content length: 150000 OK -- ID: 18 -- Status: "200" -- Content length: 150000 OK -- ID: 01 -- Status: "200" -- Content length: 150000 OK -- ID: 30 -- Status: "200" -- Content length: 150000 OK -- ID: 40 -- Status: "200" -- Content length: 150000 OK -- ID: 05 -- Status: "200" -- Content length: 150000 Update: thanks stemm for the hint with the wait_workers. I've combined your and mine code but same behaviour :( -module(crawler). -define(BASE_URL, "http://46.4.117.69/"). -export([start/0, send_reqs/0, do_send_req/2]). start() -> ibrowse:start(), proc_lib:spawn(?MODULE, send_reqs, []). to_url(Id) -> ?BASE_URL ++ integer_to_list(Id). fetch_ids() -> lists:seq(1, 50). send_reqs() -> spawn_workers(fetch_ids()). spawn_workers(Ids) -> %% collect reference to each worker Refs = [ do_spawn(Id) || Id <- Ids ], %% wait for response from each worker wait_workers(Refs). wait_workers(Refs) -> lists:foreach(fun receive_by_ref/1, Refs). receive_by_ref(Ref) -> %% receive message only from worker with specific reference receive {Ref, done} -> done end. do_spawn(Id) -> Ref = make_ref(), proc_lib:spawn_link(?MODULE, do_send_req, [Id, {self(), Ref}]), Ref. do_send_req(Id, {Pid, Ref}) -> io:format("Requesting ID ~p ... ~n", [Id]), Result = (catch ibrowse:send_req(to_url(Id), [], get, [], [], 10000)), case Result of {ok, Status, _H, B} -> io:format("OK -- ID: ~2..0w -- Status: ~p -- Content length: ~p~n", [Id, Status, length(B)]), %% send message that work is done Pid ! {Ref, done}; Err -> io:format("ERROR -- ID: ~p -- Error: ~p~n", [Id, Err]), %% repeat request if there was error while fetching a page, do_send_req(Id, {Pid, Ref}) %% or - if you don't want to repeat request, put there: %% Pid ! {Ref, done} end. Running the crawler forks fine for a handful of files, but then the code even doesnt fetch the entire files (file size each 150000 bytes) - he crawler fetches some files partially, see the following web server log :( 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /10 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /1 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /3 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /8 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /39 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /7 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /6 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /2 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /5 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /50 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /9 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /44 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /38 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /47 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /49 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /43 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /37 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /46 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /48 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /36 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /42 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /41 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /45 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /17 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /35 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /16 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /15 HTTP/1.1" 200 17020 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /21 HTTP/1.1" 200 120360 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /40 HTTP/1.1" 200 117600 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /34 HTTP/1.1" 200 60660 "-" "-" Any hints are welcome. I have no clue what's going wrong there :(

  Read the article

• How to switch between the upper and lower pane in emacs?

  - by Anthony Kong

  I am using the erlang mode in Aquamacs. The mode, by default, creates a new pane and buffer "*erlang*" when I hit C-C C-K to compile an erlang file. (as seen in the attached screen shot) What is the easiest way to switch between these two panes? I do not think "C-x b" is applicable in this case because 'C-X b' then "*erlang" is slow considering I have to switch between my files and the erlang shell rather frequently.

  Read the article

• How to make Processes Run Parallel in Erlang?

  - by Ankit S

  Hello, startTrains() -> TotalDist = 100, Trains = [trainA,trainB ], PID = spawn(fun() -> train(1,length(Trains)) end), [ PID ! {self(),TrainData,TotalDist} || TrainData <- Trains], receive {_From, Mesg} -> error_logger:info_msg("~n Mesg ~p ~n",[Mesg]) after 10500 -> refresh end. so, I created Two Processes named trainA, trainB. I want to increment these process by 5 till it gets 100. I made different processes to make each of the train (process) increments its position parallely. But I was surprised to get the output sequentially i.e process trainA ends then process trainB starts. But I want to increment themselves at simultaneously. I want to run processes like this trainA 10 trainB 0 trainA 15 trainB 5 .... trainA 100 trainB 100 but I m getting trainA 0 .... trainA 90 trainA 95 trainA 100 trainA ends trainB 0 trainB 5 trainB 10 ..... trainB 100 How to make the processes run parallel/simultaneously? Hope you get my Q's. Please help me.

  Read the article

• Cannot spawn an erlang supervisor from the shell.

  - by drfloob

  I've implemented a gen_server and supervisor: test_server and test_sup. I want to test them from the shell/CLI. I've written their start_link functions such that their names are registered locally. I've found that I can spawn the test_server from the command line just fine, but a spawned test_sup does nothing whatsoever. Why is this? For example, I can spawn a test_server by executing: 1> spawn(test_server, start_link, []). <0.39.0> 2> registered(). [...,test_server,...] I can interact with the server, and everything appears fine. However, if I try to do the same thing with test_sup, no new names/Pids are registered, and it looks like my test_server was not spawned at all. I'd assume I coded an error in my supervisor, but this method of starting my supervisor works perfectly fine: 1> {ok, Pid}= test_sup:start_link([]). {ok, <0.39.0>} 2> unlink(Pid). true 3> registered(). [...,test_server,test_sup,...] Why is it that I can spawn a gen_server but not a supervisor?

  Read the article

• Erlang OTP application design

  - by Toby Hede

  I am struggling a little coming to grips with the OTP development model as I convert some code into an OTP app. I am essentially making a web crawler and I just don't quite know where to put the code that does the actual work. I have a supervisor which starts my worker: -behaviour(supervisor). -define(CHILD(I, Type), {I, {I, start_link, []}, permanent, 5000, Type, [I]}). init(_Args) -> Children = [ ?CHILD(crawler, worker) ], RestartStrategy = {one_for_one, 0, 1}, {ok, {RestartStrategy, Children}}. In this design, the Crawler Worker is then responsible for doing the actual work: -behaviour(gen_server). start_link() -> gen_server:start_link(?MODULE, [], []). init([]) -> inets:start(), httpc:set_options([{verbose_mode,true}]), % gen_server:cast(?MODULE, crawl), % ok = do_crawl(), {ok, #state{}}. do_crawl() -> % crawl! ok. handle_cast(crawl}, State) -> ok = do_crawl(), {noreply, State}; do_crawl spawns a fairly large number of processes and requests that handle the work of crawling via http. Question, ultimately is: where should the actual crawl happen? As can be seen above I have been experimenting with different ways of triggering the actual work, but still missing some concept essential for grokering the way things fit together. Note: some of the OTP plumbing is left out for brevity - the plumbing is all there and the system all hangs together

  Read the article

• Overuse of guards in Erlang?

  - by dagda1

  Hi, I have the following function that takes a number like 5 and creates a list of all the numbers from 1 to that number so create(5). returns [1,2,3,4,5]. I have over used guards I think and was wondering if there is a better way to write the following: create(N) -> create(1, N). create(N,M) when N =:= M -> [N]; create(N,M) when N < M -> [N] ++ create(N + 1, M). Thanks, Paul

  Read the article

• Erlang ODBC parameter query with null parameters

  - by Schlomer

  Is it possible to pass null values to parameter queries? For example Sql = "insert into TableX values (?,?)". Params = [{sql_integer, [Val1]}, {sql_float, [Val2]}]. % Val2 may be a float, or it may be the atom, undefined odbc:param_query(OdbcRef, Sql, Params). Now, of course odbc:param_query/3 is going to complain if Val2 is undefined when trying to match to a sql_float, but my question is... Is it possible to use a parameterized query, such as: Sql = "insert into TableY values (?,?,?,?,?,?,?,?,?)". with any null parameters? I have a use case where I am dumping a large number of real-time data into a database by either inserting or updating. Some of the tables I am updating have a dozen or so nullable fields, and I do not have a guarantee that all of the data will be there. Concatenating a SQL together for each query, checking for null values seems complex, and the wrong way to do it. Having a parameterized query for each permutation is simply not an option. Any thoughts or ideas would be fantastic! Thank you!

  Read the article

• Decode JSON with mochijson2 in Erlang

  - by Jon Romero

  I have a var that has some JSON data: A = <<"{\"job\": {\"id\": \"1\"}}">>. Using mochijson2, I decode the data: Struct = mochijson2:decode(A). And now I have this: {struct,[{<<"job">>,{struct,[{<<"id">>,<<"1">>}]}}]} I am trying to read (for example), "job" or "id". I tried using struct.get_value but it doesn't seem to work. Any ideas?

  Read the article

• Erlang: Find intersections in a ets table

  - by Yadira Suazo

  I have an ets with the next items: [at, {other_place}, me], [other_place, {place}, {other_place}]], [at, {place}, me], [on, {surface}, {object}], [small, {object}] And I have the list [[at, door, me],[on, floor, chair],[small, bannanas]] I need to compare every item in the ets table to an item in the list and if the first one is the same atom, replace the items in round brackets. So if I have [at, door, me], it matches with [at, {other_place}, me], I have to change {other_place} for the atom door in all the ets table.

  Read the article

• Erlang : flattening a list of strings

  - by ErJab

  I have a list like this: [["str1","str2"],["str3","str4"],["str5","str6"]] And I need to convert it to ["str1", "str2", "str3", "str4", "str5", "str6"] How do I do this? The problem is that I'm dealing with lists of strings, so when I do lists:flatten([["str1","str2"],["str3","str4"],["str5","str6"]]) I get "str1str2str3str4str5str6" However, if the elements of the original list where just atoms, then lists:flatten would have given me what I needed. How do I achieve the same with strings?

  Read the article

• functional languages (erlang, f#, haskell, scala)

  - by Michael Ellick Ang

  Hi, 1) Are functional languages suited for web applications development ? 2) Are functional languages suited for business/erp/crm type of applications ?

  Read the article

• Erlang - Eccentricity with accented characters and string literal

  - by erevfall

  Hey, I am trying to implement a function to differentiate between french vowels and consonnants. It should be trivial, let's see what I wrote down : -define(vowels,"aeiouyàâéèêëôù"). is_vowel(Char) -> C = string:to_lower(Char), lists:member(C,?vowels). It's pretty simple, but it behaves incorrectly : 2> char:is_vowel($â). false While the interpreted version works well : 3> C = string:to_lower($â), lists:member(C,"aeiouyàâéèêëôù"). true What's going on ?

  Read the article

• erlang - startup script

  - by demas

  To start my program I do the next sequence: $ erl > c(module1). > c(module2). > c(modulen). modulen:start(). Is there any possibility to create script that allow me to launch my program ?

  Read the article

• Erlang rb module

  - by Justin

  When looking up messages in a sasl log using rb:list() or rb:show(), rb seems to dump the output in the console and return 'ok'; is there any way to configure rb to get it to return the actual log message ? Thanks

  Read the article

• Erlang: simple refactoring

  - by alexey

  Consider the code: f(command1, UserId) -> case is_registered(UserId) of true -> %% do command1 ok; false -> not_registered end; f(command2, UserId) -> case is_registered(UserId) of true -> %% do command2 ok; false -> not_registered end. is_registered(UserId) -> %% some checks Now imagine that there are a lot of commands and they are all call is_registered at first. Is there any way to generalize this behavior (refactor this code)? I mean that it's not a good idea to place the same case in all the commands.

  Read the article

• Erlang bit syntax variable issue

  - by Jimmy Ruska

  Is there any way to format this so it's a valid expression, without adding another step? <<One:8,_:(One*8)>> = <<1,9>>. * 1: illegal bit size These work <<One:8,_:(1*8)>> = <<1,9>>. <<1,9>> <<Eight:8,_:Eight>> = <<8,9>>. <<8,9>> I'm trying to parse a binary with nested data with list comprehensions instead of stacking accumulators.

  Read the article

• Erlang - list comprehensions - populating records

  - by tbikeev

  I have a simple record structure consisting of a header (H) and a list of the data lines (D) 1:N. All header lines must start with a digit. All data lines have a leading whitespace. There also might be some empty lines (E) in between that must be ignored. L = [H, D, D, E, H, D, E, H, D, D, D]. I would like to create a list of records: -record(posting,{header,data}). using list comprehension. Whats the best way to do it?

  Read the article

• How can I set up a fault-tolerant web-service built with Erlang/OTP?

  - by Jonas

  I would like to setup a fault-tolerant web-service. I will build the web-service with Erlang/OTP. At the beginning the web-service will be hosted on a few VPS. Each VPS has its own IP-address, and I can use more if IPs if I need. I would like to have the domain name pointing to a single IP-address. How can setup my Erlang/OTP-application to be fault-tolerant behind a single IP-address? Do I need to use VLAN? Is there a way my Erlang/OTP-application can use heartbeats and handle virtual IP-addresses to route the traffic? or how should I solve this problem?

  Read the article

• Log out of an SSH session into Erlang VM without stopping the VM or leaving stale processes

  - by Garret Smith

  I have an Erlang application running as a daemon, configured as an SSH server. I can connect to it with an SSH client and I get the standard Erlang REPL. If I 'q().' I shut down the Erlang VM, not the connection. If I close the connection ('~.' for OpenSSH, close the window in PuTTY) some processes remain under the sshd_sup/ssh_system_xx_sup tree. These appear to be stale shell processes. I do not see any exported function in the shell module that would let me shut down the shell (and therefore the SSH connection) without affecting the entire VM. How should I be logging out of the SSH session to not leave stale processes in the VM?

  Read the article

• Is there a module that implements an efficient array type in Erlang?

  - by dsmith

  I have been looking for an array type with the following characteristics in Erlang. append(vector(), term()) O(1) nth(Idx, vector()) O(1) set(Idx, vector(), term()) O(1) insert(Idx, vector(), term()) O(N) remove(Idx, vector()) O(N) I normally use a tuple for this purpose, but the performance characteristics are not what I would want for large N. My testing shows the following performance characteristics... erlang:append_element/2 O(N). erlang:setelement/3 O(N). I have started on a module based on the clojure.lang.PersistentVector implementation, but if it's already been done I won't reinvent the wheel.

  Read the article

• Current state of Erlang web development? Frameworks, Template languages

  - by pommonico

  Django is to Python as Rails is to Ruby. What is the current front-runner for Erlang? Recent Erlang web activity has resulted in many new frameworks but they are all still moving targets. ErlyWeb is very far along but now seems to be dead. If you were to invest significant capital into a new web product (i.e. not a hobby, you use your own money and you use other's investment in you with expectation of payback), would you choose Erlang as a web platform? If so, what framework and technologies would you choose?

  Read the article

• Side effect-free interface on top of a stateful library

  - by beta

  In an interview with John Hughes where he talks about Erlang and Haskell, he has the following to say about using stateful libraries in Erlang: If I want to use a stateful library, I usually build a side effect-free interface on top of it so that I can the use it safely in the rest of my code. What does he mean by this? I am trying to think of an example of how this would look, but my imagination and/or knowledge is failing me.

  Read the article

• What's the difference between "green threads" and Erlang's processes?

  - by DR

  After reading about Erlang's lighweight processes I was pretty much sure that they were "green threads". Until I read that there are differences between green threads and Erlang's processes. But I don't get it. What are the actual differences?

  Read the article

• Any tips on how to build Erlang Beam files?

  - by Zubair

  I am currently using: c(module_name) : to build my Erlang files ones by one, and I was wondering about how other people handle the build process for Erlang when they have multiple files

  Read the article

• Is there any documentation for the Cassandra Erlang interface?

  - by Zubair

  I have looked everywhere, and to use Cassandra from Erlang you end up having to download (amongst others): boost thrift : and then you have generate the erlang library by hand, and then copy lib files and beams files. Once you have the whole thing working there is absolutely zero documentation anywhere. If anyone could show me some user friendly documentation it would be much appreciated.

  Read the article

< Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >