Is running "milli"-benchmarks a good idea?

Posted by Konstantin Weitz on Programmers See other posts from Programmers or by Konstantin Weitz
Published on 2012-06-25T22:09:12Z Indexed on 2012/06/26 3:23 UTC
Read the original article Hit count: 477

Filed under:

benchmarking

|

caliper

I just came across the Caliper project and it looks very nice. Reading the introduction to microbenchmarks, one gets the feeling that the developers would not suggest to use the framework if the benchmark takes longer than a second or so. I looked at the code and it looks like a RuntimeOutOfRangeException is actually thrown if a scenario takes longer than 10s to execute.

Could you explain to me what the problems are with running larger benchmarks?

My motivation for using Caliper was to compare two join-algorithm implementations. Those will definitely run for quite some time and will do some disk IO, yet running the entire database would make it hard to do the comparison, because the configuration of the algorithms and the visualization of the results would be a pain.

© Programmers or respective owner

Related posts about benchmarking

How to run benchmarking on MySQL?

as seen on Stack Overflow - Search for 'Stack Overflow'
My server has installed MySQL Server 5.1. I would like to run benchmarking on the MySQL, but I couldn't found sql-bench, which is Benchmark Suite provided by MySQL. The MySQL Benchmark Suite seem like complicated to be install or setup into my server. I need one can be direct setup to test the benchmark… >>> More
Codeigniter benchmarking, where are these ms coming from?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm in the process of benchmarking my website. class Home extends Controller { function Home() { parent::Controller(); $this->benchmark->mark('Constructor_start'); $this->output->enable_profiler(TRUE); $this->load->library ('MasterPage'); … >>> More
Benchmarking PHP page load times

as seen on Stack Overflow - Search for 'Stack Overflow'
How could I measure the time taken to load a page (with various different PHP statements)? Somewhat like the stats available here - http://talks.php.net/show/drupal08/24 >>> More
Graph representation benchmarking

as seen on Stack Overflow - Search for 'Stack Overflow'
Currently am developing a program that solves (if possible) any given labyrinth of dimensions from 3X4 to 26x30. I represent the graph using both adj matrix (sparse) and adj list. I would like to know how to output the total time taken by the DFS to find the solution using one and then the other method… >>> More
Database Trends & Applications column: Database Benchmarking from A to Z

as seen on SQL Blog - Search for 'SQL Blog'
Have you heard of the monthly print and web magazine Database Trends & Applications (DBTA)? Did you know I'm the regular columnist covering SQL Server ? For the past six months, I've been writing a series of articles about database benchmarking culminating in the latest article discussing my three… >>> More

Related posts about caliper

Is running "milli"-benchmarks a good idea?

as seen on Programmers - Search for 'Programmers'
I just came across the Caliper project and it looks very nice. Reading the introduction to microbenchmarks, one gets the feeling that the developers would not suggest to use the framework if the benchmark takes longer than a second or so. I looked at the code and it looks like a RuntimeOutOfRangeException… >>> More
How to disable auto-assume in Google Chrome

as seen on Super User - Search for 'Super User'
I've noticed that whenever I sign in in Google Chrome version 21.0 it automatically assumes and it automatically knows what I want to search. I basically use the address bar to search(my default search engine is Google) I don't go type Google.com on the address bar and then search from there. So what… >>> More