PostgreSQL lots of writes
- by strife911
Hi,
I am using postgreSQL for a scientific application (unsupervised clustering). The python program is multi-threaded so that each thread manages its own postmaster process (one per core). Hence, their is a lot of concurrency.
Each thread-process loop infinitely though two SQL queries. The first is for reading, the second is for writing. The read operation considers 500 time the amount of rows the write operation considers.
Here is the output of dstat:
----total-cpu-usage---- ------memory-usage----- -dsk/total- --paging-- --io/total-
usr sys idl wai hiq siq| used buff cach free| read writ| in out | read writ
4 0 32 64 0 0|3599M 63M 57G 1893M|1524k 16M| 0 0 | 98 2046
1 0 35 64 0 0|3599M 63M 57G 1892M|1204k 17M| 0 0 | 68 2062
2 0 32 66 0 0|3599M 63M 57G 1890M|1132k 17M| 0 0 | 62 2033
2 1 32 65 0 0|3599M 63M 57G 1904M|1236k 18M| 0 0 | 80 1994
2 0 31 67 0 0|3599M 63M 57G 1903M|1312k 16M| 0 0 | 70 1900
2 0 37 60 0 0|3599M 63M 57G 1899M|1116k 15M| 0 0 | 71 1594
2 1 37 60 0 0|3599M 63M 57G 1898M| 448k 17M| 0 0 | 39 2001
2 0 25 72 0 0|3599M 63M 57G 1896M|1192k 17M| 0 0 | 78 1946
1 0 40 58 0 0|3599M 63M 57G 1895M| 432k 15M| 0 0 | 38 1937
I am pretty sure I could write more often than that for I have seen it write up to 110-140M on dstat. How can I optimize this process?
