The concept and usage of data grids are becoming very popular in this days since this type of technology are evolving very fast with some cool lead products like Oracle Coherence. Once for a while, developers need an programmatic way to calculate
the total size of a specific cache that are residing in
the data grid. In this post, I will show how to accomplish this using Oracle Coherence API. This example has been tested with 3.6, 3.7 and 3.7.1 versions of Oracle Coherence.
To start
the development of this example, you need to create a POJO ("Plain Old Java Object") that represents a data structure that will hold user data. This data structure will also create an internal fat so I call that should increase considerably
the size of each instance in
the heap memory. Create a Java class named "Person" as shown in
the listing below.
package com.oracle.coherence.domain;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Random;
@SuppressWarnings("serial")
public class
Person implements Serializable {
private String firstName;
private String lastName;
private List<Object> fat;
private String email;
public Person() {
generateFat();
}
public Person(String firstName, String lastName,
String email) {
setFirstName(firstName);
setLastName(lastName);
setEmail(email);
generateFat();
}
private void generateFat() {
fat = new ArrayList<Object>();
Random random = new Random();
for (int i = 0; i < random.nextInt(18000); i++) {
HashMap<Long, Double> internalFat = new HashMap<Long, Double>();
for (int j = 0; j < random.nextInt(10000); j++) {
internalFat.put(random.nextLong(), random.nextDouble());
}
fat.add(internalFat);
}
}
public String getFirstName() {
return firstName;
}
public void setFirstName(String firstName) {
this.firstName = firstName;
}
public String getLastName() {
return lastName;
}
public void setLastName(String lastName) {
this.lastName = lastName;
}
public String getEmail() {
return email;
}
public void setEmail(String email) {
this.email = email;
}
}
Now let's create a Java program that will start a data grid into Coherence and will create a cache named "People", that will hold people instances with sequential integer keys. Each
person created in this program will trigger
the execution of a custom constructor created in
the People class that instantiates an internal fat (the random amount of data generated to increase
the size of
the object) for each
person. Create a Java class named "CreatePeopleCacheAndPopulateWithData" as shown in
the listing below.
package com.oracle.coherence.demo;
import com.oracle.coherence.domain.Person;
import com.tangosol.net.CacheFactory;
import com.tangosol.net.NamedCache;
public class CreatePeopleCacheAndPopulateWithData {
public static void main(String[] args) {
// Asks Coherence for a new cache named "People"...
NamedCache people = CacheFactory.getCache("People");
// Creates three people that will be putted into
the data grid. Each
person
// generates an internal fat that should increase its size in terms of bytes...
Person pessoa1 = new Person("Ricardo", "Ferreira", "
[email protected]");
Person pessoa2 = new Person("Vitor", "Ferreira", "
[email protected]");
Person pessoa3 = new Person("Vivian", "Ferreira", "
[email protected]");
// Insert three people at
the data grid...
people.put(1, pessoa1);
people.put(2, pessoa2);
people.put(3, pessoa3);
// Waits for 5 minutes until
the user runs
the Java program
// that calculates
the total size of
the people cache...
try {
System.out.println("---> Waiting for 5 minutes for
the cache size calculation...");
Thread.sleep(300000);
} catch (InterruptedException ie) {
ie.printStackTrace();
}
}
}
Finally, let's create a Java program that, using
the Coherence API and JMX, will calculate
the total size of each cache that
the data grid is currently managing.
The approach used in this example was retrieve every cache that
the data grid are currently managing, but if you are interested on an specific cache,
the same approach can be used, you should only filter witch cache will be looked for. Create a Java class named "CalculateTheSizeOfPeopleCache" as shown in
the listing below.
package com.oracle.coherence.demo;
import java.text.DecimalFormat;
import java.util.Map;
import java.util.Set;
import java.util.TreeMap;
import javax.management.MBeanServer;
import javax.management.MBeanServerFactory;
import javax.management.ObjectName;
import com.tangosol.net.CacheFactory;
public class CalculateTheSizeOfPeopleCache {
@SuppressWarnings({ "unchecked", "rawtypes" })
private void run() throws Exception {
// Enable JMX support in this Coherence data grid session...
System.setProperty("tangosol.coherence.management", "all");
// Create a sample cache just to access
the data grid...
CacheFactory.getCache(MBeanServerFactory.class.getName());
// Gets
the JMX server from Coherence data grid...
MBeanServer jmxServer = getJMXServer();
// Creates a internal data structure that would maintain
//
the statistics from each cache in
the data grid...
Map cacheList = new TreeMap();
Set jmxObjectList = jmxServer.queryNames(new ObjectName("Coherence:type=Cache,*"), null);
for (Object jmxObject : jmxObjectList) {
ObjectName jmxObjectName = (ObjectName) jmxObject;
String cacheName = jmxObjectName.getKeyProperty("name");
if (cacheName.equals(MBeanServerFactory.class.getName())) {
continue;
} else {
cacheList.put(cacheName, new Statistics(cacheName));
}
}
// Updates
the internal data structure with statistic data
// retrieved from caches inside
the in-memory data grid...
Set<String> cacheNames = cacheList.keySet();
for (String cacheName : cacheNames) {
Set resultSet = jmxServer.queryNames(
new ObjectName("Coherence:type=Cache,name=" + cacheName + ",*"), null);
for (Object resultSetRef : resultSet) {
ObjectName objectName = (ObjectName) resultSetRef;
if (objectName.getKeyProperty("tier").equals("back")) {
int unit = (Integer) jmxServer.getAttribute(objectName, "Units");
int size = (Integer) jmxServer.getAttribute(objectName, "Size");
Statistics statistics = (Statistics) cacheList.get(cacheName);
statistics.incrementUnit(unit);
statistics.incrementSize(size);
cacheList.put(cacheName, statistics);
}
}
}
// Finally... print
the objects from
the internal data
// structure that represents
the statistics from caches...
cacheNames = cacheList.keySet();
for (String cacheName : cacheNames) {
Statistics estatisticas = (Statistics) cacheList.get(cacheName);
System.out.println(estatisticas);
}
}
public MBeanServer getJMXServer() {
MBeanServer jmxServer = null;
for (Object jmxServerRef : MBeanServerFactory.findMBeanServer(null)) {
jmxServer = (MBeanServer) jmxServerRef;
if (jmxServer.getDefaultDomain().equals(DEFAULT_DOMAIN) || DEFAULT_DOMAIN.length() == 0) {
break;
}
jmxServer = null;
}
if (jmxServer == null) {
jmxServer = MBeanServerFactory.createMBeanServer(DEFAULT_DOMAIN);
}
return jmxServer;
}
private class Statistics {
private long unit;
private long size;
private String cacheName;
public Statistics(String cacheName) {
this.cacheName = cacheName;
}
public void incrementUnit(long unit) {
this.unit += unit;
}
public void incrementSize(long size) {
this.size += size;
}
public long getUnit() {
return unit;
}
public long getSize() {
return size;
}
public double getUnitInMB() {
return unit / (1024.0 * 1024.0);
}
public double getAverageSize() {
return size == 0 ? 0 : unit / size;
}
public String toString() {
StringBuffer sb = new StringBuffer();
sb.append("\nCache Statistics of '").append(cacheName).append("':\n");
sb.append(" - Total Entries of Cache -----> " + getSize()).append("\n");
sb.append(" - Used Memory (Bytes) --------> " + getUnit()).append("\n");
sb.append(" - Used Memory (MB) -----------> " + FORMAT.format(getUnitInMB())).append("\n");
sb.append(" - Object Average Size --------> " + FORMAT.format(getAverageSize())).append("\n");
return sb.toString();
}
}
public static void main(String[] args) throws Exception {
new CalculateTheSizeOfPeopleCache().run();
}
public static final DecimalFormat FORMAT = new DecimalFormat("###.###");
public static final String DEFAULT_DOMAIN = "";
public static final String DOMAIN_NAME = "Coherence";
}
I've commented
the overall example so, I don't think that you should get into trouble to understand it. Basically we are dealing with JMX.
The first thing to do is enable JMX support for
the Coherence client (ie, an JVM that will only retrieve values from
the data grid and will not integrate
the cluster) application. This can be done very easily using
the runtime "tangosol.coherence.management" system property. Consult
the Coherence documentation for JMX to understand
the possible values that could be applied.
The program creates an in memory data structure that holds a custom class created called "Statistics".
This class represents
the information that we are interested to see, which in this case are
the size in bytes and in MB of
the caches. An instance of this class is created for each cache that are currently managed by
the data grid. Using JMX specific methods, we retrieve
the information that are relevant for calculate
the total size of
the caches. To test this example, you should execute first the CreatePeopleCacheAndPopulateWithData.java program and after the CreatePeopleCacheAndPopulateWithData.java program.
The results in
the console should be something like this:
2012-06-23 13:29:31.188/4.970 Oracle Coherence 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Loaded operational configuration from "jar:file:/E:/Oracle/Middleware/oepe_11gR1PS4/workspace/calcular-tamanho-cache-coherence/lib/coherence.jar!/tangosol-coherence.xml"
2012-06-23 13:29:31.219/5.001 Oracle Coherence 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Loaded operational overrides from "jar:file:/E:/Oracle/Middleware/oepe_11gR1PS4/workspace/calcular-tamanho-cache-coherence/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
2012-06-23 13:29:31.219/5.001 Oracle Coherence 3.6.0.4 <D5> (thread=Main Thread, member=n/a): Optional configuration override "/tangosol-coherence-override.xml" is not specified
2012-06-23 13:29:31.266/5.048 Oracle Coherence 3.6.0.4 <D5> (thread=Main Thread, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
Oracle Coherence Version 3.6.0.4 Build 19111
Grid Edition: Development mode
Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
2012-06-23 13:29:33.156/6.938 Oracle Coherence GE 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Loaded Reporter configuration from "jar:file:/E:/Oracle/Middleware/oepe_11gR1PS4/workspace/calcular-tamanho-cache-coherence/lib/coherence.jar!/reports/report-group.xml"
2012-06-23 13:29:33.500/7.282 Oracle Coherence GE 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Loaded cache configuration from "jar:file:/E:/Oracle/Middleware/oepe_11gR1PS4/workspace/calcular-tamanho-cache-coherence/lib/coherence.jar!/coherence-cache-config.xml"
2012-06-23 13:29:35.391/9.173 Oracle Coherence GE 3.6.0.4 <D4> (thread=Main Thread, member=n/a): TCMP bound to /192.168.177.133:8090 using SystemSocketProvider
2012-06-23 13:29:37.062/10.844 Oracle Coherence GE 3.6.0.4 <Info> (thread=Cluster, member=n/a): This Member(Id=2, Timestamp=2012-06-23 13:29:36.899, Address=192.168.177.133:8090, MachineId=55685, Location=process:244, Role=Oracle, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2) joined cluster "cluster:0xC4DB" with senior Member(Id=1, Timestamp=2012-06-23 13:29:14.031, Address=192.168.177.133:8088, MachineId=55685, Location=process:1128, Role=CreatePeopleCacheAndPopulateWith, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2)
2012-06-23 13:29:37.172/10.954 Oracle Coherence GE 3.6.0.4 <D5> (thread=Cluster, member=n/a): Member 1 joined Service Cluster with senior member 1
2012-06-23 13:29:37.188/10.970 Oracle Coherence GE 3.6.0.4 <D5> (thread=Cluster, member=n/a): Member 1 joined Service Management with senior member 1
2012-06-23 13:29:37.188/10.970 Oracle Coherence GE 3.6.0.4 <D5> (thread=Cluster, member=n/a): Member 1 joined Service DistributedCache with senior member 1
2012-06-23 13:29:37.188/10.970 Oracle Coherence GE 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Started cluster Name=cluster:0xC4DB
Group{Address=224.3.6.0, Port=36000, TTL=4}
MasterMemberSet
(
ThisMember=Member(Id=2, Timestamp=2012-06-23 13:29:36.899, Address=192.168.177.133:8090, MachineId=55685, Location=process:244, Role=Oracle)
OldestMember=Member(Id=1, Timestamp=2012-06-23 13:29:14.031, Address=192.168.177.133:8088, MachineId=55685, Location=process:1128, Role=CreatePeopleCacheAndPopulateWith)
ActualMemberSet=MemberSet(Size=2, BitSetCount=2
Member(Id=1, Timestamp=2012-06-23 13:29:14.031, Address=192.168.177.133:8088, MachineId=55685, Location=process:1128, Role=CreatePeopleCacheAndPopulateWith)
Member(Id=2, Timestamp=2012-06-23 13:29:36.899, Address=192.168.177.133:8090, MachineId=55685, Location=process:244, Role=Oracle)
)
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0, BitSetCount=0
)
)
TcpRing{Connections=[1]}
IpMonitor{AddressListSize=0}
2012-06-23 13:29:37.891/11.673 Oracle Coherence GE 3.6.0.4 <D5> (thread=Invocation:Management, member=2): Service Management joined
the cluster with senior service member 1
2012-06-23 13:29:39.203/12.985 Oracle Coherence GE 3.6.0.4 <D5> (thread=DistributedCache, member=2): Service DistributedCache joined
the cluster with senior service member 1
2012-06-23 13:29:39.297/13.079 Oracle Coherence GE 3.6.0.4 <D4> (thread=DistributedCache, member=2): Asking member 1 for 128 primary partitions
Cache Statistics of 'People':
- Total Entries of Cache -----> 3
- Used Memory (Bytes) --------> 883920
- Used Memory (MB) -----------> 0.843
- Object Average Size --------> 294640
I hope that this post could save you some time when calculate
the total size of Coherence cache became a requirement for your high scalable system using data grids. See you!