The concept and usage of data grids are becoming very popular in this days since this type of technology are evolving very fast with some cool lead products like
Oracle Coherence. Once for a while, developers need an programmatic way to calculate the total size of a specific cache that are residing in the data grid. In this post, I will show how to accomplish this using
Oracle Coherence API. This example has been tested with 3.6, 3.7 and 3.7.1 versions of
Oracle Coherence.
To start the development of this example, you need to create a POJO ("Plain Old Java Object") that represents a data structure that will hold user data. This data structure will also create an internal fat so I call that should increase considerably the size of each instance in the heap memory. Create a Java class named "Person" as shown in the listing below.
package com.
oracle.coherence.domain;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Random;
@SuppressWarnings("serial")
public class Person implements Serializable {
private String firstName;
private String lastName;
private List<Object> fat;
private String email;
public Person() {
generateFat();
}
public Person(String firstName, String lastName,
String email) {
setFirstName(firstName);
setLastName(lastName);
setEmail(email);
generateFat();
}
private void generateFat() {
fat = new ArrayList<Object>();
Random random = new Random();
for (int i = 0; i < random.nextInt(18000); i++) {
HashMap<Long, Double> internalFat = new HashMap<Long, Double>();
for (int j = 0; j < random.nextInt(10000); j++) {
internalFat.put(random.nextLong(), random.nextDouble());
}
fat.add(internalFat);
}
}
public String getFirstName() {
return firstName;
}
public void setFirstName(String firstName) {
this.firstName = firstName;
}
public String getLastName() {
return lastName;
}
public void setLastName(String lastName) {
this.lastName = lastName;
}
public String getEmail() {
return email;
}
public void setEmail(String email) {
this.email = email;
}
}
Now let's create a Java program that will start a data grid into Coherence and will create a cache named "People", that will hold people instances with sequential integer keys. Each person created in this program will trigger the execution of a custom constructor created in the People class that instantiates an internal fat (the random amount of data generated to increase the size of the object) for each person. Create a Java class named "CreatePeopleCacheAndPopulateWithData" as shown in the listing below.
package com.
oracle.coherence.demo;
import com.
oracle.coherence.domain.Person;
import com.tangosol.net.CacheFactory;
import com.tangosol.net.NamedCache;
public class CreatePeopleCacheAndPopulateWithData {
public static void main(String[] args) {
// Asks Coherence for a new cache named "People"...
NamedCache people = CacheFactory.getCache("People");
// Creates three people that will be putted into the data grid. Each person
// generates an internal fat that should increase its size in terms of bytes...
Person pessoa1 = new Person("Ricardo", "Ferreira", "
[email protected]");
Person pessoa2 = new Person("Vitor", "Ferreira", "
[email protected]");
Person pessoa3 = new Person("Vivian", "Ferreira", "
[email protected]");
// Insert three people at the data grid...
people.put(1, pessoa1);
people.put(2, pessoa2);
people.put(3, pessoa3);
// Waits for 5 minutes until the user runs the Java program
// that calculates the total size of the people cache...
try {
System.out.println("---> Waiting for 5 minutes for the cache size calculation...");
Thread.sleep(300000);
} catch (InterruptedException ie) {
ie.printStackTrace();
}
}
}
Finally, let's create a Java program that, using the Coherence API and JMX, will calculate the total size of each cache that the data grid is currently managing. The approach used in this example was retrieve every cache that the data grid are currently managing, but if you are interested on an specific cache, the same approach can be used, you should only filter witch cache will be looked for. Create a Java class named "CalculateTheSizeOfPeopleCache" as shown in the listing below.
package com.
oracle.coherence.demo;
import java.text.DecimalFormat;
import java.util.Map;
import java.util.Set;
import java.util.TreeMap;
import javax.management.MBeanServer;
import javax.management.MBeanServerFactory;
import javax.management.ObjectName;
import com.tangosol.net.CacheFactory;
public class CalculateTheSizeOfPeopleCache {
@SuppressWarnings({ "unchecked", "rawtypes" })
private void run() throws Exception {
// Enable JMX support in this Coherence data grid session...
System.setProperty("tangosol.coherence.management", "all");
// Create a sample cache just to access the data grid...
CacheFactory.getCache(MBeanServerFactory.class.getName());
// Gets the JMX server from Coherence data grid...
MBeanServer jmxServer = getJMXServer();
// Creates a internal data structure that would maintain
// the statistics from each cache in the data grid...
Map cacheList = new TreeMap();
Set jmxObjectList = jmxServer.queryNames(new ObjectName("Coherence:type=Cache,*"), null);
for (Object jmxObject : jmxObjectList) {
ObjectName jmxObjectName = (ObjectName) jmxObject;
String cacheName = jmxObjectName.getKeyProperty("name");
if (cacheName.equals(MBeanServerFactory.class.getName())) {
continue;
} else {
cacheList.put(cacheName, new Statistics(cacheName));
}
}
// Updates the internal data structure with statistic data
// retrieved from caches inside the in-memory data grid...
Set<String> cacheNames = cacheList.keySet();
for (String cacheName : cacheNames) {
Set resultSet = jmxServer.queryNames(
new ObjectName("Coherence:type=Cache,name=" + cacheName + ",*"), null);
for (Object resultSetRef : resultSet) {
ObjectName objectName = (ObjectName) resultSetRef;
if (objectName.getKeyProperty("tier").equals("back")) {
int unit = (Integer) jmxServer.getAttribute(objectName, "Units");
int size = (Integer) jmxServer.getAttribute(objectName, "Size");
Statistics statistics = (Statistics) cacheList.get(cacheName);
statistics.incrementUnit(unit);
statistics.incrementSize(size);
cacheList.put(cacheName, statistics);
}
}
}
// Finally... print the objects from the internal data
// structure that represents the statistics from caches...
cacheNames = cacheList.keySet();
for (String cacheName : cacheNames) {
Statistics estatisticas = (Statistics) cacheList.get(cacheName);
System.out.println(estatisticas);
}
}
public MBeanServer getJMXServer() {
MBeanServer jmxServer = null;
for (Object jmxServerRef : MBeanServerFactory.findMBeanServer(null)) {
jmxServer = (MBeanServer) jmxServerRef;
if (jmxServer.getDefaultDomain().equals(DEFAULT_DOMAIN) || DEFAULT_DOMAIN.length() == 0) {
break;
}
jmxServer = null;
}
if (jmxServer == null) {
jmxServer = MBeanServerFactory.createMBeanServer(DEFAULT_DOMAIN);
}
return jmxServer;
}
private class Statistics {
private long unit;
private long size;
private String cacheName;
public Statistics(String cacheName) {
this.cacheName = cacheName;
}
public void incrementUnit(long unit) {
this.unit += unit;
}
public void incrementSize(long size) {
this.size += size;
}
public long getUnit() {
return unit;
}
public long getSize() {
return size;
}
public double getUnitInMB() {
return unit / (1024.0 * 1024.0);
}
public double getAverageSize() {
return size == 0 ? 0 : unit / size;
}
public String toString() {
StringBuffer sb = new StringBuffer();
sb.append("\nCache Statistics of '").append(cacheName).append("':\n");
sb.append(" - Total Entries of Cache -----> " + getSize()).append("\n");
sb.append(" - Used Memory (Bytes) --------> " + getUnit()).append("\n");
sb.append(" - Used Memory (MB) -----------> " + FORMAT.format(getUnitInMB())).append("\n");
sb.append(" - Object Average Size --------> " + FORMAT.format(getAverageSize())).append("\n");
return sb.toString();
}
}
public static void main(String[] args) throws Exception {
new CalculateTheSizeOfPeopleCache().run();
}
public static final DecimalFormat FORMAT = new DecimalFormat("###.###");
public static final String DEFAULT_DOMAIN = "";
public static final String DOMAIN_NAME = "Coherence";
}
I've commented the overall example so, I don't think that you should get into trouble to understand it. Basically we are dealing with JMX. The first thing to do is enable JMX support for the Coherence client (ie, an JVM that will only retrieve values from the data grid and will not integrate the cluster) application. This can be done very easily using the runtime "tangosol.coherence.management" system property. Consult the Coherence documentation for JMX to understand the possible values that could be applied. The program creates an in memory data structure that holds a custom class created called "Statistics".
This class represents the information that we are interested to see, which in this
case are the size in bytes and in MB of the caches. An instance of this class is created for each cache that are currently managed by the data grid. Using JMX specific methods, we retrieve the information that are relevant for calculate the total size of the caches. To test this example, you should execute first the CreatePeopleCacheAndPopulateWithData.java program and after the CreatePeopleCacheAndPopulateWithData.java program. The results in the console should be something like this:
2012-06-23 13:29:31.188/4.970
Oracle Coherence 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Loaded operational configuration from "jar:file:/E:/Oracle/Middleware/oepe_11gR1PS4/workspace/calcular-tamanho-cache-coherence/lib/coherence.jar!/tangosol-coherence.xml"
2012-06-23 13:29:31.219/5.001
Oracle Coherence 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Loaded operational overrides from "jar:file:/E:/Oracle/Middleware/oepe_11gR1PS4/workspace/calcular-tamanho-cache-coherence/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
2012-06-23 13:29:31.219/5.001
Oracle Coherence 3.6.0.4 <D5> (thread=Main Thread, member=n/a): Optional configuration override "/tangosol-coherence-override.xml" is not specified
2012-06-23 13:29:31.266/5.048
Oracle Coherence 3.6.0.4 <D5> (thread=Main Thread, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
Oracle Coherence Version 3.6.0.4 Build 19111
Grid Edition: Development mode
Copyright (c) 2000, 2010,
Oracle and/or its affiliates. All rights reserved.
2012-06-23 13:29:33.156/6.938
Oracle Coherence GE 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Loaded Reporter configuration from "jar:file:/E:/Oracle/Middleware/oepe_11gR1PS4/workspace/calcular-tamanho-cache-coherence/lib/coherence.jar!/reports/report-group.xml"
2012-06-23 13:29:33.500/7.282
Oracle Coherence GE 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Loaded cache configuration from "jar:file:/E:/Oracle/Middleware/oepe_11gR1PS4/workspace/calcular-tamanho-cache-coherence/lib/coherence.jar!/coherence-cache-config.xml"
2012-06-23 13:29:35.391/9.173
Oracle Coherence GE 3.6.0.4 <D4> (thread=Main Thread, member=n/a): TCMP bound to /192.168.177.133:8090 using SystemSocketProvider
2012-06-23 13:29:37.062/10.844
Oracle Coherence GE 3.6.0.4 <Info> (thread=Cluster, member=n/a): This Member(Id=2, Timestamp=2012-06-23 13:29:36.899, Address=192.168.177.133:8090, MachineId=55685, Location=process:244, Role=Oracle, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2) joined cluster "cluster:0xC4DB" with senior Member(Id=1, Timestamp=2012-06-23 13:29:14.031, Address=192.168.177.133:8088, MachineId=55685, Location=process:1128, Role=CreatePeopleCacheAndPopulateWith, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2)
2012-06-23 13:29:37.172/10.954
Oracle Coherence GE 3.6.0.4 <D5> (thread=Cluster, member=n/a): Member 1 joined Service Cluster with senior member 1
2012-06-23 13:29:37.188/10.970
Oracle Coherence GE 3.6.0.4 <D5> (thread=Cluster, member=n/a): Member 1 joined Service Management with senior member 1
2012-06-23 13:29:37.188/10.970
Oracle Coherence GE 3.6.0.4 <D5> (thread=Cluster, member=n/a): Member 1 joined Service DistributedCache with senior member 1
2012-06-23 13:29:37.188/10.970
Oracle Coherence GE 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Started cluster Name=cluster:0xC4DB
Group{Address=224.3.6.0, Port=36000, TTL=4}
MasterMemberSet
(
ThisMember=Member(Id=2, Timestamp=2012-06-23 13:29:36.899, Address=192.168.177.133:8090, MachineId=55685, Location=process:244, Role=Oracle)
OldestMember=Member(Id=1, Timestamp=2012-06-23 13:29:14.031, Address=192.168.177.133:8088, MachineId=55685, Location=process:1128, Role=CreatePeopleCacheAndPopulateWith)
ActualMemberSet=MemberSet(Size=2, BitSetCount=2
Member(Id=1, Timestamp=2012-06-23 13:29:14.031, Address=192.168.177.133:8088, MachineId=55685, Location=process:1128, Role=CreatePeopleCacheAndPopulateWith)
Member(Id=2, Timestamp=2012-06-23 13:29:36.899, Address=192.168.177.133:8090, MachineId=55685, Location=process:244, Role=Oracle)
)
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0, BitSetCount=0
)
)
TcpRing{Connections=[1]}
IpMonitor{AddressListSize=0}
2012-06-23 13:29:37.891/11.673
Oracle Coherence GE 3.6.0.4 <D5> (thread=Invocation:Management, member=2): Service Management joined the cluster with senior service member 1
2012-06-23 13:29:39.203/12.985
Oracle Coherence GE 3.6.0.4 <D5> (thread=DistributedCache, member=2): Service DistributedCache joined the cluster with senior service member 1
2012-06-23 13:29:39.297/13.079
Oracle Coherence GE 3.6.0.4 <D4> (thread=DistributedCache, member=2): Asking member 1 for 128 primary partitions
Cache Statistics of 'People':
- Total Entries of Cache -----> 3
- Used Memory (Bytes) --------> 883920
- Used Memory (MB) -----------> 0.843
- Object Average Size --------> 294640
I hope that this post could save you some time when calculate the total size of Coherence cache became a requirement for your high scalable system using data grids. See you!
