[perfsonar-user] perfSONAR 4.0 RC2 Cassandra logs have fatal errors

[Michael Petry]

cassandra leaves a fatal error in its logs. Is this expected behavior?

/var/log/cassandra/cassandra.log


Btw: I've tried using the


INFO 11:00:15,857 Cassandra version: 2.0.17

INFO 11:00:15,857 Thrift API version: 19.39.0

INFO 11:00:15,861 CQL supported versions: 2.0.0,3.1.7 (default: 3.1.7)

INFO 11:00:15,874 Loading persisted ring state

INFO 11:00:15,890 Starting up server gossip

INFO 11:00:15,893 Enqueuing flush of Memtable-***@1959711969(375/3750
serialized/live bytes, 13 ops)

INFO 11:00:15,904 Writing Memtable-***@1959711969(375/3750
serialized/live bytes, 13 ops)

INFO 11:00:15,931 Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-24-Data.db (313 bytes)
for commitlog position ReplayPosition(segmentId=1479312014388,
position=179710)

INFO 11:00:15,932 Loading settings from
file:/etc/cassandra/default.conf/cassandra.yaml

ERROR 11:00:16,101 Fatal configuration error

org.apache.cassandra.exceptions.ConfigurationException: localhost/
127.0.0.1:7000 is in use by another process. Change
listen_address:storage_port in cassandra.yaml to values that do not
conflict with other services

at
org.apache.cassandra.net.MessagingService.getServerSockets(MessagingService.java:492)

at
org.apache.cassandra.net.MessagingService.listen(MessagingService.java:438)

at
org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:692)

at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:622)

at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:514)

at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:424)

at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:554)

at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:643)

localhost/127.0.0.1:7000 is in use by another process. Change
listen_address:storage_port in cassandra.yaml to values that do not
conflict with other services

Fatal configuration error; unable to start server. See log for stacktrace.

INFO 11:00:16,121 Announcing shutdown

ERROR 11:00:16,122 Exception in thread
Thread[StorageServiceShutdownHook,5,main]

java.lang.IllegalArgumentException: Invalid token for Murmur3Partitioner.
Got true but expected a long value (unsigned 8 bytes integer).

at
org.apache.cassandra.dht.Murmur3Partitioner$1.fromString(Murmur3Partitioner.java:190)

at
org.apache.cassandra.service.StorageService.getTokensFor(StorageService.java:1470)

at
org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:1532)

at
org.apache.cassandra.service.StorageService.onChange(StorageService.java:1354)

at
org.apache.cassandra.gms.Gossiper.doOnChangeNotifications(Gossiper.java:1171)

at
org.apache.cassandra.gms.Gossiper.addLocalApplicationStateInternal(Gossiper.java:1401)

at
org.apache.cassandra.gms.Gossiper.addLocalApplicationStates(Gossiper.java:1416)

at
org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1406)

at org.apache.cassandra.gms.Gossiper.stop(Gossiper.java:1432)

at
org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:583)

at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)

at java.lang.Thread.run(Thread.java:745)

[***@oit-82-33 cassandra]$


Mike

[Dan Doyle]

Hi Mike,

Do you already have an instance of cassandra running? I have seen similar cases where you "start" cassandra on a machine where it's still running, particularly where the pidfile gets out of sync and so the init scripts think things are okay.

You may want to look at something like `ps aux | grep cassandra` and kill any existing processes that may be running, that start again via the init script.

Dan Doyle
GlobalNOC Software Developer
1-812-856-3892

Post by Michael Petry
cassandra leaves a fatal error in its logs. Is this expected behavior?
/var/log/cassandra/cassandra.log
Btw: I've tried using the
INFO 11:00:15,857 Cassandra version: 2.0.17
INFO 11:00:15,857 Thrift API version: 19.39.0
INFO 11:00:15,861 CQL supported versions: 2.0.0,3.1.7 (default: 3.1.7)
INFO 11:00:15,874 Loading persisted ring state
INFO 11:00:15,890 Starting up server gossip
INFO 11:00:15,931 Completed flushing /var/lib/cassandra/data/system/local/system-local-jb-24-Data.db (313 bytes) for commitlog position ReplayPosition(segmentId=1479312014388, position=179710)
INFO 11:00:15,932 Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
ERROR 11:00:16,101 Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: localhost/127.0.0.1:7000 <http://127.0.0.1:7000/> is in use by another process. Change listen_address:storage_port in cassandra.yaml to values that do not conflict with other services
at org.apache.cassandra.net.MessagingService.getServerSockets(MessagingService.java:492)
at org.apache.cassandra.net.MessagingService.listen(MessagingService.java:438)
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:692)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:622)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:514)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:424)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:554)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:643)
localhost/127.0.0.1:7000 <http://127.0.0.1:7000/> is in use by another process. Change listen_address:storage_port in cassandra.yaml to values that do not conflict with other services
Fatal configuration error; unable to start server. See log for stacktrace.
INFO 11:00:16,121 Announcing shutdown
ERROR 11:00:16,122 Exception in thread Thread[StorageServiceShutdownHook,5,main]
java.lang.IllegalArgumentException: Invalid token for Murmur3Partitioner. Got true but expected a long value (unsigned 8 bytes integer).
at org.apache.cassandra.dht.Murmur3Partitioner$1.fromString(Murmur3Partitioner.java:190)
at org.apache.cassandra.service.StorageService.getTokensFor(StorageService.java:1470)
at org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:1532)
at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1354)
at org.apache.cassandra.gms.Gossiper.doOnChangeNotifications(Gossiper.java:1171)
at org.apache.cassandra.gms.Gossiper.addLocalApplicationStateInternal(Gossiper.java:1401)
at org.apache.cassandra.gms.Gossiper.addLocalApplicationStates(Gossiper.java:1416)
at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1406)
at org.apache.cassandra.gms.Gossiper.stop(Gossiper.java:1432)
at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:583)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.lang.Thread.run(Thread.java:745)
Mike

[Michael Petry]

Only one process running when I looked, but this was straight off the ISO
behavior. That was my concern. There may be a synchronization issue
hidden in the startup scripts/configs.

The cassandra log shows:

/usr/sbin/cassandra: line 155: /var/run/cassandra/cassandra.pid: No such
file or directory


and there is no /var/run/cassandra


Looks like something in the setup on the ISO needs tweaking.



Mike

Post by Dan Doyle
Hi Mike,
Do you already have an instance of cassandra running? I have seen similar
cases where you "start" cassandra on a machine where it's still running,
particularly where the pidfile gets out of sync and so the init scripts
think things are okay.
You may want to look at something like `ps aux | grep cassandra` and kill
any existing processes that may be running, that start again via the init
script.
Dan Doyle
GlobalNOC Software Developer
1-812-856-3892
cassandra leaves a fatal error in its logs. Is this expected behavior?
/var/log/cassandra/cassandra.log
Btw: I've tried using the
INFO 11:00:15,857 Cassandra version: 2.0.17
INFO 11:00:15,857 Thrift API version: 19.39.0
INFO 11:00:15,861 CQL supported versions: 2.0.0,3.1.7 (default: 3.1.7)
INFO 11:00:15,874 Loading persisted ring state
INFO 11:00:15,890 Starting up server gossip
serialized/live bytes, 13 ops)
serialized/live bytes, 13 ops)
INFO 11:00:15,931 Completed flushing /var/lib/cassandra/data/
system/local/system-local-jb-24-Data.db (313 bytes) for commitlog
position ReplayPosition(segmentId=1479312014388, position=179710)
INFO 11:00:15,932 Loading settings from file:/etc/cassandra/default.
conf/cassandra.yaml
ERROR 11:00:16,101 Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: localhost/
127.0.0.1:7000 is in use by another process. Change
listen_address:storage_port in cassandra.yaml to values that do not
conflict with other services
at org.apache.cassandra.net.MessagingService.getServerSockets(
MessagingService.java:492)
at org.apache.cassandra.net.MessagingService.listen(
MessagingService.java:438)
at org.apache.cassandra.service.StorageService.prepareToJoin(
StorageService.java:692)
at org.apache.cassandra.service.StorageService.initServer(
StorageService.java:622)
at org.apache.cassandra.service.StorageService.initServer(
StorageService.java:514)
at org.apache.cassandra.service.CassandraDaemon.setup(
CassandraDaemon.java:424)
at org.apache.cassandra.service.CassandraDaemon.activate(
CassandraDaemon.java:554)
at org.apache.cassandra.service.CassandraDaemon.main(
CassandraDaemon.java:643)
localhost/127.0.0.1:7000 is in use by another process. Change
listen_address:storage_port in cassandra.yaml to values that do not
conflict with other services
Fatal configuration error; unable to start server. See log for stacktrace.
INFO 11:00:16,121 Announcing shutdown
ERROR 11:00:16,122 Exception in thread Thread[
StorageServiceShutdownHook,5,main]
java.lang.IllegalArgumentException: Invalid token for Murmur3Partitioner.
Got true but expected a long value (unsigned 8 bytes integer).
at org.apache.cassandra.dht.Murmur3Partitioner$1.
fromString(Murmur3Partitioner.java:190)
at org.apache.cassandra.service.StorageService.getTokensFor(
StorageService.java:1470)
at org.apache.cassandra.service.StorageService.handleStateNormal(
StorageService.java:1532)
at org.apache.cassandra.service.StorageService.onChange(
StorageService.java:1354)
at org.apache.cassandra.gms.Gossiper.doOnChangeNotifications(
Gossiper.java:1171)
at org.apache.cassandra.gms.Gossiper.addLocalApplicationStateIntern
al(Gossiper.java:1401)
at org.apache.cassandra.gms.Gossiper.addLocalApplicationStates(
Gossiper.java:1416)
at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(
Gossiper.java:1406)
at org.apache.cassandra.gms.Gossiper.stop(Gossiper.java:1432)
at org.apache.cassandra.service.StorageService$1.runMayThrow(
StorageService.java:583)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.lang.Thread.run(Thread.java:745)
Mike

[Andrew Lake]

This post might be inappropriate. Click to display it.

[Michael Petry]

over at https://issues.apache.org/jira/browse/CASSANDRA-10920

the quick fix is to add the following the cassandrea init.d file

start)
# Cassandra startup
echo -n "Starting Cassandra: "
+ pid_dir=`dirname $pid_file`
+ [ -d $pid_dir ] || install -m 755 -o $CASSANDRA_OWNR -d $pid_dir
su $CASSANDRA_OWNR -c "$CASSANDRA_PROG -p $pid_file" > $log_file 2>&1

Post by Andrew Lake
Hi,
It’s a known bug in the cassandra RPM on CentOS 7. Cassandra creates
/var/run/cassandra on install, but that directory is cleaned-out on reboot
by the OS. Really the cassandra init script should be creating that
directory, not the RPM. We don't actually control that RPM, but we might
have to roll our own or tweak some boot scripts if they don’t fix that.
Thanks,
Andy
Only one process running when I looked, but this was straight off the ISO
behavior. That was my concern. There may be a synchronization issue
hidden in the startup scripts/configs.
/usr/sbin/cassandra: line 155: /var/run/cassandra/cassandra.pid: No such
file or directory
and there is no /var/run/cassandra
Looks like something in the setup on the ISO needs tweaking.
Mike

Post by Dan Doyle
Hi Mike,
Do you already have an instance of cassandra running? I have seen similar
cases where you "start" cassandra on a machine where it's still running,
particularly where the pidfile gets out of sync and so the init scripts
think things are okay.
You may want to look at something like `ps aux | grep cassandra` and kill
any existing processes that may be running, that start again via the init
script.
Dan Doyle
GlobalNOC Software Developer
1-812-856-3892
cassandra leaves a fatal error in its logs. Is this expected behavior?
/var/log/cassandra/cassandra.log
Btw: I've tried using the
INFO 11:00:15,857 Cassandra version: 2.0.17
INFO 11:00:15,857 Thrift API version: 19.39.0
INFO 11:00:15,861 CQL supported versions: 2.0.0,3.1.7 (default: 3.1.7)
INFO 11:00:15,874 Loading persisted ring state
INFO 11:00:15,890 Starting up server gossip
serialized/live bytes, 13 ops)
serialized/live bytes, 13 ops)
INFO 11:00:15,931 Completed flushing /var/lib/cassandra/data/system
/local/system-local-jb-24-Data.db (313 bytes) for commitlog position
ReplayPosition(segmentId=1479312014388, position=179710)
INFO 11:00:15,932 Loading settings from file:/etc/cassandra/default.co
nf/cassandra.yaml
ERROR 11:00:16,101 Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: localhost/
127.0.0.1:7000 is in use by another process. Change
listen_address:storage_port in cassandra.yaml to values that do not
conflict with other services
at org.apache.cassandra.net.MessagingService.getServerSockets(M
essagingService.java:492)
at org.apache.cassandra.net.MessagingService.listen(MessagingSe
rvice.java:438)
at org.apache.cassandra.service.StorageService.prepareToJoin(St
orageService.java:692)
at org.apache.cassandra.service.StorageService.initServer(Stora
geService.java:622)
at org.apache.cassandra.service.StorageService.initServer(Stora
geService.java:514)
at org.apache.cassandra.service.CassandraDaemon.setup(Cassandra
Daemon.java:424)
at org.apache.cassandra.service.CassandraDaemon.activate(Cassan
draDaemon.java:554)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraD
aemon.java:643)
localhost/127.0.0.1:7000 is in use by another process. Change
listen_address:storage_port in cassandra.yaml to values that do not
conflict with other services
Fatal configuration error; unable to start server. See log for stacktrace.
INFO 11:00:16,121 Announcing shutdown
ERROR 11:00:16,122 Exception in thread Thread[StorageServiceShutdownH
ook,5,main]
java.lang.IllegalArgumentException: Invalid token for
Murmur3Partitioner. Got true but expected a long value (unsigned 8 bytes
integer).
at org.apache.cassandra.dht.Murmur3Partitioner$1.fromString(
Murmur3Partitioner.java:190)
at org.apache.cassandra.service.StorageService.getTokensFor(Sto
rageService.java:1470)
at org.apache.cassandra.service.StorageService.handleStateNorma
l(StorageService.java:1532)
at org.apache.cassandra.service.StorageService.onChange(Storage
Service.java:1354)
at org.apache.cassandra.gms.Gossiper.doOnChangeNotifications(Go
ssiper.java:1171)
at org.apache.cassandra.gms.Gossiper.addLocalApplicationStateIn
ternal(Gossiper.java:1401)
at org.apache.cassandra.gms.Gossiper.addLocalApplicationStates(
Gossiper.java:1416)
at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(G
ossiper.java:1406)
at org.apache.cassandra.gms.Gossiper.stop(Gossiper.java:1432)
at org.apache.cassandra.service.StorageService$1.runMayThrow(St
orageService.java:583)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnab
le.java:28)
at java.lang.Thread.run(Thread.java:745)
Mike

[Valentin Vidic]

Post by Michael Petry
over at https://issues.apache.org/jira/browse/CASSANDRA-10920
the quick fix is to add the following the cassandrea init.d file
start)
# Cassandra startup
echo -n "Starting Cassandra: "
+ pid_dir=`dirname $pid_file`
+ [ -d $pid_dir ] || install -m 755 -o $CASSANDRA_OWNR -d $pid_dir
su $CASSANDRA_OWNR -c "$CASSANDRA_PROG -p $pid_file" > $log_file 2>&1

We can probably make use of tmpfiles.d boot service,
for example postgresql has:

$ cat /usr/lib/tmpfiles.d/postgresql.conf
d /var/run/postgresql 0755 postgres postgres -