thread_risks

Risk of Threads

• https://www.wikiwand.com/en/articles/Safety_and_liveness_properties
• Safety Hazards
  • Causes Thread safety be compromised?
  • Safety Property: “bad things” don’t happen during execution of a program
  • Examples:
    • Race condition
• Liveness Hazards
  • When an activity gets into a state such that it is permanently unable to make forward progress
  • Liveness Property: “good things” do happen (eventually)
  • Liveness does not concern about the time bound, but safety concerns about happening bad thing in a tight time bound
  • Examples:
    • Deadlock
    • Starvation
    • Livelock
    • Missed Signals
• Performance Hazards
  • Causes Poor service time, responsiveness, throughput, resource consumption, scalability
  • Examples:
    • Context Switching
    • Multithreading overheads
• Thread Interference
• Memory Consistency Errors

Race Condition

• Result depends on the relative ordering of the execution of multiple threads
• Examples
  • Read-modify-write
  • Check-then-act

Deadlock

• aka Deadly embrace
• A deadlock is when two or more threads are stuck waiting for the same resource to become available.
• Java Programs cannot recover from deadlocks
• Database systems are designed to detect and recover from deadlock
• Following is an example of lock-ordering deadlock, hence should be avoided
  • cook1 thread waits for bowl and cook2 thread waits for spoon to free indefinitely, hence reaches a deadlock.

public class Kitchen {
	public static final Object spoon = new Object();
	public static final Object bowl = new Object();
 
	public static void main() {
		Thread cook1 = new Thread(() -> {
            synchronized (bowl) {
                System.out.println("Cook1 acquired bowl");
                synchronized (spoon) {
                    System.out.println("Cook1 acquired spoon");
                }
            }
        });
 
        Thread cook2 = new Thread(() -> {
            synchronized (spoon) {
                System.out.println("Cook2 acquired spoon");
                synchronized (bowl) {
                    System.out.println("Cook2 acquired bowl");
                }
            }
        });
 
        cook1.start();
        cook2.start();
	}
}

• Types
  • Lock ordering deadlock — example is above
    • Dynamic lock ordering deadlock — lock objects are dynamic (for example: locks are argument of methods)
  • Resource deadlock
    • Thread starvation deadlock

Necessary Conditions for deadlock

• Taken from OS concepts by Silberschatz
  • Mutual Exclusion
    • At least one resource must be held in non-sharable mode
  • Hold and Wait
    • A process must hold one resource and waiting to acquire another resource
  • No Preemption
    • Resources cannot be preempted
    • i.e. Resources can only be released by process holding it
  • Circular wait
    • A set $\{P_0,P_1,P_2,P_3,...,P_n\}$ of processes must exist such that
      • $P_0$ is waiting for the resource held by $P_1$
      • $P_1$ is waiting for the resource held by $P_2$
      • …
      • $P_n$ is waiting for the resource held by $P_0$
• Simpler Version
  • Mutual Exclusion: Only one thread can access a particular resource at a time.
  • Hold and Wait: A thread holds one lock and waits for another.
  • No Preemption: Resources can be released only by the thread holding them.
  • Circular Wait: A set of threads are waiting for each other in a circular chain.

Detect Deadlocks

• Using Thread dump analysis

Avoid Deadlocks

• Avoid nested lock: A program that never request more than one lock cannot experience lock ordering deadlock
• Timeout while attempting to lock
  • Example: ReentrantLock’s timed tryLock method
• Lock Ordering: Always acquire locks in fixed global order

Starvation

• Starvation describes a situation where a thread is unable to gain regular access to shared resources and is unable to make progress
• usually when shared resources are made unavailable for long periods by “greedy” threads
• The most commonly starved resource is CPU cycles
• Examples:
  • inappropriate use of thread priorities
  • non-terminating loops/waits that do not release the lock

Livelock

• Livelock is a form of liveness failure in which a thread, while not blocked, still cannot make progress because it keeps retrying an operation that will always fail
• The solution to variety of livelock problems is to introduce randomness to the retry mechanism
• Example:
  • When two stations in ethernet network try to send packet on shared carrier at the same time, packets collide. Retrying sending packets will again cause collision
    • Ethernet protocol includes exponential backoff after repeated collisions
  • In transactional messaging, if rollback happens due to any issue, the application puts the message back at the head of the queue

Thread issues

• symptoms:
  • deadlocks
  • high CPU usage
• diagnosis:
  • analyze thread dumps

Thread dump

• Complete picture of all threads in your JVM and what they’re doing at the moment in time in which the “picture” was taken
• It includes
  • stack trace for each running thread
  • locking information on which locks are held by which thread
  • which thread is BLOCKED and waiting to acquire which lock
  • deadlock information by calculating is-waiting-for graph for threads
• tools to capture thread dump:
  • Java Mission Control (JMC)
  • Java Flight Record (JFR)
  • jcmd
  • jstack
  • jvisualvm
• ~5 thread dumps can be collected within 10 seconds to understand if they are waiting or blocked
• Online thread dump analyzer:
  • jstack.review
  • Spotify thread dump analyzer
• Info about thread to look for:
  • name
  • ID
  • Daemon status
  • OS thread ID
  • Status
  • Timing (cpu and elapsed)
  • Stack trace
• Thread Statuses to look out for:
  • RUNNABLE
  • BLOCKED
  • WAITING
  • TIMED_WAITING
• Daemon marked threads are system threads and avoid analyzing them
• CPU time is idle but elapsed time is increasing across several thread dumps, means thread is not doing anything
• Taking thread dumps:

# List processes
# Find and use java process PID
ps -a
 
# jcmd
jcmd <pid> Thread.print
 
# jstack
jstack <pid>