Systematic Stress Testing of Concurrent Programs

Systematic Stress Testing of Concurrent Programs

Concurrency Testing Challenges, Algorithms, and Tools Madan Musuvathi Microsoft Research Concurrency is HARD A concurrent program should Function correctly Maximize throughput Finish as many tasks as possible Minimize latency Respond to requests as soon as possible While handling nondeterminism in the environment Concurrency is Pervasive Concurrency is an age-old problem of computer

science Most programs are concurrent At least the one that you expect to get paid for, anyway Solving the Concurrency Problem We need Better programming abstractions Better analysis and verification techniques Better testing methodologies Weakest Link Testing is more important than you think My first-ever computer program: Wrote it in Basic Not the worlds best programming language With no idea about program correctness

I didnt know first-order logic, loop-invariants, I hadnt heard about Hoare, Dijkstra, But still managed to write correct programs, using the write, test, [debug, write, test]+ cycle How many of you have written a program > 10,000 lines? written a program, compiled it, called it done without testing the program on a single input? written a program, compiled it, called it done without testing the program on few interesting inputs? Imagine a world where you cant pick the inputs during testing

You write the program Check its correctness by staring at it int factorial ( int x ) { int ret = 1; while(x > 1){ ret *= x; x --; } return ret; } Give the program to the computer The computer tests on inputs of its choice factorial(5) = 120 factorial(5) = 120 the next 100 times factorial(7) = 5040

The computer runs this program again and again on these inputs for a week The program didnt crash and therefore it is correct This is the world of concurrency testing You write the program Parent_thread() { if (p != null) { p = new P(); Set (initEvent); } } Child_thread(){ if (p != null) { Wait (initEvent);

} } Check its correctness by staring at it Give the program to the computer The computer generates some interleavings The computer runs this program again and again on these interleavings The program didnt crash and therefore its is correct Demo How do we test concurrent software today

CHESS Proposition Capture and expose nondeterminism to a scheduler Threads can run at different speeds Asynchronous tasks can start at arbitrary time in the future Hardware/compiler can reorder instructions Explore the nondeterminism using several algorithms Tackle the astronomically large number of interleavings Remember: Any algorithm is better than no control at all CHESS in a nutshell CHESS is a user-mode scheduler Controls all scheduling nondeterminism Replace the OS scheduler Guarantees: Every program run takes a different thread interleaving

Reproduce the interleaving for every run Download CHESS source from http://chesstool.codeplex.com/ CHESS architecture Unmanaged Program Win32 Wrappers Windows CHESS Exploration Engine CHESS

Scheduler Managed Program .NET Wrappers CLR Every run takes a different interleaving Reproduce the interleaving for every run Running Example Thread 1 Lock (l); bal += x; Unlock(l); Thread 2

Lock (l); t = bal; Unlock(l); Lock (l); bal = t - y; Unlock(l); Introduce Schedule() points Thread 1 Schedule(); Lock (l); bal += x; Schedule(); Unlock(l); Thread 2 Schedule(); Lock (l);

t = bal; Schedule(); Unlock(l); Schedule(); Lock (l); bal = t - y; Schedule(); Unlock(l); Instrument calls to the CHESS scheduler Each call is a potential preemption point First-cut solution: Random sleeps Thread 1 Sleep(rand());

Lock (l); bal += x; Sleep(rand()); Unlock(l); Thread 2 Sleep(rand()); Lock (l); t = bal; Sleep(rand()); Unlock(l); Sleep(rand()); Lock (l); bal = t - y; Sleep(rand()); Unlock(l); Introduce random sleep at

schedule points Does not introduce new behaviors Sleep models a possible preemption at each location Sleeping for a finite amount guarantees starvation-freedom Improvement 1: Capture the happens-before graph Thread 1 Schedule(); Lock (l); bal += x;Sleep(5) Schedule(); Unlock(l);

Thread 2 Schedule(); Schedule(); Lock (l); (l); Lock bal; tt == bal; Schedule(); Schedule(); Unlock(l); Unlock(l); Schedule(); Schedule(); Lock (l); Lock bal = (l);

t - y; bal = t - y;Sleep(5) Schedule(); Schedule(); Unlock(l); Unlock(l); Delays that result in the same happens-before graph are equivalent Avoid exploring equivalent interleavings Improvement 2:

Understand synchronization semantics Thread 1 Schedule(); Lock (l); bal += x; Schedule(); Unlock(l); Thread 2 Schedule(); Schedule(); Lock (l); (l); Lock bal; tt == bal; Schedule(); Unlock(l);

Schedule(); Unlock(l); Schedule(); Lock (l); Schedule(); bal = t - y; Lock (l); Schedule(); bal = t - y; Unlock(l); Schedule(); Unlock(l); Avoid exploring delays that are impossible Identify when threads can make progress

CHESS maintains a run queue and a wait queue Mimics OS scheduler state CHESS modes: speed vs coverage Fast-mode Introduce schedule points before synchronizations, volatile accesses, and interlocked operations Finds many bugs in practice Data-race mode Introduce schedule points before memory accesses Finds race-conditions due to data races Captures all sequentially consistent (SC) executions CHESS Design Choices Soundness Any bug found by CHESS should be possible in the field

Should not introduce false errors (both safety and liveness) Completeness Any bug found in the field should be found by CHESS In theory, we need to capture all sources of nondeterminism In practice, we need to effectively explore the astronomically large state space Capture all sources of nondeterminism? No. Scheduling nondeterminism? Yes Timing nondeterminism? Yes Controls when and in what order the timers fire Nondeterministic system calls? Mostly

CHESS uses precise abstractions for many system calls Input nondeterminism? No Rely on users to provide inputs Program inputs, return values of system calls, files read, packets received, Good tradeoff in the short term But cant find race-conditions on error handling code Capture all sources of nondeterminism? No. Hardware relaxations? Yes Hardware can reorder instructions Non-SC executions possible in programs with data races Sober [CAV 08] can detect and explore such non-SC executions Compiler relaxations? No

Very few people understand what compilers can do to programs with data races Far fewer than those who understand the general theory of relativity Schedule Exploration Algorithms Two kinds Reduction algorithms Explore one out of a large number equivalent interleavings Prioritization algorithms Pick interesting interleavings before you run out of resources Remember: anything is better than nothing

Schedule Exploration Algorithms Reduction Algorithms Enumerating Thread Interleavings Using Depth-First Search Thread 1 Thread 2 x = 1; y = 1; x = 2; y = 2; Explore (State s) { T = set of threads in s;

foreach x = 1;t in T { s = schedule t in s Explore(s); y = 1; } } x = 2; y = 2; 0,0 2,0 1,0 1,1 1,0

2,0 2,2 2,1 2,1 2,2 1,1 1,2 1,2 2,2

2,2 2,1 1,2 1,1 1,1 Behaviorally equivalent interleavings x = 1; x = 1; y = 2; if(x == 1) {

equiv if(x == 1) { y = 2; y = 3; } Reach the same final state (x = 1, y = 3) y = 3; } Behaviorally inequivalent interleavings x = 1; x = 1; y = 2;

if(x == 1) { equiv if(x == 1) { y = 3; } y = 3; } y = 2; Reach different final states (1, 3) vs (1,2) Behaviorally inequivalent interleavings if(x == 1) { x = 1; x = 1; if(x == 1) { equiv y = 3; }

y = 2; y = 2; Dont necessarily have to reach different states Execution Equivalence Two executions are equivalent if they can be obtained by commuting independent operations x=1 r1 = y r2 = y r3 = x

x=1 r2 = y r1 = y r3 = x r2 = y x=1 r1 = y r3 = x r2 = y

x=1 r3 = x r1 = y Formalism Execution is a sequence of transitions Each transition is of the form tid: thread performing the transition var: the memory location accessed in the transition op: READ | WRITE | READWRITE Two steps are independent if They are executed by different threads and Either they access different variable or READ the same

variable Equivalence makes the schedule space a Directed Acyclic Graph Thread 1 Thread 2 x = 1; y = 1; x = 2; y = 2; 0,0 2,0 1,0 1,1

1,0 2,0 2,2 2,1 2,1 2,2 1,1 1,2 1,2

2,2 2,2 2,1 1,2 1,1 1,1 DFS in a DAG (CS 101) HashTable visited; Explore (Sequence s) { T = set of threads enabled in S; foreach t in T {

s = s . ; Explore(s); s = canon(s); if (s in visited) continue; } if (s in visited) continue; visited.Add(s); } visited.Add(s); Explore(s); } Explore(s); } } } Sleep sets algorithm explores a DAG without maintaining the table

Sleep Set Algorithm Thread 1 Thread 2 x = 1; y = 1; x = 2; y = 2; 0,0 2,0 1,0 Identify transitions that take you to visited states

1,1 1,0 2,0 2,2 2,1 2,2 1,1 1,2 2,2

2,1 1,2 1,1 Sleep Set Algorithm Explore (Sequence s, sleep C) { T = set of transitions enabled in s; T = T C; foreach t in T { C=C+t s = s . t ; C = C {transitions dependent on t} Explore(s, C); } }

Summary Sleep sets ensure that a stateless execution does not explode a DAG into a tree Persistent Set Reduction Thread 1 Thread 2 x = 1; x = 2; y = 1; y = 2;

With Sleep Sets Thread 1 Thread 2 x = 1; x = 2; y = 1; y = 2; With Persistent Sets Thread 1 Thread 2 x = 1; x = 2;

y = 1; y = 2; Assumption: we are only interested in the reachability of final states (for instance, no global assertions) Persistent Sets A set of transitions P is persistent in a state s, if In the state space X reachable from s by only exploring transitions not in P Every transition in X is independent with P P persists in X

It is sound to only explore P from s s x With Persistent Sets Thread 1 Thread 2 x = 1; x = 2; y = 1; y = 2;

Dynamic Partial-Order Reduction Algorithm [Flanagan & Godefroid] Identifies persistent sets dynamically After execution a transition, insert a schedule point before the most recent conflict Thread 1 Thread 2 y = 1; x = 1; x = 2; z = 3; y=1 x=1

x=2 x=2 z=3 x=1 z=3 Schedule Exploration Algorithms Prioritization Algorithms Schedule Prioritization Preemption bounding Few preemptions are sufficient for finding lots of bugs Preemption sealing Insert preemptions where you think bugs are Random

If you dont have additional information about the state space, random is the best Still do partial-order reduction Concurrency Correctness Criterion CHESS checks for various correctness criteria Assertion failures Deadlocks Livelocks Data races Atomicity violations (Deterministic) Linearizability violations Concurrency Correctness Criterion Linearizability Checking in CHESS

Motivation Writing good test oracles is hard Thread 1 Thread 2 Bank.Add($20) Bank.Withdraw($20); Assert(Bank.Balance() == ?) Motivation Writing good test oracles is hard Thread 1 Thread 2 q.AddFirst(10) q.AddLast(20) q.RemoveLast() q.RemoveFirst()

Assert(q.IsEmpty()) Is this a correct assertion to check for? Now what if there are 5 threads each performing 5 queue operations We want to magically Check if a Bank behaves like a Bank should do Check if a queue behaves like a queue Answer: Check for linearizability Linearizability The correctness notion closest to thread safety A concurrent component behaves as if it is protected by a single global lock

Each operation appears to take effect instantaneously at some point between the call and return The Problem with Linearizability Checking Need a sequential specification Imagine writing a sequential specification for your operating system Instead, check if a component is linearizable with respect to some deterministic specification This can be done automatically Generate the sequential specification by inserting a global lock LineUp: Two-Phase method

For a given test: First, generate the sequential specification Enumerate serial executions of the test Record all observed histories Assume the generated histories are the intended behaviors of the component Second, check linearizability with respect to the generated specification Enumerate fully concurrent executions Test history against compatibility with serial executions Line-Up on the Bank Example Thread 1 Thread 2 Bank.Add($20) Bank.Withdraw($20);

Assert( Bank.Balance() == 20 || Bank.Balance() == 0 ) Serial executions imply that the final balance can be 20 or 0 Concurrent executions should satisfy the assertion Line-Up guarantees Full Completeness: If Line-Up reports a violation, the implementation is not linearizable with respect to any deterministic specification. Restricted Soundness: If the implementation is not linearizable with respect to any deterministic specification, there

exists a test on which Line-Up will report a violation. Linearizability Violations Non-linearizable histories can reveal implementation errors (e.g. incorrect synchronization) The nonlinearizable behavior below was caused by a bug in .NET 4.0 (accidental lock timeout). Thread 1 Thread 2 Add 200 return TryTake Add 200 return TryTake

Return 200 return empty Generalizing Linearizability Some operations may block. e.g. semaphore.acquire() Blocking can be good (expected behavior) or bad (bug). Original definition of linearizability does not make this distinction. Blocking is always o.k. We generalized definition to be able to catch bad blocking

A buggy counter implementation class Counter{ int count = 0; bool b = false; Lock lock = new Lock(); void inc() { b = true; lock.acquire(); count = count + 1; lock.release(); b = false; } void get() { lock.acquire(); t = count; if(!b) lock.release(); return t;

} } Stuck History: inc call get call get 1 inc ret inc call Results Each letter is a separate root cause Questions

(A) Incorrect use of CAS causes state corruption. (B) RemoveLast() uses an incorrect lock-free optimization. (C) Call to SemaphoreSlim includes a timeout parameter by mistake. (D) ToArray() can livelock when crossing segment boundaries. Note that the harness for this class performs a particular pre-test sequence (add 31 elements, remove 31 elements). (E) Insufficient locking: thread can get preempted while trying to set an exception. (F) Barrier is not a linearizable data type. Barriers block each thread until all threads have entered the barrier, a behavior that is not equivalent to any serial execution. (G) Cancel is not a linearizable method: The effect of the cancellation can be delayed past the operation return, and in fact even past subsequent operations on the same thread. (H) Count() may release a lock it does not own if interleaved with Add(). (I) Bag is nondeterministic by design to improve performance: the returned value can depend on the specific interleaving. (J) Count may return 0 even if the collection is not empty. The specification of the Count method was weakened after Line-Up detected this behavior.

(K) TryTake may fail even if the collection is not empty. The specification of the TryTake method was weakened after Line-Up detected this behavior. (L) SetResult() throws the wrong exception if the task is already reserved for completion by somebody else, but not completed yet. Results: Phase 1 / Phase 2 Outline Preemption bounding Makes CHESS effective on deep state spaces Fair stateless model checking Sober FeatherLite Concurrency Explorer Outline

Preemption bounding Makes CHESS effective on deep state spaces Fair stateless model checking Makes CHESS effective on cyclic state spaces Enables CHESS to find liveness violations (livelocks) Sober FeatherLite Concurrency Explorer Concurrent programs have cyclic state spaces Thread 1 L1: while( ! done) { L2: Sleep(); } Thread 2

! done L1 ! done L2 done L1 done L2 M1: done = 1; Spinlocks Non-blocking algorithms Implementations of synchronization primitives

Periodic timers A demonic scheduler unrolls any cycle adinfinitum Thread 1 while( ! done) { Sleep(); } Thread 2 ! done done = 1; ! done ! done

! done done done done Depth bounding Prune executions beyond a bounded number of steps ! done ! done ! done Depth bound

! done done done done Problem 1: Ineffective state coverage Bound has to be large enough to reach the deepest bug Typically, greater than 100 ! done synchronization operations ! done

Every unrolling of a cycle redundantly explores reachable state space ! done Depth bound ! done Problem 2: Cannot find livelocks Livelocks : lack of progress in a program Thread 1 temp = done; while( ! temp)

{ Sleep(); } Thread 2 done = 1; Key idea Thread 1 while( ! done) { Sleep(); } Thread 2 ! done ! done

done done done = 1; This test terminates only when the scheduler is fair Fairness is assumed by programmers All cycles in correct programs are unfair A fair cycle is a livelock We need a fair demonic scheduler Test Harness Concurrent Program

Avoid unrolling unfair cycles Effective state coverage Detect fair cycles Win32 API Fair Demonic Demonic Scheduler Scheduler Find livelocks What notion of fairness do we use? Weak fairness

Forall t :: GF ( enabled(t) scheduled(t) ) A thread that remains enabled should eventually be scheduled Thread 1 while( ! done) { Sleep(); } Thread 2 done = 1; A weakly-fair scheduler will eventually schedule Thread 2 Example: round-robin Weak fairness does not suffice Thread 1 Lock( l );

While( ! done) { Unlock( l ); Sleep(); Lock( l ); } Unlock( l ); en = {T1, T2} en = {T1, T2} T1: Sleep() T2: Lock( l ) T1: Lock( l ) T2: Lock( l )

Thread 2 Lock( l ); done = 1; Unlock( l ); en = { T1 } T1: Unlock( l ) T2: Lock( l ) en = {T1, T2} T1: Sleep() T2: Lock( l ) Strong Fairness Forall t :: GF enabled(t) GF scheduled(t) A thread that is enabled infinitely often is scheduled infinitely often

Thread 1 Lock( l ); While( ! done) { Unlock( l ); Sleep(); Lock( l ); } Unlock( l ); Thread 2 Lock( l ); done = 1; Unlock( l ); Thread 2 is enabled and competes for the lock infinitely often

Good Samaritan violation Thread yield the processor when not making progress Forall threads t : GF scheduled(t) GF yield(t) Thread 1 while( ! done) { ; } Thread 2 done = 1; Found many such violations, including one in the Singularity boot process Results in sluggish I/O behavior during bootup Results: Achieves more coverage faster

Work stealing queue with one stealer Without fairness, with depth bound With fairness 20 30 40 50 60 States Explored

1726 871 1505 1726 1307 683 Percentage Coverage 100%

50% 87% 100% 76% 40% 143 97 763 2531

>5000 >5000 Time (secs) Finding livelocks and finding (not missing) safety violations Program Lines of code Safety Bugs Work Stealing Q CDS CCR

ConcRT Dryad APE STM TPL PLINQ Singularity 4K 6K 9K 16K 18K 19K 20K 24K 24K 175K

4 1 1 2 7 4 2 4 1 26 (total) Livelocks 2 2 5

2 11 (total) Acknowledgement: testers from PCP team Outline Preemption bounding Makes CHESS effective on deep state spaces Fair stateless model checking Makes CHESS effective on cyclic state spaces Enables CHESS to find liveness violations (livelocks) Sober Detect relaxed-memory model errors Do not miss behaviors only possible in a relaxed memory model

FeatherLite Concurrency Explorer Single slide on Sober Relaxed memory verification problem Is P correct on a relaxed memory model Sober: split the problem into two parts Is P correct on a sequentially consistent (SC) machine Is P sequentially consistent on a relaxed memory model Check this while only exploring SC executions CAV 08 solves the problem for a memory model with store buffers (TSO) EC2 08 extends this approach to a general class of memory models

Outline Preemption bounding Makes CHESS effective on deep state spaces Fair stateless model checking Makes CHESS effective on cyclic state spaces Enables CHESS to find liveness violations (livelocks) Sober Detect relaxed-memory model errors Do not miss behaviors only possible in a relaxed memory model FeatherLite A light-weight data-race detection engine (<20% overhead) Concurrency Explorer

Single slide on FeatherLite Current data-race detection tools are slow Process every memory access done by the program One in 5 instructions access memory 1 billion accesses/sec Key idea: Do smart adaptive sampling of memory accesses Nave sampling does not work, need to sample both racing instructions Cold-path hypothesis: At least one of the racing instructions occurs in a cold path Races between fast-paths are most probably benign FeatherLite adaptively samples cold-paths at 100% rate and hot-paths at 0.1% rate Finds 70% of the data-races with <20% runtime overhead Existing data-race detection tools >10X overhead Outline Preemption bounding Makes CHESS effective on deep state spaces

Fair stateless model checking Makes CHESS effective on cyclic state spaces Enables CHESS to find liveness violations (livelocks) Sober Detect relaxed-memory model errors Do not miss behaviors only possible in a relaxed memory model FeatherLite A light-weight data-race detection engine (<20% overhead) Concurrency Explorer First-class concurrency debugging Concurrency explorer Single-step over a thread interleaving Inspect program states at each step

Program state = Stack of all threads + globals Limited bi-directional debugging Interleaving slices for better understanding Working on: Closer integration with the Visual Studio debugger Explore neighborhood interleavings Conclusion Dont stress, use CHESS CHESS binary and papers available at http://research.microsoft.com/CHESS Points to get across Capturing non-determinism Sync-orders, data-races, hardware interleavings

Adding elastic delay Soundness & completeness Scoping Preemptions Questions Did you find new bugs How is this different from your previous papers How is this different from previous mc efforts How is this different from Are these behaviors expected ? Thread 1 Add 10 Thread 2 Thread 3

Thread 1 TryTake Add 20 return empty return Add 10 TryTake Add 20 return return10

TryTake Thread 2 Thread 3 return return10 return TryTake return empty Linearizability Component is linearizable if all operations Appear to take effect at a single temporal point And that point is between the call and the return

As if the component was protected by a single global lock Thread 1 Add 10 Thread 2 Thread 3 TryTake TryTake return Add 20

return return10 return empty This behavior is not linearizable Thread 2 getting a 10 means that Thread 1s Add got the queue before Thread 3s Add So, when Thread 3 does a TryTake, 20 should be still in the queue Thread 1 Thread 2 Thread 3

return Add 10 TryTake Add 20 return10 return TryTake return empty Linearizable? Thread 1 Thread 2 Thread 3

return Add 10 TryTake Add 20 return20 return TryTake return empty How is Linearizability different than Seriazliability? Serializability All operations happen atomically in some serial order

Linearizability All operations happen at a single instant That instant is between the call and return Serializable behavior that is not Linearizable Thread 1 Thread 2 Add 10 return TryTake return empty Linearizability assumes that there is a global observer

that can observe that Thread 1 finished before Thread 2 started This is what makes linearizability composable Serializable behavior that is not Linearizable Thread 1 Thread 2 Add 10 return TryTake return empty Linearizability assumes that there is a global observer that can observe that Thread 1 finished before Thread

2 started This is what makes linearizability composable Serializability does not compose Thread 1 Thread 2 Add 10 return TryTake return empty Add 10 return TryTake return empty

The behavior of the blue queue and green queue are individually serializable But, together, the behavior is not serializable To make this all the more confusing Database concurrency control ensures that transactions are linearizable Even though the literature only talks about serializability Quote from Jim Gray: When a transaction finishes, the state of the database immediately reflects the updates of the transaction The commit point of a transaction is guaranteed

between the transaction begin and end When using a two-phase locking protocol, for instance Standard definition of Linearizability Is a little more general than my interpretation as if protected by a single global lock Sometimes, a concurrent implementation can have more behaviors than a sequential implementation Example: a set implemented as a queue A sequential version will be FIFO even order does not matter for a set For performance, a concurrent version can break the FIFO ordering but still maintain the set abstraction Define a sequential specification

A Sequential Specification (A fancy word for something you already know but dont usually think about) Each object has a state the sequence of elements in the queue Each operation has a precondition and a postcondition Precondition: if the queue is not empty Postcondition: Remove will remove the first element in queue Another example Precondition: True Postcondition: TryTake will Return false if the queue is empty and leave the state unchanged Otherwise, return true and remove the first element in the queue

Recently Viewed Presentations

  • Sister Annette slater - Razor Planet

    Sister Annette slater - Razor Planet

    Sister Mary Bell . Sister rose lott. Sister Courtney bradford. Sister Gloria Williams . Sister rosettahayes. Sister Margaret beam . ... Sister Myrtle Mcivor. Sister Margaret gray . Sister Velma Williams . Sister Janice wilkins. Sister Rosalyn zeigler. Sister Cathy...
  • Concept English Units and Lessons:

    Concept English Units and Lessons:

    R. How will we rethink or revise? ... fluency, reading accuracy, comprehension, and writing instruction each day. Margaret Mooney says balanced literacy can be achieved by (1) reading ... and Apollo missions. After astronaut Ed White perished in a capsule...
  • AGENDA - websites.rcc.edu

    AGENDA - websites.rcc.edu

    Plain View. Harris v. United States. A legal term describing the ready visibility of objects that might be seized as evidence during a search by police in the absence of a search warrant specifying the seizure of those objects. Legal...
  • Philippians - Having a Life of Purpose

    Philippians - Having a Life of Purpose

    Philippians 2:1-4. Therefore if you have any encouragement from being united with Christ, if any comfort from his love, if any common sharing in the Spirit, if any tenderness and compassion, then make my joy complete by being like-minded, having...
  • End-Stage Renal Disease

    End-Stage Renal Disease

    Anemia Treatment of ESRD Hemodialysis-remove metabolic wastes by diffusion as blood is pumped through dialysis machine Fistula: Joined artery and vein that allows vascular access to patient's blood 3-4 hr sessions, 3 times/week , $46,000/yr Dialysis(cont.) Peritoneal Dialysis-peritoneal ...
  • South Yorkshire Netball

    South Yorkshire Netball

    4th court added to the league - Sheffield Concord Sports Centre . Umpire Mentoring Programme. Shorter league to allow for Summer Competition to take place between May and July. Umpire Expenses - paid by teams on the evening. ... South...
  • The Rise in Spiritual Travel

    The Rise in Spiritual Travel

    Born in India, studied in a Christian missionary school, a Muslim by faith, now living in Buddhist Thailand Amateur student of the role of religions in world affairs Write a column in the Bangkok Post Follower of One, Believer in...
  • Cuyamaca College Assessment

    Cuyamaca College Assessment

    Test scores will be available on WebAdvisor within 24-48 hours of completing your Assessment test. Complete your Online Orientation (if you haven't already) Complete your Online Advising. You will need to complete your Online Advising session on WebAdvisor upon viewing...