diff options
| author | Puranjay Mohan <puranjay@kernel.org> | 2026-06-24 06:23:52 -0700 |
|---|---|---|
| committer | Paul E. McKenney <paulmck@kernel.org> | 2026-07-24 16:57:29 -0700 |
| commit | ec5ee5417b00278fa050b38d878b76f1365420fb (patch) | |
| tree | fb6cd8c1d6a60256075a2c16b50a339ece36c724 | |
| parent | fde2b18817e4f32b92c1e9881356fbc314983a7a (diff) | |
rcu: Advance callbacks for expedited GP completion in rcu_core()
Even when rcu_pending() triggers rcu_core(), the normal callback
advancement path through note_gp_changes() -> __note_gp_changes() bails
out when rdp->gp_seq == rnp->gp_seq (no normal GP change). Since
expedited GPs do not update rnp->gp_seq, rcu_advance_cbs() is never
called and callbacks remain stuck in RCU_WAIT_TAIL.
Add a direct callback advancement block in rcu_core() that checks for GP
completion via rcu_segcblist_nextgp() combined with
poll_state_synchronize_rcu_full(). When detected, trylock rnp and call
rcu_advance_cbs() to move completed callbacks to RCU_DONE_TAIL. Wake the
GP kthread if rcu_advance_cbs() requests a new grace period.
Uses trylock to avoid adding contention on rnp->lock. If the lock is
contended, callbacks will be advanced on the next tick.
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
| -rw-r--r-- | kernel/rcu/tree.c | 17 |
1 files changed, 17 insertions, 0 deletions
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 3541feed9557..f6809ccedfc1 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2885,6 +2885,23 @@ static __latent_entropy void rcu_core(void) /* Update RCU state based on any recent quiescent states. */ rcu_check_quiescent_state(rdp); + /* Advance callbacks if an expedited GP has completed. */ + if (!rcu_rdp_is_offloaded(rdp) && rcu_segcblist_is_enabled(&rdp->cblist)) { + struct rcu_gp_seq gp_state; + + if (rcu_segcblist_nextgp(&rdp->cblist, &gp_state) && + poll_state_synchronize_rcu_full(&gp_state)) { + guard(irqsave)(); + if (raw_spin_trylock_rcu_node(rnp)) { + bool needwake = rcu_advance_cbs(rnp, rdp); + + raw_spin_unlock_rcu_node(rnp); + if (needwake) + rcu_gp_kthread_wake(); + } + } + } + /* No grace period and unregistered callbacks? */ if (!rcu_gp_in_progress() && rcu_segcblist_is_enabled(&rdp->cblist) && !rcu_rdp_is_offloaded(rdp)) { |