Coroutines

2026-11-18 · lightweight threads

notes / DOCS

The only purpose i see is

  1. non blocking i/o -> do something useful during io (which threads would also guarantee ig but scheduler overhead)

    actually no, even with threads its still blocking because that particular thread is blocked. async io avoids even that

  2. coroutines -> just functions that can run inbetween each other, doesn't necessarily have to be io

    a better definition ig would be A coroutine is a subroutine (function) that can be suspended and resumed. It is suspended by the await expression and resumed once the await expression is resolved.

    Quote

    Coroutines are an alternative that is provided by the Python language and runtime (standard interpreter) and further supported by the asyncio module. They are suited to non-blocking I/O with subprocesses and sockets, however, blocking I/O and CPU-bound tasks can be used in a simulated non-blocking manner using threads and processes under the covers.1

  3. async stuff -> any sort of lazy computation i suppose

OS thread overhead2

for example

#include <thread>

int main() {
  int counter{0};
  auto mythread = std::thread([&] {
    for (auto i{0}; i < 100000u; i++)
      counter++;
  });
  mythread.join();
}
❯ g++ test.cpp -o test.out
❯ perf stat -r 15 -d ./test.out

 Performance counter stats for './test.out' (15 runs):

                 0      context-switches:u               #      0.0 cs/sec  cs_per_second
                 0      cpu-migrations:u                 #      0.0 migrations/sec  migrations_per_second
               139      page-faults:u                    #  66671.1 faults/sec  page_faults_per_second  ( +-  0.25% )
              2.08 msec task-clock:u                     #      nan CPUs  CPUs_utilized         ( +-  5.89% )
             2,333      L1-dcache-load-misses:u          #      4.9 %  l1d_miss_rate            ( +- 32.60% )  (56.66%)
            13,212      branch-misses:u                  #      2.9 %  branch_miss_rate         ( +-  7.06% )
           558,152      branches:u                       #    267.7 M/sec  branch_frequency     ( +-  4.11% )
         2,306,669      cpu-cycles:u                     #      1.1 GHz  cycles_frequency       ( +-  4.44% )  (49.86%)
     <not counted>      instructions:u                   #      nan instructions  insn_per_cycle  ( +- 16.63% )  (0.00%)
     <not counted>      stalled-cycles-frontend:u        #      nan frontend_cycles_idle        ( +-100.00% )  (0.00%)

       0.002428192 +- 0.000144113 seconds time elapsed  ( +-  5.93% )

The same thing without threads

int main() {
  int counter{0};
  for (auto i{0}; i < 100000u; i++) {
    counter++;
  }
}
❯ g++ test.cpp -o test.out
❯ perf stat -r 15 -d ./test.out

 Performance counter stats for './test.out' (15 runs):

                 0      context-switches:u               #      0.0 cs/sec  cs_per_second
                 0      cpu-migrations:u                 #      0.0 migrations/sec  migrations_per_second
               131      page-faults:u                    #  72461.6 faults/sec  page_faults_per_second  ( +-  0.23% )
              1.81 msec task-clock:u                     #      nan CPUs  CPUs_utilized         ( +-  5.32% )
             2,972      L1-dcache-load-misses:u          #      4.8 %  l1d_miss_rate            ( +- 26.60% )  (9.04%)
            13,205      branch-misses:u                  #      2.8 %  branch_miss_rate         ( +-  6.74% )  (66.79%)
           560,235      branches:u                       #    309.9 M/sec  branch_frequency     ( +-  3.15% )
         2,233,337      cpu-cycles:u                     #      1.2 GHz  cycles_frequency       ( +-  4.74% )  (90.96%)
         3,124,955      instructions:u                   #      1.9 instructions  insn_per_cycle  ( +- 20.29% )  (33.21%)
     <not counted>      stalled-cycles-frontend:u        #      nan frontend_cycles_idle        ( +-100.00% )  (0.00%)

       0.002108552 +- 0.000113730 seconds time elapsed  ( +-  5.39% )

slight improvement thoda sa ig, around the cost of creating one thread

Thread Creation

Thread creation is relatively cheap

// Source - https://stackoverflow.com/a/27764581
// Posted by Nafnlaus, modified by community. See post 'Timeline' for change history
// Retrieved 2026-09-13, License - CC BY-SA 3.0
#include <thread>

int main(int argc, char** argv)
{
  for (volatile int i = 0; i < 500000; i++)
    std::thread([](){}).detach();
  return 0;
}

on my machine that's about 23 microseconds per thread whcih is pretty meh, not bad

Context Switch

The real cost of threads comes from context switching and scheduler overhead. Every syscall/interrupt handler throttles IPC and takes some time to come back to normal

So threads are bad when

  1. short computations
  2. result needed in flow control -> will block main thread
  3. need stack memory which apparently can not be "decommitted"

ig they're fine for longer cpu bound tasks because apparently asyncio doens't improve that

Event loops

The linux scheduler to my knowledge treats threads as separate processes and schedules them as such (lot more nuance but whatever), they each get their own PCBs and must be context switched between

To replace OS overhead we need our own scheduler implementation, the event loop

It gets events from some sort of event provider, and schedules them for execution according to whatever policy -> choosing the next best event that is ready

Control goes to the event loop whenever the current task is blocked/waiting/finished. That task's state must be saved in those cases.

Javascript Event Loop

js event loop

The main code acts as the event provider on a single thread. Promises/Browser APIs etc are treated as callbacks on the callback queue and executed when the main call stack is empty

  1. JS Event loop
  2. same thing with images

Python AsyncIO

The event loop

def run_forever(self):
  """Run until stop() is called"""
  self._check_closed()
  if self.is_running():
    raise RuntimeError("Event loop is running")
  self._set_coroutine_wrappper(self._debug)
  self._thread_id = threading.get_ident()
  try:
    while True:
      self._run_once()
      if self._stopping
        break
  finally:
    self._stopping = False
    self._thread_id = None
    self._set_coroutine_wrappper(False)

Single Step

def _run_once(self):
  """Run one full iteration of the event loop.

  This calls all currently ready callbacks, polls for I/O,
  schedules the resulting callbacks, and finally schedules
  'call_later' callbacks.
  """

  sched_count = len(self._scheduled)
  if (sched_count > _MIN_SCHEDULED_TIMER_HANDLES and
      self._timer_cancelled_count / sched_count >
          _MIN_CANCELLED_TIMER_HANDLES_FRACTION):
      # Remove delayed calls that were cancelled if their number
      # is too high
      new_scheduled = []
      for handle in self._scheduled:
          if handle._cancelled:
              handle._scheduled = False
          else:
              new_scheduled.append(handle)

      heapq.heapify(new_scheduled)
      self._scheduled = new_scheduled
      self._timer_cancelled_count = 0
  else:
      # Remove delayed calls that were cancelled from head of queue.
      while self._scheduled and self._scheduled[0]._cancelled:
          self._timer_cancelled_count -= 1
          handle = heapq.heappop(self._scheduled)
          handle._scheduled = False

  timeout = None
  if self._ready or self._stopping:
      timeout = 0
  elif self._scheduled:
      # Compute the desired timeout.
      timeout = self._scheduled[0]._when - self.time()
      if timeout > MAXIMUM_SELECT_TIMEOUT:
          timeout = MAXIMUM_SELECT_TIMEOUT
      elif timeout < 0:
          timeout = 0

  event_list = self._selector.select(timeout)
  self._process_events(event_list)
  # Needed to break cycles when an exception occurs.
  event_list = None

  # Handle 'later' callbacks that are ready.
  now = self.time()
  # Ensure that `end_time` is strictly increasing
  # when the clock resolution is too small.
  end_time = now + max(self._clock_resolution, math.ulp(now))
  while self._scheduled:
      handle = self._scheduled[0]
      if handle._when >= end_time:
          break
      handle = heapq.heappop(self._scheduled)
      handle._scheduled = False
      self._ready.append(handle)

  # This is the only place where callbacks are actually *called*.
  # All other places just add them to ready.
  # Note: We run all currently scheduled callbacks, but not any
  # callbacks scheduled by callbacks run this time around --
  # they will be run the next time (after another I/O poll).
  # Use an idiom that is thread-safe without using locks.
  ntodo = len(self._ready)
  for i in range(ntodo):
      handle = self._ready.popleft()
      if handle._cancelled:
          continue
      if self._debug:
          try:
              self._current_handle = handle
              t0 = self.time()
              handle._run()
              dt = self.time() - t0
              if dt >= self.slow_callback_duration:
                  logger.warning('Executing %s took %.3f seconds',
                                  _format_handle(handle), dt)
          finally:
              self._current_handle = None
      else:
          handle._run()
  handle = None  # Needed to break cycles when an exception occurs.

At each step the event loop has 4 responsibilities

  1. call the currently scheduled callbacks

    for handle in self._scheduled:
        if handle._cancelled:
            handle._scheduled = False
        else:
            new_scheduled.append(handle)
  2. poll for i/o

    event_list = self._selector.select(timeout)
    self._process_events(event_list)
    # Needed to break cycles when an exception occurs.
    event_list = None

    Selectors: Reactors and Proactors

    io selectors are faster on windows because those guys choose from a thread pool at the kernel level, on unix its recommended to use uvloop

  3. schedule remaining callbacks (handled above)

  4. schedule call_later callbacks (method to append functions to event loop queue)

    # Handle 'later' callbacks that are ready.
    now = self.time()
    # Ensure that `end_time` is strictly increasing
    # when the clock resolution is too small.
    end_time = now + max(self._clock_resolution, math.ulp(now))
    while self._scheduled:
        handle = self._scheduled[0]
        if handle._when >= end_time:
            break
        handle = heapq.heappop(self._scheduled)
        handle._scheduled = False
        self._ready.append(handle)

Goroutines

Go's interpretation of coroutines. src code

Work Stealing Go Scheduler

this is nice

All of these interact, and are overall managed by the scheduler

diagram

type schedt struct {
  // ...

    lock mutex

    midle        listHeadManual // idle m's waiting for work
    nmidle       int32          // number of idle m's waiting for work
    nmidlelocked int32          // number of locked m's waiting for work
    mnext        int64          // number of m's that have been created and next M ID
    maxmcount    int32          // maximum number of m's allowed (or die)
    nmsys        int32          // number of system m's not counted for deadlock
    nmfreed      int64          // cumulative number of freed m's

    pidle        puintptr // idle p's

    // Global runnable queue.
    runq gQueue

    // Global cache of dead G's.
    gFree struct {
        lock    mutex
        stack   gList // Gs with stacks
        noStack gList // Gs without stacks
    }

    // ...
}

work stealing

What is work stealing?

src code

// Finds a runnable goroutine to execute.
// Tries to steal from other P's, get g from local or global queue, poll network.
// tryWakeP indicates that the returned goroutine is not normal (GC worker, trace
// reader) so the caller should try to wake a P.
func findRunnable() (gp *g, inheritTime, tryWakeP bool) {
    mp := getg().m

    // The conditions here and in handoffp must agree: if
    // findRunnable would return a G to run, handoffp must start
    // an M.

top:
    // ...

    // Try to schedule a GC worker.
    if gcBlackenEnabled != 0 {
        gp, tnow := gcController.findRunnableGCWorker(pp, now)
        if gp != nil {
            return gp, false, true
        }
        now = tnow
    }

    // Check the global runnable queue once in a while to ensure fairness.
    // Otherwise two goroutines can completely occupy the local runqueue
    // by constantly respawning each other.
    if pp.schedtick%61 == 0 && !sched.runq.empty() {
        lock(&sched.lock)
        gp := globrunqget()
        unlock(&sched.lock)
        if gp != nil {
            return gp, false, false
        }
    }

    // local runq
    if gp, inheritTime := runqget(pp); gp != nil {
        return gp, inheritTime, false
    }

    // global runq
    if !sched.runq.empty() {
        lock(&sched.lock)
        gp, q := globrunqgetbatch(int32(len(pp.runq)) / 2)
        unlock(&sched.lock)
        if gp != nil {
            if runqputbatch(pp, &q); !q.empty() {
                throw("Couldn't put Gs into empty local runq")
            }
            return gp, false, false
        }
    }

    // Poll network.
    // This netpoll is only an optimization before we resort to stealing.
    // We can safely skip it if there are no waiters or a thread is blocked
    // in netpoll already. If there is any kind of logical race with that
    // blocked thread (e.g. it has already returned from netpoll, but does
    // not set lastpoll yet), this thread will do blocking netpoll below
    // anyway.
    // We only poll from one thread at a time to avoid kernel contention
    // on machines with many cores.
    if netpollinited() && netpollAnyWaiters() && sched.lastpoll.Load() != 0 && sched.pollingNet.Swap(1) == 0 {
        // ...
    }

    // Spinning Ms: steal work from other Ps.
    //
    // Limit the number of spinning Ms to half the number of busy Ps.
    // This is necessary to prevent excessive CPU consumption when
    // GOMAXPROCS>>1 but the program parallelism is low.
    if mp.spinning || 2*sched.nmspinning.Load() < gomaxprocs-sched.npidle.Load() {
        if !mp.spinning {
            mp.becomeSpinning()
        }

        gp, inheritTime, tnow, w, newWork := stealWork(now)
        if gp != nil {
            // Successfully stole.
            return gp, inheritTime, false
        }
        if newWork {
            // There may be new timer or GC work; restart to
            // discover.
            goto top
        }

        now = tnow
        if w != 0 && (pollUntil == 0 || w < pollUntil) {
            // Earlier timer to wait for.
            pollUntil = w
        }
    }

    // We have nothing to do.

    // Delicate dance: thread transitions from spinning to non-spinning
    // state, potentially concurrently with submission of new work. We must
    // drop nmspinning first and then check all sources again (with
    // #StoreLoad memory barrier in between). If we do it the other way
    // around, another thread can submit work after we've checked all
    // sources but before we drop nmspinning; as a result nobody will
    // unpark a thread to run the work.
    //
    // This applies to the following sources of work:
    //
    // * Goroutines added to the global or a per-P run queue.
    // * New/modified-earlier timers on a per-P timer heap.
    // * Idle-priority GC work (barring golang.org/issue/19112).
    //
    // If we discover new work below, we need to restore m.spinning as a
    // signal for resetspinning to unpark a new worker thread (because
    // there can be more than one starving goroutine).
    //
    // However, if after discovering new work we also observe no idle Ps
    // (either here or in resetspinning), we have a problem. We may be
    // racing with a non-spinning M in the block above, having found no
    // work and preparing to release its P and park. Allowing that P to go
    // idle will result in loss of work conservation (idle P while there is
    // runnable work). This could result in complete deadlock in the
    // unlikely event that we discover new work (from netpoll) right as we
    // are racing with _all_ other Ps going idle.
    //
    // We use sched.needspinning to synchronize with non-spinning Ms going
    // idle. If needspinning is set when they are about to drop their P,
    // they abort the drop and instead become a new spinning M on our
    // behalf. If we are not racing and the system is truly fully loaded
    // then no spinning threads are required, and the next thread to
    // naturally become spinning will clear the flag.
    //
    // Also see "Worker thread parking/unparking" comment at the top of the
    // file.
    // ...
}

Besides the optimisations over the previous scheduler, if

  1. local runqs are empty
  2. global runq is empty

the idle os threads (mspinning) will steal work from busy P's to reduce CPU load, therefore work stealing

Old Go Runtime Scheduler

  1. Runtime alone has several goroutines
    1. garbage collection
    2. scheduler
    3. main user's code g0
  2. create upto GMAXPROCS threads (M) as required
    • if an M is blocked the scheduler wakes up another one from the thread pool
  3. available M's pick up a G from the scheduler's global runnable queue

Shortcomings:

Goroutine Stacks

Uses split/segmented stacks and copying stacks cloudflare blog

Comparisions

  1. Memory:
    • lighter stacks
  2. Context Switch:
    • scheduler cooperatively/preemptively
  3. Concurrent IO
    • goroutines scale higher because more goroutines and less blockgin

  1. asyncio dev
  2. threads vs asyncio overhead
  3. scalable go scheduler designer doc
  4. coroutine stacks design doc
  5. GCCGO stack design
  6. split stack performance
  7. coroutine package fro go

  1. python asyncio docs↩︎

  2. OSX thread costs↩︎