# Performance

* It's about time and the software system's ability to meet timing requirements
  * When events occur marking the passage of time
    * like
      * interrupts
      * messages
      * requests from users or other systems
      * clock events
    * the system, or some element of the system, must respond to them in time.
* Example
  * web-based system
    * the desired response might be expressed as number oftransactions that can be processed in a minute.
  * engine control system
    * the response might be the allowable variation in the firing time.
* performance has been the driving factor in system architecture for most of history, due to hardware constraints
  * This has compromised other quality attributes
  * As the price/performance ratio of hardware continues to plummet and the cost of developing software continues to rise, other attributes are gaining more importance
* all systems have performance requirements
  * even if they are not expressed
  * Example, word processor has the expectation not to wait an hour to see a character on the screen after pressing the button
* Performance is often linked to **scalability**
  * **increasing your system's capacity for work, while still performing well**
  * scalability is making your system easy to change in a particular way, and so is a kind of modifiability

## Genearl Scenario

* it begins with an event arriving at the system
  * Events can arrive in
    * periodic - predictable patterns
      * arrive predictably at regular time intervals.
      * ie an event may arrive every 10 millisecond
      * most often seen in real-time systems
    * Stochastic - mathematical distributions
      * events arrive according to some probabilistic distribution
    * Sporadic  unpredictable patterns
      * These can be captured like:
        * we might know that at most 600 events will occur in a minute
        * That there will be at least 200 milliseconds between the arrival of any two events.
      *   But hard to specific

          \-
* Responding correctly to the event requires resources (including time) to be consumed
  * can be measured by the following:
    * Latency
      * **The time between the arrival ofthe stimulus and the system's response to it**
    * Deadlines in processing
      * In the engine controller, for example, the fuel should ignite when the cylinder is in a particular position, thus introducing a processing deadline
    * The throughput of the system
      * **usually given as the number of transactions the system can process in a unit of time.**
    * The jitter of the response
      * is the allowable variation in latency
    * The number of events not processed because the system was too busy to respond
* While this is happening, the system may be simultaneously servicing other events
* Scenario
  * Source of stimulus
    * The stimuli arrive either from external (possibly multiple) or internal sources.
  * Stimulus
    * The stimuli are the event arrivals. The arrival pattern can be periodic, stochastic, or sporadic, characterized by numeric parameters
  * Artifact
    * The artifact is the system or one or more of its components.
  * Environment
    * The system can be in various operational modes, such as normal, emergency, peak load, or overload.
  * Response
    * The system must process the arriving events. This may cause a change in the system environtnent (e.g., from normal to overload mode).
  * Response measure
    * The response measures are the time it takes to process the arriving events (latency or a deadline), the variation in this time (jitter), the number of events that can be processed within a particular time interval (throughput), or a characterization ofthe events that cannot be processed (miss rate)

### Concurrency

* operations occurring in parallel
  * either on the same processor or multiple processors
* Concurrency occurs any time your system creates a new thread,
  * because threads, by definition, are independent sequences of control
* Multi-tasking on your system is supported by independent thread
* Multiple users are simultaneously supported on your system through the use ofthreads.
* Concurrency occurs any time your system is executing on more than one processors
  * mulit core on same box or different computers
* Scheduling is important when processing multiple threads on same processor
* Allowing operations to occur in parallel improves performance, because delays introduced in one thread allow the processor to progress on another thread
  * because of the interleaving phenomenon (race condition), concurrency must also be carefully managed
  * race conditions can occur when there are two threads of control and there is shared state
* The management of concurrency frequently comes down to managing how state is shared
  * Solutions:
    * for preventing race conditions is to use locks to enforce sequential access to state
    * to partition the state based on the thread executing a portion of code
* Race conditions are one of the hardest types ofbugs to discover

## Tactics

* The goal of performance tactics is to generate a response to an event arriving at the system within some time-based constraint.
  * The event can be single or a stream and is the trigger to perform computation.
  * tactics control the time within which a response is generated
* At any instant during the period after an event arrives but before the system's response fo it is complete,
  * either
    * the system is working to respond to that event or
    * the processing is blocked for some reason
* two basic contributors to the response time:
  * processing time (when the system is working to respond)
    * Processing consumes resources, which takes time
    * Events are handled by the execution of one or more components, whose time expended is a resource
      * Hardware resources
        * CPU, data stores, network communication bandwidth, and tnemory
      * Software resources
        * include entities defined by the system under design.
        * For example, buffers must be managed and access to critical sections must be made sequential
          * **A critical section is a section of code in a multi-threaded system in which at most one thread may be active at any time**
    * Different resources behave differently as they become saturated
      * as their utilization approaches their capacity
      * For example
        * as a CPU becomes more heavily loaded, performance usually degrades fairly steadily
        * when you start to run out of memory, at some point the page swapping becomes overwhelming and performance crashes suddenly
  * Blocked time (when the system is unable to respond)
    * A computation can be blocked because
      * of contention for some needed resource, because the resource is unavailable or
        * Many resources can only be used by a single client at a time
        * that other clients must wait for access to those resources, until the client has finished using the resource
          * ie a resource can only process 5 requests at a time
        * Multiple streams vying for the same resource or different events in the same stream vying for the same resource contribute to latency.
        * The more contention for a resource, the more likelihood of latency being introduced
      * Availability ofresources
        * computation cannot proceed if a resource is unavailable
        * Unavailability may be caused by the resource being offline or by failure of the component
        * Unavailability may be caused by the resource being offline or by failure of the component
      * the computation depends on the result of other computations that are not yet available
        * A computation may have to wait because
          * it must synchronize with the results of another computation or
          * it is waiting for the results of a computation that it initiated
        * This can increase with network latency or other hardware being performance
* Type of tactics
  * Control resource demand.
    * This tactic operates on the demand side to produce smaller demand on the resources that will have to service the events
    * Tactic: Manage sampling rate
      * reduce the sampling frequency at which a stream of environmental data is captured, then demand can be reduced, typically with some attendant loss of fidelity
      * maintain predictable levels oflatency
      * decide whether having a lower fidelity but consistent stream of data is preferable to losing packets of data
    * Tactic: Limit event response.
      * When discrete events arrive at the system too rapidly to be processed, then the events must be queued until they can be processed
      * Because these events are discrete, it is typically not desirable to "downsample" them
      * may choose to process events only up to a set maximum rate, thereby ensuring more predictable processing when the events are actually processed
      * could be triggered by a queue size or processor utilization measure exceeding some warning level
      * if it is unacceptable to lose any events, then you must ensure that your queues are large enough to handle the worst case.
      * If you choose to drop events,
        * then you need to choose a policy for handling this situation:
          * Do you log the dropped events, or simply ignore them?
          * Do you notify other systems, users, or administrators?
    * Tactics: Prioritize events
      * If not all events are equally important, you can impose a priority scheme that ranks events according to how important it is to service them
      * If there are not enough resources available to service them when they arise, low-priority events might be ignored
      * Ignoring events consumes minimal resources (including time), and thus increases performance compared to a system that services all events all the time
    * Tactic: Reduce overhead
      * The use of intermediaries increases the resources consumed in processing an event stream, and so removing them improves latency
      * Separation of concerns, another linchpin of modifiability, can also increase the processing overhead necessary to service an event if it leads to an event being serviced by a chain of components rather than a single component
      * The context switching and intercomponent communication costs add up,
        * especially when the components are on different nodes on a network
      * is to co-locate resources
        * Co-location may mean hosting cooperating components on the same processor to avoid the time delay of network communication
        * putting the resources in the same runtime software component to avoid even the expense of a subroutine call
        * perform a periodic cleanup of resources that have become inefficient
        * execute single-threaded servers (for simplicity and avoiding contention) and split workload across them
    * Tactic: Bound execution times
      * Place a limit on how much execution time is used to respond to an event
      * Timeouts
      * For iterative, data-dependent algorithms, limiting the number of iterations is a method for bounding execution times.
        * cost is usually a less accurate computation.
    * Tactic:  Increase resource efficiency
      * Improving the algorithms used in critical areas will decrease latency
  * Manage resources.
    * This tactic operates on the response side to make the resources at hand work more effectively in handling the demands put to them
    * Tactic: Increase resources
      * Faster processors, additional processors, additional memory, and faster networks all have the potential for reducing latency
      * Can cost, at the very top limit may not be cost effective
    * Tactic: Introduce concurrency
      * If requests can be processed in parallel, the blocked time can be reduced
      * Concurrency can be introduced by
        * processing different streams of events on different threads
        * by creating additional threads to process different sets of activities
      * scheduling policies can be used to achieve the goals you find desirable
      * Different scheduling policies may
        * maximize fairness (all requests get equal time)
        * throughput (shortest time to finish first)
        * other goals
    * Tactic: Maintain multiple copies of computations
      * Multiple servers in a client-server pattern are replicas of computation
      * purpose of replicas is to reduce the contention that would occur if all computations took place on a single server
      * **A load balancer is a piece of software that assigns new work to one of the available duplicate servers**
        * criteria for assignment vary but can be as simple as round-robin or assigning the next request to the least busy server.
    * Tactic: Maintain multiple copies of data
      * **Caching is a tactic that involves keeping copies of data (possibly one a subset of the other) on storage with different access speeds.**
        * The different access speeds may be inherent (memory versus secondary storage) or may be due to the necessity for network communication.
      * Data replication involves keeping separate copies of the data to reduce the contention from multiple simultaneous accesses
      * Because the data being cached or replicated is usually a copy of existing data, **keeping the copies consistent and synchronized** becomes a responsibility that the system must assume
      * What to cache needs to be decided
        * Some caches operate by merely keeping copies of whatever was recently requested, but it is also possible to predict users' future requests based on patterns of behavior, and begin the calculations or prefetches necessary to comply with those requests before the user has made them.
    * Tactic: Bound queue sizes
      * This controls the maximum number of queued arrivals and consequently the resources used to process the arrivals
      * need to adopt a policy for what happens when the queues overflow and decide if not responding to lost events is acceptable.
    * Tactic: Schedule resources
      * Whenever there is contention for a resource, the resource must be scheduled
      * understand the characteristics of each resource's use and choose the scheduling strategy that is compatible with it

## Scheduling Policies

* A scheduling policy conceptually has two parts
  * a priority assignment
    * ie FIFO
    * it can be tied to the deadline ofthe request or its semantic importance.
    * Competing criteria for scheduling include
      * optimal resource usage
      * request importance
      * minimizing the number of resources used
      * minimizing latency
      * maximizing throughput
      * preventing starvation to ensure fairness
  * dispatching
* A high-priority event stream can be dispatched only ifthe resource to which it is being assigned is available.
  * Sometimes this depends on preempting the current user of the resource
    * preemption options :
      * can occur anytime
      * can occur only at specific preemption points
      * executing processes cannot be preempted
* Scheduling policies
  * First-in/first-out
    * treat all requests for resources as equals and satisfy them in turn
    * Issues
      * one request will be stuck behind another one that takes a long time to generate a response
        * Ok if all requests are equal, but not it some have different priorities
  * Fixed-priority scheduling
    * assigns each source of resource requests a particular priority and assigns the resources in that priority order.
    * Issues
      * it admits the possibility of a lower priority, but important, request taking an arbitrarily long time to be serviced, because it is stuck behind a series of higher priority requests.
    * prioritization strategies
      * Semantic importance
        * Each stream is assigned a priority statically according to some domain characteristic of the task that generates it
      * Deadline monotonic
        * is a static priority assignment that assigns higher priority to streams with shorter deadlines.
        * This scheduling policy is used when streams of different priorities with real-time deadlines are to be scheduled
      * Rate monotonic
        * is a static priority assignment for periodic streams that assigns higher priority to streams with shorter periods.
        * This scheduling policy is a special case of deadline monotonic but is better known and more likely to be supported by the operating system.
      * Dynamic priority scheduling
        * Round-robin
          * is a scheduling strategy that orders the requests and then, at every assignment possibility, assigns the resource to the next request in that order.
          * A special form of round-robin is a cyclic executive, where assignment possibilities are at fixed time intervals
        * Earliest-deadline-first
          * assigns priorities based on the pending requests with the earliest deadline
        * Least-slack-first
          * This strategy assigns the highest priority to the job having the least "slack time," which is the difference between the execution time remaining and the time to the job's deadline.
* For a single processor and processes that are preemptible (it is possible to suspend processing of one task in order to service a task whose deadline is drawing near)
  * the earliest-deadline and least-slack scheduling strategies are optimal.
* Static scheduling
  * A cyclic executive schedule is a scheduling strategy where the preemption points and the sequence of assignment to the resource are determined offline. The runtime overhead of a scheduler is thereby obviated.

## Checklist for Performance

* Allocation of Responsibilities
  * Determine the system's responsibilities that will involve
    * heavy loading
    * have time-critical response requirements
    * are heavily used
    * impact portions of the system where
      * heavy load's occur
      * time-critical events occur.
  * For those responsibilities
    * Identify the processing requirements of each respons1bility,
    * determine whether they may cause bottlenecks
  * identify additional responsibilities to recognize and process requests appropriately, including
    * Responsibilities that result from a thread of control crossing process or processor boundaries
    * Responsibilities to manage the threads of control-allocation and deallocation of threads, malntalnlng thread pools, and so forth
    * Responslbilities for scheduling shared resources or managing performance-related artifacts such as
      * queues
      * buffers
      * caches
  * For the responsibiliti.es and resources you identified, ensure that the required performance response can be met
* Coordination Model
  * Determine the elements of the system that must coordinate with each other-directly or indirectly and choose communication and coordination mechanisms that do the following:
    * Support any introduced
      * concurrency (for example, is it thread safe?)
      * event prioritization
      * scheduling strategy
    * Ensure that the required performance response can be delivered
    * Can capture periodic, stochastic, or sporadic event arrivals, as needed
    * Have the appropriate properties of the communication mechanlsms;
      * for example:
        * stateful
        * stateless
        * synchronous
        * asynchronous
        * guaranteed delivery
        * throughput
        * latency
* Data Model
  * Determine those port1;ons of the data model that will be
    * heavily loaded
    * have time-critical response requirements
    * are heavily used
    * impact portions of the system where heavy loads
    * time-critical events occur
  * For those data abstractions, determine the following:
    * Whether maintaining multiple copies of key data would benefit performance
    * Whether partitioning data would benefit performance
    * Whether reducing the processing requirements for the creatlon, initialization, persistence, manipulation, translation, or destruction of the enumerated data abstractions is possible
    * Whether adding resources to reduce bottlenecks for the creation, Initialization, persistence, manipulatton, translation, or destruction of the enumerated data abstractions is feasible
* Mapping among Architectural Elements
  * Where heavy network loading will occur, determine whether co-locating some components will reduce loading and improve overall efficiency.
  * Ensure that components with heavy computation requirements are assigned to processors with the most processing capacity
  * Determine where introducing concurrency (that is, allocattng a piece of functionallty to two or more copies of a component running simultaneously) is feasible and has a significant positive effect on performance
  * Determine whether the choice of threads of control and their associated responsibilities introduces bottlenecks.
* Resource Management
  * Determine which resources in your system are critical for performance.
  * For these resources, ensure that they will be monitored and managed under normal and overloaded system operation. For example:
    * System elements that need to be aware of, and manage, time and other performance-critical resources
    * Process/thread models
    * Prioritization of resources and access to resources
    * Scheduling and looking strategies
    * Deploying additional resources on demand to meet increased loads
* Binding Time
  * For each element that will be bound after compile time, determine time following:
    * Time necessary to complete the binding
    * Additional overhead introduced by using the late binding mechanism
  * Ensure that these values do not pose unacceptable performance penalties on the system
* Choice of Technology
  * Will your choice of technology let you set and meet hard, real time deadlines?
  * Do you know its characteristics under load and its limits?
  * Does your choice of technology g'ive you the ability to set the following
    * Scheduling policy
    * Priorities
    * Policies for reducing demand
    * Allocation of portions of the technology to processors
    * Other performance-related parameters
  * Does your choice of technology introduce excessive overhead for heavily used operations?
