General Web Performance Process Ideas

Take a baseline first and measure your starting point.
Work on one change at a time and re-validate

Front-end

Analyze the waterfall and find gaps in time between loading landmarks. I like WebPageTest
- Test on different devices and geographic locations
Use Search Console to identify pages with bad web vitals metrics.
- Look for pages that have the same types of issues so the solution is magnified across the site
Use CrUX to get bad pages and filter by device, etc.
- Same process as search console
Use PageSpeed to get performance tips and basic metrics.
Use Yellow Lab Tools to find issues with compression and requests.
Use Treo to see the distribution of CWV data and compare it.
Reduce blocking and see if you can self-host critical assets (CSS, JS, Font, LCP element if image).
- Add measurements with performance observer to measure blocking time for the <head> or individual components
Get critical assets as early as you can in the waterfall.
- 103 Early hints
- Resource hints
- fetchpriority=high
De-prioritize things not needed for the first paint
- defer, async, type="module" on script tags
- fetchpriority=low
- Dynamic imports
- Lazy loading
Use RUM tools to analyze the last mile
- Where is it slow?
- Under what conditions?
Profile in DevTools
- Find and fix long tasks
  - Yield to the main thread
- Annotate things in devtools for discussion
- Highlight painting and layout shifts. Throttle it to slow down the shifts and help highlight issues.

// Track annotation example
// from https://dzone.com/articles/showing-long-animation-frames-in-your-devtools
const observer = new PerformanceObserver((list) => {
  list.getEntries().forEach((entry) => {
    // Measure the LoAF
    performance.measure('LoAF', {
      start: entry.startTime,
      duration: entry.duration,
      detail: {
        devtools: {
          dataType: 'track-entry',
          track: 'Long Animation Frames',
          color: 'primary',
          tooltipText: 'Long Animation Frame',
          properties: [
            ['blockingDuration', entry.blockingDuration],
            ['firstUIEventTimestamp', entry.firstUIEventTimestamp],
            ['renderStart', entry.renderStart],
            ['styleAndLayoutStart', entry.styleAndLayoutStart]
          ]
        }
      }
    });
  });
});

Backend

Use the page cache hit rate from server side metrics or CDN to find misses and potential improvements
Examine server-side performance with APM. See if your server load is spiking or if servers are scaled accordingly.
Find slow DB queries. Add anything above a threshold to a searchable log (e.g. > 50ms)
Find object cache misses for slow for frequent queries on the backend. Use in-memory caching for these.
Add the Server-Timing header and consume on the front-end.