For PC power supplies
How many milliseconds? Remember that your power supply is at near zero volts twice (on the AC) for at least 1 millisecond every 50th or 60th of a second. If you are talking of 20 milli seconds or less then you should have no issues since all but the worse/unser-powered powersupply will last for 100 or more milliseconds.
If you are running your power supply on the limit of its power capabilities then depending on its design 100mSec might be its limit.
Personally I have never had an issue in the past before I used a UPS with 1/4 second (250mSec) dropouts and typically 1/2 sec to 1 sec was handled by the power supply. Yes occasionally a possum would cause a AC dropout when it contacted to overhead wires (or caused them to touch each other. Screen would flash, lights drop out and back again but the computer kept going and the router too.
An online UPS is supposed to be
AC in ==> Power supply to charge battery
Battery ==> inverter to supply power to device
An offline UPS is supposed to be
While AC in has over a minimum voltage
AC in ==> Power supply to supply Battery
AC in ==> Power device
When AC power drops in voltage below the set minimum it switches to
Battery ==> inverter to power device
Now it only detects low/no voltage after approx 50 milliseconds and has an inverter startup time rated in 100mSec or more.
This is still within specs for decent computer supplies.
Easy to solve, you can buy 5.5Volt 1 Farad capacitors (now many farads) on each Pi would keep the Pi running for perhaps seconds. These are not really expensive at all. 1F 5.5VDC Super Capacitor | Jaycar Electronics
You can also get 1F capacitors used for car systems, only useful is you were using common 12/5V lines for all the Pis
EDIT: here is a test. Switch off then back on the (AC mains) power switch and gauge how long before the Pi is stopped, resets, whatever. The quickest you can switch off then on again would likely be more than 100 milliseconds