Network Outage Diagnostic Checklist

First 5 Minutes: Stabilize and Observe

Do these in order. Stop here if any step restores service.

1. Note the time things stopped working and what exactly is broken. Are certain devices still working? Internet connection, Drawbridge, camera systems, Mythric, PLCs, etc.

2. Ask: "Did anything change in the last 24 hours?" Power blip, storm, breaker tripped, someone moved cables, contractor onsite, a device was rebooted, a new device was plugged in, internet was out earlier

3. Visual Check. Walk to the network rack/closet. Take a phone photo of every device's front lights (router, switch(es), Drawbridge server, Mythric, NVR). Make note of any that are not powered on.

4. Open a support ticket with Drawbridge Support and include: time of outage, what's broken, what's still working, and any answers to #1-3

5. Do NOT power-cycle the router or any devices yet. Rebooting a device is the last resort, particularly if there is partial functionality, as any reboot may result in changed IPs or further degradation. Save it for after the symptom-based diagnosis below.

Terms and Definitions

• Router: Often a small network box provided by an ISP. Often only has a few (<5) Ethernet ports. Responsible for handing out IP addresses. Example Front. Example Back.

• Switch: A network switch is a hardware traffic controller of a local network (LAN). It connects multiple devices—like computers, printers, and gaming consoles—together using Ethernet cables, allowing them to share data, resources, and internet access. It often has many Ethernet ports (24-48). It only connects local on-site devices. Example

Step 1: Identify the Symptom (highest-value step)

This is the single most important section. What's broken tells you where to look. Find the row that matches your situation based on your symptoms:

Step 2: Internet / ISP / Router

Use this section if everything seems dead or you can't reach the internet at all.

• Check internet on a phone using cellular data (not Wi-Fi) — baseline confirms web pages should load

• Check internet on a phone using the venue Wi-Fi — if Wi-Fi works but venue devices don't, the router is fine and the problem is downstream (skip to §3)

• Check the router's Power status light — should be solid (color varies by brand, often green).

• Check the router's "Internet" or "WAN" status light — should be solid (color varies by brand). If off or red/amber, the ISP is the problem. Call and see if they have an outage.

• If the ISP confirms an outage, you're done with diagnosis — wait it out, and notify Drawbridge Support so they know to expect downtime

Step 3: Main Switch / LAN

Use this section if the internet works but venue devices are dark.

• Find your main network switch (the box with many ethernet ports — typically near the router)

• Check the switch's power LED — if dark, the switch is unpowered. Check the power brick, outlet, and surge protector

• Look at the Ethernet port lights:

  • Green + amber/orange blinking = healthy connection with traffic

  • Green only, no blinking = link is up, no traffic (device on the other end may be off)

  • Amber only = often a slower speed link, can still be fine

  • Both off = no physical link (cable issue or device off)

• Count the port lights vs. what should be plugged in. If half the ports are lit and the other half are dark even though cables are in them → ⚠️ partial switch failure — proceed to §5

• Take a phone photo of the switch's lights and send to support

• If the switch appears fully dead → unplug its power, count to 30, plug back in. Wait 60 seconds. If it doesn't come back, it needs replacement (see §7).

Step 4: Drawbridge or Mythric Server

Use this section if a specific server's web app won't load from GM PCs.

1. Is the server physically powered on? Front-panel power LED should be lit

2. Check the ethernet port on the back of the server:

   1. Green + amber/orange = good

   2. Both off = cable, port, or NIC problem

   3. Only green, no amber blink = link up, no traffic (server may have lost IP)

3. Reseat the ethernet cable at both ends (server side AND switch side). Listen/feel for the click

4. Try a different ethernet cable from your spares

5. Try a different port on the switch

6. Try a different switch or directly into the router

7. From a GM PC, try to ping the server. Open Command Prompt (Windows key, type cmd, Enter) and type ping <server-IP> (your Drawbridge or Mythric IP — keep these recorded in §8 below)

   1. Replies coming back → network is fine, the web service may be down → contact support

   2. "Request timed out" → server is unreachable on the network

   3. "Destination host unreachable" → routing issue, often switch or VLAN

8. Power-cycle the server. Press the power button briefly — if it shuts down gracefully, wait 30 seconds, press again to start. If unresponsive and still powered on after 30 seconds, hold the power button for 10 seconds to force off, wait 30 seconds, press briefly to start. Allow 3–5 minutes for full boot before testing.

Step 5: Switches, NVRs, Cameras, PLCs, PoE

This is the section the incident this plan is based on lived in. Read carefully.

The "Partial Switch Failure" Test

A switch can be half-alive. Some ports work, others don't, even with the same cables and devices. This is sneaky because if even one or two devices on the switch still work, it looks fine. Test it:

• Take a known-working device (e.g., a GM PC currently online via that switch, or a laptop)

• Unplug it from its current working port and plug it into a port that has a non-working device — does the working device now lose its connection? If yes, the port is dead, the switch is partially failed and must be replaced

• Do the reverse: take a non-working device and plug it into a port that has a known-working device. Does it come back to life? If yes, same conclusion — switch is bad

• Critical lesson from past incidents: "But my cameras still work on this switch" is not proof the switch is healthy. Half-dead switches happen, especially after a power blip

PoE (Power over Ethernet)

Cameras, in-game speakers, some access points, and some PLC modules draw their power from the ethernet cable. If you replace a PoE switch with a non-PoE switch, those devices will go dark even though the network works fine.

• Before buying a replacement switch, confirm: "Is this switch PoE-capable, and does it have enough PoE budget (watts) for my cameras + speakers?"

• PoE switches are usually labeled "PoE" or "PoE+" on the front. If you're unsure, look at the model number and check the spec sheet

• If you only have a non-PoE switch on hand temporarily, you can chain it: plug non-PoE devices (GM PCs, servers, PLCs that don't use PoE) into the new switch, keep cameras/speakers on the old switch if any of its ports still work, or pick up PoE injectors as a temporary fix

Cameras specifically (RTSP)

• Check the NVR(s) — power LED, screen output, network port lights

• Check each camera's LED if visible — most have a small light indicating power and link

• A single camera down is almost always cable, port, or that specific camera — not network-wide

• All cameras down = NVR is down, PoE switch is down, or NVR lost network

PLCs specifically (TCP)

• PLCs talk over TCP to Drawbridge — if Drawbridge is offline they all look offline

• If Drawbridge is up but PLCs are offline → switch or PLC power problem

• One PLC offline, others fine → that PLC's power supply, cable, or port