Unscheduled Negishi Outage
Link to update at December 17, 2024 8:06am EST UPDATE:
Space on the NVMe-tier of Negishi Scratch has been cleared sufficiently that scheduling was resumed last night around 9pm. The Negishi cluster is no longer in an outage state.
Link to update at December 16, 2024 4:54pm EST UPDATE:
Work continues on bringing Negishi back to normal operation. Engineers have identified the source of the problem, the fast flash tier of the scratch filesystem, and are currently applying a fix. Scheduling will remain paused while the fix runs through the night, and will be readdressed at 8am tomorrow.
Link to original posting ORIGINAL:
The Negishi Scratch file system began experiencing issues around 11am EST today (December 16).
The issue manifests as a "No space left on device" warning on new file creation. Reading and writing existing files is not affected. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed.
We will provide an update by 5pm EST (December 16).