Rejected 64 address records at 9:12 a.m. because my state-code validator ([1]{2}$) balked at 'NY ’ — one invisible passenger at the end. I felt justified until TRIM cleaned it and I logged ‘user error: whitespace’ in the audit note; anyone else documenting ghosts like this?
I logged ‘user error: whitespace’ in the audit note; anyone else documenting I switched to “rejected; tail=U+00A0 (NBSP), bytes=C2 A0” and a preview like ▷NY␠◁… Normalize (NFC) and map NBSP/NNBSP/ZWSP to space before the [A-Z]{2}$ check — TRIM alone missed most of ours, and I’m seeing U+00A0 more than U+200B.
My take: I’d lean toward the simplest next step and see if it changes anything this week — if not, you’ve got a clear case to escalate. What would block you from trying that?
But same here — after a 9:12 a.m. purge I started logging the trailing char’s codepoint with a visible mark, like ‘NY·’, then run a Unicode-aware trim that strips \p{Zs} and \p{Cf} before validation. Tiny caveat: your ‘([1]{2}$)’ looks like it’s checking for two ones; did you mean letters? Want a link to the little prefilter I use?
, I’ve had a 9:12 a.m.-style mass reject too — my fix was a pre-save sanitizer that runs NFKC, strips Bidi control/zero-width chars, and highlights the final grapheme in red if it isn’t two A-Z so users see the ‘ghosts’ before submit. I also log the paste source when an invisible tail shows up (caught Outlook injecting a BOM), @lgreenwood12, so it’s easy to block those on paste with a tiny handler. Caveat: NFKC can fold look-alikes, so I limit it to the 2-char state field.
For two-letter states I stopped hard-rejects and moved to a dropdown; the paste handler trims any separators and I still stash the raw field in an audit column with a quiet ‘normalized=true’ tag… If you keep regex, ditch ([1]{2}$) and use ^\p{Lu}{2}\p{Z}*$ with the u flag so the tail matches and you trim before save. @mariah_87, I keep both raw and cleaned so the audit shows what arrived and what shipped.
I stopped chasing ghosts by validating a generated column in Postgres that uppercases and trims Unicode whitespace, while keeping the raw field for audit; the constraint targets the normalized value and our rejects vanished (ref: UAX #44: Unicode Character Database). Small caveat: make the regex u-aware or hair spaces sneak through.
I added a focus-only ‘show invisibles’ toggle that renders trailing whitespace as a middle dot and labels zero‑widths (ZWSP, NBSP), so ‘NY·’ jumps out before submit; when it happens, I auto-trim on blur and add the codepoints (e.g., U+00A0) to the audit note instead of just ‘user error: whitespace’. Tiny caveat: if your validator really is ([1]{2}$), you might want [A-Z]{2} with optional trailing space to avoid another 9:12 a.m. pileup.
I log the exact code points when normalization changes a state code — e.g., ‘ghost: U+00A0’ or ‘U+200B’ — using ICU (NFKC + map \p{Zs} to space), which quickly surfaces where they’re coming from; handy set: Unicode Utilities: UnicodeSet. Tiny caveat: I soft‑fail the first time and show a hint, because hard‑rejecting a zero‑width feels like tripping on invisible glitter.