🌐 Also available in: 繁體中文 →

Deleted, Yet Still There: Notes from HackerRank Orchestrate

I joined HackerRank Orchestrate, where the task was to audit whether "deleted" personal data was really gone. I can barely write Python, so I built the whole system with Claude Code. This post covers how I did it, what the data revealed, and what AI can and can't do in a problem like this.

I recently joined HackerRank Orchestrate (October 2026), a 24-hour challenge. It was my first time, and all I knew beforehand was that agent-assisted coding was allowed, which felt pretty novel. The problem was called Ghost Data Still Exists (challenge repo). The setting is a commerce platform: a user asks for their account to be deleted, the deletion workflow reports “complete”, but their data is still scattered across database replicas, logs, support attachments, search indexes, the analytics platform and backups. Worse, some sync jobs and backup restores write deleted data back.

The job was to build an audit agent. For each deletion request, decide whether the deletion is really done (COMPLETE, INCOMPLETE, or NOT_APPLICABLE when it isn’t a deletion request at all), then list the remaining records, the remediation steps, and how to verify them afterwards. There were 518 requests, plus 32 samples with answers.

This post is about how I built it. But the part I enjoyed most came halfway through: reverse-engineering how the platform works from the data, and finding its design flaws one by one. That’s in the second half.

I can’t really write Python

Let me get this out of the way. I normally write Go; I can read Python, not much more. Claude Code wrote the entire pipeline. My job was to set the direction, define the rules, check whether the results were right, and argue with it when they weren’t.

The challenge required submitting the full chat transcript, so every turn and every decision is in the log. Looking back at it, the most valuable part isn’t the code. It’s the moments where I said no.

The approach: trust only evidence

The core idea of the challenge in one line: the status the workflow reports is a claim, not evidence. Even if a request says COMPLETE, you have to find its real state at audit time in the evidence yourself.

The pipeline has these steps:

1dataset ─► split evidence into units ─► link to the person ─► attribute by timing ─► contested cases ─► apply policy ─► output.csv
2          (SQL, JSON, logs,             (current and old       (records with no       (rules first,
3           PDFs, screenshots)            emails, phones)        identifier left)       LLM last)
  1. Check whether it’s really a deletion request. Some requests say “remove me from your marketing lists, keep my account”. That isn’t a deletion.
  2. Link every record to the person. Start from the request email, add the old emails and phones the user changed over time, and expand: identifiers find records, and resource IDs in those records find logs and activities. No matching on names, because many people share a name.
  3. Decide the state from evidence. Only things actually observed count: readback logs, database snapshots, admin screenshots. And only evidence captured before the audit time. Evidence older than the request doesn’t count either; it only shows the state before deletion.
  4. Apply the deletion policy. Some records must be deleted, some redacted, some anonymized, and security audit records must be kept. An account tombstone keeps the user_id but must drop the name, email and phone.

The hardest part: once deleted, you can’t tell whose it was

This is the most interesting part of the whole challenge. Once an account is deleted and analytics data is anonymized, there’s no identifier left on the record. So whose record is it?

A normal system could trace it back through logs, but the logs here are incomplete too. The deletion job’s records carry only a resource ID, no user. A trace_id only works inside one service and can’t follow a request across services. The deletion workflow itself has no log at all. There’s no thread tying the data together: once a request is split into a dozen steps, no field can put them back together as one person. (More on this in the reverse-engineering section.)

The only clue left is the deletion job’s timing. From records I could already identify, I learned roughly when each stage of the deletion job runs relative to the “reported” time. If a record was processed inside exactly one request’s time window, it goes to that request.

But many requests are processed in the same batch, so the windows overlap. Two more rules help:

  • Results must agree. The result of the primary account deletion matches the request status almost one to one (about 97%): if the request is COMPLETE, that step is SUCCESS. Candidates that don’t match are dropped.
  • The rhythm must fit. The gaps between one request’s job events, in seconds, are very regular and fall into a few narrow ranges. Candidates that don’t fit the rhythm are dropped too.

Both rules are learned only from records with a confirmed identity, never from the sample answers. Only what’s still undecidable goes to the LLM.

Why the LLM only comes in at the last step

I was asked this in the interview. My answer at the time: the logs themselves don’t have enough information, so the LLM can’t decide either, and scripts are more efficient.

Looking at the numbers afterwards, that holds up. The rules settled 273 attributions, and only 56 contested cases were left for the LLM. Those cases had the same timing, the same result and the same status; the information just wasn’t there. The LLM abstained on 54 of them, and only one of its answers was used.

There’s another reason I didn’t mention then: every conclusion in a deletion audit has to be explainable. A rule can say exactly why a record belongs to a request; an LLM can’t. And one wrong guess causes two errors: one request gains a step that isn’t its own, and another loses the step it should have. So I set the LLM to abstain by default. Better to leave it empty than to guess.

No hard-coded answers

During the challenge I had Claude start from the data it could already prove, use it to find the matching time ranges and widen the search, and check the results against the samples. Hard-coding answers was not allowed.

It traced every remaining failure back to its source in the data and fixed most of them. The samples ended at 29 of 32 cases matching exactly, with 100% precision. The last 3 steps used resource IDs that don’t exist in any file participants can see, PDFs and screenshots included. The only way to get them was to write the answers into the code.

So that’s where it stopped. I think it was the most important decision of the whole process.

In the final scoring, output.csv got 28.5 / 30. With nothing hard-coded, it looks like this audit approach does work.

Evidence has strength

At some point I realised the answers alone weren’t enough; I also needed to know how much each answer could be trusted. So the results are graded into three levels:

LevelMeaningWhat’s needed
ConfirmedLinked to the person by an identifier, and the state was observedNothing; the evidence verifies it
InferredAttributed by timing or rules, or the state wasn’t observed directlyOptional: more logs would make it verifiable
UndeterminedA processed resource couldn’t be pinned to a requestMore logs required

Checked on the samples, all 160 confirmed steps and all 30 inferred steps were correct. Of the 518 real requests, 55 cases were undetermined, and the report lists exactly which job records would settle each one.

output.csv has a fixed format that can’t hold any of this, so it goes into a separate evidence report.

The system, reverse-engineered from the data

This was the most fun part of the challenge for me. We got 4,000+ system activity records, assorted service logs and database snapshots, but no architecture diagram. I asked Claude to reverse-engineer how the platform works using only records with a confirmed identity: no inferred attributions, no sample answers.

Normal data flows

Counting every “source → action → target” in the activity records, the platform’s data flows look like this (counts in parentheses):

 1profile_service ──UPDATE(440)──► account_service ─┬─INDEX(400)────► search_service
 2                                 (primary DB)      ├─SYNC(360)─────► analytics_platform
 3                                                   ├─REFRESH(284)──► cache_service
 4                                                   ├─SYNC(56)──────► replication cluster / failover replica
 5                                                   └─ export_service ─EXPORT(155)─► object_store
 6support_platform ─UPDATE(232)─► search_service
 7notification_service ─SEND_EMAIL(368)─► email_delivery_service
 8security_service ─RETAIN(196)─► archive_service (audit records kept long-term)
 9integration_service (connectors) ─SYNC(31)─► search_service   ← writes data back
10backup_service ─RESTORE(1)─► search_service                   ← also writes data back

One account’s data is copied into search, analytics, cache, replicas and exports, and support tickets get indexed into search as well. The search service only receives; it’s purely derived data. The problem is the last two lines: connectors and backup restores bypass the primary database and write straight into the search index.

The deletion workflow

Next, I lined up the time of each deletion step against the request’s “reported” time to see when each stage typically runs:

StageRelative to the report (median)
Delete primary account−59 min
Handle tombstone, legal hold−28 min
Redact support records−21 min
Disable connectors−15 min
Block backup restores−7 min
Report status0
Check connectors, check restore access (check only, no action)+49 to +54 min
Delete search, export and support records, anonymize analytics+47 to +94 min

The deletion job clearly runs in two modes, and the mode almost decides the outcome:

ModeJudged completeJudged incomplete
Act, then report: disable connectors, block backups, report last411
Report, then act: report right after the account deletion, delete downstream later, only check connectors and backups0350

Of the 350 report-then-act cases, 160 reported “complete”. The job’s own results can’t be trusted either: one kind of connector check says “connector confirmed still enabled” in its message, yet its result field says SUCCESS. In the whole workflow, only the primary account deletion’s result agrees with the request status.

Flaw one: syncing bypasses the primary database

The usual design makes the primary database the single source of truth, with search, analytics and cache derived only from it. Delete from the primary, sync once, and every copy disappears with it.

This platform’s search index has three writers:

Path into the search indexCountResult
Indexed from the primary DB400Expected
Connector writes back directly31Every following delete failed
Backup restored straight into the index1Account deleted, yet the search document came back

With more than one writer, a deletion has to be made against each of them, and missing one brings the data back. Of the deletes that followed the 32 write-backs, 31 were FAILED and 1 PARTIAL. Not one succeeded.

That single backup restore was especially interesting. Claude first suggested it might be caused by disaster recovery. That didn’t sound right to me: a real restore would restore the account in the primary database first, mark it as “deletion pending”, and let the syncer update search. Why would anyone write a backup straight into the search index? We went back and checked:

DatePrimary DBSearch indexBackup
Apr 26Backup contains this user’s data
Apr 27User requests deletionStill restorable
Jul 26Retention period ends
Aug 22Account deleted, no restore at allBackup writes the search document backRestored
Sep 15Lookup returns 404Search document still thereStill restorable, no deletion block

The result is an orphan record with no source: the account no longer exists in the primary database, yet the search index still finds it. And across the whole period there was only this one restore, affecting a single record, with no sign of any failover. It looks more like a targeted restore or an index rebuild than disaster recovery.

Flaw two: no unified per-user operations log

Where one person’s data was copied and whether it was deleted is scattered everywhere, in different formats:

  • The system activity records carry only a resource ID, no user.
  • In the service logs, 56% of lines carry an email, 14% a user_id, and 30% no identifier at all.
  • trace_ids never cross services: not one of 421 traces spans two services.
  • The deletion workflow has no log of its own, only that one line of result in the activity records.

With an operations log keyed by user_id, recording every create, copy, sync and delete, a deletion is just a walk through that log, and verification is checking every entry was removed. The whole challenge wouldn’t need timing inference, or an LLM. What my pipeline really does is piece that log back together after the fact. The 55 cases it couldn’t piece together are the cost of not having it.

What AI can and can’t do

The biggest takeaway: in this challenge, all AI can really do is help guess. The system’s own records are incomplete; when a piece of data is missing, no model can bring it back. And a deletion audit is exactly the kind of thing you can’t base on guesses.

So my conclusion isn’t “AI is great” or “AI is useless”. It’s about putting it in the right place:

  • Rules and code make the decisions: repeatable, explainable, and free.
  • AI writes code and investigates: Claude Code read evidence in every format and wrote the whole pipeline for me. I couldn’t have done that part myself.
  • Where the data isn’t enough, say you don’t know: and list what’s needed.

If I did it again

After the challenge ended I kept thinking that a closer-to-real-world approach would be a loop:

1 pipeline decides ──► evidence report (what's undetermined, what's needed)
2       ▲                              │
3       │                              ▼
4 feed new evidence back ◄──── agent queries logs and reads back databases from the list

Hand the evidence-gathering step to an agent, and you have a harness loop. The key to the split: the agent gathers evidence; the deterministic rules still decide. The agent can read but not delete; every piece of data it brings back records where it came from; every round must reduce the undetermined cases, and with no progress it stops and hands over to a person.

That way the AI doesn’t have to guess. It goes and fetches the missing data. It’s the same idea as my previous post on agent tool_calls: AI can take part, but decisions and irreversible steps need a rule or a person guarding them.

Murmur

The organisers say this is a problem that has actually happened in the real world. But please, if you build a microservice architecture, have trace logs! Without logs, proving anything is really hard.

I once heard a funny story: an engineer on another team broke the logging library, and every log in production vanished. In the end, everyone could only find bugs by psychic powers, haha.

Have you run into something like this? The psychic work used to run on human intelligence; now artificial intelligence can lend a hand… great.

Postscript

The final results are out: I placed 674th out of 4,094, roughly the top 16%.

HackerRank Orchestrate October 2026 certificate, final rank #674 of 4,094

It was my first time, and there’s still room to improve. I’ll keep at it next time.