Behavioral interview

Hi, I'm Alex. This is the behavioral round, about thirty minutes. I'll ask about four situations you've actually been in, and for each one I'll dig into the details: what you did yourself, what happened, numbers where you have them. Real examples work better than what you would do in theory. I may cut in to move us along, that's just about time. At the end there's time for your questions. Before we start, what are you working on these days?

Um, hi. So I'm in my last year at Lakeview State, computer science. Right now, um, most of my time goes into the capstone, you know, which is a course-planning app for students, and I'm also a TA for the intro data structures class, which is, uh, mostly grading linked lists at midnight. And last summer I interned at a company called Brightwater Analytics, on their data platform team.

Okay. What does the TA job involve, day to day? How many students, and what part do you run yourself?

I run one lab section a week, about thirty students, and I hold office hours on Thursdays. Honestly the office hours are the part I like, because you see exactly where people get stuck. Usually it's pointers, or recursion, or both at once.

And the capstone. It's a team of students, so I'm curious about the split. What's your part in it, and what do the others own?

I do most of the backend, the API and the database, and the part that checks whether a schedule is valid, prerequisites, time clashes, that kind of thing. There are four of us, two mostly on the frontend. We demo it to the department in December, so, um, it's getting real now.

Thanks. Let's start. The first one is technical, and I'm interested in how you worked, not only the answer. Tell me about the hardest technical problem you debugged or solved. Walk me through how you approached it.

Okay, so, the hardest one was at Brightwater. The team runs a nightly import: vendors drop files, around forty files a night, and a set of workers loads them into Postgres, and then the dashboards for the account managers read from those tables. The problem was that about one night in five, actually, it was closer to one in six when I counted later, the import finished green, no errors, but the dashboard numbers came out a few percent low. Like two to four percent of rows missing. And nobody could reproduce it in staging. The engineer who had been looking at it was moving to another team, so my lead asked me to own it in my third week.

Okay. You had a symptom that showed up one night in several and never in staging. Where did you start, and how did you narrow it down?

So the first thing I did was get a good night and a bad night side by side. I wrote a small script that compared row counts per vendor file between the source file and what ended up in the table. And on the bad nights the missing rows were always from the two biggest vendors, never the small ones. That told me it wasn't random. Then I added logging around the batch writer, which worker picked up which file and when, and waited for the next bad night. It took four days. And on that night two workers had picked up the same big file, about six minutes apart.

Mm-hm. Two workers on the same file. The import had a lock, I assume. Why would two workers pick up the same file anyway?

Our lock was a row in Postgres with a timestamp, and a worker treated the lock as stale after five minutes. The big files took longer than five minutes to load, so a second worker thought the first one had died, took the lock, and started the same file. And the load did delete-then-insert for the file's date range, so the second worker's delete removed rows the first worker had already written. Then the first one kept going, but with a gap. It only happened when the file was big enough, which is why staging, with small sample files, never showed it.

How did you confirm that was the cause, and not just something that happened that night?

Um, good question, because at first I only had the one night. So I tried to reproduce it on purpose. In staging I made a synthetic vendor file about the size of the biggest real one, and I added a sleep in the writer so the load took seven minutes instead of four. Then I started two workers. And the second one took the lock at the five minute mark and the table came out short, by roughly the rows the first one had written in its last batch. I ran it three times and it happened every time. With the sleep removed, it never happened. So that was the point where I was confident.

Mm-hm. Before the fix, there were probably some wrong turns. What did you try that did not work, and how long did each one cost you?

Uh, quite a lot, honestly. My first guess was that the vendors sent partial files, so I spent about two days adding checksum checks against the manifest they send. They all passed, which was annoying, but it ruled it out. Then, after I found the double pickup, my first fix was just raising the stale timeout to thirty minutes. And my lead, Priya, pointed out in the review that it only moves the problem to the next bigger file, and one vendor was growing like ten percent a month. So that didn't ship.

Okay. Raising the timeout was rejected. So what did you put in place so it doesn't happen again, and why that over the alternatives?

Two things. The lock became a heartbeat: the worker renews it every thirty seconds, and a lock only counts as stale after three missed heartbeats, so a slow worker keeps it as long as it's alive. And I changed the write from delete-then-insert to an upsert keyed by the vendor's row id, so even if two workers ever do load the same file, the second one can't delete anything. Plus a check at the end of the run that compares source and table row counts per file and pages if they differ by more than zero point one percent.

What happened after it shipped? How did you know it was fixed and not just a quiet stretch?

Zero short nights in the six weeks I was still there. That's forty-two nightly runs. Before the fix we'd had about eight short nights in the same length of window. And the row count check fired once, but that was a vendor actually sending a truncated file, so it was doing its job.

Earlier you said about one night in six. Eight short nights in forty-two runs is closer to one in five. Which is it?

Oh. Um, you're right, those don't match. The eight in forty-two is the win- the window I actually counted from the logs before the fix. The one in six, I mean, I think I was remembering a longer stretch, the whole quarter, but I didn't count that one properly. So the number I'd stand behind is eight in forty-two, about one in five.

Okay. You keep mentioning your lead. Split it for me: which parts of this were yours, start to finish, and which were hers?

The investigation was mine, the comparison script, the logging, finding the double pickup. The heartbeat design was mine too, but Priya pushed on the upsert part, she said fix the lock and also make the write safe, don't rely on one guard. So I'd say the idea of two layers came from her and I built both. I also wrote up the root cause on the team wiki and presented it at their Friday demo, which was, um, terrifying, but fine.

How did you roll the change out? This was a nightly job other people depended on.

Behind a flag, per vendor. I turned on the new lock and the upsert for the two small vendors first, for three nights, and compared the row counts with the old path. Then the two big ones. The upsert was slower, about fifteen percent longer for the biggest file, which we accepted, it still finished well before the dashboards refresh at six. And I kept the old code path for two weeks so we could flip back.

Not technically, but for you personally, what was the hardest part of the whole thing?

Honestly, the waiting. You add logging and then nothing happens for four nights, and you start to doubt your theory, and people ask in standup if there's news. I think I got a bit too quiet in that period. I didn't want to say I don't know yet every day. Priya told me later that saying I don't know yet, here's what I'm waiting for, is a perfectly good update.

Two account managers had already sent wrong numbers to clients. Did anyone tell them what happened, and who did that?

Priya did the client side, that wasn't something an intern would do. But she asked me to write the explanation for the account managers, in plain language, no lock or heartbeat. Which nights were affected, which dashboards, roughly how much the numbers were off, and that it was fixed. I wrote it, she edited maybe two sentences, and it went out from her. I also made a list of the eight affected dates so they could check whether anything they'd sent used them.

If the same kind of intermittent problem landed on you next week, what would you do differently from the start?

I'd add the logging on day one instead of day five. I spent two days on the checksum theory because it felt likely, but I had no data for it. Now when something is intermittent, I instrument first and guess second.

Okay. Next one. It can be from the internship again, or the capstone, or something outside both. Tell me about a time you took on a part of a project that nobody had asked you to do.

Hmm. Let me think. Okay, so, also at Brightwater. The team had a Slack channel for data quality alerts. Every check that failed posted there. And when I joined, it was getting around forty alerts a day, and nobody read it. People had muted it. Which meant when something real fired, like a missing file, it got lost. Nobody owned the channel, it was just there. So in my fourth week, while I was waiting for bad nights on the import, I started looking at it.

It wasn't your channel and you were three weeks in, working on something else. Why did you step in?

Partly selfish, I was watching the channel anyway for my import alerts, and I couldn't find mine in the noise. And partly because I asked two engineers if anyone was fixing it, and both said, yeah, someone should. So it was clearly nobody's.

Walk me through what you actually did, step by step, from the first day you looked at it.

I exported two weeks of alerts, about five hundred and sixty, into a spreadsheet, and grouped them by the check that fired. Three checks caused about eighty percent of them. One was a null-rate check on a column that is legitimately empty on weekends. One was a freshness check with a one hour threshold on a table that only updates every three hours. And one was a duplicate check that fired on every retry. I wrote a one page proposal: fix the thresholds on those three, add a weekend rule, and route the rest by owner. I showed it to Priya before touching anything, because those checks belonged to other people.

Okay. Proposals are one thing. What was the impact once it was in place, in numbers if you have them?

After the changes it went from about forty a day to about six a day, averaged over the next three weeks. And two people unmuted the channel, which, I mean, is not a metric, but I noticed. The real one was that a missing vendor file got caught the same morning, about a week later, which before would have been found by an account manager two days later.

Of the three fixes, which one did most of the work?

The weekend null rule, by far. That check alone was about half of all the alerts, it fired every Saturday and Sunday on every partition. The freshness threshold was maybe a quarter, and the duplicate check on retries was the smallest, but it was the most annoying one because it fired in bursts of ten at three in the morning.

Three days is real time for an intern. What did it cost you or your regular work, and who noticed?

It took about three days in total, spread out. My own ticket, the lock fix, slipped by about three days, and I hadn't, um, told Priya in advance that I was doing this, so when she asked about the lock in standup I had to say, uh, it's late because I was doing alerts. She was fine with the alerts work but not with hearing about it in standup. Fair enough.

Those checks belonged to other engineers, some more senior than you. Did any of the check owners push back, and what did you do about it?

One did. The person who owned the freshness check wanted to keep the one hour threshold because a downstream report depends on it. We ended up keeping one hour on weekdays mornings only, when the report runs, and three hours otherwise. I'd say that was a compromise he proposed, not me.

You mentioned a third part, routing the rest of the alerts by owner. Did that actually happen, all of it?

Partly. Um, I set up the routing for my team's checks, so they went to the right person's channel. For the other two teams I wrote the config and opened the pull requests, but one of them was still open when my internship ended. I don't know if it got merged. I should have handed it to someone by name instead of just leaving it in review.

Would you take something like that on again, the same way?

Yes, but I'd tell my lead before I start, not after, and I'd give it a time box, like two days, and say what I'm pushing back. The work itself I'd do the same. Starting from the data, the five hundred and sixty alerts, is what got people to agree. Nobody argues with a spreadsheet of their own alerts.

Okay. Tell me about a time you disagreed with a teammate about how to build something. What happened?

Sorry, could you repeat that? I was still half in the alerts story.

Tell me about a time you disagreed with a teammate about how to build something. What happened?

Hmm. Okay. Um, so in my capstone, the course-planning app, there are four of us, and, uh, we had a disagreement about, like, caching. My teammate, Marco, wanted to put Red- Redis in front of, you know, everything from the start, and I, I mean, I didn't think we needed it yet. So, um, yeah, we disagreed about that.

Mm-hm. Before we get to how it ended, what was your specific position, and what was it based on?

That it was too early. Like, we didn't have users yet, so I thought it was, um, premature. It would add complexity, and, you know, we had other things to do. I think I said something like, let's not optimize before we know it's slow.

And his position? As he would have put it.

Um, that it would be slow? I think he said the catalog page was going to be slow because it, uh, loads all the sections. I'm honestly not sure what his whole reasoning was.

Okay. He thought the page would be slow and you thought it was premature. What did you do to understand his point of view?

Um, honestly, I didn't really. We argued about it in the group chat for a while, it got a bit long, and then we kind of agreed to try it on one page and see.

Trying it on one page is a specific compromise. Who suggested it, and how did the group get there?

I think it was, um, our fourth teammate, Dana? Or, uh, maybe it came up in the meeting, you know, when we were all there. I don't remember exactly. It was kind of a group thing.

And then what did you personally do on the caching work?

Um, so, we set it up together, kind of. Marco did the Redis part, and we, like, we rev- we reviewed it, and, uh, we tested the page afterwards. I mean, I think I reviewed his pull request.

What was your specific part?

Reviewing it, mainly. And I changed the API so the catalog endpoint returned the whole list at once, which made it easier to cache. But that was a small change. Mostly it was his.

Okay. How did it end, for the project and for the decision itself?

We ended up caching the course catalog page, and it got a lot faster, so I think everyone was happy with that.

How much faster? Do you have a number, before and after?

I don't have the exact number. It felt a lot faster. Marco measured it, I think. Like, it went from noticeably slow to basically instant.

Same situation, same teammate, starting tomorrow. Would you handle it differently now?

Yeah, I'd probably communicate better. I think, I mean, it just took longer than it needed to.

Communicate better can mean a lot of things. What would it look like, concretely, the first time the topic came up?

Um... probably talking instead of the group chat. And, yeah, maybe asking him why earlier. I guess I was more focused on being right about it.

If Marco were telling this story, what would he say you did wrong?

Hmm. Probably that I dismissed it without looking. That I said premature optimization like it was a rule, and didn't ask what he'd seen. I don't know, maybe. We never really talked about it after.

You're still on the same team for a few more months. How are things with Marco now?

Fine, I think. We still work together on the project. It wasn't a big fight or anything, it was more like a long annoying thread. Yeah. It's fine.

Last one. Pick something where the feedback actually stung a little, if you have one. Tell me about the most useful critical feedback you got on a project or a piece of coursework. What did you do with it?

Um... let me think. Okay. During the internship, my first design doc for the lock change came back from Priya with one comment at the top: all solution, no problem. I had three pages on how the heartbeat works, the timing, the edge cases, and one line on why it mattered. Nobody reading it knew how often the import was short or what it cost the account managers.

One comment at the top of a three page doc. How did you react when you first heard it, honestly?

I was a bit defensive, to be honest. I thought the fix was the interesting part, and I'd spent a lot of time on it. I think I replied something like, the problem is in the ticket. And she said, nobody reads the ticket. Which, um, is true.

And the thing is, the heartbeat part really was subtle, because you have to think about clock skew between workers, and, uh, what happens if the database itself is slow to answer the renewal, so I had this whole section on

Let me stop you there. What did you do with her comment?

Right, sorry.

I rewrote it with the problem first. A short timeline of the incidents, eight short nights in six weeks, how many rows, and that two account managers had sent wrong numbers to a client because of it. Then the fix, then what I'd rejected, the longer timeout. The first draft needed two review meetings and still had open questions. The rewrite was approved in one.

What did the first paragraph of the new version say, roughly?

Something like: on eight of the last forty-two nights, the vendor import silently lost between two and four percent of rows from the two largest vendors. The dashboards showed the lower numbers and at least two client reports used them. The cause is a lock that expires while a large file is still loading. This doc proposes a fix that removes the cause and makes a double load harmless. Then everything else came after that.

That's one document. What specifically changed afterwards, in how you work?

Now every doc I write starts with a problem paragraph: what broke, how often, and what it costs. I did the same for my capstone report, and for the alerts proposal I mentioned, that one page started with five hundred and sixty alerts in two weeks.

How do you know it worked? What tells you the change made a difference?

Um, I guess... the capstone report got the best grade in our section? But I'm not sure that's because of the structure. The alerts proposal got approved without changes, so maybe that. I don't really have a clean way to measure it.

You're a TA now, so you're on the other side of this too. Have you given feedback like that to someone else since, and how did it land?

Actually, yes. As a TA I grade the project write-ups, and I started leaving the same kind of comment, what problem does this solve, before the comments about the code. A couple of students said it helped. Though I'm, uh, a bit nicer about it than all solution, no problem.

Most people get some feedback they agree with and still don't act on. Was there anything like that from the internship?

Um... yes, probably. In my final review Priya said I should push back more in design discussions, that I'd go quiet when a senior engineer disagreed, even when I had the data. I agree with it, but I don't think I've really changed that yet. In the capstone I'm the one with the most backend experience, so it hasn't been tested the same way.

Before we finish, I want to go back to one thing. The caching disagreement: you said Marco measured the speed-up. Did his measurement change your view of who was right?

Hmm. I think so. If it went from slow to instant then he was right about that page, at least. I still think Redis everywhere would have been too much. But I never really looked at his numbers, so I can't say for sure.

Okay. That's all the questions I had. We have a few minutes left. Do you have any questions for me, about the role, the process, anything?

Yes, two, if that's okay. First, for new grads on an engineering team, what does the first project usually look like? Is it something small and contained, or are you thrown into the main system?

It varies by team. Often it's something contained enough to finish in a few weeks but touching the real system, so you learn the codebase, the review process and how things get deployed. Some teams start people on bugs or on-call shadowing first. I can't speak for a particular team; the hiring team can tell you what it looks like for a specific role. Anything else?

Yeah. In behavioral rounds like this one, what makes a story land better? Like, is it more about the result or the process?

In general, interviewers want both, but anchored in what you personally did: the situation in a sentence or two, your own decisions and why, what came of it, ideally with a number, and what you'd change. I can't say how this conversation went. Your report will cover that.

Okay, fair. Thanks.

That's time. Thanks for your time today. You'll get your report in a few minutes.